Building intelligent market research surveys with text similarity search

Implemented owned by Peltarion
3y ago update

Summary

Dealing with data is part of daily life for the global market research company Ipsos, and for a company already skilled in data processes, the potential with adding AI is tremendous. So we decided to team up with Ipsos to bring their domain expertise and our AI experience together, to see what good we could create. We quickly realized there are so many areas to bring value to with AI, but one of them is “Question Library Similarity Search”.

Objectives

Drawing conclusions from data is really the core in market research, so the step to using AI to automate is fairly small. If you have datasets of images or text, you can easily use them to classify different factors such as geography, demography, ethnicity, and connect them with habits, opinions, and much more.

The majority of data that market research companies possess is text. 
Text data that can be classified into different categories, text data where correlations are found with other text data, and text data with a past which makes it possible to predict the future. All of which AI can help, to make things go faster and be more accurate.

Solution

Asking questions could be a really simple task, but knowing how to professionally build up a survey with the “right” questions is a whole scientific field of its own. For instance - you can ask “How old are you?” and you get the answer “42”.  But if you read the answer two years later it’s no longer correct. Instead, asking “What year were you born?” with the answer ”1978”, could be a much more valuable answer to keep for the records.  

Same question in two different ways, where the latter will remain useful also in a few years...

For the Question Library Similarity Search, we used the questions from 90’000 projects, resulting in som 600’000 raw questions as input and used state of the art NLP (Natural Language Processing) techniques to provide users with similar questions, to present good alternatives to choose from.

Outcome

By systemizing the process of finding new alternatives for questions with help of AI, Ipsos can raise the quality of how their research is done, and maintain a high and consistent standard across the globe. And - in many cases save a massive amount of time. All of which will give Ipsos a better competitor advantage.

If you’re curious to hear more about the project, here’s a good video of Ipsos and Peltarion talking about "Similarity Assessment Using Bert in Market Research - Ipsos Digital", at Re Work’s Deep Learning Summit in London.

pages

Resources

2022-12-13 18:44 Video

How can AI help in market research? Using the AI language model, BERT, Peltarion and Ipsos Digital together made it possible to create smart surveys using similarity search. Being able to draw conclusions from data is essential in market research, thus being able to apply AI to this process is relatively easy, as the data already exists. If you have datasets of images or text, you can easily use them to classify different factors such as geography, demography, ethnicity, and connect them with habits, opinions, and much more. 

Watch the presentation by Anders Arpteg, Head of Research at Peltarion, and Andrea Postoaca, CEO Ipsos Digital at ReWork 2019.

Attributes

Information Technology, Media
Marketing & Communications
More Efficient, Smarter Product or Service
Professional
Discovery, Prediction, Language
CNN, DNN, NLP, Supervised, Transformer, Self/Unsupervised
Unstructured Data