Computational Consumer Insights

Computational Consumer Insights:
Using AI and Machine Learning
to Understand Consumer Preferences
Summary
The globalization and digitalization of the world have given
consumers more information and choices than ever before. At the
same time, it has become easier for businesses to collect consumer
data from online surveys and other sources. Thus, Consumer
Insights has become a key pillar of business strategy. The objective
of consumer data analysis is to find valuable business insights. We
view Consumer Insights as an information game where the goal is to
uncover as much relevant information as possible about consumers,
with the least costs and efforts. We believe that human-driven
technology is the key to enhancing human capabilities in data
insights, by leveraging computational and algorithmic power. We
use Artificial Intelligence (AI) and Machine Learning to create
the next-generation tools for Consumer Insights. AI and Machine
Learning must be used in conjunction with Human insights to
create value, and we call this Human-Driven Artificial Intelligence
(HD-AI). HD-AI can also be used to optimize survey design and
data collection. Asking the right questions increases the quality
of consumer data, which in turns improves business insights. Our
technology can also automatically process free-text data from
open-ended survey questions.
February 2016
c 2016 Business Interactive Games Inc. All rights reserved. http://www.bi-games.com
Copyright We view Consumer Insights as a game where companies try to uncover as much relevant
information as possible about consumers, with the least costs and efforts.
Consumer Insights
as a Game
1
thods, such as web surveys - with
multiple-choice questions, openended questions, or choice tasks have become common ways to obThe globalization and digitaliza- tain consumer data. While busition of the world have given con- nesses can in principle greatly besumers more information and choi- nefit from more data, this is conces than ever before. To remain tingent on their ability to extract
competitive in today’s landscape, valuable business insights from
businesses have to become more these data. We use Artificial Inconsumer-focused to better meet telligence (AI) and Machine Learntheir needs. At the heart of a ing to create the next-generation
consumer-centric approach is the tools for Consumer Insights.
quest to understand consumer preWe view Consumer Insights as
ferences, perceptions, and behav- a game where companies try to
iors (which products they want, uncover as much relevant informahow much they are willing to pay, tion as possible about consumers,
how much they like a brand, etc). with the least costs and efforts
Consumer Insights is therefore ne- (Fig. 1). One needs to solve an incessary for a successful business formation maximization and cost
strategy.
minimization problem using new
At the same time, it has be- computational techniques. The
come easier for businesses to col- ultimate goal of the game is to
lect large and diverse sets of con- capture consumer utility functions
sumer data from sources such as (perceptions, preferences, choices
transaction data, loyalty programs, and behaviors). These functions
or consumer surveys. Online me- are predictive quantitative mo-
CONSUMER INSIGHTS AS A GAME
Collect
Data
Consumers
Integrate &
Visualize Data
Analyze Data &
Derive Insights
OBJECTIVE
HD-AI
Consumer Analytics
Capture consumer
utility functions
Automatic
Free-Text Analysis
• Perceptions
• Preferences
• Choices & Behaviors
?
Ask questions
Dynamic Survey Design
Fig. 1: Our innovations in Consumer Insights
dels of consumer choices, with
which we develop Computational
Economics models and Computational Business Games. For more
information, refer to our white paper “Computation Strategy: Using Games for Decision Making”.
We believe that human-driven
technology is the key to enhancing human capabilities in data
insights by leveraging computational and algorithmic power. Using AI and Machine Learning, we
propose three new techniques for
Consumer Insights:
Human-Driven AI (HD-AI)
Automatically discover all significant relations in data to improve
human insights
Dynamic Survey Design
Leverage HD-AI to adjust survey
questions during data collection
and increase data quality
Automatic Free-Text
Analysis
Automatically extract normalized
and quantitative data from freeform text inputs
Human-Driven AI
The objective of consumer data
analysis is to find valuable business insights. One type of output from statistical data analysis
is correlations (‘Variable A is dependent on Variable B’), which
yield insights about cause/effect
relationships (‘someone who buys
shampoo is likely to buy condi-
c 2016 Business Interactive Games Inc. All rights reserved. http://www.bi-games.com
Copyright We use Artificial Intelligence (AI) and Machine Learning to create the next-generation
tools for Consumer Insights.
tioner’). Another type of output
is clusters (‘Values of Variables A,
B, and C occur together’), which
yield insights about consumer segments.
We use custom AI and Machine Learning algorithms to automatically identify all significant
correlations and clusters in consumer datasets. While correlation does not imply causation, the
lack of correlation always implies
the lack of causation. This means
that if a variable does not exhibit
a significant relationship to another variable, it can be ignored.
Our technology is based on
cutting-edge research in hierarchical Bayesian networks, tree learning algorithms, and constraint preference encodings. Our tools automatically identify a small set of
potentially interesting data relationships from many thousands of
possibilities, so that human analysis can quickly hone in on the
most valuable business insights
(Fig. 2).
Fundamentally, no algorithm
or machine can yield data insights
on its own, without human intelligence. We believe that AI and
Machine Learning must always be
used in conjunction with Human
insights to create value, and we
call this approach Human-Driven
Artificial Intelligence (HD-AI).
Dynamic Online
Surveys
Beyond data analytics, HD-AI
can also be used to optimize the
design of online surveys used for
Consumer Insights. We view survey design and data collection as
an information-maximization and
cost-minimization problem. The
objective is to learn information
from each survey question, but
each question has a cost. It generally costs money to pay people
to answer questions and longer
surveys take more time to complete. Moreover, the attention of
respondents also typically diminishes with the length of the survey. Therefore, it is important to
only ask questions that give valuable insights, rather than wasting
valuable respondent time on useless questions. Asking the right
questions increases the quality of
1
Artificial
Intelligence
Use range of Machine
Learning techniques to
discover ALL significant
variable correlations and
clusters in datasets.
Sets of correlations
between variables
consumer data, which in turns
improves business insights.
We propose a new dynamic
approach to online surveys where
HD-AI analytics are applied on
the data after an initial set of
responses have been collected, to
refine survey questions. The goal
is to automatically test for potential variable correlations (or the
lack thereof) and adjust the set
of survey questions based on the
results. Questions that are highly
unlikely to yield insights are removed and replaced with other
questions. Consider a question
where 97 out of the first 100 respondents selected answer A. At
this point, we can be very confident that the answer is almost
always A. Therefore, asking this
question again to other respondents would be unlikely to yield
any new insight, and another potentially valuable question could
be asked instead.
2
Human
Intelligence
Use business expertise to
explain causation
relationships underlying
correlations found by
Machine Learning.
Business insights and
recommendations
Fig. 2: HD-AI Consumer Analysis
c 2016 Business Interactive Games Inc. All rights reserved. http://www.bi-games.com
Copyright 2
Our technology automatically turns free-text survey responses into normalized quantitative data that can be analyzed by statistical tools.
In essence, the automated approach of HD-AI reduces the cost
and time of data analysis, making
it easier for humans to test the
quality of survey data “on the fly”
and adjust survey questions accordingly. This testing process
can be repeated throughout data
collection to hone in on the most
valuable set of questions – leading to higher quality consumer
data. Better data begets better
insights.
3
time and labor-intensive to analyze even for small sample sizes.
Typos, multiple-word inputs, lack
of word space all make free-form
text questions hard to analyze
with common statistical tools. It
requires human workers to manually read, classify, and count text
responses.
Our technology based on Natural Language Processing (NLP)
and Computational Linguistics can
process free text fully automatically. It turns free text inputs
into normalized quantitative data
Automatic Free-Text that can be analyzed by statistical tools. Consider the example
Analysis
of asking respondents to name
Consumer surveys often ask open- their favorite brand. From the
ended questions where respondents list of misspellings, abbreviations,
enter their own answers as free unnecessary words, or incorrect
text. While free-form text ques- phrases, our algorithm automatitions remove the bias of multiple- cally identifies the intended brand
choice questions, as there is no names and groups them for frepre-defined set of answer choices, quency analysis, with no prior
they are also traditionally very knowledge of brands expected in
the data (Fig. 3).
In many respects, AI and Machine Learning are powerful tools
to extract insights from data. In
the game of Consumer Insights,
the better you are at using these
tools, the closer you get to the
goal of understanding consumer
utility functions.
Authors
Aurélie Mei-Hoa Beaumel is the
CEO of BIG. She has BA/MS degrees from Stanford University in
Economics and Cognitive Science,
and has multi-year experience in International Business Strategy.
Thomas Dillig is the President of
BIG. He has a PhD from Stanford
University in Computer Science, and
has a decade of experience in cuttingedge Computer Science research at
top academic institutions.
For more information, visit our
website at www.bi-games.com and
contact us at [email protected].
“What is your favorite brand?”
50 shades of Louis Vuitton
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
louie v
louie vaton
louie vatton
louie vitton
louie voutton
louie vuitton
louie vuttion
louie vutton
louisvutton
louis vatton
louis veton
louis vetton
louis vinton
louis viotton
louis vitaan
louis vouton
louis vouttan
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
louis vouttion
louis voutton
louis vuiton
louis vuittion
louise vuitton
lous vouitton
lous vuitton
lousi vuitton
louis vitan
louis viton
louis vittan
louis vittion
louis vitton
louis vittuon
louis viuton
louis viutton
louis voiuttan
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
luis vuttion
louis votton
louis vouitan
louis vouiton
louis vouitton
louis voutain
louis voutin
louis vouton
luie viton
luis vatton
luis vitton
luis vittons
luis vouton
luis vuiton
luis vuitoon
luis vuitton
BIG Automatic
Free-Text Technology
Output
Louis Vuitton
n = 50
Algorithm groups all
(mis)spellings of a brand
for frequency analysis
Brand Name
Frequency
Fig. 3: Example of Automatic Free-Text Analysis
c 2016 Business Interactive Games Inc. All rights reserved. http://www.bi-games.com
Copyright