MarketTools External Presentation Template

Once a “Bad” Respondent, Always a “Bad”
Respondent
How to Identify and Exclude “Bad” Respondents from Research Studies
Nallan Suresh, MarketTools Inc.
CASRO Online Conference
Las Vegas, Mar 3-4, 2011
©2011 MarketTools, Inc. All Rights Reserved. These materials
may not be reproduced without permission from MarketTools, Inc.
Who Is MarketTools?
Founded
Origins
Broad Client
Base
December 1997
Pioneered the first solutions for online surveys,
sample data quality, and communities
80% of the Fortune 500 including:
“MarketTools
scored higher
than any other
vendor on its
technology.”
- Forrester
Research
Global
Presence
460 employees in 10 locations across the
United States, Canada, United Kingdom, and India
Headquarters in San Francisco, CA
“MarketTools is
one of our
„10 to watch‟.”
- Outsell
Strong
Financials
Forecasting ~$90M in 2011 sales
Record profitability 2009, 2010, and 2011
© 2011 MarketTools, Inc. All Rights Reserved. These materials may not be reproduced without permission from MarketTools, Inc.
2
Who Are Chronically Unengaged Respondents And
Why Are They Important?
•
Chronically unengaged respondents or “Bad” respondents can be
identified when tracked across surveys
•
•
•
The quality of the data from these respondents is shown to be
compromised
•
•
•
•
Unengaged respondents (typically 2-5% of sample) can be identified at a survey
level and quarantined for the survey
About 20% of respondents who have ever been flagged as unengaged can be
qualified as “chronically unengaged
These respondents provide more satisficing responses
Are more likely to select “middle of the road” responses
Are more likely to display much higher intent to buy
We can set threshold to identify chronically unengaged respondents
and exclude them from future surveys
•
•
These respondents tend to be heavier survey takers
More number of surveys are in jeopardy of having poorer data quality
© 2011 MarketTools, Inc. All Rights Reserved. These materials may not be reproduced without permission from MarketTools, Inc.
3
Phase I: Once A “Bad” Respondent …
What is a “bad” or unengaged respondent?
Is there a demographic bias in these respondents?
Do they have an impact on data quality?
© 2011 MarketTools, Inc. All Rights Reserved. These materials may not be reproduced without permission from MarketTools, Inc.
4
Unengaged Respondents Are Those That Speed And
Straightline On Several Pages
Speeders spend less time on
page than norm
Straightliners pick same
answer choice and speed
2s
Time Spent On Page
Unengaged for survey if they:
•
•
Speed on > 40% of pages or
Straightline on >25% grid-question pages
© 2011 MarketTools, Inc. All Rights Reserved. These materials may not be reproduced without permission from MarketTools, Inc.
5
Analysis Was Performed On A Robust Set Of Data
Over 35000 interviews
Several Product/Service
Categories
Spanning Several Months
Attitudinal questions – general or product
Intent to buy
Survey Rating
© 2011 MarketTools, Inc. All Rights Reserved. These materials may not be reproduced without permission from MarketTools, Inc.
6
Unengaged Respondents More Likely To Be Male, Younger
Gender
Age
30-40%
greater
Typically 2-5% of respondents per survey are unengaged
© 2011 MarketTools, Inc. All Rights Reserved. These materials may not be reproduced without permission from MarketTools, Inc.
7
Unengaged Respondents Tend To “Satisfice”
They rate surveys higher
They select highest intent
to buy
20%
greater
% of respondents that select the choice
*Data was re-balanced for age and gender
© 2011 MarketTools, Inc. All Rights Reserved. These materials may not be reproduced without permission from MarketTools, Inc.
8
On Attitudinal Questions Unengaged Respondents
Tend To Appear More Apathetic
They pick the middle choice
Or split between middle & high
or
% of respondents that select the choice
© 2011 MarketTools, Inc. All Rights Reserved. These materials may not be reproduced without permission from MarketTools, Inc.
9
Do We Need To Set Different Time Thresholds For
Marking Younger Respondents As Speeders?
Not at all.
Unengaged <35
Yr Old
Respondents
based on
population
benchmarks are
still different from
engaged <35 Yr
Old Respondents
% of respondents that select the choice
© 2011 MarketTools, Inc. All Rights Reserved. These materials may not be reproduced without permission from MarketTools, Inc.
10
Phase II: … Always A “Bad” Respondent?
What is a “chronically” unengaged respondent?
Is there a demographic bias in these respondents?
Do we need to exclude them from future surveys?
© 2011 MarketTools, Inc. All Rights Reserved. These materials may not be reproduced without permission from MarketTools, Inc.
11
Analysis Was Performed On Data Collected Over Time
Over 20000 unengaged
respondents
Spanning Several Months
Several Product/Service
Categories
Across over 1600 surveys
Tracked these
respondents over time
and surveys
© 2011 MarketTools, Inc. All Rights Reserved. These materials may not be reproduced without permission from MarketTools, Inc.
12
%Pages Sped - On surveys they are not “caught”
Unengaged Respondents Are More Likely To Speed
“Under The Radar” On Other Surveys As Well
Unengaged at least once
Always engaged
Just because they
don’t cross the
threshold does not
mean they are
engaged – their data
could still be
compromised!
% Surveys unengaged - On surveys they are “caught”
© 2011 MarketTools, Inc. All Rights Reserved. These materials may not be reproduced without permission from MarketTools, Inc.
13
How Do We Identify Respondents As “Chronically”
Unengaged?
We can set a threshold to identify chronically unengaged respondents and
exclude them from future surveys
Occasionally
Unengaged
Chronically
Unengaged
Dropout Rate
# Surveys Taken
© 2011 MarketTools, Inc. All Rights Reserved. These materials may not be reproduced without permission from MarketTools, Inc.
14
Chronically Unengaged Respondents Tend To Be Younger
Than Occasionally Unengaged Ones
© 2011 MarketTools, Inc. All Rights Reserved. These materials may not be reproduced without permission from MarketTools, Inc.
15
How Many Survey Takers Did We Tag As Chronically
Unengaged In This Study?
© 2011 MarketTools, Inc. All Rights Reserved. These materials may not be reproduced without permission from MarketTools, Inc.
16
Conclusions
> Chronically unengaged respondents or “bad” respondents
can be identified when tracked across surveys
> The quality of the data from these respondents is shown to
be compromised
> We can set a threshold to identify chronically unengaged
respondents and exclude them from future surveys
© 2011 MarketTools, Inc. All Rights Reserved. These materials may not be reproduced without permission from MarketTools, Inc.
17
Thank you.
© 2011 MarketTools, Inc. All Rights Reserved. These materials may not be reproduced without permission from MarketTools, Inc.
18