PDF of Complete Presentation - Measuring the User Experience

Top Ten Myths
about Usability
Thomas S. Tullis, Ph.D.
VP, Usability & User Insight
Fidelity Investments
[email protected]
Simmons College
November 12, 2009
1
2
Original photo: http://www.adamsavage.com/images/pix/mythbusters.jpg
Myth #1:
Usability is Just
Common Sense
Curved Barrel
Machine Gun
Life Magazine, 1953
For shooting around
corners, where you can’t
3
see.
If usability is just
common sense,
then why do we
have elevators like
this…
4
Which of the following
methods for re-ordering
fields (e.g., columns of a
table) is the most effective?
Which is the least
effective?
5
6
7
8
A
B
C
D
E
F
G
9
Mean Task Times (secs)
(Error bars represent the 90% confidence interval)
The Loser
14
12
10
8
6
4
2
0
A.
Drag/Drop
On
B.
Drag/Drop
Between
C. Icons
D. Radio
Buttons
E. Dropdown
Menus
F. One Entry
G. Four
Field
Entry Fields
The Winners
10
Actual Times vs. Predictions
6
C. Icons
r = -.07
Predictions (Lower=Better)
F. One Entry Field
5
G. Four Entry Fields
D. Radio Buttons
4
E. Drop-down Menus
3
A. Drag/Drop On
B. Drag/Drop Between
2
4
5
6
7
8
9
10
11
12
13
Actual Times (secs)
Tullis, T.S. (1993). Is User Interface Design Just Common Sense? Proceedings of the Fifth
International Conference on Human-Computer Interaction, (HCI International '93), Orlando, Florida,
USA, August 8-13, 1993, Volume 2. Elsevier 1993, ISBN 0-444-89540-X, pp. 9-14.
11
Myth #2: Usability is Just About
Making Things Look Nice
Sleeper (1973)
12
Beautiful doesn’t necessarily mean usable.
Usable doesn’t necessarily mean beautiful.
But beautiful doesn’t preclude usable,
nor does usable preclude beautiful.
Teapot theme inspired by Don Norman’s The Design of Everyday Things 13
Which of these sites
is more visually
appealing?
14
15
ACSI Score vs. Visual Appeal Rating
4.00
Neiman Marcus
Banana Republic
Visual Appeal Rating
3.50
Apple
r = 0.21
3.00
2.50
Amazon
2.00
1.50
NewEgg
1.00
65
70
75
80
85
90
ACSI Score
Tullis, T.S., & Tullis, C.M. (2007). Statistical Analyses of E-commerce Websites: Can a Site Be
Usable and Beautiful? Proceedings of HCI International 2007 Conference, Beijing.
16
17
Myth #3:
Usability
Can’t Be
Measured
18
Photo: http://www.flickr.com/photos/minieng/2309432198/
Tobii 1750 eyetracking monitor
Video
19
Which of these designs
for a messaging area
would attract the most
attention?
20
Design A
21
Design B
22
Design C
23
Design D
24
Design A
25
Design B
26
Design C
27
Design D
28
% of Participants Who Fixated Area for a Total of
At Least 500 msec
60%
50%
40%
30%
20%
10%
0%
Design A
Design B
Design C
Design D
29
Some traditional usability metrics:
Percent Task Success
100%
96.5%
Mean
92.7%
97.8%
Task
Time (secs)
86.4%
90%
80%
70%
60%
140
120
40%
20%
100
10%
0%
4.5
4.2
4.1
4.0
Frequency Distribution
of SUS Scores
4.0
4.0
3.5
2.9
18%
3.0
16%
60
2.5
14%
12%
40
NASA
2.0
10%
20
Wikipedia
8%
1.5
Task 1
Task 2
Task 3
Task 4
Task 5
06%
1.0
Task 2
Task 3
Task 4
Task 5
4% Task 1
Task 1
Task 2
Task 3
Task 4
Task 5
2%
0%
0-10 11- 21- 31- 41- 51- 61- 71- 81- 9120 30 40 50 60 70 80 90 100
80
% of Participants
20%
Mean Task Ease Rating (1-5, Higher=Better)
160
50%
30%
79.1%
180
SUS Score
30
Some less-traditional usability metrics:
Trimmel, M., Meixner-Pendleton, M., & Haring, S. (2003). Stress response caused
by system response time when searching for information on the Internet :
Psyochophysiology in ergonomics. Human Factors, Vol. 45, No. 4, pp. 615–621.
31
Some less-traditional usability metrics:
Who can tell me what this is?
Hint: This is what measures it!
Kapoor, A., Mota, S., & Picard, R. (2001). Towards a learning companion that recognizes affect.
In AAAI Fall Symposium, November.
32
Some less-traditional usability metrics:
Reynolds, C.J. (2005). Adversarial Uses of Affective Computing and Ethical Implications. Ph.D.
Thesis, MIT.
33
Myth #4:
Usability
Means
Usability
for the
“Average”
Person
Photo:
http://www.it4education.at/fubbcontent/lektionen
/E/adjectives/adjquiz/tall_short.jpg
34
Several years ago, I was
building a pool shed…
One of the decisions I
had to make was how
tall to make the
35
doorway.
What’s the average height of an American?
About 66.5”
So if I make the doorway about 67” high…
36
But I don’t want
President Obama
to bump his head
in the doorway
when he comes
over to swim at
my house…
Photo:
http://www.daylife.com/photo/08SRdCmdKh0M3
37
Height (inches)
Height Percentiles, in inches, for Americans (2002)
75
74
73
72
71
70
69
68
67
66
65
64
63
62
61
60
59
58
57
56
55
Women
Men
5th
10th
15th
25th
50th
75th
85th
90th
95th
Percentile
Source: http://www.cdc.gov/nchs/data/ad/ad361.pdf
38
What’s the Real Answer?
• Doors are a standard height: 80”
• That’s not the 95th percentile of the
population height– it’s more like the
99.9th percentile.
• Architecture has evolved to a level of
maturity where they have:
– Standards
– Standard building blocks
• And it’s much cheaper to make the
doorway 80” high from the beginning
than to go back later and retrofit one in!
39
So unless my
good buddy
Yao Ming
comes over
for a swim, I
should be OK!
Bottom Line: We
should be making
the “doors” to our
websites and other
products “taller”.
40
Original photo: http://www8.garmin.com/yao/content/media.html
In the year 2000 U.S.
census, what percentage
of Americans reported
that they have a
disability?
A.
B.
C.
D.
5%
10%
15%
20%
41
42
How it might look to
someone with cataracts
43
A Wikipedia article to
someone with cataracts.
44
But how it can look to them
by enlarging the text using
browser text sizing.
45
How does someone who’s blind use the web?
Demo
Using screen-reading software.
46
So who is your
“average user”?
Or is it this 68-year-old man
who is a college graduate,
has a disability, has
broadband access, and
speaks English at home?
Is it this 45-year-old woman
who is a high school
graduate, has no disabilities,
dial-up access, and speaks
English at home?
Data from the Year
2000 Census about
100 Representative
Americans…
47
My Stunning Conclusion from 6 Years of
Graduate Training in Human Factors and
30+ Years of Work Experience in the
Usability Field…
People are different!
And to achieve true usability for the
broadest range of people, we need to
pay attention to those differences in the
design of our products and services.
48
Myth #5: Usability doesn’t have
any real impact on our world.
Photo credit: http://static.panoramio.com/photos/original/8071466.jpg
49
Three Mile Island, March 29, 1979
Photos: http://www.animatedsoftware.com/hotwords/control_room/control_room.htm
50
This valve was stuck open,
leading to loss of a large
amount of reactor coolant,
which caused a partial core
meltdown.
The indicator light showing
the status of the valve was
actually showing what the
valve had been commanded
to do, not its actual state.
Bottom Line: Poor usability almost caused a nuclear meltdown.
Drawing source: http://www.nrc.gov/reading-rm/doc-collections/fact-sheets/tmi-2.gif
51
Another Example…
In a quick study done by Canadian researchers right after
the U.S. election, they found that 7% of shoppers “voted” for
the wrong candidate when using this kind of butterfly ballot.
Sinclair, R.C., Mark, M.M., Mooer, S.E., Lavis, C.A., & Soldat, A.S. (2000) Nature, 408, pp 665-666.
52
And…
53
Photo: http://www.usagold.com/wallpaper/coins_stack_43_lg.jpeg
Myth #6:
Usability costs
too much.
54
“Formal” Usability Testing…
55
There’s a wide range of options…
Heuristic Evaluation or Expert
Review
Photo: http://images.inmagine.com/img/inspirestock/ispi084/ispi084131.jpg
Photo: http://www.ergosign.de/en/user-centereddesign/usability-evaluation/expert-color.jpg
“Drive-by” usability testing
“Café” studies (or
Starbucks, mall, etc)
Photo: http://staging.mddailyrecord.com/generationjd/files/2009/08/fe0791126d7ef6ba_man-laptop-starbucks.jpg
56
Lab Studies
From 5-6
participants to
many more
Simmons College Usability Lab: http://gslis.simmons.edu/usability/equipment.html
57
Online Usability
Studies
58
Online Usability Studies
• Conducted over the web.
• Users participate from their homes or
offices at a time they choose.
• They’re asked to go through some tasks
using a website or prototype.
• Data collected:
– Task success
– Task times
– Subjective ratings
– Clickstream data*
* Depending upon the tool being used.
59
Results of Obama vs. McCain Website
Evaluation
Total effort to do this
online study: about 6
person-hours.
Efficiency: % Success per Minute
90%
80%
70%
60%
Overall Ratings
50%
40%
7.0
20%
10%
0%
Obama
McCain
Rating (1-7, Higher=Better)
30%
6.0
5.0
Obama
4.0
McCain
3.0
2.0
1.0
Visual Appeal
Details: http://www.measuringux.com/Obama-McCain/index.htm
Ease of Finding
60
Some Tools for Conducting
Online Usability Studies
Or you can “Roll Your Own” either very cheaply or for free:
http://www.measuringux.com/RollingYourOwnOnlineStudy.ppt
61
Online Usability Studies
To be published
January 29, 2010, by
Morgan Kaufmann.
62
Myth #7:
This must
be usable
because
we built it
in Flash
(or AJAX,
etc)
Life magazine, 1950
63
Using the latest
cool technology
doesn’t make
something usable.
Inventor Hugo Gernsback with his
T.V. Glasses.
Photo: Alfred Eisenstaedt/Time &
Life Pictures/Getty Images
Jan 01, 1963
64
The default settings in many
new tools are often terrible from
a usability perspective.
65
66
Myth #8:
Usability is
a fad. This
too shall
pass.
Life Magazine
Photo: Bill Ray
Jan 01, 1972
67
The B-17 “Flying Fortress”
Usability’s heritage is in the human factors
field, which began to emerge as a
discipline during World War II.
68
Photo: http://www.military.cz/usa/air/war/bomber/b17/b17_infl.jpg
B-17 coming in for a landing,
with its landing gear down and
flaps fully extended.
Photo: http://lh5.ggpht.com/_1QUegZfskfE/R9QtQ6aREfI/AAAAAAAABco/jVkwCu04BOU/IMG_0007.jpg
69
But they would sometimes
come in like this…
And crash land like this…
70
Photo: http://www.historyarticles.com/B17_COCKPIT.jpg
The B-17 Cockpit
One of these controls
extends the flaps.
Another virtually
identical one raises
the landing gear.
71
As a result of the work of the
human-factors engineers, shapecoded controls were introduced
into cockpits:
72
In the 1970’s, computers started to
introduce new challenges for human
factors people to solve…
73
So the early 1970’s was
really the birth of
interactive systems,
which led to the first
book, in 1973, on the
design of these systems
from the perspective of
the user…
(As long as the
user was male!)
74
Early Pointing Devices
75
And then in the 1980’s along
came personal computers
and solved all the usability
problems…
76
And then in the 1990’s along came the web and once again
solved all the usability problems…
A web page from 1992
by Tim Berners-Lee
Bottom Line: As technologies
evolve, they create new
usability problems that didn’t
exist before. Usability issues
aren’t going away, just evolving.
77
Graphic: http://konstructr.com/wp-content/uploads/2009/08/mpj043588000001.jpg
Myth #9: Usability doesn’t
impact the bottom line.
78
If usability doesn’t impact the bottom line,
then someone should tell Microsoft…
Just a few of the
billboard ads at
South Station,
Boston, advertising
Windows 7.
79
Usability ROI Case Study
Task Times for Old vs. New Site
Average Time (sec)
300
250
They calculated that
the usability
improvements
yielded an annual
savings of $1.2
million, plus an
additional $550k of
revenue annually.
200
Old Site
New Site
150
100
50
Nu
rs
e
lic
en
Fo
St
ur
at sur
ye e M e
ar
a
co p *
lle
Vo St ge s
a
te
*
r r te b
eg
i rd
is
tra *
U.
St
S.
tio
ar
l
tin eg n *
is
g
la
a
t
b
Ro
us ors
ad
in
co ess
Hu nd
*
itio
nt
in
ns
g
*
lic
en
se
*
0
Withrow, J., Brinck, T., and Speredelozzi, A.
Comparative Usability Evaluation for an eGovernment Portal, Diamond Bullet Design
Report, #U1-00-2, Ann Arbor, MI. Dec. 2000.
Task
Redesign of a
state government
portal.
80
Myth #10:
Usability only
applies to
computers
and web
sites.
81
The good…
Oxo measuring
cup in action! 82
The bad…
Photo: http://www.treehugger.com/bad-packaging-design-individually-wrapped-bananas-teaser-photo.jpg
The good…
83
The good…
The bad…
PaperPro springloaded stapler.
Plastic packages that require a
special tool to open.
84
The bad…
Any clock radio that requires a
user manual to operate.
Photo credits: ThisIsBroken.com
85
The good and the bad in the same
product…
The serial number on
the back of the iPod
that some apps
require you to enter.
86
To Recap: Top 10 Myths about Usability
•
•
•
•
•
•
•
•
•
•
Myth #1: Usability is Just Common Sense
Myth #2: Usability is Just About Making Things Look Nice
Myth #3: Usability Can’t Be Measured
Myth #4: Usability Means Usability for the “Average”
Person
Myth #5: Usability Doesn’t Have Any Real Impact on our
World
Myth #6: Usability Costs Too Much
Myth #7: This Must Be Usable Because We Built It in
Flash (or AJAX, etc)
Myth #8: Usability is a Fad. This Too Shall Pass.
Myth #9: Usability Doesn’t Impact the Bottom Line
Myth #10: Usability Only Applies to Computers and Web
Sites.
87
A summary of this presentation will be
available from:
http://www.MeasuringUX.com/UsabilityMyths/
Thank You!
88