What Would MacGyver Do?

 W2
Track 10/7/2009 11:30 AM "What Would MacGyver Do?"
Presented by:
Harry Robinson
Microsoft
Brought to you by: 330 Corporate Way, Suite 300, Orange Park, FL 32073 888‐268‐8770 ∙ 904‐278‐0524 ∙ [email protected] ∙ www.sqe.com
Harry Robinson
Harry Robinson is Principal Software Development Engineer in Test for Microsoft's Bing team.
He has twenty years of software development and testing experience at AT&T Bell Labs,
Hewlett-Packard, Microsoft, and Google as well as time spent in the startup trenches. While at
Bell Labs, Harry created a model-based testing system that won the 1995 AT&T Award for
Outstanding Achievement in the Area of Quality. At Microsoft, he pioneered the model-based
test generation technology, which won the Microsoft Best Practice Award in 2001. Harry holds
two patents for software test automation methods, maintains the site www.model-basedtesting.org, and speaks and writes frequently on software testing and automation issues.
9/9/2009
What Would MacGyver Do?
Harry Robinson
Microsoft
Principal SDET, Bing
What Would Ed Smylie Do? Harry Robinson
Microsoft
Principal SDET, Bing
1
9/9/2009
April 14, 1970
“The contraption wasn't very handsome, but it worked.”
‐ James Lovell, Apollo 13
June 1, 2009
bing!
2
9/9/2009
Smart thumbs
yipes!
Popular now
Deals on underwater cameras
Plan your next vacation today
Obama education speech · <tacky phrase here> · Laura Bush · Tropica…
3
9/9/2009
Autosuggest
Deals on underwater cameras
Plan your next vacation today
Obama education speech · Congressional hearing · Laura Bush · Trop…
Spell correction
“porn viedo”
“porn viedo”
“porn video”
4
9/9/2009
Images/Shopping
We considered the usual options
•
•
•
•
•
Testers
Automation scripts
A huge test lab
Relevance judges
Bug bashes
… as well as some not‐so‐usual options
• Fuzzers
• Crowdsourcing
C
d
i
• Logged user queries
Desperation tends to make one sort of flexible – Angus MacGyver
5
9/9/2009
Welcome to the monkey trap
“The trap consists of a hollowed‐out coconut chained to a stake. The coconut has some rice inside which can be grabbed through a small hole. The hole is big enough so that the monkey's hand can go in, but too small for his fist with rice in it to come out. The k
h i
di
dd l t
d
monkey reaches in and is suddenly trapped ...
… what general advice would you give the poor monkey?
Well, he should … see if things he thought were important really were important and, well, stop yanking and just stare at the coconut for a while.
Before long he should get a nibble from a little fact
wondering if he is interested in it.”
‐Robert Pirsig
Zen and the Art of Motorcycle Maintenance
When something's broken, the easiest thing is to throw it away, Wh
hi ' b k
h
i
hi i
h
i
forget about it. But if you just step back and take a look at what you've got, you find a totally different way for it to work.
‐ Angus MacGyver
6
9/9/2009
Components of creativity
Expertise
Motivation
Creativity
Creative‐thinking skills
“Creative thinking … refers to how people approach problems and solutions – their capacity to put existing ideas together in new combinations.”
‐ Teresa Amabile
Motivating creativity in organizations
Motivation
7
9/9/2009
Expertise
•
•
•
•
•
•
Lawyers
Project managers
Domain experts
Feature testers
Senior testers
Automation specialists
Conceptual blocks
“… mental walls that block the problem‐solver from correctly perceiving a problem or conceiving its solution.” ‐ James L. Adams
Conceptual Blockbusting
p
g
Perceptual
•
•
•
•
Stereotyping
Difficulty isolating the problem
Delimiting the problem too closely
Saturation
Cultural
•
•
•
•
Taboos
Humor (lack of)
Reason vs intuition
Tradition vs change
Emotional
•
•
•
•
•
Fear of risk
No appetite for chaos
Judging rather than generating ideas
Inability to incubate
Complacency
Environmental
•
•
•
•
Supportive team
Supportive management
Incorporating feedback
Hidden agendas
8
9/9/2009
Voices in our heads: automation
“Automated tests represent the automation of a manual process.” – Harold F. Tipton, Micki
Harold F Tipton Micki Krause
“The most important benefit of automated testing over conventional manual testing is the minimization of costs over repeated tests.” – Markus Helfen, Michael Lauer
“We
We have found that it is essential for tests to execute very quickly; our target is have found that it is essential for tests to execute very quickly; our target is
under 30 seconds for the typical test run.” – Shaun Smith, Gerard Mezaros
The less you know sometimes, the better. – Angus MacGyver
Voices in our heads: new bugs
“I have never been convinced that finding ‘new’ bugs is a realistic expectation for test automation.” – I.M. Testy
“After you have run your automated tests for the first time, you are done finding new bugs. Never again will you find a new issue. ” – Steve Rowe
“Running automated test scripts can not be used to find new bugs in the software …” – Cordell Vail
9
9/9/2009
Voices in our heads: test oracles
“Oracle: A mechanism to produce the
p
predicted outcomes to compare with p
p
the actual outcomes of the software under test.” – BS 7925‐1
“A test oracle is a mechanism for predicting the expected results.”
‐ Andreas Spillner; Tilo Linz; Hans Schaefer
“In real testing the outcome is predicted and documented before the test is run.” – Boris Beizer
i
i
Challenge assumptions
•
•
•
•
•
•
•
Is it necessary to predict the outcome of a test?
Must an automated test behave like a human?
Must an automated test behave like a human?
Can random inputs be used well?
Does a test program really need to run quickly?
How do humans judge if a result is correct?
How efficient must a test program be?
Why can’tt a test program be used for exploratory testing?
Why can
a test program be used for exploratory testing?
Never laugh at what you don't know. – Angus MacGyver
10
9/9/2009
Blockbusting
•
•
•
•
•
•
•
Challenge assumptions
Investigate the problem
Question all constraints
Look at object attributes
Brainstorm ideas
Who else has done something like this?
Imagine you have unlimited resources …
Imagination is the most important thing the human mind has. – Angus MacGyver
Part 1: Break the problem up
11
9/9/2009
Part 2: Smart thumbs/Popular now
• Smart thumbs:
– Create explicit domain
• Popular now:
– Validate upon data entry
Validate upon data entry
Part 3: Autosuggest
• Bug bash?
• Generate and check?
G
t
d h k?
– Input from
• Log data
• Exhaustive combinations
– Check with
• Homemade utility
Homemade utility
• PoliCheck
• Simple solution: Check source file
12
9/9/2009
Part 4: Spell correction
• Generate and check
– Input from
• Log data
• Mutated combinations
– Check with
• Homemade utility
• PoliCheck
Part 5: Images/Shopping
• Generate and check
– Input from
• Log data
• Exhaustive combinations
– Check with
• PoliCheck
• Homemade utilities
• Autonomation (people + machines)
If you don't have the right equipment for the job, you just have to make it yourself.
– Angus MacGyver
13
9/9/2009
What should you do?
• Approach test as an engineering problem
• Study up on
– Test approaches
– Your product
– Available tools
– Creative problem‐solving
• Testing books: Beizer, Binder, Jorgensen
• Papers: Test generation, heuristic oracles, combinatorics
h
l
b
Always carry around a knapsack to pick up things you find along the way. – Angus MacGyver
April 19, 2005
Ed Smylie accepts the first Great Moments in Engineering award
on behalf of NASA’s Crew Systems Division who improvised an air filter
that kept astronauts alive on the ill‐fated mission 35 years ago.
Any problem can be solved with a little ingenuity. – Angus MacGyver
14
9/9/2009
The most important tool for survival isn't duct tape or a Swiss Army Knife, it's your wits. Always keep them about you.
– Angus MacGyver
15
10/7/2009
What Would MacGyver Do?
Harry Robinson
Microsoft
Principal SDET, Bing
What Would Ed Smylie Do? Harry Robinson
Microsoft
Principal SDET, Bing
1
10/7/2009
April 14, 1970
“The contraption wasn't very handsome, but it worked.”
‐ James Lovell, Apollo 13
June 1, 2009
bing!
2
10/7/2009
Smart thumbs
yipes!
Popular now
Deals on underwater cameras
Plan your next vacation today
Obama education speech · tacky phrase goes here · Laura Bush · Trop…
3
10/7/2009
Autosuggest
Deals on underwater cameras
Plan your next vacation today
Obama education speech · Congressional hearing · Laura Bush · Trop…
Spell correction
“porn viedo”
“porn viedo”
“porn video”
4
10/7/2009
Images/Shopping
We considered the usual options
•
•
•
•
•
Testers
Automation scripts
A huge test lab
Relevance judges
Bug bashes
… as well as some not‐so‐usual options
• Fuzzers
• Crowdsourcing
C
d
i
• Logged user queries
Desperation tends to make one sort of flexible – Angus MacGyver
5
10/7/2009
Welcome to the monkey trap
“The trap consists of a hollowed‐out coconut chained to a stake. The coconut has some rice inside which can be grabbed through a small hole. The hole is big enough so that the monkey's hand can go in, but too small for his fist with rice in it to come out. The monkey reaches in and is suddenly trapped ...
k
h i
di
dd l t
d
Welcome to the monkey trap
“The trap consists of a hollowed‐out coconut chained to a stake. The coconut has some rice inside which can be grabbed through a small hole. The hole is big enough so that the monkey's hand can go in, but too small for his fist with rice in it to come out. The k
h i
di
dd l t
d
monkey reaches in and is suddenly trapped ...
… what general advice would you give the poor monkey?
6
10/7/2009
Welcome to the monkey trap
“The trap consists of a hollowed‐out coconut chained to a stake. The coconut has some rice inside which can be grabbed through a small hole. The hole is big enough so that the monkey's hand can go in, but too small for his fist with rice in it to come out. The k
h i
di
dd l t
d
monkey reaches in and is suddenly trapped ...
… what general advice would you give the poor monkey?
Well, he should … see if things he thought were important really were important and, well, stop yanking and just stare at the coconut for a while.
Before long he should get a nibble from a little fact
wondering if he is interested in it.”
‐Robert Pirsig
Zen and the Art of Motorcycle Maintenance
When something's broken, the easiest thing is to throw it away, Wh
hi ' b k
h
i
hi i
h
i
forget about it. But if you just step back and take a look at what you've got, you find a totally different way for it to work.
‐ Angus MacGyver
7
10/7/2009
Components of creativity
Expertise
Motivation
Creativity
Creative‐thinking skills
“Creative thinking … refers to how people approach problems and solutions – their capacity to put existing ideas together in new combinations.”
‐ Teresa Amabile
Motivating creativity in organizations
Motivation
8
10/7/2009
Expertise
•
•
•
•
•
•
Lawyers
Project managers
Domain experts
Feature testers
Senior testers
Automation specialists
Creative‐thinking skills
“Conceptual blocks are mental walls that block the problem‐solver from correctly perceiving a problem or conceiving its solution.”
‐ James L. Adams
Conceptual Blockbusting
Perceptual
•
•
•
•
Stereotyping
Difficulty isolating the problem
Delimiting the problem too closely
Saturation
Cultural
•
•
•
•
Taboos
Humor (lack of)
Reason vs intuition
Tradition vs change
Emotional
•
•
•
•
•
Fear of risk
No appetite for chaos
Judging rather than generating ideas
Inability to incubate
Complacency
Environmental
•
•
•
•
Supportive team
Supportive management
Incorporating feedback
Hidden agendas
9
10/7/2009
Ping‐Pong Pipe Problem
A steel pipe is embedded in the concrete floor of a bare room… The inside diameter is 0.06” larger than the diameter of the ping‐pong ball (1.50”) that is resting at the bottom of the pipe. You are one of a group of six people in the room, along with the following objects:
• 100 feet of clothesline
• A box of granola
• A light bulb
• A file
• A carpenter’s hammer
• A wire coat hanger
• A chisel
• A monkey wrench
List as many ways as you can think of to get the ball out of the pipe without damaging the ball, the pipe or the floor.
from Conceptual Blockbusting
by James L. Adams
Ping‐Pong Pipe Problem
A steel pipe is embedded in the concrete floor of a bare room… The inside diameter is 0.06” larger than the diameter of the ping‐pong ball (1.50”) that is resting at the bottom of the pipe. You are one of a group of six people in the room, along with the following objects:
• 100 feet of clothesline
• A box of granola
• A light bulb
• A file
• A carpenter’s hammer
• A wire coat hanger
• A chisel
• A monkey wrench
List as many ways as you can think of to get the ball out of the pipe without damaging the ball, the pipe or the floor.
from Conceptual Blockbusting
by James L. Adams
Notice that having tools can distract you from a good solution!
10
10/7/2009
So, Are Tools Bad?
MacGyver Gets Lazy
http://xkcd.com/444/
You can do more with a gun that just fire it. – Angus MacGyver
Voices in our heads: automation
“Automated tests represent the automation of a manual process.” – Harold F. Tipton, Micki
Harold F Tipton Micki Krause
“The most important benefit of automated testing over conventional manual testing is the minimization of costs over repeated tests.” – Markus Helfen, Michael Lauer
“We
We have found that it is essential for tests to execute very quickly; our target is have found that it is essential for tests to execute very quickly; our target is
under 30 seconds for the typical test run.” – Shaun Smith, Gerard Mezaros
The less you know sometimes, the better. – Angus MacGyver
11
10/7/2009
Voices in our heads: new bugs
“I have never been convinced that finding ‘new’ bugs is a realistic expectation for test automation.” – I.M. Testy
“After you have run your automated tests for the first time, you are done finding new bugs. Never again will you find a new issue. ” – Steve Rowe
“Running automated test scripts can not be used to find new bugs in the software …” – Cordell Vail
Voices in our heads: test oracles
“Oracle: A mechanism to produce the
p
predicted outcomes to compare with p
p
the actual outcomes of the software under test.” – BS 7925‐1
“A test oracle is a mechanism for predicting the expected results.”
‐ Andreas Spillner; Tilo Linz; Hans Schaefer
“In real testing the outcome is predicted and documented before the test is run.” – Boris Beizer
i
i
12
10/7/2009
Challenge assumptions
Is it necessary to predict the outcome of a test?
Must an automated test behave like a human?
Must an automated test behave like a human?
Can random inputs be used well?
Does a test program really need to run quickly?
How do humans judge if a result is correct?
How efficient must a test program be?
Why can’tt test automation be used for exploratory testing?
Why can
test automation be used for exploratory testing?
•
•
•
•
•
•
•
Never laugh at what you don't know. – Angus MacGyver
Blockbusting
•
•
•
•
•
•
•
•
Challenge assumptions
Investigate the problem
Question all constraints
Look at object attributes
Brainstorm ideas
Who else has done something like this?
Imagine you have unlimited resources …
I
Imagine you just arrived from Mars
i
j
i df
M …
Imagination is the most important thing the human mind has. – Angus MacGyver
13
10/7/2009
Part 1: Break the problem up
Part 2: Smart thumbs/Popular now
• Smart thumbs:
– Create explicit domain
• Popular now:
– Validate upon data entry
Validate upon data entry
14
10/7/2009
Part 3: Autosuggest
• Bug bash?
• Generate and check?
G
t
d h k?
– Input from
• Log data
• Exhaustive combinations
– Check with
• Homemade utility
Homemade utility
• PoliCheck
• Simple solution: Check source file
Part 4: Spell correction
• Generate and check
– Input from
• Log data
• Mutated combinations
– Check with
• Homemade utility
• PoliCheck
15
10/7/2009
Part 5: Images/Shopping
• Generate and check
– Input from
• Log data
• Exhaustive combinations
– Check with
• PoliCheck
• Homemade utilities
• Autonomation (people + machines)
If you don't have the right equipment for the job, you just have to make it yourself.
– Angus MacGyver
Part 6: Bing overall
• Use Best‐of‐Breed as a test oracle
Commercial
filter
Results
Do these
results differ?
web
Results
16
10/7/2009
Narrator: “It had been an unavoidable accident; one of those rare times when fog forms right out of mid‐air and in five minutes the visibility goes from ten miles down to zero. There had been no airport nearby; he couldn’tt land. Unavoidable.
zero. There had been no airport nearby; he couldn
land. Unavoidable.”
Narrator: “It had been an unavoidable accident; one of those rare times when fog forms right out of mid‐air and in five minutes the visibility goes from ten miles down to zero. There had been no airport nearby; he couldn’tt land. Unavoidable.
zero. There had been no airport nearby; he couldn
land. Unavoidable.”
Outlaw: “[Your] rule number eight told him never to land away from an airport. In his last five minutes of visibility, he flew over eight hundred thirty‐seven landing places –
smooth fields and level pastures – but they were not ‘designated airports’ … so he didn’t even think about landing … 17
10/7/2009
Narrator: “It had been an unavoidable accident; one of those rare times when fog forms right out of mid‐air and in five minutes the visibility goes from ten miles down to zero. There had been no airport nearby; he couldn’tt land. Unavoidable.
zero. There had been no airport nearby; he couldn
land. Unavoidable.”
“Outlaw”: “[Your] rule number eight told him never to land away from an airport. In his last five minutes of visibility, he flew over eight hundred thirty‐seven landing places –
smooth fields and level pastures – but they were not ‘designated airports’ … so he didn’t even think about landing … “It’s up to us to keep flight alive in a world of airplane drivers – the people who come out of your school, concerned only with moving straight and level from airport to airport. We’re trying to keep a few real pilots left in the air. “
Narrator: “It had been an unavoidable accident; one of those rare times when fog forms right out of mid‐air and in five minutes the visibility goes from ten miles down to zero. There had been no airport nearby; he couldn’tt land. Unavoidable.
zero. There had been no airport nearby; he couldn
land. Unavoidable.”
“Outlaw”: “[Your] rule number eight told him never to land away from an airport. In his last five minutes of visibility, he flew over eight hundred thirty‐seven landing places –
smooth fields and level pastures – but they were not ‘designated airports’ … so he didn’t even think about landing … “It’s up to us to keep flight alive in a world of airplane drivers – the people who come out of your school, concerned only with moving straight and level from airport to airport. We’re trying to keep a few real pilots left in the air. “
‐ Richard Bach
A Gift of Wings
18
10/7/2009
What can you do?
• Approach test as an engineering problem
• Study up on
– Test approaches
– Your product
– Available tools
– Creative problem‐solving
• Testing books: Beizer, Binder, Jorgensen
• Papers: Test generation, heuristic oracles, combinatorics
h
l
b
Always carry around a knapsack to pick up things you find along the way. – Angus MacGyver
April 19, 2005
Ed Smylie accepts the first Great Moments in Engineering award
on behalf of NASA’s Crew Systems Division who improvised an air filter
that kept astronauts alive on the ill‐fated mission 35 years ago.
Any problem can be solved with a little ingenuity. – Angus MacGyver
19
10/7/2009
The most important tool for survival isn't duct tape or a Swiss Army Knife, it's your wits. Always keep them about you.
– Angus MacGyver
The most important tool for survival isn't duct tape or a Swiss Army Knife, it's your wits. Always keep them about you.
– Angus MacGyver
Thanks for attending!
20