slides

From the Lab to the Web:
Experiments on the Internet
By John Morgan
University of California, Berkeley
Economic Science Association World Meeting
Copenhagen, Denmark
10 July 2010
Overview
“The Internet is a tidal wave. It changes the rules. It is an incredible
opportunity as well as incredible challenge. I am looking forward to
your input on how we can improve our strategy to continue our track
record of incredible success.”
--Bill Gates, The Internet Tidal Wave Memo, May 26, 1995
1.
2.
3.
4.
5.
Found Experiments
DIY
Connections
A Lab without Walls
Going Native
Found Experiments
“Beautiful as the chance encounter
of a sewing machine and an
umbrella on a dissecting table.”
--Comte de Lautréamont
Transforming ordinary websites into
settings for controlled experiments.
Marcel Duchamp, Fountain, 1917
Photograph: Alfred Steiglitz
A Found Experimental “Lab”
eBay was never built
to be a “lab”
 eBay decides on the
rules of the game
 eBay changes its
rules from time to
time
 Perfect for running
certain experiments

Revenue Equivalence Theorem

Two auction forms are said to be revenue
equivalent if they produce the same expected
revenues in equilibrium.
◦ Key implication: Auctions which differ only in
their reserve price and shipping charge are
revenue equivalent when reserve + shipping
are the same.
◦ Ebay lets you vary the opening bid (reserve
price) and the shipping charge.
Hossain and Morgan (2006)
◦ Become eBay sellers
◦ Matched pairs of popular CDs and Xbox
games
 Varied opening bid and shipping charge
 Maintained (theoretical) revenue equivalence
◦ Controls
 Reputation
 Presentation
 Shipping terms
Low Reserve Treatments: CDs
Low Reserve Treatments: Revenue Comparison
18.00
16.00
14.00
Revenues
12.00
10.00
8.00
6.00
4.00
2.00
0.00
Music
Ooops! I Did it Again
Serendipity
O Brother Where
Art Thou?
Greatest Hits - Tim A Day Without Rain
McGraw
Automatic for the
People
Title of CD
Free Shipping
Source: Hossain and Morgan, BEJEAP 2006
$4 Shipping
Everyday
Joshua Tree
Unplugged in New
York
High Reserve Treatments: Xbox
50.00
45.00
40.00
35.00
Revenues
30.00
25.00
20.00
15.00
10.00
5.00
0.00
Halo
Wreckless
Circus Maximus
Max Payne
Genma
Onimusha
Project Gotham
Racing
NBA 2K2
NFL 2K2
Title of Xbox Gam e
$2 Shipping
Source: Hossain and Morgan, BEJEAP 2006
$6 Shipping
NHL 2002
WWF Raw
Testing Behavioral Models
◦ Gabaix and Laibson (2006) suggest that sellers
will optimally want to shroud (hide) aspects
of a product’s price.
 i.e. hotel mini-bar charges
◦ Brown, Hossain, and Morgan (QJE, 2010)
examined this in Taiwan, Ireland, and the US
 Sold identical batches of iPods
 Shrouded shipping charge in description of item
versus disclosing it in header
◦ Key result: Shrouding attracts more bidders,
increases revenues on higher shipping
DIY (Do It Yourself)

The Internet is a tidal wave. It changes the
rules.
◦ Low barriers to entry
◦ Loose organization
◦ Build your own lab on the fly
The Magic of Usenet

Usenet: Early bulletin
board service
◦ Many specialized groups
◦ Informal “trading post”
using many types of
auctions.

Magic: The Gathering
is a popular game in
the 90s.
◦ <rec.games.deckmaster>
Lucking-Reiley (1999)

Theory:
◦ Under IPV auction model, various auction
forms (1st price, Dutch, 2nd price, English) are
revenue equivalent.

Experiment:
◦ Becomes a Magic card seller
◦ (DIY) Chooses various auction forms for
identical cards
◦ Compares revenues
Dutch versus First-Price Auctions
Dutch auctions outperform first-price auctions
Source: Lucking-Reiley, AER 1999
Movielens

Chen, et al. (AER, 2009)
◦ (DIY) Website run by University of Minnesota
◦ Influence messaging to clients to help fix freerider problem of providing recommendations.
◦ Personalized social information seems to help.
Connections
◦ What is the link between Found and DIY
experiments and traditional laboratory
experiments?
◦ Substitutes?

“One must take great care when claiming that patterns measured in the
experimental economics laboratory are shared broadly by agents in certain realworld markets.” –Levitt and List (2007)
◦ Complements?
 Behavioral features magnified in the found
experiments (shipping charges)
 Revenue patterns reversed (Dutch perform worse
than 1st price in the lab)
Competing Auction Sites

Early 2000s
◦ EBay, Amazon, and Yahoo all compete in US
online auctions
◦ EBay and Amazon differ in bid ending rules
 Hard close (eBay) versus soft close (Amazon)
◦ Roth and Ockenfels (2002) observe:
 More last minute bidding (sniping) on eBay
 Possibly lower revenues for eBay sellers
From the Web to the Lab

Key problem: Amazon and eBay differ in
many ways other than ending rule
◦ Apples to apples comparison

Ariely, Roth, and Ockenfels (2005)
compare ending rules in the lab
◦ Treatments:
 Hard versus Soft Close
 Varied probability a snipe bid was transmitted
Results
More late bidding with a hard close
More revenues with a soft close
Source: Ariely, Roth, and Ockelfels, RAND, 2005
And Back to the Web
Yahoo offered the seller a toggle for a
hard versus soft close at its auction site
 Brown and Morgan (2009) took
advantage of this “found experiment”
 Sold late 19th and early 20th century US
Silver Dollars

◦ Slabbed and graded matched pairs
◦ Identical reputable seller
Does the ending rule matter?
Mean Revenues by Coin
90
80
70
60
50
Revenues
40
30
20
10
0
Soft Close
Hard Close
1
2
3
4
5
6
7
8
Coin
No difference in revenues by ending rule.
No difference in the timing of bids.
Source: Brown and Morgan, JPE 2009
Connections: Reputation

Reputation affects trust in laboratory
experiments
◦ Worry:
 Limited scope for reputation
 Reputation is a social construction

Resnick, et al. (2006)
◦ Reputation in the “wild”
 Matched pairs of collectible postcards
 Varied seller reputation (both positive and negative)
◦ Reputation increased buyer willingness to pay by
8.1%
A Lab Without Walls

Limitations of the
traditional lab:
◦ How representative are
sophomores?
◦ I want a lot of subjects.
◦ I don’t have a lot of
money?
◦ I want to study long-run
effects.
◦ Subjects are suspicious
of any randomizations
try.
Amazon Mechanical Turk
HITS
• Lots of “workers”
• Cheaper than undergrads
• Can employ over time
• Amazon reputation
Mason and Watts (2009)

Used AMT to recruit
for image sorting
task
◦ Varied difficulty of task
◦ Varied rate of pay for
task completion
◦ Varied quota versus
piece rate pay scheme
Going Native

“Bucket testing” is part of the DNA for
many online companies
◦ Yahoo, eBay, Google, Microsoft
Opportunities to “go native” and analyze
this data
 The company itself becomes the
laboratory.

Search Experiments
These are
paid search
results. How
does
competition
affect clicks?
Going Native
(Reiley, Li and Lewis (2010)
Randomly varied the number of north ads
displayed for a given search result.
 Tracked clickthrough rates (CTR) for
each ad.

◦ Competition should drive down CTR when
more ads are shown.
◦ Behavioral heuristics drive CTR up by making
top ad seem more relevant.
Results
Source: Reiley, Li, and Lewis, APM 2010
Going Native
Colonel Blotto Games for Fun
Project Tycoon: Yahoo Games platform for experiments.
Going live by 1 August
Competition

Platform competition can easily result in a
single dominant player
◦ Declining average cost, network effects

How can we deduce whether platforms
are co-existing or tipping (slowly)?
◦ Ellison, et al. (2004) offer a theory
◦ Key implication:
 No profitable deviation for sellers across platforms.
Testing the Tipping Hypothesis

Brown and Morgan (2009) became sellers
on competing platforms—eBay and Yahoo
◦ Identical products, conditions, descriptions
◦ Reputable sellers on both platforms.
◦ Tested two hypotheses:
 Revenue equalization
 Buyer-seller ratio equalization
Yahoo versus eBay revenue
comparison.
Mean Revenues by Coin
120
100
80
Revenues 60
eBay
Yahoo
40
20
0
1
2
3
4
5
6
Coin
28% Higher Revenue on eBay
7
8
Yahoo versus eBay number of bidders
comparison.
Mean Number of Bidders by Coin
8
7
6
5
Number of
4
Bidders
3
2
1
0
eBay
Yahoo
1
2
3
4
5
6
7
Coin
50% more buyers per item on eBay
8
Conclusions
An ever-changing sandbox
 Play a role
 Build it yourself
 Complements not substitutes
 Get out of the tower
