Research - David M. Pennock

Transcription

Research
Prediction Market Science &
Technology at Yahoo!
David M. Pennock
Mike Dooley, Tej Kasturi, Bernard Mangold, Havi Hoffman
Yiling Chen, Chao-Hsien Chu, Sandip Debnath, Rael Dornfest,
Joan Fiegenbaum, Gary Flake, Lance Fortnow, Brian Galebach,
Lee Giles, Joe Kilian, Steve Lawrence, Tracy Mullen, Rahul Sami,
Emile Servan-Schreiber, Michael Wellman, Justin Wolfers
Research
Prediction markets
• Futures market designed to elicit a forecast
about some future event
• Leverages “wisdom of crowds”, extracting and
combining information from distributed sources
• Have been used successfully to
• Predict election outcomes [IEM, 1988-]
• Predict corporate metrics (sales, product release
times, …) [HP, 2001-] [MSFT, Eli Lily, Intel,
Siemens] [GOOG, 2005-]
• Predict movie box office returns [HSX], news
[NewsFutures], scientific conjectures [FX], sporting
events, judicial nominations, economic numbers, …
[inTrade], real estate [HedgeStreet], many others
Research
Prediction markets research @ Y!
2002-2005
• Computational aspects & mechanism design
n
4
• n events, 2n combinations, 22 poss. bets!
• Algorithms & computational complexity
• Leverage independence (“compact markets”)
• “Betting boolean-style”: Generic bidding language
3 • New exchange mechanism: dynamic pari-mutuel market;
Cross btw stock market and horse race betting;
Ideal for huge numbers of futures and low liquidity
common in derivatives trading and gambling
2 • Empirical analyses of real-$/play-$ markets;
Does money matter?
• Academic: 6 pubs; 4 patents; 2 workshops
1 • Practical: Search futures & Tech Buzz Game
Research
Search Futures &
Tech Buzz Game
Research
Search futures
•
•
•
•
Search data: What people worldwide are thinking about today
Search futures: What people will be thinking about tomorrow
Billions of advertising/business dollars ride on answers
Research questions
• Is search buzz predictable? If so, what factors promote accuracy?
(trading mechanism, currency [real/fantasy], subsidies, dividend
rates, noise bots, #traders, competence of traders, ... )
• What types of search concepts? What types of trends?
(sudden, cyclical, growth, burst)
• How to handle huge numbers of (combinatorial/conditional)
markets and low liquidity
• Which traders do well? (buzz traders, buy/hold traders, day
traders, noise traders, affinity traders, cheaters)
• Can machine learning post-processing boost accuracy?
• How to combat search spam, manipulation
• How to hedge marketing/business risks with real-$ search/PPC
futures
Research
http://buzz.research.yahoo.com
•
•
•
Yahoo!,O’Reilly launched Buzz Game 3/05 @ETech
Research testbed for investigating search futures
Buy “stock” in hundreds of technologies
•
Earn dividends based on actual search “buzz”
•
•
API interface
Exchange mechanism is new Yahoo!R invention
Cross btw stock market and horse race betting
Research
Exchange Interface
Dynamic Parimutuel “Market Maker”
Research
Technology forecasts
• iPod phone
• What’s next?
Google Calendar?
price
search
buzz
8/28: buzz gamers
begin bidding
up iPod phone
8/29: Apple
invites press
to “secret”
unveiling
9/7: Apple
announces
Rokr
9/8-9/18: searches
for iPod phone soar;
early buyers profit
• Another Apple unveiling
10/12; iPod Video?
9am 10/5
Research
Forecast
accuracy
Early lessons
learned
• Average forecast error
across 352 stocks
• Market closing deadline
focuses traders
• Dividend levels matter
• Intelligent strategies work
forecast error
rapidly declines
as traders zero in
on correct
predictions
end of phase 1
contest period
• Randomized bots lost
money to real traders
• Contest winner followed
optimal buzz trading
strategy (prices ∝ √buzz);
Went from 4th to 1st place in
final days
• Forecast error does
decrease over time
Research
2002-2005
n
4
Does money matter?
Research
Does Money Matter?
Research
Real markets vs. market games
HSX
average
log
score
arbitrage
closure
IEM
Research
Real markets vs. market games
HSX
FX, F1P6
probabilistic
forecasts
actual
forecast source
F1P6 linear scoring
F1P6 F1-style scoring
betting odds
F1P6 flat scoring
F1P6 winner scoring
100
expected
value
forecasts
50
20
10
5
489 movies
2
1
1
2
5
10
20
50
100
estimate
avg log score
-1.84
-1.82
-1.86
-2.03
-2.32
Research
Does money matter?
Play vs real, head to head
Experiment
• 2003 NFL Season
• ProbabilitySports.com
Online football forecasting
competition
• Contestants assess
probabilities for each game
• Quadratic scoring rule
• ~2,000 “experts”, plus:
• NewsFutures (play $)
• Tradesports (real $)
•
Results:
• Play money and real
money performed
similarly
• 6th and 8th respectively
• Markets beat most of
the ~2,000 contestants
• Average of experts
came 39th (caveat)
Used “last trade” prices
Electronic Markets, Emile ServanSchreiber, Justin Wolfers, David
Pennock and Brian Galebach
TradeSports: Correlation=0.96
NewsFutures: Correlation=0.94
90
Prices: TradeSports and NewsFutures
100
TradeSports Prices
100
80
70
60
Fitted Value: Linear regression
45 degree line
75
50
50
40
25
30
20
10
0
0
0
10
20
30
40
50
60
70
Trading Price Prior to Game
80
90
0
100
20
40
60
NewsFutures Prices
n=416 over 208 NFL games.
Correlation between TradeSports and NewsFutures prices = 0.97
Data are grouped so that prices are rounded to the nearest ten percentage points; n=416 teams in 208 games
Prediction Performance of Markets
Relative to Individual Experts
Rank
Observed Frequency of Victory
Research
Prediction Accuracy
Market Forecast Winning Probability and Actual Winning Probability
0
20
40
60
80
100
120
140
160
180
200
220
240
260
280
300
NewsFutures
Tradesports
1
2
3
4
5
6
7
8
9 10 11 12 13 14
Week into the NFL season
80
100
Research
Does money matter?
Play vs real, head to head
ProbabilityFootball Avg
TradeSports
(real-money)
NewsFutures
(play-money)
Difference
TS - NF
0.443
0.439
0.436
0.003
(0.012)
(0.011)
(0.012)
(0.016)
0.476
0.468
0.467
0.001
(0.025)
(0.023)
(0.024)
(0.033)
Average Quadratic Score
9.323
12.410
12.427
-0.017
= 100 - 400*( lose_price2 )
(4.75)
(4.37)
(4.57)
(6.32)
Average Logarithmic Score
-0.649
-0.631
-0.631
0.000
= Log(win_price)
(0.027)
(0.024)
(0.025)
(0.035)
Mean Absolute Error
= lose_price
[lower is better]
Root Mean Squared Error
= ?Average( lose_price2 )
[lower is better]
[higher is better]
[higher (less negative) is better]
Statistically:
TS ~ NF
NF >> Avg
TS > Avg
Research
2002-2005
n
4
Does money matter?
Research
Dynamic Parimutuel Market
Research
What is a pari-mutuel market?
A B
• E.g. horse racetrack style wagering
• Two outcomes:
A
B
• Wagers:
Research
A B
√A
• Two outcomes:
B
• Wagers:
Research
A B
√A
• Two outcomes:
B
• Wagers:
Research
• Before outcome is revealed, “odds” are
reported, or the amount you would win per
dollar if the betting ended now
• Horse A: $1.2 for $1; Horse B: $25 for $1; … etc.
• Strong incentive to wait
•
•
•
•
payoff determined by final odds; every $ is same
Should wait for best info on outcome, odds
⇒ No continuous information aggregation
⇒ No notion of “buy low, sell high” ; no cash-out
Dynamic pari-mutuel
market
Standard PM: Every $1 bet is the same
 DPM: Value of each $1 bet varies
depending on the status of wagering at the
time of the bet
 Encode dynamic value with a price

– price is $ to buy 1 share of payoff
– price of A is lower when less is bet on A
– as shares are bought, price rises; price is for an
infinitesimal share; cost is integral
Research
Pari-mutuel market
Basic idea
1
1
1
1
1
1
1
1
1
1
1
1
Research
Dynamic pari-mutuel market
Basic idea
1
1
0.2
0.4
1.1
1.3
1.6
0.9
2
5
2.
3
Research
How are prices set?
• A price function pi(n) gives the
instantaneous price of an
infinitesimal additional share beyond
the nth
n
• Cost of buying n shares: ∫0 pi(n) dn
• Different reasonable assumptions
lead to different price functions
Research
Price functions
Share type
Constraint/
Assumption
Result
Losing money
p1 = P 2
p2 = P 1
Closed form
cost() & shares()
Losing money
pi/pj = Mi/Mj
Closed form
cost() & shares()
All money
pi/pj = Mi/Mj
Closed form
shares() ;
Numeric cost()
All money
pi/pj = Si/Sj
Closed form
cost() & shares()
Research
2002-2005
n
4
Does money matter?
Research
Computational Aspects:
Complex Betting
[Thanks: Wolfers, Fortnow]
Market combinatorics
Probability
Bush wins each state
90 to
80 to
70 to
60 to
50 to
40 to
30 to
20 to
10 to
0 to
100 (14)
90 (6)
80 (3)
70 (6)
60 (2)
50 (5)
40 (2)
30 (2)
20 (7)
10 (4)
Source: www.Tradesports.com; 3/26/2004.
[Thanks: Wolfers, Fortnow]
Market combinatorics






Combinatorial markets
Singly exponential
Buy/sell multiple
securities
simultaneously
Buy 2 TX & sell 4 CT,
pay $10
Exactly analogous to
combinatorial auctions
250 possible “bundles”
of securities




Compound markets
Doubly exponential
Pr(CA ^ AZ) ?
Pr(Elec | FL) ?
Pr((IL^NJ)∨(¬IL^¬NJ)) ?
Not derivable as a
Probability
linear combinations
of
base securities
50
2
2 possible functions
“Only” 250 securities
neededSource:
to www.Tradesports.com;
span space
3/26/2004.
Bush wins each state


90 to
80 to
70 to
60 to
50 to
40 to
30 to
20 to
10 to
0 to
100 (14)
90 (6)
80 (3)
70 (6)
60 (2)
50 (5)
40 (2)
30 (2)
20 (7)
10 (4)
Compound markets I:
Brute force



I am entitled to:
$1 if A1&A2&…&An
I am entitled to:
$1 if A1&A2&…&An
I am entitled to:
$1 if A1&A2&…&An
I am entitled to:
$1 if A1&A2&…&An
I am entitled to:
$1 if A1&A2&…&An
I am entitled to:
$1 if A1&A2&…&An
I am entitled to:
$1 if A1&A2&…&An
I am entitled to:
$1 if A1&A2&…&An
In principle, markets in all possible combinations will get
you everything you want
In practice, this is infeasible
It’s also unnatural
Compound markets II:
Leverage independence
Structure market according to unanimously
agreed-upon independencies
E4
$1 if E6|E3E5
E5
E1
E6
E2
E3
$1 if E6|E3Ê5
$1 if E6|Ê3E5
$1 if E6|Ê3Ê5
[Pennock & Wellman 2000]
Compound markets II:
Leverage independence

CARA & Markov indep ⇒ risk-neutral indep

If all agents have CARA, then market structured as
n
TRIANGULATE[∪ i=1 MORALIZE(Di)] is op complete
E4
E5
E1
E6
E2


E3
$1 if E6|E3E5
$1 if E6|Ê3E5
$1 if E6|E3Ê5
$1 if E6|Ê3Ê5
Can still yield exponential savings (“compact sec. markets”)
This example: 19 vs. 63
[Pennock & Wellman 2000]
Compound markets III:
High-level bidding language

A bidding language: write your own security
I am entitled to:





$1 if Boolean_fn | Boolean_fn
For example
I am entitled to:
$1 if A1 | A2
I am entitled to:
$1 if (A1&A7)||A13 | (A2||A5)&A9
I am entitled to:
$1 if A1&A7
Offer to buy/sell q units of it at price p
Let everyone else do the same
Auctioneer must decide who trades with whom at
what price… How? (next)
More concise/expressive; more natural
The matching problem




There are many possible matching rules for the
auctioneer
A natural one: maximize trade subject to
no-risk constraint
trader gets $$ in state:
A1A2 A1A2 A1A2 A1A2
Example:
– buy 1 of $1 if A1
for $0.40
– sell 1 of $1 if A1&A2 for $0.10
– sell 1 of $1 if A1&A2 for $0.20
0.60 0.60 -0.40 -0.40
-0.90 0.10 0.10 0.10
0.20 -0.80 0.20 0.20
No matter what happens,
auctioneer cannot lose
money
-0.10 -0.10 -0.10 -0.10

|

Another way to look at it:
Logical reduction
|
Example:
– buy 1 of $1 if A1
for $0.40
– sell 1 of $1 if A1&A2 for $0.10
– sell 1 of $1 if A1&A2 for $0.20
= sell $1 if A1
||

Clear match btw buy and sell|
for $0.3








Divisible orders: will accept any q* ≤ q
Indivisible: will accept all or nothing
Let Ω=all possible combinations; |Ω|=2n
Let αi be fraction of order i filled
Let Υiω be payoff for order i in state ω
Div. MP: Does ∃αi∈[0,1], ∀ω∈Ω, -∑αiΥiω≥0
Indiv. MP: Does ∃αi∈{0,1}, ∀ω∈Ω, -∑αiΥiω≥0
Optimizations
– max trade; max percent orders filled
(at least 1 αi > 0)
– max auctioneer utility subject to no-risk
– max auctioneer utility -- with risk (“market maker”)
Divisible vs. indivisible



Sell 1 of A1 at $0.50
Buy 1 of (A1&A2) | (A1 || A2) at $0.50
Buy 1 of A1|A2 at $0.40
A1A2 A1A2 A1A2 A1A2
-0.50 -0.50 0.50 0.50
0.50 -0.50 -0.50 0
0
0.60 0
-0.40
0
-0.40
0
0.10



Buy 1 of (A1&A2) | (A1 || A2) at $0.50
Buy 1 of A1|A2 at $0.40
A1A2 A1A2 A1A2 A1A2
-0.50 -0.50 0.50 0.50
0.50 -0.50 -0.50 0
0
0.60 0
-0.40
0
-1
0
0.50



Buy 1 of (A1&A2) | (A1 || A2) at $0.50
Buy 1 of A1|A2 at $0.40
A1A2 A1A2 A1A2 A1A2
-0.50 -0.50 0.50 0.50
0.50 -0.50 -0.50 0
0
0.60 0
-0.40
-0.50
0.10
0.50
0.10



Buy 1 of (A1&A2) | (A1 || A2) at $0.50
Buy 1 of A1|A2 at $0.40
A1A2 A1A2 A1A2 A1A2
-0.50 -0.50 0.50 0.50
0.50 -0.50 -0.50 0
0
0.60 0
-0.40
0.50
0.10 -0.50 -0.40



Buy 1 of (A1&A2) | (A1 || A2) at $0.50
Buy 1 of A1|A2 at $0.40
A1A2 A1A2 A1A2 A1A2
3/5 x -0.50 -0.50 0.50 0.50
3/5 x 0.50 -0.50 -0.50 0
0.60 0
-0.40
1x 0
divisible
0
0
0
-0.10
match!
Complexity results


Divisible orders: will accept any q* ≤ q
Indivisible: will accept all or nothing
LP
# events
O(log n)
O(n)

divisible
polynomial
co-NP-complete
reduction from SAT
Natural algorithms
reduction from X3C
indivisible
NP-complete
Σ2p complete
reduction from T∃∀BF
– divisible: linear programming
– indivisible: integer programming;
logical reduction?
Fortnow; Kilian; Sami
Open questions

Other matching rules
– maximize utility subject to no-risk
– maximize utility (market maker)

What to do with the surplus
– can be in cash and “leftover” securities
– auctioneer keeps surplus
– surplus is shared back among traders, auctioneer; how?

Trader optimization problem
– how to choose securities, p’s, q’s, subject to limits/penalties for
number, complexity of bids
– ultimately a game-theoretic question


Approximate algorithms, heuristics
Incentive properties
Research
2002-2005
n
4
Does money matter?

Research - David M. Pennock

Transcription

Similar documents

Contents

Nigel Mawer QPM WPBSA Vice Chair and Chairman of the

Dermascope, May 2014

SAERA 2015

norsk Rikstoto

to infinity and beyond! - Division of Student Affairs

Beadling clinic - Monroeville Soccer Club

THE BEST - Kate Schelter

Bios Buzz March 18, 2016 - Rollins School of Public Health

Productfolder