Two Models of Collec ve Intelligence

Two Models of Collec-ve Intelligence Sco5 E Page University of Michigan Santa Fe Ins-tute Lu Hong Loyola University-­‐Chicago Physics: What are the features and laws of the recorded-­‐informa-on universe? Social Science: How do people relate to, seek, and use informa-on? Engineering: How can access to recorded informa-on be made most rapid and effec-ve? Marcia Bates: UCLA Model 1
Problem
Model 2
Model 3
Background Background: Capitalism vs Socialism and the iConference Background Y Model? Background Y Model?: Logic, Explain, Predict, Act, Design Background Y Model? Collec-ve Intelligence Background Y Model? Collec-ve Intelligence: Problem Solving vs Predic-on Background Y Model? Collec-ve Intelligence Two Models of Predic-on Background Y Model? Collec-ve Intelligence Two Models of Predic-on: Generated and Interpreted Signals Background Y Model? Collec-ve Intelligence Two Models of Predic-on Many Model Thinking Background: Capitalism vs Socialism iConference Oscar Lange Gerard Debreu Leo’s Big Idea (as told by Stanley) Leo Hurwicz Stanley Reiter Environment Environment SWF/SCC Outcome Environment SWF/SCC Message Space Outcome Environment SWF/SCC Message Space Outcome Environment SWF/SCC Message Space Outcome 40
35
30
25
20
15
10
5
0
Manufacturing Employment %
1950
1960
1970
1980
1990
2000
2010
iConference Physics: What are the features and laws of the recorded-­‐informa-on universe? Social Science: How do people relate to, seek, and use informa-on? Engineering: How can access to recorded informa-on be made most rapid and effec-ve? Marcia Bates: UCLA Physics: What are the features and laws of the recorded-­‐informa-on universe? Environment SWF/SCC Message Space Outcome Social Science: How do people relate to, seek, and use informa-on? Environment SWF/SCC Message Space Outcome Engineering: How can access to recorded informa-on be made most rapid and effec-ve? Effec-ve Environment SWF/SCC Message Space Outcome (Rapid) Y Model? Logic, Explain, Predict, Act, Design Logic Carl Simon Dig the well before you are thirsty. Dig the well before you are thirsty. Don’t cross the bridge un>l you come to it. Explain Charles Doering Predict Marissa Eisenberg Act Wells Fargo Wachovia Merrill Lynch Bear Stearns Morgan Stanley Bank of America AIG Ci-group Lehman Brothers Goldman Sachs JP Morgan Chase Design Jenna Bednar Coverage: Federal Gov’t
encroachment
shirking
State A
State B
burden-shifting
Y Model? Logic Explain Predict Act Design Predict Logic Explain Act Design Collec-ve Intelligence Problem Solving vs Predic-on Collec>ve Intelligence 2015 hGp://sites.lsa.umich.edu/
collec>veintelligence/ Santa Clara, Marriot, May 31 – June 2, 2015 (s-ll accep-ng posters) Problem Solving wikipedia Predic-on Two Models of Predic-on Generated and Interpreted Signals Generated Signal: disturbance or interference (social scien-sts/sta-s-cians) Interpreted Signal: predic-on from a model (computer scien-sts/psychologists) Generated Signals Outcome disturbance Signal Generated 101 Outcome: θ in Θ Signal: si Distribu-on: f(si|θ) UCSC L L -­‐ ε L + ε Crowd’s Predic-on: n
1
si
∑
n i=1
Error: 2 (si -­‐ θ)
Average Error: n
1
2
(s
−
θ
)
∑
i
n i=1
Crowd Error: 2 (c -­‐ θ)
Diversity: n
1
2
(si − c)
∑
n i=1
Diversity Predic-on Theorem Crowd Error = Average Error -­‐ Diversity Diversity Predic-on Theorem n
n
1
1
2
2
(c − θ ) = ∑ (si − θ ) − ∑ (si − c)
n i=1
n i=1
2
Diversity Predic-on Theorem Crowd Error = Average Error -­‐ Diversity 0.6 = 2,956.0 -­‐ 2955.4 Three’s a Crowd Goncola Abecasis 18. Supplementary&Figures&
18
Table 1
Role of Staff and FOMC Forecasts in Predicting Actual Values
Constant
Staff Forecast
FOMC Forecast
2/3 set
Broad
Sanger
R2
UMich
Average discordance at non−homref genotypes
0.11
0.1
Inflation
0.09
(1) OLS
–0.20 (0.22)
1.10 (0.39)
–0.10 (0.37)
0.86
0.08
(2) WLS
–0.26 (0.11)
1.40 (0.25)
–0.38 (0.25)
0.93
0.07
Unemployment
(3) OLS
0.26 (0.41)
0.97 (0.38)
–0.03 (0.40)
0.79
(4) WLS
0.21 (0.38)
0.78 (0.37)
0.17 (0.34)
0.89
0.06
0.05
Real Growth
0.04
(5) OLS
0.43 (0.36)
0.25 (0.49)
0.63 (0.52)
0.44
0.03
(6) WLS
0.52 (0.58)
0.17 (0.88)
0.67 (0.90)
0.50
0.02
0.01
Notes: The dependent variable is the actual value of the variable being forecast.
Standard errors are in parentheses. The weighted least squares regressions use
Newey-West standard errors.
10
20
30
40
50
60
Allele count from genotyping chip
70
80
90
Supplementary Figure 1. Average discordance between primary and consensus low
Generated 501 Mean of i’s signal: µi(θ) Bias of i’s signal: bi = (µi(θ) -­‐ θ) Variance of i’s signal: vi =E[(µi(θ) -­‐ θ)]2 n
1
∑ bi
Average Bias B = n i=1
n
1
∑vi
Average Variance V = n i=1
€
n
1
Average Covariance C = ∑ ∑ E[si − µi ][s j − µ j ]
€
€
n(n − 1)
i=1 j ≠ i
Bias Variance Decomposi-on 1
n −1
E[SqE(c)] = B + V +
C
n
n
2
€
Resolving the Paradox Diversity Predic>on Theorem: Error = error Diversity = realized diversity Bias Variance Decomposi>on: Variance = error Nega-ve covariance = expected diversity Interpreted Signals A5ributes Feature Space Model Predic-on Binary Interpreted Signals Model Set of objects |X|=N Set of outcomes S = {G,B} Interpreta-on: a par--on of X Interpre-ve Signal Example Experience
Charisma H MH ML L H G G G B MH G G G B G ML G B B B L B G B B Experience Interpreta-on 75 % Correct H G G G B G B MH G G G B G G B ML G B B B B L B G B B B Interpreted Signals Accuracy: Number of boxes Diversity: Different boxes Charisma Interpreta-on 75% Correct H MH ML L G G G B G G B G G G B B B B G B B G B G B B Balanced Interpreta-on 75% Correct Extreme on one measure. Moderate on the other G B G G B G B G B G G B B G B B G G B B Vo-ng Outcome H MH ML L H GGB GGG GBG BGB MH GGG GGB GBB G GBG ML BGG BBG BBB BBG L BGB BGG BBG BBB Reality H MH ML L H G G G B MH G G G B G ML G B B B L B G B B Claim: A collec'on of interpreted signals makes the correct predic'on if an only if the the outcome func+on can be wri1en as a sum of the individual models. Agent 1 Outcome Func-on Agent 2 Agent 1 Outcome Func-on Agent 2 Are Two Different Than One? Predict Logic Explain Act Design Predict Logic Explain Act Design Predict Logic Explain Act Design Generated Signals Collec-ve accuracy comes from error cancella>on Difficulty = average error Diversity = correla-on Interpre-ve Signals Collec-ve accuracy comes from adding diverse models Difficulty depends on nonlinearity Correla-on comes from diversity Predict Logic Explain Act Design Engineering: How can access to recorded informa-on be made most rapid and effec-ve? Effec-ve Environment SWF/SCC Message Space Outcome (Rapid) Weigh-ng Weigh-ng by Accuracy Accuracy A = 1/σ2 Weigh-ng by Accuracy Accuracy A = 1/σ2 Weights: wi= Ai/(A1+A2+A3+A4…+An) E
⇤ ⌃
M
i=1
⇥ ⇥2 ⌅ ⌃
M
M ⌃
⌃
A2i
Ai Aj
si
=
(si )+
(si , sj )
⇧M
⇧M
⇧
M
2
2
A
(
A
)
(
A
)
j=1 j
k=1 k
k=1 k
i=1
i=1 j=i
Ai
Accuracies: 1, 0.5, 0.25 Equally weighted: Error = 7/9 Accuracy weighted: Error = 4/7 {} “Use equal weights unless you have strong evidence to support unequal weigh-ng of forecasts” (Armstrong 2001) Six years of data
Half million users
17,700 movies
Data divided into (training, testing)
Testing Data dived into (probe, quiz, test)
Singular Value Decomposition
Each movie represented by a vector: (p1,p2,p3,p4…pn) Each person represented by a vector: (q1,q2,q3,q4…qn) Robert Bell Chris-na and David Romer BellKor
Best Model: 6.8%
BellKor
107 models combined: 8.4%
BellKor’s Pragmatic Chaos
Combined Models: 10.1% Enter ``The Ensemble’’
23 Teams
30 Countries
And The Winner is…
RMSE for The Ensemble:
RMSE for Bellkor
0.85671
0.85670
Ensemble
Bellkor
10.06%
10.06%
Ensemble
Bellkor
50-50 Combination
10.06%
10.06%
10.19%
WEIGHT vs RMSE 0.4 0.3 0.2 0.1 0 0.855 -­‐0.1 -­‐0.2 0.86 0.865 0.87 0.875 0.88 V(a,b,c,d) = a + b + c + d Model 1: a+b+c Model 2: b+c+d Model 3: b+c “Op-mal” Weigh-ng: 1,1,-­‐1 Predict Logic Explain Act Design Amplifica-on Bias: IARPA K Predic-ons {P1,P2,P3,..Pk} with Pi < Pi+1 Aggregate Predic-on: P*(P1,P2,P3,..Pk) QUESTION: Is P* in [P1 , Pk]? Proposi+on: With generated signals the op'mal predic'on lies in the range of the predic'ons. Proposi+on: With interpreted signals the op'mal predic'on need not lie in the range of the predic'ons. A is true if x1 + x2 + x3 > 1.5 Each xi in {0,1} equally likely Forecaster 1 sees x1= 1 so P1= 0.75 Forecaster 2 sees x2= 1 so P2= 0.75 P* = 1 Many Model Thinking Generated and Interpreted Signals Physics: What are the features and laws of the recorded-­‐informa-on universe? Social Science: How do people relate to, seek, and use informa-on? Engineering: How can access to recorded informa-on be made most rapid and effec-ve? Marcia Bates: UCLA Model
Problem
``All Models Are Wrong’’ - George Box
Model 1
Problem
Model 2
Model 3
``Hence, our truth is the intersec-on of independent lies.’’ - Richard Levins
?