Elici%ng Informa%on from the Crowd a Part of the EC 13 Tutorial on Social Compu%ng and User- Generated Content

Transcription

1 Elici%ng Informa%on from the Crowd a Part of the EC 13 Tutorial on Social Compu%ng and User- Generated Content Yiling Chen Harvard University June 16, 2013

2 Roadmap Elici%ng informa%on for events with verifiable outcomes Proper scoring rules Market scoring rules Elici%ng unverifiable informa%on Peer predic%on Bayesian Truth Serum Robust Bayesian Truth Serum

3 How to Evaluate Weather Forecasts p 1 p Specifica%on of a scale of goodness for weather forecasts the forecaster may oten find himself in the posi%on of choosing to let it (the verifica%on system) do the forecas%ng for him by hedging or playing the system. one essen%al criterion for sa%sfactory verifica%on is that the verifica%on scheme should influence the forecaster in no undesirable way [Brier 1950]

4 The Brier Scoring Rule [Brier 1950] p 1 p S(p, sunny) = 1 (1 p) 2 S(p, rainy) = 1 p 2 Expected score of predic%on S(p, q) =qs(p, sunny) + (1 =1 q(1 p) 2 (1 q)p 2 p =1 q + q 2 (p q) 2 given belief q q)s(p, rainy) Predic%ng q maximizes the expected score arg max S(p, q) =q p S(p, q) increases as p q decreases.

5 The Brier Scoring Rule State Predic%on! 2 {1, 2,...,n} p =(p 1,p 2,...,p n ) Brier score for predic%on in state! is Also called the quadra%c scoring rule p S(p,!) =a! b e! p 2!- th indicator vector Parameters, b >0. Affine transforma%on does not change incen%ves.

6 Other Strictly Proper Scoring Rules Logarithmic scoring rule S(p,!) = log p! Spherical scoring rule S(p,!)= p! p

7 Proper Scoring Rules Scoring rule S(p,!) Expected score for predic%on given belief q : predic%on belief S(p, q) := nx!=1 p q! S(p,!) A scoring rule S(p,!) is proper if and only if S(q, q) S(p, q). It is strictly proper if the inequality is strict unless q = p. Proper scoring rules are dominant- strategy incen%ve compa%ble for risk- neutral agents.

8 Geometric Interpreta%on Brier Scoring Rule S(p, sunny) = 1 (1 p) 2 G(p) =S(p, p) =(p 1/2) 2 +3/4 S(p, rainy) = 1 p Bregman Divergence D G (q, p) =G(q) G(p) rg(p)(q p) G G(q) =S(q, q) 0.85 S(p, sunny) G(p) =S(p, p) S(p, q) 0.7 S(p, rainy) 0.65 q p Probability for Sunny

9 McCarthy, Savage Characteriza%on [McCarthy 1956][Savage 1971] A scoring rule S(p,!) S(p,!)=G(p) is (strictly) proper if and only if rg(p) p + rg! (p) where G is a (strictly) convex func%on and rg(p) is a subgradient of G and rg! (p) is its!-th component., Expected score S(p, p) = X! p! (G(p) rg(p) p + rg! (p)) = G(p) In terms of Bregman Divergence, for differen%able G, S(p,!) =G(p) rg(p) p + rg(p) e! = G(e! ) (G(e! ) G(p) rg(p) (e! p)) = G(e! ) D G (e!, p)

10 Common Strictly Proper Scoring Rules Brier scoring rule S(p,!) =1 e! p 2 G(p) = p 2 D G (q, p) = q p 2 Logarithmic scoring rule S(p,!) = log p! G(p) = X! p! log p! D G (q, p) = X! q! log q! p! Spherical scoring rule G(p) = p S(p,!)= p! p D G (q, p) = q q p p

11 Other Work on Scoring Rules Proper scoring rules for con%nuous variables [Matheson and Winkler 1976; Gnei%ng and RaTery 2007] Proper scoring rules for proper%es of distribu%ons (e.g. mean, quin%les) [Savage 1971; Lambert et al. 2008; Abernethy and Frongillo 2012] Scoring rules for more complex environments Agents can affect the event outcome [Shi et al. 2009; Bacon et al. 2012] A decision will be made based on the elicited informa%on [Othman and Sandholm. 2010; Chen and Kash 2011; Bou%lier 2012]

12 One Expert à Mul%ple Experts

13 Market Scoring Rules (MSR) [Hanson 2003, 2007] Sequen%al, shared version of proper scoring rules Reward improvements on predic%on Select a proper scoring rule S(p,!) Market opens with an ini%al predic%on Par%cipants sequen%ally change the market predic%on Par%cipant who changes the predic%on from p t 1 to p t receives payment S(p t,!) S(p t 1,!) p 0

14 An Example: LMSR Market S(p,!) = 5 log p! t 5log(0.9)- 5log(0.5) 5log(0.6)- 5log(0.5) 5log(0.8)- 5log(0.6) 5log(0.4)- 5log(0.8) 5log(0.9)- 5log(0.4) 5log(0.1)- 5log(0.5) 5log(0.4)- 5log(0.5) 5log(0.2)- 5log(0.4) 5log(0.6)- 5log(0.2) 5log(0.1)- 5log(0.6) The mechanism only pays the final par%cipant!

15 Proper%es of MSR Bounded loss (subsidy) Eg. Loss bound of LMSR with S(p,!) =b log p! is b log n Incen%ve compa%ble for myopic par%cipants Recent work studies informa%on aggrega%on in MSR with forward- looking par%cipants [Chen et al. 2010; Ostrovsky 2012; Gao et al. 2013] Market as a Bayesian extensive- form game Condi%ons under which informa%on is fully aggregated in the market

16 Predic%on and Trading p 1 p $1 iff Buy this contract if Sell this contract if price <p price >p

17 MSR as Automated Market Makers [Chen and Pennock 07] One contract for each outcome Payments are determined by a cost poten%al func%on C(Q) Q i is the current number of shares of the contract for outcome! that have been purchased Current cost of purchasing a bundle R of shares is C(Q + R) Instantaneous prices $1 iff! C(Q) p! Market Predic%on

18 Example LMSR with S(p,!) =b log p! Cost func%on C(Q) =b log X! e Q!/b Price func%ons p! (Q) = log e Q!/b P! 0 e Q! 0 /b

19 LMSR Equivalence C(Q) = log X! e Q! Profit of a par%cipant who changes to in state Q Q! ( Q! Q! ) (C( Q) C(Q)) =( Q! C( Q)) (Q! C(Q)) p! (Q) = = (log e Q! log X! 0 e Q!0 ) (log e Q! log X! 0 e Q! 0 ) e Q! P! 0 e Q! 0 S(p,!) = log p! e Q! e Q! = log P! e Q log P 0! 0! e Q 0! 0 = log p! ( Q) log p! (Q) = S(p( Q),!) S(p(Q),!)

20 Cost Func%on MM [Abernethy et al. 2013] Strictly Proper MSR An MSR using a strictly proper scoring rule The expected scoring func%on The corresponding cost func%on MM G(p) =S(p, p) S(p,!) C(Q) = sup p2 n Q p G(p) Conjugate Duality p(q) =rg(q) = arg max p2 n Q p G(p) p(q) Profit equivalence holds when interior of the probability simplex is in the rela%ve

21 Elici%ng Unverifiable Informa%on

22 Unverifiable Informa%on Is this website age- appropriate for children? Yes No

23 The General Sesng State E.g. whether this website is really age- appropriate or not m 2 agents i E.g. whether agent i Each agent! 2 {1, 2,...,n} receives a private signal s i 2 S thinks the website age- appropriate Prior distribu%on Pr(!,s 1,...,s m ) is common knowledge for all agents Goal: truthfully revealing their private signal strict Bayesian Nash equilibrium s i is a

24 The Peer Predic%on Mechanism [Miller et al. 2005] Require that the mechanism knows the prior Each agent has a reference agent Payment S(Pr(s j ŝ i ), ŝ j ) s i ŝ i Mechanism Pr(s j ŝ i ) s j ŝ j Pr(s 1,...,s m ) Pr(s i ŝ j ) Payment S(Pr(s i ŝ j ), ŝ i )

25 The Peer Predic%on Mechanism [Miller et al. 2005] S(p, ŝ) is a strictly proper scoring rule Technical assump%on: Stochas%c Relevance Pr(s j s i ) 6= Pr(s j s 0 i), 8s i 6= s 0 i Truthful repor%ng is a strict Bayesian Nash equilibrium. Payment S(Pr(s j ŝ i ), ŝ j ) Requiring the mechanism know the prior is undesirable! s i s j ŝ i ŝ j Mechanism Pr(s 1,...,s m ) Pr(s j ŝ i ) Pr(s i ŝ j ) Payment S(Pr(s i ŝ j ), ŝ i )

26 Bayesian Truth Serum (BTS) [Prelec 2004] Is this website age- appropriate for children? (Yes, No) Each agent is asked for two reports The mechanism calculates Score for agent i: x i Informa%on report: is an indicator vector, having value 1 at its k-th element if agent i reports signal k Predic%on report: y i is agent i s predic%on of the frequency of reported signals in the popula%on x k = lim n!1 1 n nx i=1 X k x i k x i k log x k ȳ k + X k log ȳ k = lim n!1 x k log yi k x k 1 n nx log y i k i=1

27 Bayesian Truth Serum (BTS) Technical assump%ons: condi%onal independence of signals and stochas%c relevance When n!1, truthful repor%ng is a strict Bayesian Nash equilibrium. For sufficiently large n, truthful repor%ng is a strict Bayesian Nash equilibrium. But depends on the prior. Informa%on Score X k x i k log x k ȳ k + X k x k log yi k x k Predic%on Score Log scoring rule! Truthful predic%on reports

28 BTS: Surprisingly Common True signal maximizes X Consider an agent with yes signal k x i k log x k ȳ k in expecta%on Predic%on for "Yes" Predic%on for "No" 0 Prior Agents with "Yes" Agents with "No" Geometric mean

29 Robust Bayesian Truth Serum (RBTS) [Witkowski and Parkes 2012a] Works for small popula%ons RBTS assumes binary signals Each agent S = {0, 1} is asked for two reports i Informa%on report: reported signal x i 2 {0, 1} Predic%on report: y i 2 [0, 1] is the predic%on of the frequency of signal 1 Each agent has two reference agents

30 Robust Bayesian Truth Serum (RBTS) [Witkowski and Parkes 2012a] s i s j S(y 0 i, x k )+S(y i, x k ), S being the Brier scoring rule (x i,y i ) Mechanism (x j,y j ) y 0 { y j + d if x i =1 i = yj d if x i =0 s k (x k,y k ) d =min{y j, 1 y j } Truthful repor%ng is a strict Bayesian Nash equilibrium for n 3.

31 Shadowing S is the Brier scoring rule Informa%on Score S(y 0 i, x k )+S(y i, x k ) Predic%on Score y 0 { y j + d if x i =1 i = yj d if x i =0 Shadowing d =min{y j, 1 y j } 0 y 1 j x d d x Predic%on of agent i given signal 0 and y j Predic%on of agent i given signal 1 and y j

32 Other Related Work Many improvements on the original peer predic%on [Jurca and Fal%ngs 2006, 2007, 2008, 2011] Private prior peer predic%on [Witkowski and Parkes 2012b] RBTS for non- binary signals [Radanovic and Fal%ngs, 2013]

33 Ques%ons? hwp://yiling.seas.harvard.edu

34 References J. Abernethy, Y. Chen, and J. W. Vaughan, Efficient Market Making via Convex Op%miza%on, and a Connec%on to Online Learning. ACM Transac%ons on Economics and Computa%on. Vol. 1, no. 2, pp. 12:1-12:39. J. D. Abernethy and R. M. Frongillo A Characteriza%on of Scoring Rules for Linear Proper%es. COLT 12. D. F. Bacon, Y. Chen, I. A. Kash, D. C. Parkes, M. Rao, and M. Sridharan Predic%ng Your Own Effort. AAMAS 12. C. Bou%lier Elici%ng forecasts from self- interested experts: scoring rules for decision makers. AAMAS 12. G. W. Brier Verifica%on of forecasts expressed in terms of probability. Monthly Weather Review 78(1): 1-3. Y. Chen, S. Dimitrov, R. Sami, D. M. Reeves, D. M. Pennock, R. D. Hanson, L. Fortnow, and R. Gonen Gaming predic%on markets: equilibrium strategies with a market maker. Algorithmica, 58: Y. Chen and D. M. Pennock, A U%lity Framework for Bounded- Loss Market Makers. UAI'07. Y. Chen and I. A. Kash Informa%on elicita%on for decision making. AAMAS 11.

35 References (Cont.) X. A. Gao, J. Zhang, Y. Chen What You Jointly Know Determines How You Act - Strategic Interac%ons in Predic%on Markets, EC 13. T. Gnei%ng and A. E. RaTery Strictly proper scoring rules, predic%on, and es%ma%on. Journal of the American Sta%s%cal Associa%on 102, 477, R. D. Hanson Combinatorial informa%on market design. Informa%on Systems Fron%ers 5(1), R. D. Hanson Logarithmic market scoring rules for modular combinatorial informa%on aggrega%on. Journal of Predic%on Markets, 1(1), R. Jurca and B. Fal%ngs, Minimum Payments that Reward Honest Reputa%on Feedback. EC'06. R. Jurca and B. Fal%ngs, Collusion Resistant, Incen%ve Compa%ble Feedback Payments. EC'07. R. Jurca and B. Fal%ngs.Incen%ves For Expressing Opinions in Online Polls, EC'08. R. Jurca and B. Fal%ngs Incen%ves for Answering Hypothe%cal Ques%ons. Workshop on Social Compu%ng and User Generated Content at EC'11. N. S. Lambert, D. M. Pennock, and Y. Shoham Elici%ng proper%es of probability distribu%ons. EC '08. J. E. Matheson and R. L. Winkler Scoring rules for con%nuous probability distribu%ons. Management Science 22,

36 References (Cont.) J. McCarthy Measures of the value of informa%on. PNAS: Proceedings of the Na%onal Academy of Sciences of the United States of America, 42(9): N. Miller, P. Resnick, and R. Zeckhauser 2005, Management Science, 51(9): A. Othman and T. Sandholm Decision rules and decision markets. AAMAS 10. M. Ostrovsky Informa%on aggrega%on in dynamic markets with strategic traders. Econometrica, 80(6): D. Prelec, A Bayesian Truth Serum for Subjec%ve Data. Science, 306 (5695): G. Radanovic and B. Fal%ngs, A Robust Bayesian Truth Serum for Non- Binary Signals. AAAI 13. L. J. Savage Elicita%on of personal probabili%es and expecta%ons. Journal of the American Sta%s%cal Associa%on, 66(336): P. Shi, V. Conitzer, and M. Guo Predic%on mechanisms that do not incen%vize undesirable ac%ons. WINE 09. J. Witkowski and D. C. Parkes, 2012a. A Robust Bayesian Truth Serum for Small Popula%ons. AAAI'12. J. Witkowski and D. C. Parkes, 2012b. Peer Predic%on without a Common Prior. EC'12.