Crowdsourcing contests

Transcription

1 December 8, 2012

2 Table of contents 1 Introduction 2 Related Work 3 Model: Basics 4 Model: Participants 5 Homogeneous Effort 6 Extensions

3 Table of Contents 1 Introduction 2 Related Work 3 Model: Basics 4 Model: Participants 5 Homogeneous Effort 6 Extensions

4 Problem Contests as crowdsourcing: specific case Problem of soliciting high quality answers Entry is costly; All-pay analogy Authors focus on online knowledge-sharing forums.

5 Problem Contests as crowdsourcing: specific case Problem of soliciting high quality answers Entry is costly; All-pay analogy Authors focus on online knowledge-sharing forums.

6 Problem Contests as crowdsourcing: specific case Problem of soliciting high quality answers Entry is costly; All-pay analogy Authors focus on online knowledge-sharing forums.

7 Problem

8 Problem

9 Problem Question Is a contest the right structure for eliciting desirable outcomes in such Q &A forums, in the presence of strategic participants acting selshly to maximize their own utility?

10 Table of Contents 1 Introduction 2 Related Work 3 Model: Basics 4 Model: Participants 5 Homogeneous Effort 6 Extensions

11 Related Work Two classes of related work: 1 General contest design

12 Related Work Two classes of related work: 1 General contest design 2 Optimal design of crowdsourcing contests

13 Related Work: General Contest Design E. Maskin. Mechanism Design: How to Implement Social Goals Reverse Game Theory, i.e. Mechanism Design B. Moldovanu, A. Sela. The Optimal Allocation of Prizes in Contests A. Glazer, R. Hassin. Optimal Contests All-pay auctions Linear or concave cost functions single prize Convex cost functions sometimes requires multiple prizes

14 Related Work: Optimal Design of Crowdsourcing Contests A. Ghosh, P. Hummel. A Game-Theoretic Analysis of Rank-Order Mechanisms for User-Generated Content. Focus on rank-order and proportional mechanisms Agents desire attention; allocate attention based on votes N. Archak, A. Sunararajan. Optimal Design of Crowdsourcing Contests Asymptotic results, real payments Utility is quality of top k submissions A. Ghosh, P. McAfee. Crowdsourcing with Endogenous Entry Endogenous entry, no difference in ability

15 Related Work: Key Differences Is optimal outcome implementable, not what is optimal among implementable outcomes Endogenous entry Very general mechanism designer utility

16 Table of Contents 1 Introduction 2 Related Work 3 Model: Basics 4 Model: Participants 5 Homogeneous Effort 6 Extensions

17 Overview General Framework Best contribution mechanism Agents Mechanism designer

18 Overview General Framework Best contribution mechanism Agents Mechanism designer

19 Overview General Framework Best contribution mechanism Agents Mechanism designer

20 Overview General Framework Best contribution mechanism Agents Mechanism designer

21 General Framework Goal: Model social computing applications Specifically online Q&A forums, and other settings with a best contribution mechanism n agents (may) make a contribution to the mechanism Mechanism designer derives some value from submissions Mechanism designer allocates prizes (virtual internet points) to submitters

22 General Framework Goal: Model social computing applications Specifically online Q&A forums, and other settings with a best contribution mechanism n agents (may) make a contribution to the mechanism Mechanism designer derives some value from submissions Mechanism designer allocates prizes (virtual internet points) to submitters

23 General Framework Goal: Model social computing applications Specifically online Q&A forums, and other settings with a best contribution mechanism n agents (may) make a contribution to the mechanism Mechanism designer derives some value from submissions Mechanism designer allocates prizes (virtual internet points) to submitters

24 General Framework Goal: Model social computing applications Specifically online Q&A forums, and other settings with a best contribution mechanism n agents (may) make a contribution to the mechanism Mechanism designer derives some value from submissions Mechanism designer allocates prizes (virtual internet points) to submitters

25 General Framework Goal: Model social computing applications Specifically online Q&A forums, and other settings with a best contribution mechanism n agents (may) make a contribution to the mechanism Mechanism designer derives some value from submissions Mechanism designer allocates prizes (virtual internet points) to submitters

26 Best Contribution Mechanism: Payments The class of best contribution mechanisms can be characterized as mechanisms of the form M B (P B, P C ). P B is the payment made to the best submission P C [0, P B ) is the payment made to all other submissions

27 Best Contribution Mechanism: Payments The class of best contribution mechanisms can be characterized as mechanisms of the form M B (P B, P C ). P B is the payment made to the best submission P C [0, P B ) is the payment made to all other submissions But how do we rank the submissions?

28 Best Contribution Mechanism: Rankings We describe the process by which the mechanism M B chooses the best submission by a function: π : Q Q N 1 N [0, 1] Then π(q i, q i, m) is the probability that a submission with quality q i will win when there are m other submissions with quality q i.

29 Best Contribution Mechanism We assume π must be: Increasing in q i Decreasing in m For any fixed m and qi, the set of values of q i for which π is discontinuous at q i = qi has measure zero.

30 Best Contribution Mechanism We assume π must be: Increasing in q i Decreasing in m For any fixed m and qi, the set of values of q i for which π is discontinuous at q i = qi has measure zero.

31 Best Contribution Mechanism We assume π must be: Increasing in q i Decreasing in m For any fixed m and qi, the set of values of q i for which π is discontinuous at q i = qi has measure zero.

32 Best Contribution Mechanism We assume π must be: Increasing in q i Decreasing in m For any fixed m and qi, the set of values of q i for which π is discontinuous at q i = qi has measure zero.

33 Best Contribution Mechanism We assume π must be: Increasing in q i Decreasing in m For any fixed m and qi, the set of values of q i for which π is discontinuous at q i = qi has measure zero. While quality is deterministic, π does not have to pick the submission of highest quality!

34 Best Contribution Mechanism: Questions Any questions?

35 Best Contribution Mechanism: Questions Any questions? Randomly selected submission gets $10; all others get $1

36 Best Contribution Mechanism: Questions Any questions? Randomly selected submission gets $10; all others get $1 Top three quality submissions get prize of $10; all others gets $0

37 Best Contribution Mechanism: Questions Any questions? Randomly selected submission gets $10; all others get $1 Top three quality submissions get prize of $10; all others gets $0 Contest run for 10 minutes. Probability of a submission with quality q submitted at time t being sele q (10 t) i q i (10 t i )

38 Best Contribution Mechanism: Motivation Why do we study the best contribution mechanism?

39 Best Contribution Mechanism: Motivation Why do we study the best contribution mechanism? Many online knowledge forums and contests are structured this way

40 Best Contribution Mechanism: Motivation Why do we study the best contribution mechanism? Many online knowledge forums and contests are structured this way More complicated mechanisms require additional effort by mechanism designer

41 Best Contribution Mechanism: Motivation Why do we study the best contribution mechanism? Many online knowledge forums and contests are structured this way More complicated mechanisms require additional effort by mechanism designer Anything else?

42 Table of Contents 1 Introduction 2 Related Work 3 Model: Basics 4 Model: Participants 5 Homogeneous Effort 6 Extensions

43 Agents Each agent has a type a i drawn from a atomless distribution with a strictly increasing CDF F. This represents their ability. F is common knowledge, but a i private information If agent i chooses to submit, it generates a submission of quality q(a i, e i ) q is non-decreasing in a i.

44 Agents Each agent has a type a i drawn from a atomless distribution with a strictly increasing CDF F. This represents their ability. F is common knowledge, but a i private information If agent i chooses to submit, it generates a submission of quality q(a i, e i ) q is non-decreasing in a i.

45 Agents Each agent has a type a i drawn from a atomless distribution with a strictly increasing CDF F. This represents their ability. F is common knowledge, but a i private information If agent i chooses to submit, it generates a submission of quality q(a i, e i ) q is non-decreasing in a i.

46 Agents Each agent has a type a i drawn from a atomless distribution with a strictly increasing CDF F. This represents their ability. F is common knowledge, but a i private information If agent i chooses to submit, it generates a submission of quality q(a i, e i ) q is non-decreasing in a i.

47 Agents: Two Frameworks 1 Homogeneous Effort 2 Endogeneous Effort

48 Agents: Two Frameworks 1 Homogeneous Effort 2 Endogeneous Effort

49 Agents: Homogeneous Effort Agents choose an action α i {C, R, N} 1 C: Contribute a submission to the mechanism. They incur a constant cost of c C > 0 (resulting from investing the same effort e) 2 R: Rate some non-empty subset of the submissions. They incur a constant cost of c R 0 3 N: Do nothing, and incur no cost } Their expected utility is then: u i = p i (α, q) c(α i )

50 Agents: Endogeneous Effort Same as before, except that if an agent chooses action C, they also must choose some effort e i [0, 1]. Incur cost of c(e i ), continuously differentiable and increasing in e i Quality of submission q(a i, e i ) is continuously differentiable and increasing in e i

51 Agents: Motivation for Frameworks When is each framework reasonable?

52 Agents: Motivation for Frameworks When is each framework reasonable? Homogeneous effort:

53 Agents: Motivation for Frameworks When is each framework reasonable? Homogeneous effort: Static knowledge question - you either know or you don t know

54 Agents: Motivation for Frameworks When is each framework reasonable? Homogeneous effort: Static knowledge question - you either know or you don t know Endogenous Effort:

55 Agents: Motivation for Frameworks When is each framework reasonable? Homogeneous effort: Static knowledge question - you either know or you don t know Endogenous Effort: Researchable question

56 Agents: Motivation for Frameworks When is each framework reasonable? Homogeneous effort: Static knowledge question - you either know or you don t know Endogenous Effort: Researchable question Work-based question

57 Agents: Motivation for Frameworks When is each framework reasonable? Homogeneous effort: Static knowledge question - you either know or you don t know Endogenous Effort: Researchable question Work-based question Ambiguous question

58 Agents: Motivation for Frameworks When is each framework reasonable? Homogeneous effort: Static knowledge question - you either know or you don t know Endogenous Effort: Researchable question Work-based question Ambiguous question Do both of these models seem reasonable and natural?

59 Agents: Other possible frameworks? Can you think of any other reasonable ways to model the quality of submissions?

60 Agents: Other possible frameworks? Can you think of any other reasonable ways to model the quality of submissions? Nondeterministic quality Quality doesn t depend on ability Subjective quality

61 Subjective Quality

62 Mechanism Designer The objective function of the mechanism designer can be broken into two pieces: Utility derived from submissions

63 Mechanism Designer The objective function of the mechanism designer can be broken into two pieces: Utility derived from submissions Cost associated with awarding prizes

64 Mechanism Designer: Revenue The mechanism designer, given m submissions with quality (q 1,..., q m ), receives utility of V (m, q 1,..., q m ) where V is non-decreasing in q i for any fixed m. What about m?

65 Mechanism Designer: Revenue V is nonmonotone in m! Why is this significant?

66 Mechanism Designer: Revenue V is nonmonotone in m! Why is this significant? This allows for situations where: V (1, q 1 ) > V (1, q 1, q 1 )

67 Mechanism Designer: Revenue V is nonmonotone in m! Why is this significant? This allows for situations where: V (1, q 1 ) > V (1, q 1, q 1 ) V (2, q 1, q 2 ) > V (1, q 1 ) + V (1, q 2 )

68 Mechanism Designer: Cost What cost does the mechanism designer incur?

69 Mechanism Designer: Cost What cost does the mechanism designer incur? It is allocating virtual internet points. Virtual internet points are free!

70 Mechanism Designer: Cost What cost does the mechanism designer incur? It is allocating virtual internet points. Virtual internet points are free! Thus, the mechanism s designer s objective is just to maximize V.

71 Mechanism Designer: Questions Any questions? Highest quality submission Total quality of submissions Number of submissions Difference between the highest and lowest quality submission

72 Mechanism Designer: Questions Any questions? Highest quality submission Total quality of submissions Number of submissions Difference between the highest and lowest quality submission

73 Mechanism Designer: Questions Any questions? Highest quality submission Total quality of submissions Number of submissions Difference between the highest and lowest quality submission

74 Mechanism Designer: Questions Any questions? Highest quality submission Total quality of submissions Number of submissions Difference between the highest and lowest quality submission

75 Table of Contents 1 Introduction 2 Related Work 3 Model: Basics 4 Model: Participants 5 Homogeneous Effort 6 Extensions

76 Homogeneous Case Overview Introduction of solution concept Existence of symmetric equilibrium Uniqueness of equilibrium Implementability of equilibrium Asymmetric suboptimal equilibria

77 Homogeneous Case Solution Concept Our goal is to show that, given any V, we can create a best contribution mechanism that implements the optimal outcome (the outcome that maximizes V ). Definition A mechanism is said to implement an outcome in a particular solution concept, if there is an equilibrium under that solution concept which results in the desired outcome. The solution concept we will be using is the Bayes-Nash Equilibrium.

78 Homogeneous Case Existence of Symmetric Equilibrium: Definitions First, we will prove that any best contribution mechanism M B (P B, P C ) has a symmetric equilibrium. Definition Define a threshold strategy to be a strategy where there is some constant ai such that agent i will contribute iff it has ability a i ai.

79 Homogeneous Case Existence of Symmetric Equilibrium: Best Response Note that regardless of the strategies of all other agents, agent i s best response is always a threshold strategy: 1 Agent i s expected payout from rating or doing nothing is independent of its ability a i.

80 Homogeneous Case Existence of Symmetric Equilibrium: Best Response Note that regardless of the strategies of all other agents, agent i s best response is always a threshold strategy: 1 Agent i s expected payout from rating or doing nothing is independent of its ability a i. 2 Agent i s expected payout from submitting is strictly increasing in a i.

81 Homogeneous Case Existence of Symmetric Equilibrium: Best Response Note that regardless of the strategies of all other agents, agent i s best response is always a threshold strategy: 1 Agent i s expected payout from rating or doing nothing is independent of its ability a i. 2 Agent i s expected payout from submitting is strictly increasing in a i. 3 Let a l i be the lowest ability at which agent i will submit; the expected payout of submitting is higher than rating or doing nothing.

82 Homogeneous Case Existence of Symmetric Equilibrium: Best Response Note that regardless of the strategies of all other agents, agent i s best response is always a threshold strategy: 1 Agent i s expected payout from rating or doing nothing is independent of its ability a i. 2 Agent i s expected payout from submitting is strictly increasing in a i. 3 Let a l i be the lowest ability at which agent i will submit; the expected payout of submitting is higher than rating or doing nothing. 4 Then for all a i > a l i agent i must always prefer to submit as well.

83 Homogeneous Case Existence of Symmetric Equilibrium: Best Response Note that regardless of the strategies of all other agents, agent i s best response is always a threshold strategy: 1 Agent i s expected payout from rating or doing nothing is independent of its ability a i. 2 Agent i s expected payout from submitting is strictly increasing in a i. 3 Let a l i be the lowest ability at which agent i will submit; the expected payout of submitting is higher than rating or doing nothing. 4 Then for all a i > a l i agent i must always prefer to submit as well. 5 Thus the threshold strategy with a l i as the threshold is its best response.

84 Homogeneous Case Existence of Symmetric Equilibrium: Restriction Thus, we know that in every equilibrium, all agents must be playing threshold strategies. Furthermore, if we restrict our game to only allowing threshold strategies, any equilibrium we find there is an equilibrium in our full game. Our restricted game is: 1 Symmetric 2 Has a compact and Hausdorff strategy space Thus, our restricted game has a symmetric mixed equilibrium by 1, and so does our full game. 1 J.G. Becker, D.S. Damianov, On the Existence of Symmetric Mixed Strategy Equilibria, Economics Letters, 90(1), 84-87, 2006.

85 Homogeneous Case Uniqueness of Symmetric Equilibrium: Definitions Next, we will prove that any best contribution mechanism M B (P B, P C ) has a unique symmetric equilibrium. Definitions Define P(C >0 a) to be the probability that an agent observes at least one submission from the other n 1 agents if everyone plays a threshold strategy with threshold a. Define P(W a) to be the probability that an agent with ability a wins when all agents are playing a threshold strategy with threshold a. Both P(C >0 a) and P(W a) are continuous; P(C >0 a) is strictly decreasing in a and P(W a) is strictly increasing in a.

86 Homogeneous Case Uniqueness of Symmetric Equilibrium: Expected Payout of Rating An agent will receive: u R (a ) = (P R C R )P(C >0 a ) when all agents play according to a threshold strategy with threshold a. This is continuous and non-increasing in a.

87 Homogeneous Case Uniqueness of Symmetric Equilibrium: Expected Payout of Contributing An agent with ability a will receive: u C (a ) = (P B C C )P(W a ) + (P C C C )(1 P(W a )) when all agents play according to a threshold strategy with threshold a. This is continuous and strictly increasing in a.

88 Homogeneous Case Uniqueness of Symmetric Equilibrium: Equilibrium Type 1 Now consider the function u R (a) u C (a) If there is some a such that u R (a) u C (a) 0 and some u R (a ) u C (a ) 0 then by the intermediate value theorem there is some a such that u R (a ) u C (a ) = 0. Furthermore, we know that a a a because u R is non-increasing and u C is strictly increasing. From this, it is clear that threshold a is the unique symmetric equilibrium.

89 Homogeneous Case Uniquness of Symmetric Equilibrium: Equilibrium Type 2 If u R (a) u C (a) > 0 for all a [0, 1], then the unique symmetric equilibrium is to always rate. If u R (a) u C (a) < 0 for all a [0, 1], then the unique symmetric equilibrium is to always contribute.

90 Homogeneous Case Implementability Proof Overview Recall implementability concept. Proof has two parts. For any V, there exists threshold â that maximizes V in expectation For any â, there exists a pair (p B, p C ) such that the optimal threshold under (p B, p C ) is â. Restriction to symmetric strategy optimization.

91 Homogeneous Case Implementability Let σ be a symmetric strategy of any sort across agents. σ induces some probability λ(σ) across agents participation. Choice of threshold â(σ) = F 1 (1 λ(σ)) trivially replicates σ (continuity of F) Agent quality under â stochastically dominates σ; by monotonicity, maximizes E[V ].

92 Homogeneous Case Implementability If p B c C < 0 and p C c C < 0 then all agents rate, a = 1 If p B c C > p C c C > p R c R all agents contribute; a = 0. Thresholds are continuous in (p B, p C ); Intermediate Value Theorem means that any â is implementable by some (p B, p C ) Therefore, there is some (p B, p C ) that maximizes designer s utility.

93 Homogeneous Case Implementability Value of Result Authors emphasize generality of result. Result has few direct assumptions (what are they?) What structural assumptions does it rely on?

94 Homogeneous Case Comparative Statics Cost of contribution can be determined over time by designers Rewards both increasing in n.

95 Homogeneous Case Asymmetric equilibrium Proof Overview Proof details omitted here. General concept; exists framing of problem such that agent 1 always contributes and others play threshold strategy Hinges on concept that expected payoff from rating when no other agent is contributing is 0. Significantly weakens uniqueness result above; not discussed extensively in paper.

96 Endogeneous Effort Existence Once more, restrict discussion to threshold strategies. Reduction of problem to case where draw of ability determines participation Follows from compactness

97 Endogeneous Effort Optimality Authors construct an optimal case for e i = 1. Follows from proof above on implementability of threshold strategies. But is this reasonable?

98 Endogeneous Effort Optimality Authors construct an optimal case for e i = 1. Follows from proof above on implementability of threshold strategies. But is this reasonable? Neglecting e i (0, 1) seems strange

99 Endogeneous Effort Implementability: Negative Result Definition A Perfectly Ranking mechanism satisfies the criterion π(q i, q i, m) = 1 iff q i > q j i, j Proposition: When a mechanism ranks perfectly, there is no equilibrium that maximizes the designer s utility.

100 Endogeneous Effort Implementability: Negative Result Suppose there exists some optimal threshold â Suppose all other agents are maximizing utility of designer already Consider agent with ability a = â. Since rankings are perfect, agents with higher ability will certainly win. Therefore, equilibrium for this agent is to exert 0 effort.

101 Endogeneous Effort Implementability: Negative Result Result seems weak; why focus on e {0, 1}? Also reliant on the form of designer utility. However, intuition is sound; perfect ranking dissuades possible low-ability high-effort contributors.

102 Endogeneous Effort Implementability: Positive Result Good news! Adverse incentives can be avoided Authors introduce noise to the system to give everyone a chance. Noise condition allows assertion that given c (e i ) q i e i π q i (p B p C )(1 F (â) n 1 ) when p B = c(0), utility maximizing equilbrium is implementable

103 Endogeneous Effort Implementability: Positive Result Critical Condition when p B = c(0) c (e i ) q i e i π q i (p B p C )(1 F (â) n 1 ) Given one other contribution, agent s marginal contribution gets utility b i (q i, q i ) = π(q i, q i )pb + (1π(q i, q i ))p C ve b i = q i π q i (p B p C ) q i e i Probability of another contribution, conditional on threshold strategy, is 1 F (â) n1

104 Endogeneous Effort Implementability: Positive Result Critical Condition when p B = c(0) c (e i ) q i e i π q i (p B p C )(1 F (â) n 1 ) If we know that the critical condition holds for p B = c(0), it holds a fortiori for greater p B. Therefore, agents have positive benefit for exertion of effort; conditional on contribution, e i = 1. Authors then show that there exists some equilibrium in which agents participate iff a i a for some a < â and exert effort e i = 1 conditional on participating.

105 Endogeneous Effort Implementability: Positive Result Theorem proved c (e i ) q i e i π q i (p B p C )(1 F (â) n 1 ) when p B = c(0) implies that mechanism designer can implement optimal outcome. π Key assertion in proof: q i strictly positive. Trick is to design noisy regime such that this holds.

106 Endogeneous Effort Implementability: Significance Result initially seems damning; lazy agents cannot be extracted from bed unless noise is introduced.

107 Endogeneous Effort Implementability: Significance Result initially seems damning; lazy agents cannot be extracted from bed unless noise is introduced. Do not fear. Negative result depends on strong conception of optimality Comprehensive equilibrium analysis would be more convincing

108 Table of Contents 1 Introduction 2 Related Work 3 Model: Basics 4 Model: Participants 5 Homogeneous Effort 6 Extensions

109 Extensions Agents Different quality funcitons. Incentivized both by internet points and attention Strategic rating Mechanism designer has real costs Other mechanisms, with similarly general mechanism designer objective function