Strategic Learning in Teams

Transcription

1 Stategic Leaning in Teams Nicolas Klein This vesion: August, 03 Abstact This pape analyzes a two-playe game of stategic epeimentation with thee-amed eponential bandits in continuous time. Playes play bandits of identical types, with one am that is safe in that it geneates a known payoff, wheeas the likelihood of the isky ams yielding a positive payoff is initially unknown. When the types of the two isky ams ae pefectly negatively coelated, the efficient policy is an equilibium if and only if the stakes ae high enough. If the negative coelation is impefect and stakes ae high, thee eists an equilibium that leads to efficiency fo optimistic enough pio beliefs. Keywods: Stategic Epeimentation, Thee-Amed Bandit, Eponential Distibution, Poisson Pocess, Bayesian Leaning, Makov Pefect Equilibium, R&D Teams. JEL Classification Numbes: C73, D83, O3. Univesité de Montéal and CIREQ. Mailing addess: Univesité de Montéal, Dépatement de Sciences Économiques, C.P. 68 succusale Cente-ville; Montéal, H3C 3J7, Canada. Telephone: ; kleinnic@yahoo.com.

2 Intoduction When faced with two competing hypotheses, economic agents often have to stike a balance between optimally using thei cuent infomation on the one hand, and investing in the poduction of new infomation on the othe hand. When doing so, they have to take into account the impact of thei decisions not just on themselves, but on thei patnes and competitos also; indeed, the latte may benefit fom the infomation a given agent poduces. Think e.g. of two juisdictions, o two hospitals, which ae faced with a cetain disease that could eithe be caused by a vius o by a bacteium. Now, eithe hospital has to decide which of the two competing hypotheses to investigate, e.g. by administeing eithe an antibiotic o an anti-vial dug. Once one of them has found out which hypothesis is tue, both benefit fom the discovey. Thus, while the costs of epeimentation have to be bone pivately, any infomation an agent poduces is a public good. This makes fo a situation in which a playe s epeimentation decisions ae stategic, in that they affect the othe playe s payoffs. I model this tade-off as a thee-amed stategic bandit poblem. Specifically, I conside two playes opeating thee-amed eponential bandits in continuous time. One am is safe in that it yields a known flow payoff, wheeas the othe two ams ae isky, i.e. they can be eithe good o bad. The isky ams ae meant to symbolize two mutually incompatible hypotheses. In the baseline case, I assume that it is common knowledge that eactly one of them is good. In Section 6, I etend the analysis to the case in which thee is some chance that both hypotheses may be bad, while maintaining the assumption that the two hypotheses cannot be tue at the same time; i.e. it could be that the disease is neithe caused by a vius no by a bacteium (it could e.g. be genetic, but it is cetainly not caused by both. Playes ae playing eact cabon copies of the same bandit machine; conditional on the state of the wold, daws ae iid between the playes (i.e. playes ae playing so called eplica bandits. The bad isky am neve yields a positive payoff, wheeas a good isky am yields positive payoffs afte eponentially distibuted times. As the epected payoff of a good isky am eceeds that of the safe am, playes will want to know which isky am is good. As eithe playe s actions, as well as the outcomes of his epeimentation, ae pefectly publicly obsevable, thee is an incentive fo playes to fee-ide on the infomation the othe playe is poviding; infomation is a public good. Obsevability, togethe with a common pio, implies that the playes beliefs agee at all times. As only a good isky am can eve yield a positive payoff, all the uncetainty is esolved as soon as eithe playe has a beakthough on a isky am of his and beliefs become See Klein & Rady (0. Fo an oveview of the bandit liteatue, see Begemann & Välimäki (008.

3 degeneate at the tue state of the wold. As all the payoff-elevant stategic inteaction is captued by the playes common belief pocess, I estict playes to use stationay Makov stategies with thei common posteio belief as the state vaiable, thus making my esults diectly compaable to those in the pevious stategic epeimentation liteatue. Playes have to bea epeimentation costs pivately; the benefit, by contast, is public. Thee ae thus obvious incentives fo playes to fee-ide on thei patne s epeimentation. Indeed, inefficiency because of fee-iding has been a staple esult of the liteatue on stategic epeimentation with bandits. Using distibutional assumptions simila to ous, Kelle et al. (005 e.g. have shown that in the game with positively coelated bandits, the efficient benchmak is neve sustainable in equilibium. 3 In Klein & Rady (0, howeve, full efficiency, egading both the amount and the speed of epeimentation, is an equilibium if, and only if, stakes ae below a cetain theshold. In the pesent setting, by contast, I show that, if playes know that eactly one isky am is good, fee-iding incentives can be completely ovecome if and only if the stakes eceed a cetain theshold; in this case, thee eists an efficient equilibium. The esult etends to the impefect negative coelation setting of Section 6 in the sense that, fo high enough stakes, thee eists an equilibium that leads to playes behaving efficiently povided thei pio beliefs ae optimistic enough. The est of the pape is stuctued as follows: Section eviews some elated liteatue; Section 3 intoduces the model; Section 4 analyzes the utilitaian planne s poblem; Section 5 analyzes the non-coopeative game, ehibiting a necessay and sufficient condition fo the eistence of an efficient equilibium; Section 6 investigates the obustness of ou main esult to a moe geneal coelation stuctue; and Section 7 concludes. Some auiliay esults and poofs ae povided in the Appendi. Related Liteatue Chattejee & Evans (004 analyze a teasue-hunting game in discete time, whee it is common knowledge that eactly one of seveal pojects is good. As in my model, they allow playes to switch pojects at any point in time. The game ends as soon as one of the playes finds the teasue. As in my model, they find that, fo high enough stakes, thee eists an efficient equilibium. The foces leading to this esult ae vey diffeent fom those at wok in my model, though. The nub is that the Chattejee & Evans (004 game also involves payoff etenalities, in the sense that if playe finds the teasue it is lost to playe. In actual fact, thei winne takes all, and hence intenalizes all the social 3 See also Bolton & Hais (999,000, and Kelle & Rady (00.

4 gains of his discovey. In my model, by contast, etenalities ae puely infomational in natue; when one agent makes a discovey, the othe agent fully pofits fom the infomation geneated by this beakthough, as he will imitate his patne s successful appoach in the futue. Thus, Chattejee & Evans (004 model may be bette-suited e.g. to the analysis of epeimentation by ival fims competing fo maket shae; mine may be moe appopiate to the case of diffeent juisdictions investigating the impact of vaious teatment options fo a paticula disease, o if e.g. one wants to analyze fee-iding incentives by scientists woking fo the same fim o in the same lab, and the like. My model lets agents choose themselves which hypothesis to eploe; Klein & Rady (0 by contast assign one hypothesis each to eithe playe. The compaison between my model and Klein & Rady (0 would suggest that delegation was a good idea if the stakes at play wee high; if the stakes ae low, howeve, it can be bette to assign one hypothesis each to eithe playe, the compaison suggests. Indeed, it is notable that, depending on the cicumstances, fims o institutions seem to pusue quite diffeent appoaches in this espect. Fo instance, subsequently to maked gowth in the numbe of its eseach laboatoies and facing inceasing competitive pessues, 3M, which aguably makes moe low-stakes poducts such as adhesives and abasives, moved to estict scientists discetion ove thei wok, which had taditionally been vey vast (see Batlett & Mohammed, 995. By contast, fims in the aguably moe high-stakes phamaceutical secto moved in the opposite diection. Swiss phamaceutical giant Novatis, fo instance, enteed into a multi-million five-yea ageement with the Depatment of Micobial and Plant Biology at Bekeley, CA, delegating poject decisions to a committee being compised of five epets, only two of whom wee Novatis employees (see Lacetea, 008 a scheme that can easonably be intepeted as a commitment device on the pat of Novatis to delegate poject choice to thei scientific patnes in academia. A somewhat simila deal had ealie been signed by Thousand Oaks, CA, based phamaceutical company Amgen and MIT; Lawle (003 quotes MIT biologist Nancy Hopkins: Thee was no attempt by eithe side to change the diection of ou basic eseach in the aftemath of the ageement. 4 The pesent pape belongs to the liteatue on stategic epeimentation with bandits. While bandit models have been analyzed as ealy as the 950s (see e.g. Bellman, 956, Badt et al, 956, Robbins, 95, thei use in economics haks back to the discete-time model of Rothschild (974. Wheeas the fist papes analyzing stategic inteaction featued a 4 The optimal allocation of eseach pojects between academia and the commecial secto is the subject of papes by Aghion et al. (008, as well as by Lacetea (008, who intepet academia as a commitment device fo pincipals not to intefee with scientists discetion. The fictions at the heat of both of these papes ely on the assumption that scientists pefeences divege fom those of economically oiented, pofitmaimizing, fims. 3

5 Bownian motion model (Bolton & Hais, 999, 000, the eponential famewok I use was fist analyzed by Pesman (990 in a single-agent setting, and has poved itself to be moe tactable (see Kelle et al, 005, Kelle & Rady, 00, Klein & Rady, 0. In this liteatue, my pape is most elated to Kelle et al. (005 and Klein & Rady (0. Kelle et al. (005 show that with eplica two-amed bandits thee does not eist an equilibium in cutoff stategies, 5 and that the amount, as well as the speed, of leaning ae inefficiently low in equilibium. Klein & Rady (0 show that with pefectly negatively coelated two-amed bandits thee ae equilibia in cutoff stategies; the long-un amount of epeimentation is always at efficient levels, though the speed of epeimentation may be too low. Howeve, thee eists an efficient equilibium if, and only if, the stakes ae below a cetain theshold. These esults etend to the case of impefect negative coelation in the sense that thee will always eist an equilibium in cutoff stategies in which the long-un amount of epeimentation will be at the efficient level. In my model, playes play eplica bandits, and both playes will have access to both types of isky am at any time. While the afoe-mentioned papes, as well as the pesent one, assume both actions and outcomes to be public infomation, Bonatti & Höne (0 analyze stategic inteaction unde the assumption that only outcomes ae publicly obsevable, while actions ae pivate infomation. Rosenbeg et al. (007, as well as Muto & Välimäki (0, analyze the two-amed poblem of public actions and pivate outcomes in discete time, assuming action choices ae ievesible. Recently, thee has also been an effot at genealization of eisting esults in the decision-theoetic bandit liteatue. Fo eample, Bank & Föllme (003, as well as Cohen & Solan (009, analyze the single-agent poblem when the undelying pocess is a geneal Lévy pocess. Camago (007 also analyzes the poblem of a single decision make, who faces a twoamed bandit with coelated ams. He consides any numbe of possible states of the wold as well as quite geneal distibutional assumptions. He shows that if the set of outcomes can be odeed in such a way that highe outcomes ae good news fo one altenative and bad news fo the othe, a cutoff policy is optimal. 6 As, in geneal, beliefs ae only patially odeed, cutoff beliefs need not be unique. In the baseline case of the pesent pape, stategic inteaction between two playes is analyzed in a setting with only two possible states of the wold, so that beliefs can be epesented by a single numbe in [0, ]; Section 6 etends the analysis to thee possible states of the wold. The pesent pape is also somewhat elated to the Moal Hazad in teams liteatue, to 5 A cutoff stategy is a stategy of the fom play isky if and only if my belief eceeds a given cutoff. 6 In this moe geneal setting, a cutoff policy is of the fom if am A is used at a given belief, then am A is used at highe beliefs as well. 4

6 which Holmstöm (98 povided the seminal contibution. He found that the intoduction of a pincipal acting as a budget beake was apt to achieve fist-best effot levels on the pat of team membes. Manso (0 fomalizes an agent s decision among an established poduction method of known yield, an innovative method with as yet unknown yields, and shiking, as a thee-amed bandit in a two-peiod pincipal-agent model. His focus is on the wage schemes a pincipal would optimally offe the agent to induce him to choose the pincipal s pefeed poduction method. In Klein (0, I show that in a continuous-time model, the availability of the known poduction method eithe endes the implementation of the unknown method impossible, o does not distot its costs at all. 3 The Model I conside a model of two playes, each of whom opeates a thee-amed bandit in continuous time. One am is safe in that it yields a known flow payoff of s > 0; both othe ams, A and B, ae isky, and in the baseline case, it is commonly known that eactly one of these isky ams is good and one is bad. The bad isky am neve yields any payoff. The good isky am yields a positive payoff with a pobability of dt if played ove a time inteval of length dt; the appetaining epected payoff incement amounts to g dt. Playes discount payoffs at the common discount ate > 0. The constants,, s and g ae common knowledge; the only uncetainty is which of the two isky ams is good. The common pio is that A is good with pobability p 0. This belief evolves based on the histoy of epeimentation and payoffs. These ae commonly obsevable and so the playes continue to have a common belief (pobability that am A is good, which we denote by p t, at all times t 0. At each point in time, both playes eceive a flow endowment of one unit of a pefectly divisible esouce. Eithe playe s objective is to maimize his own epected discounted payoffs by choosing the faction of his endowment flow that he wants to allocate to eithe isky am. Specifically, eithe playe i {, } chooses a stochastic pocess {(k i,a, k i,b (t} 0 t which is measuable with espect to the infomation filtation that is geneated by the obsevations available up to time t, with (k i,a, k i,b (t {(a, b [0, ] : a + b } fo all t; k i,a (t and k i,b (t denote the faction of the esouce devoted by playe i at time t to isky ams A and B, espectively. 7 Thoughout the game, eithe playe s actions and payoffs ae 7 Hee, putting a faction of the available esouces on a isky poject means that the pobability of getting a lump-sum ewad is educed at any moment of time, but the size of the ewad does not change. It can also be viewed as an appoimation fo a situation in which only one am can be pulled at any moment but a policy may change abitaily quickly between the ams, spending faction k i,a (k i,b of the time on am 5

7 pefectly obsevable to the othe playe. Specifically, playe i seeks to maimize his total epected discounted payoff [ ] E e t [( k i,a (t k i,b (ts + (k i,a (tp t + k i,b (t( p t g] dt, 0 whee the epectation is taken with espect to the pocesses {p t } t R+ and {(k i,a, k i,b (t} t R+. As can immediately be seen fom this objective function, thee ae no payoff etenalities between the playes; the only channel though which the pesence of the othe playe may impact a given playe is via his belief p t, i.e. via the infomation that the othe playe is geneating. Thus, ous is a game of puely infomational etenalities. As only a good isky am can eve yield a lump sum, beakthoughs ae fully evealing. Thus, if thee is a lump sum on isky am A (B at time τ, then p t = (p t = 0 at all t > τ. If thee has not been a beakthough by time τ, Bayes Rule yields p τ = p 0 e τ 0 K A,t dt p 0 e τ 0 K A,t dt + ( p 0 e τ 0 K B,t dt, whee K A,t := k,a (t + k,a (t and K B,t := k,b (t + k,b (t. Thus, conditional on no beakthough having occued, the pocess {p t } t R+ will evolve accoding to the law of motion almost eveywhee. p t = (K A,t K B,t p t ( p t As all payoff-elevant stategic inteaction is captued by the playes common posteio beliefs {p t } t R+, it seems quite natual to focus on Makov pefect equilibia with the playes common posteio belief p t as the state vaiable. As is well known, this estiction is without loss of geneality in the planne s poblem, which is studied in Section 4. A Makov stategy fo playe i is any piecewise continuous function (k i,a, k i,b : [0, ] {(a, b [0, ] : a + b }, p t (k i,a, k i,b (p t, meaning that it is continuous at all but a finite numbe of points. Following Klein & Rady (0, I say that a pai of Makov stategies is admissible if thee eists at least one well-defined solution to the coesponding law of motion of beliefs; in case of multiple solutions, the unique solution that is consistent with a discete-time appoimation is selected. given by Given an admissible stategy pai ((k,a, k,b (p t, (k,a, k,b (p t, the playes belief is A (B, fo instance. p τ = p 0 e τ 0 K A(p t dt p 0 e τ 0 K A(p t dt + ( p 0 e τ 0 K B(p t dt, 6

8 if thee has not been a beakthough by time τ, with K A (p t := k,a (p t + k,a (p t and K B (p t := k,b (p t +k,b (p t. Each admissible stategy pai (k, k = ((k,a, k,b, (k,a, k,b induces a pai of payoff functions (u, u with u i given by u i (p k, k = [ } ] E e {(k t i,a (p t p t + k i,b (p t ( p t g + [ k i,a (p t k i,b (p t ]s dt p 0 = p 0 fo each i {, }. Fo stategy pais that ae not admissible, I set u = u =. In the subsequent analysis, it will pove useful to make case distinctions based on the stakes at play, as measued by the atio of the epected payoff of a good isky am ove that of a safe am ( g, the playes impatience (as measued by the discount ate, and the s Poisson aival ate of a good isky am, which can be intepeted as the playes innate ability at finding out the tuth: I say that the stakes ae high if g 4(+ ; stakes ae s +3 intemediate if + < g < 4(+ ; stakes ae low if g +; they ae vey low if g < (+. + s +3 s + s + 4 The Planne s Poblem Fist, we investigate a benevolent utilitaian planne s solution to the two-playe poblem at hand. As the planne does not cae about the distibution of suplus, and both playes ae equally apt at finding out the tuth, all that mattes to him is the sum of esouces devoted to eithe type of isky am, K A (p t and K B (p t, espectively. Put diffeently, the planne s poblem is equivalent to that of a single agent contolling twice the esouces. In ode to make his value function compaable to that of ou playes, we nomalize the planne s value by the facto. Now, staightfowad computations show that the planne s Bellman equation is given by 8 [ u(p = s + ma {K A B A (p, u c ] [ A(p + K B B B (p, u c ]} B(p, {(K A,K B [0,] :K A +K B } whee c A (p := s pg and c B (p := s ( pg measue the myopic oppotunity costs of playing isky am A (isky am B athe than the safe am. By contast, B A (p, u := p[g u(p ( pu (p] and B B (p, u := ( p[g u(p + pu (p] measue the value of infomation gleaned fom playing isky am A (o isky am B, espectively. 9 As the Bellman equation is linea in the planne s choice vaiables, it is without loss of geneality 8 By standad aguments, if a continuously diffeentiable function solves the Bellman equation, it is the value function. 9 By the standad pinciple of smooth pasting, the planne s payoff function fom playing an optimal policy is once continuously diffeentiable. 7

9 fo me to estict attention to cone solutions, fo which it is staightfowad to deive closed-fom solutions fo the value function, which ae ehibited in Appendi A. The optimal policy depends on whethe the stakes at play, as measued by the atio g (+, eceed the theshold of o not. Note that g (+ if and only if s + s + p, whee p p :=. It will be convenient futhemoe to define the odds atio Ω(p := s (+(g s+s and ū := + (+ planne s solution is summaized in the following poposition: g, the planne s payoff when investing one unit each pe isky am. The Poposition 4. (Planne s Solution If g < (+, the planne will invest all his esouces in am A on (p, ]; in am B on [0, p ; and in the safe am on [ p, p ]. s + The coesponding payoff function is given by [ ] g p + p p +p (Ω(pΩ(p if p p, u(p = s [ if p p p, ( ] g p + p p +( p Ω(p if p p Ω(p. p, If g > (+, the planne will invest all his esouces in am A on (, ], and in am s + B on [0,. At p =, he will split his esouces equally between the isky ams. The coesponding payoff function is given by u(p = { g [ p + g [ p + + Eithe solution is optimal if g s = (+ +. ] + pω(p if p, ( pω(p ] if p. Poof: Optimality is established by a standad veification agument, which is in Klein (0 and omitted hee. Note that if the stakes ae vey low, thee is no option value to the initially less pomising isky am, since the planne will neve make use of it. As is easily veified, the optimal solution in this case implies incomplete leaning. Indeed, let us suppose that it is isky am A that is good. Then, if the initial pio p 0 is in [0, p, we have that lim t p t = p with pobability. If p 0 [ p, p ], then p t = p 0 fo all t, since the planne will always play safe. If p 0 (p, ], it is staightfowad to show that the belief will convege to p with pobability Ω(p 0 Ω(p, while the tuth will be found out (i.e. the belief will jump to with the counte-pobability. Hence, thee is always a positive pobability that the tue state of the wold will not be found out, i.e. leaning is incomplete. 8

10 If g s > (+ +, howeve, which is the case if and only if ū > s, the planne will neve avail himself of the option to play safe; his solution will ensue that leaning be complete, i.e. that the tuth will eventually be found out with pobability. As the planne does not cae which of the isky ams is good, the solution is symmetic aound p =. At the switch point p = itself, the planne s actions ae pinned down by the need to ensue a well-defined law of motion of the state vaiable. Thus, thee is now an option value to the initially less pomising isky poject, as the planne will make use of it with stictly positive pobability, whateve his initial belief p 0 (0, may be. At the knife-edge case of g = (+, the planne is indiffeent ove all thee ams at s + p =. Yet, in ode to ensue a well-defined time path of beliefs, he has to set K A( = K B ( [0, ]. The single-agent optimum has the same stuctue as the planne s solution; all that changes in the elevant diffeential equations is that is eplaced by. Of couse, the elevant cutoffs will also change as a esult: In the single-agent poblem, complete leaning will obtain fo g > +; fo g < +, the agent will switch fom isky am A to the safe s + s + am at the cutoff belief p s := > (+g s p, and fom isky am B to the safe am at p. Thus, wheneve the stakes ae below the elevant theshold, the second isky option does not play a ole; hence, it is not supising that the same cutoff will be applied as in the poblem with two-amed eponential bandits, whee p and p ae the elevant cutoffs in the single-agent poblem and the planne s poblem with two eplica bandits, espectively, as Poposition 3. in Kelle et al. (005 shows. 5 Equilibia of the Non-Coopeative Game Poceeding as befoe, I find that the Bellman equation fo playe i (i j is given by 0 u i (p = s + k j,a B A (p, u i + k j,b B B (p, u i + ma {k i,a [B A (p, u i c A (p] + k i,b [B B (p, u i c B (p]}. {(k i,a,k i,b [0,] :k i,a +k i,b } As playes ae pefectly symmetic in that they ae opeating two eplicas of the same bandit, the Bellman equation fo playe j looks eactly the same. On account of the linea 0 By the smooth pasting pinciple, playe i s payoff function fom playing a best esponse is once continuously diffeentiable on any open inteval on which (k j,a, k j,b (p is continuous. If (k j,a, k j,b (p ehibits a jump at p, u i (p, which is contained in the definitions of B A and B B, is to be undestood as the one-sided deivative in the diection implied by the motion of beliefs. In eithe instance, standad esults imply that if fo a cetain fied (k j,a, k j,b, the payoff function geneated by the policy (k i,a, k i,b solves the Bellman equation, then (k i,a, k i,b is a best esponse to (k j,a, k j,b. 9

11 stuctue of the optimization poblem, we can estict ou attention to the nine pue stategy pofiles, along with thee indiffeence cases pe playe. Each of these cases leads to a fistode odinay diffeential equation (ODE. A leading case is ehibited in Appendi A; a full oveview can be found in Klein (0. The lineaity of the poblem povides us with a poweful tool to deive necessay conditions fo a cetain stategy combination ((k,a, k,b, (k,a, k,b to be consistent with mutually best esponses on an open set of beliefs. As an eample, suppose playe is playing (, 0. If playe s best esponse is given by (, 0, it follows immediately fom the Bellman equation that it must be the case that B A (p, u c A (p and B A (p, u B B (p, u c A (p c B (p fo all p in the open inteval in question. Moeove, we know that in the open inteval in question, the playe s value function satisfies p( pu (p + (p + u (p = ( + pg, which can be plugged into the two inequalities above, yielding a necessay condition fo (k,a, k,b = (, 0 to be a best esponse to (k,a, k,b = (, 0. Poceeding in this manne fo the possible pue-stategy combinations gives us necessay conditions fo a cetain puestategy combination to be consistent with mutually best esponses on an open inteval of beliefs. I epot these necessay conditions as an auiliay esult in Appendi A. It is notewothy that, as we can see immediately fom the Bellman equation, a playe only has to bea the oppotunity costs of his own epeimentation, while the benefits accue to both, which indicates the pesence of fee-iding incentives. Fo futue efeence, I define the myopic cutoff belief p m := s g by c A(p m = 0. A playe who was only inteested in maimizing his cuent payoff would switch fom isky am A (B to the safe am at p m ( p m. Such fee-iding incentives would also appea in a much simple model without uncetainty. Suppose the safe am, which, when pulled, yielded a payoff flow of s, was paied with an am that was known to be good, and to yield a lump sum of h, to be shaed equally by both playes, accoding to a known Poisson distibution with paamete (k,t + k,t, whee (k,t, k,t [0, ] denoted the popotion of his unit endowment flow that eithe playe devoted to the Poisson am at instant t. As is easy to veify, efficiency would equie both playes to pull the good am if h > s; the playes would be willing to do so in Makov equilibium, howeve, only if h s. 3 Hence, efficiency would pevail in equilibium fo As we keep playe j s stategy (k j,a, k j,b fied on an open inteval of beliefs, playe i s value function u i (i j is of class C on that open inteval. Theefoe, by standad aguments, u i solves the Bellman equation on the open inteval in question. I am indebted to an anonymous efeee fo pointing this out. 3 As hee the Poisson am is known to be good, i.e. beliefs ae degeneate, the Makov estiction essentially 0

12 vey lage and vey small h. Fo s < h < s, howeve, efficiency would equie both playes to use the good am, but they would both pefe to play safe in equilibium. Yet if thee is some chance that the Poisson am is bad, and neve yields any payoffs, then in the case of many unsuccessful daws, playes beliefs may each a level of pessimism such that the epected payoff fom pulling the Poisson am entes the fee-iding egion, howeve lage h may be. As even vey optimistic playes know that this is going to happen with positive pobability at some point down the oad, thei equilibium utility is lowe than in the efficient benchmak. This is why in these sots of poblems, thee is typically too little epeimentation in equilibium. 4 By contast, I find that the planne s solution is compatible with equilibium if and only if stakes eceed a cetain theshold. This may at fist glance seem supising given that, in contast to Chattejee & Evans (004, a playe does not fully intenalize the benefits of his discovey. Indeed, playes povide a positive infomational etenality though thei epeimentation; any infomation a playe geneates helps his patne make bette decisions in tun. This is the eason why efficiency is not sustainable in equilibium in Kelle et al. (005. In Klein & Rady (0, this calculation changes, though, when the stakes ae so low that the playes espective single-agent cutoffs do not ovelap: In this case, the moe pessimistic playe will neve play isky unde any cicumstances, which the moe optimistic playe will anticipate, and hence behave efficiently. Howeve, the efficient equilibium disappeas as soon as the elevant single-agent cutoffs ovelap and fee-iding incentives kick in again. While it is not supising that the utilitaian planne, who now has moe options, should always be doing bette than the planne in Klein & Rady (0, who could not tansfe esouces between the two types of isky am, it may seem somewhat supising that, fo high stakes, the playes should now be able to achieve even this highe efficient benchmak, while they could not achieve the lowe benchmak in the pefectly negatively coelated two-amed model in Klein & Rady (0. Indeed, with the stakes high enough, fee-iding incentives can be ovecome completely in non-coopeative equilibium, as the following poposition shows. Poposition 5. (Efficient Equilibium Thee eists an efficient equilibium if and only if g s 4(+ +3. Poof: See Appendi B. equies playes to pick one am and stick with it. 4 See Bolton & Hais (999, 000 Kelle et al. (005, Kelle & Rady (00.

13 Since playes ae playing eplica bandits, thee will neve aise a situation in which one playe is optimistic while the othe one is pessimistic; as soon as one playe finds it optimal to epeiment in isolation then so will the othe playe, and fee-iding incentives ente the pictue again. Theefoe, the Klein & Rady (0 channel effecting efficiency cannot be at wok hee, whateve the stakes might be. Fo high stakes, a diffeent channel will kick in, though: On account of pefect negative coelation between the isky ams, playes will neve simultaneously be vey pessimistic about both pospects. Hence, fo stakes above a cetain theshold, they would neve conside the safe option. This is even though they ae still eeting a positive etenality, which they do not mind, howeve, as individual incentives to foeswea the safe option ae stong enough. Moeove, since thee ae no switching costs in my model, playes would use the isky am that looks momentaily moe pomising if they wee left to thei own devices. Thus, in the absence of specific incentives to deviate fom this policy, they would do what efficiency equies. In paticula, if the othe playe behaves efficiently, a playe s best esponse calls fo behaving efficiently also; i.e. thee eists an efficient equilibium. 5 In Chattejee & Evans (004 efficient equilibium, playes behave myopically. Note that this is not necessaily the case hee, as the elevant theshold above which fee-iding incentives ae totally eclipsed is lowe than (above which epeimentation becomes costless, i.e. myopically optimal, at all beliefs. This is because playes take the leaning benefit of epeimentation into account, at least to the etent it benefits the playe himself. Thus, it is no supise that the elevant theshold should be inceasing in the playes impatience, and deceasing in the infomativeness of epeimentation, as measued by. Fo beliefs below this theshold, one can still constuct a symmetic Makov pefect < g < 4(+, such an equilibium featues s +3 equilibium. 6 If stakes ae intemediate, i.e. + + complete leaning, as efficiency equies. If the stakes ae low, i.e. g s +, thee eists + an equilibium in which playes eclusively use the safe am on [ p, p ]. Hence, this equilibium implies incomplete leaning, while efficiency equies complete leaning fo g s > (+. + Thus, while ou analysis would unambiguously suggest that, if stakes wee high, delegating poject choice to the agents was a good idea since it inceased epeimentation intensities, this conclusion need not hold fo g < +. Fo this case, Klein & Rady (0 s + have shown that if agents ae assigned one of the pojects each, the unique equilibium 5 Holmstöm (98 shows that a team cannot poduce efficiently in the absence of a budget-beaking pincipal, on account of payoff etenalities between team membes. By contast, my analysis shows that, in a model with puely infomational etenalities in which playes can choose whethe to investigate a given hypothesis o its negation, the efficient solution is an equilibium if the stakes at play eceed a cetain theshold. 6 See a pevious vesion fo details (Klein, 0.

14 featues an epeimentation intensity of fo the moe pomising poject thoughout [0, p (p, ]. By contast, in the equilibium fo low stakes we have mentioned hee and which is discussed in moe detail in Klein (0, the oveall epeimentation intensity inceases continuously fom 0 at p to at some ˆp > p (deceases continuously fom at ˆp to 0 at p. Thus, fo initial beliefs just above p, fo instance, the ate of epeimentation may be highe if scientists do not have the feedom to choose the hypothesis they ae woking on. Hence, if the stakes ae low, as aguably they might be at a company like 3M, it might make sense to estict scientists discetion, whilst delegation would seem advisable in moe high-stakes sectos, such as phamaceutics, fo instance. 6 Genealized Pessimism Heetofoe, we have assumed that playes ae cetain that eactly one of thei isky ams is good. The pupose of this section is to investigate whethe ou main esult that fee-iding incentives can be ovecome fo stakes eceeding a cetain theshold etends to situations of genealized pessimism à la Klein & Rady (0. In this setting, thee ae thee possible states of the wold: Eithe only am A is good, o only am B is good, o neithe am is good. Thus, the types of ams A and B ae still negatively coelated, yet the negative coelation is no longe pefect. Following the constuctive appoach in Klein & Rady (0, I show that the esults fo this case will be mied: Thee does not eist an equilibium in which playes behave efficiently at all beliefs; howeve, if the stakes ae high enough and initial geneal pessimism is low, it is possible to constuct a symmetic Makov pefect equilibium pescibing efficient behavio at all beliefs that ae eached with positive pobability along the path of play. Ou state vaiable (p A, p B will now be two-dimensional, with p I,t denoting the playes (subjective pobability at time t that am I {A, B} is good. Thus, p B,0 p A,0 is an (invese measue of initial geneal pessimism. As befoe, playes i will be esticted to stationay Makov stategies (k i,a, k i,b : [0, ] {(a, b [0, ] : a + b }. Admissibility of stategies is defined in analogy to the case of pefect negative coelation. motion of beliefs ae given by and ṗ A,t = p A,t [K A,t ( p A,t K B,t p B,t ], ṗ B,t = p B,t [K B,t ( p B,t K A,t p A,t ]. The laws of As ou state space is now two-dimensional, the deivation of eplicit solutions fo playes payoff functions now equies the solution of Patial Diffeential Equations (PDE. Details 3

15 ae povided in Appendi A. The following poposition summaizes the efficient benchmak fo this case. The elevant cutoff is again given by p, as in the case of pefect coelation. When the pobability of being good dops below this theshold fo both ams, the planne will give up and hencefoth play safe. At all othe beliefs, the planne will devote all of his esouces to the am that is moe likely to be good. When both ams ae equally likely to be good, he uses both at equal intensity. Poposition 6. (The Planne s Solution The planne sets (K A, K B = (0, 0 if (p A, p B [0, p ] ; (K A, K B = (, 0 if p A > ma{p B, p }; (K A, K B = (0, if p B > ma{p A, p }; and (K A, K B = (, if p A = p B > p. Poof: Poof is by a veification agument, see Appendi B. In the following poposition, I show that, if the stakes ae high, i.e. if g s > 4(+ +3, thee eists a egion of pio beliefs which attach little enough weight to both ams being bad, such that, stating out fom these pio beliefs, playes behavio will be efficient with pobability in the equilibium I constuct. Howeve, in maked contast to the case of pefect coelation, thee neve eists an equilibium in which playes behave efficiently at all beliefs, no matte how high the stakes may be. Indeed, as in the case of pefect coelation, playes ae not willing to put in the equied effot at beliefs close to p. Yet, with impefect coelation, thee ae some points in the belief space at which the pobability that the moe pomising am will be good is discouagingly low, something that neve happens fo high enough stakes when the coelation is pefect. Hence we can have an efficient equilibium in the latte case but not in the fome. The key to the constuction of an equilibium that achieves efficient behavio given optimistic pio beliefs is that once beliefs each the 45-degee line in (p A, p B -space, deviations ae pecluded by the admissibility equiement that stategies lead to a well-defined time path of beliefs which is consistent with Bayes ule, a logic simila to that applying at the point p = in the case of pefect coelation. Thus, efficiency can be attained if and only if playes ae still optimistic enough at the moment they each the 45-degee line, which is p the case fo high enough atios B,0 p A,0. In the limiting case when p B,0 p A,0 = (pefect coelation, the 45-degee line collapses to the point p =, and playes ae still optimistic enough when they each that point if and only if the stakes ae high, i.e. g 4(+. In case this s inequality holds with stictness, thee eists an η > 0 such that if p B,0 p A,0 +3 η, playes ae still optimistic enough to do the wok efficiency equies of them when beliefs each the 45-degee line. The following poposition summaizes these findings. 4

16 Poposition 6. (Genealized Pessimism If p B,0 < p A,0, thee does not eist an efficient equilibium. If g > 4(+, thee eists an η > 0 such that, if p B,0 s +3 p A,0 η, thee eists an equilibium in which playes behavio coincides with the efficient benchmak with pobability. If g 4(+, playes behave inefficiently with positive pobability in all s +3 equilibia fo all non-degeneate pio beliefs p A,0 + p B,0 < not in [0, p ] {(p A, p B : p A = p B }. Poof: Hee, I give a vey shot sketch of the poof, the full poof being povided in Appendi B. Playe i s Bellman equation is given by (i j u(p A, p B = s + k j,a B A (p A, p B, u + k j,b B B (p A, p B, u + ma (k i,a,k i,b {k i,a [B A (p A, p B, u c A (p A ] + k i,b [B B (p A, p B, u c B (p B ]}. By symmety, we can focus on the case p A,0 p B,0. We note that if only am A is used on a time inteval [t, t + (with > 0, p A,0 > p B,0, and := p B,0 p A,0 p p c A (p A fo p A close to p (whee u P p B,τ p A,τ emains constant fo all τ [t, t +. Now, let. It is now staightfowad to show that B A (p A, p B, u P < denotes the value function of the planne s poblem. This aleady establishes that thee is no efficient equilibium if p B,0 < p A,0. ( Now let g 4(+. Fo this case, one can show that, fo each p,, B s +3 p A (p A, p B, u P < c A (p A fo p A > p B and p A close to p B. This establishes that if g 4(+, playes behave s +3 inefficiently with positive pobability in all equilibia fo all non-degeneate pio beliefs p A,0 + p B,0 < not in [0, p ] {(p A, p B : p A = p B }. Now, let g > 4(+. One shows that thee eists a neighbohood N of = such that, s +3 fo all N, B A (p A, p B, u P c A (p A and B A (p A, p B, u P c A (p A B B (p A, p B, u P c B (p B fo p A > p B and p B p A stating fom pio beliefs (p A,0, p B,0 satisfying p B,0 p A,0 thoughout with pobability. N. The poof now constucts an equilibium such that, N, playes behavio be efficient In this equilibium, actions at beliefs p A p B ae as follows: Both playes play safe if p A p. If p A = p B > p, one playe plays (, 0 while the othe plays (0, until we each (p, p. If p A > ma{p B, p } and p B p A N, both playes use am A. If p A > ma{p B, p } and N, both playes mi ove am A and the safe am close to the 45-degee line in p B p A (p A, p B -space (if p o the line p p A = p (if < p, espectively. Away fom the p 45-degee line o the line p A = p espectively, both playes use am A if p A > ma{p B, p }. If p B < p A p, both playes play safe. At beliefs p A < p B, the symmetic actions pevail, with the oles of am A and am B evesed. 5 That these stategies constitute mutually

17 best esponses is established by a veification agument, the details of which ae povided in Appendi B. 7 Conclusion I have analyzed a game of stategic epeimentation with thee-amed bandits. We have seen that when the two isky ams ae pefectly negatively coelated, the efficient solution can be sustained as a Makov pefect equilibium if and only if stakes ae high. While making my esults easily compaable with the eisting liteatue, the estiction to Makovian equilibia of couse ules out histoy-dependent play, which is familia fom discete time, yet technically athe inticate to fomalize in continuous time. Appopiate continuous-time concepts can e.g. be found in Begin & MacLeod (993. Höne et al. (03 etend the analysis of the epeimentation game with eplica two-amed Poisson bandits à la Kelle & Rady (00 to non-makovian equilibia. While a full chaacteization and analysis of non-makovian equilibia lies outside the scope of this pape, it seems clea that simple gim tigge equilibia could achieve the efficient outcome fo a substantially lage set of paametes. Indeed, fo stakes that ae not vey low, i.e. fo (+ < g < 4(+, conside + s +3 beliefs p, p (0,, with p and p vey close to 0, and, espectively. On (p, p, the payoff in the planne s solution is bounded away fom the payoff in the Makov pefect equilibium fo g < 4(+ that we have mentioned above. Thus, the theat of punishing any unilateal s +3 deviation with an immediate and indefinite evesion of play to the symmetic Makov pefect equilibium gives playes appopiate incentives to behave efficiently on (p, p. On [0, p] [ p, ], meanwhile, playes want to behave efficiently even if deviations ae ignoed. If g < s (+, howeve, this simple constuction fails, as it is now no longe the case that, away fom + p = 0 and p =, playes utility in the planne s solution is bounded away eveywhee fom that in the Makov pefect equilibium. We have seen that ou esults to some etent genealize when we additionally allow fo the possibility of both isky ams being bad; indeed, if playes initially assess the likelihood of both ams being bad as low enough, thee eists an equilibium inducing efficient behavio with pobability. In futue eseach, it could be inteesting to eploe the additional tade-offs aising when playes diffeed with espect to thei innate leaning abilities, as paameteized by the Poisson aival ate of beakthoughs. Analyzing these additional tade-offs that would appea, if, say, playe was able to lean faste on isky am A, while playe was faste with isky am B might yield insights into conditions unde which thee is ecessive, o insufficient, specialization in equilibium. 6

18 Appendi A Closed-Fom Solutions And An Auiliay Result Closed-Fom Solutions fo the Case of Pefect Negative Coelation If ((, 0, (, 0 is played, both playes value functions satisfy the following ODE: p( pu (p + (p + u(p = ( + pg, which is solved by u(p = pg + C( pω(p, whee C is some constant of integation. The same solution also applies to the planne s value function if (K A, K B = (, 0 pevails. If ((0,, (0, is played, the oles of p and p ae evesed, and both playes value functions satisfy the following ODE: p( pu (p + (( p + u(p = ( + ( pg, which is solved by u(p = ( pg + CpΩ(p. The same solution also applies to the planne s value function if (K A, K B = (0, pevails. If the playes belief feezes, as is e.g. the case when ((0,, (, 0 obtains, the playes values ae linea. Thus, in the case of ((0,, (, 0 fo instance, we have u (p = + ( p g; + u (p = + p + g. In this case, the planne s value, being the aveage of the two, is constant, + (+ g = ū. Solutions fo the othe si pue stategy pofiles, as well as the thee indiffeence cases, ae ehibited in Klein (0. An Auiliay Result The logic we discussed in Section 5 of the main tet gives us the following auiliay esult, which will be useful in the poof of Poposition 5.. Lemma A. Conside playes (i, j {, } \ {(i, i : i {, }}, and let P (0, be an open inteval of beliefs in which the action pofile emains constant, and let p P. Let k j (p = (0, 0. Then the following statements hold: If playe i s best esponse is given by k i (p = (0, 0, then u i (p = s. 7

19 If playe i s best esponse is given by k i (p = (, 0 o k i (p = (0,, then u i (p ma{s, + + g}. Let k j (p = (, 0. Then the following statements hold: If playe i s best esponse is given by k i (p = (0, 0, then +( p + g u i (p s pg. If playe i s best esponse is given by k i (p = (, 0, then u i (p ma{ +( p + g, s pg}. If playe i s best esponse is given by k i (p = (0,, then u i (p = +( p + g and p min{ p m +, +3 }. Let k j (p = (0,. Then the following statements hold: If playe i s best esponse is given by k i (p = (0, 0, then +p + g u i(p s ( pg. If playe i s best esponse is given by k i (p = (, 0, then u i (p = +p + g and p ma{pm, + +3 }. If playe i s best esponse is given by k i (p = (0,, then u i (p ma{ +p + g, s ( pg}. + As +3 < < + +3, the lemma immediately implies that in no equilibium ((, 0, (0, o ((0,, (, 0 can aise on an open inteval. If futhemoe g +( p s, and hence s pg + g fo all p [0, ], then ((, 0, (0, 0, ((0, 0, (, 0, ((0,, (0, 0 and ((0, 0, (0, cannot aise on an open inteval eithe. Eplicit Solutions fo the Case of Impefect Coelation (Section 6 If (K A, K B = (0, 0, both playes payoff function u satisfies u = s. Fo (K A, K B = (, 0, we veify that p B p A is constant. The PDE is given by p A ( p A u p A p A p B u p B + ( + p A u = ( + p A g, which, as in Klein & Rady (0, we find is solved by ( pb u = p A g + f 0 ǔ 0 (p A, p A with Fom Klein & Rady (0, we know that ( p ǔ 0 (p := ( p. p ǔ 0(p = + p p( pǔ0(p, and ǔ 0 > 0. Fo (K A, K B = (,, we veify that p B pa is given by u = + ( + (p A + p B g + f is constant and find that the playes aveage payoff 8 ( pb p A u 0 (p A + p B,

20 with u 0 (p := ( p ( p p. Fom Klein & Rady (0, we know that u 0(p = + p p( p u 0(p, and u 0 > 0. If a playe is indiffeent between am A and the safe am, his payoff function u satisfies This is solved by p A ( p A u p A p A p B u p B + p A u = ( + p A g s. u = s + + (g s + s( p A ln ( pa p A ( + f pb ( p A. p A B Poofs Poof of Poposition 5. Suppose g s 4(+ +3. What is to be shown is that the action pofiles ((, 0, (, 0 and ((0,, (0, ae mutually best esponses on (, ], and [0,, espectively. At p =, admissibility uniquely pins down a playe s esponse to the othe playe s action. By the chaacteization of efficiency (see Poposition 4., both playes espective value function if efficiency pevails is given by: [ ] g p + + u(p = pω(p if p [ ] g p + + ( pω(p if p. Now, by Lemma A., it is sufficient to show that u(p > ma{ +( p + g, s pg} on (, ], and u(p > ma{ +p + g, s ( pg} on [0,. I shall only conside the fome inteval, as the agument petaining to the latte is pefectly symmetic. Simple algeba shows that if g s 4(+ +( p +3, w(p := + g s pg eveywhee in [, ]. Since u( = w(, and u is stictly inceasing while w is stictly deceasing in (,, the claim follows. Suppose (+ + g s < 4(+ +3, and define w(p := s pg. It is now staightfowad to show, and, theefoe, by Lemma A., thee eists a neighbohood to the ight that w( > w( = u( of p = in which (, 0 is not a best esponse to (, 0. Suppose that the stakes ae vey low, i.e. g s < (+ +. Fom ou chaacteization of the efficient solution (see Poposition 4., we know that the planne s value function is given by [ ] g p + p p + p (Ω(pΩ(p if p p, u(p = s [ if p p p (, ] g p + p p + ( p Ω(p Ω(p if p p and that B A (p, u = c A(p. Fo the efficient actions to be a best esponse, it is necessay that B A c A on (p, ]. Yet, since u is of class C, we have that lim p p B A (p, u = c A(p < c A (p, as p < pm. 9,

21 Poof of Poposition 6. The planne s Bellman equation is given by [ u = s + ma {K A B A (p A, p B, u c ] [ A(p A + K B B B (p A, p B, u c ]} B(p B (K A,K B with c I (p I := s p I g fo I = A, B. B A (p A, p B, u := p A B B (p A, p B, u := p B [ g u ( p A u u + p B p A p B [ g u ( p B u u + p A p B p A By symmety, we only need to look at beliefs p A p B. We define := p B,0 that wheneve K B,t = 0, p B,t p A,t ], ], p A,0 <, and note emains constant, i.e. if p A,0 > p B,0 and (, 0 pevails, beliefs move towad the 45-degee line in (p A, p B -space along the staight line p B = ( p A. We make a case distinction depending on whethe > p p at the end. o < p p, dealing with the knife-edge case = p p Case > p p We shall fist analyze the case of > p p. As <, this implies p <, i.e. g s > (+ +. Along the 45-degee line, i.e. fo p A = p B =: p, we have seen in Appendi A that the planne s payoff function fo ou conjectued solution is given by u (p A, p B = + ( + (p A + p B g + f P ( pb Value matching at (p, p pins down f P ( =: C, and gives us with C = p <. + s + p g u 0 (p u P (p A, p B = p A u 0 (p A + p B. + ( + (p A + p B g + C u 0 (p A + p B. One veifies that C > 0 if g s > (+ +, i.e. if the stakes ae not vey low, and Let us now tun to the (, 0 egion. As we have seen in Appendi A, the payoff function hee is given by Value matching at (, gives us u P 0 = p A g + f P 0 ( pb p A ǔ 0 (p A. ( f0(ǔ P 0 = g + C u 0 (, + i.e. f P 0( = + g + + C + ( { } = + g + C u0 (. 0

22 In ode to show that the planne s behavio is indeed optimal, we fi an > p p and check that B A c A and B A c A B B c B fo all (p A, p B in the (, 0-egion along the ay defined by. Fom the closed-fom solution, we have that u P 0 p A = g + p A p A ( p A f 0(ǔ P 0 (p A + f P 0 (ǔ 0 (p A, p A and Thus, u P 0 p B = p A f P 0 (ǔ 0 (p A. B A = f P 0(ǔ 0 (p A. Hence, B A? c A p Ag + f0(ǔ P 0 (p A? s. (B. As the left-hand side is a stictly conve function in p A, we ae going (. to check optimality at p A =, and (. then check if the deivative of the left-hand side of (B. with espect to p A at p A = is non-negative (holding fied as we move along the ay given by ou. If both (. along ou paticula ay. ( ( We note that by value matching (. is equivalent to u P 0, = u P, > s. We define ũ(p := u P (p, p. Using the eplicit epession fo C, we have that and (. hold, it follows by conveity that B A c A fo all p A ũ(p = + ( + pg + s + u0 (p + p g u 0 (p. As ũ(p = s, and ũ is a stictly conve function of p fo g s > (+ +, with ũ appoaching 0 as p p, we can conclude that ũ is stictly inceasing on (p,, and hence that ũ(p > s fo all p (p, ]. Regading point (., we have that the deivative of the left-hand side of (B. with espect to p A at p A = Noting that p is non-deceasing if and only if ( + + [ = + s p C u 0 (p = p + + to ( + g + C + u 0 ( ]? g. + (B. ( g (+ +, s and that value matching at (p, p implies that ] + s, we have that C u 0 (p = +s. Thus, we can e-aange (B. [ g (+ ( ( + p [ g s + s ]? g. By diect calculation, one shows that this condition eactly binds at = p p The deivative of its left-hand side with espect to woks out as + ( g s + s s ( ( + ( ( +. as well as at =.