Analysis of the Chow-Robbins Game with Biased Coins

Transcription

1 Aalysis of the Chow-Robbis Game with Biased Cois Arju Mithal May 7, 208 Cotets Itroductio to Chow-Robbis 2 2 Recursive Framework for Chow-Robbis 2 3 Geeralizig the Lower Boud 3 4 Geeralizig the Upper Boud 3 4. Proof of Theorem A More Rigorous Upper Boud Computatioal Results for V (0, Applicatio to the Stop Whe Ahead Problem 6 7 Coclusio 8 Abstract The rules of the Chow-Robbis game are simple: flip a coi repeatedly, ad stop wheever you choose. The goal: maximize the ratio of heads to total flips. This simple problem has o kow closed form solutio, however [] gives the tightest kow bouds o the expected payoff uder optimal play. The traditioal formulatio of the game assumes a fair coi (p = q = /2. This allows for relatively straightforward aalysis. I this paper, we exted the aalysis to games with a biased coi (p = q /2. The treatmet ad approach of the aalysis is similar to [] ad [2], though the results for biased cois are origial.

2 Itroductio to Chow-Robbis Yua-Shih Chow ad Herbert Robbis formulated the problem ow kow as Chow- Robbis i 965 [3] as part of a larger aalysis of optimal stoppig rules for i.i.d. radom variables. The problem is simple: flip a coi as may times as desired, ad receive a payoff equal to the ratio of heads to total coi flips. I [], Chow ad Robbis show that there exists a stoppig rule for this game that triggers with probability. Other aalyses of the problem, developed idepedetly by Dvoretzsky, Shepp, Häggström ad Wästlud, ad Media ad Zeilberger all focus o derivig asymptotically optimal coditios of the game. I particular, [] derives bouds o differet game states aalytically ad the computatioally. [2] derives the asymptotic ratio of heads required to stop after flips have bee made. Fially, [4] shows that the optimality coditios for certai game states are still ukow ad ot prove. We ow cotiue by presetig a recursive approach to the problem. Much of the otatio used i this paper is take from [], however the extesios to biased cois are origial results. 2 Recursive Framework for Chow-Robbis Let V (h, deote the expected payoff uder optimal play from a game state with h heads i the first flips. The followig recursive formula for V (h, relates the optios of either stoppig or cotiuig from game state (h, uder a fair coi: ( h V (h +, + + V (h, + V (h, = max, ( 2 The payoff of stoppig the game at game state (h, is exactly h/. I the fair game, with probability /2 a head is observed o the + flip. So, with probability /2, we receive the expected payoff of either game state (h +, + or (h, +. Next, we will itroduce a importat probabilistic cocept which will be used extesively i the remaider of our aalysis. For the sake of geerality, we specify the Lemma idepedet of the bias p: Lemma 2.. From ay fiite game state (h,, ad for a coi with ay bias p, we ca achieve V (h, p with probability. Proof. Clearly, for h p, we ca stop the game immediately ad collect payoff of at least p. For h < p, we make use of the Strog Law of Large Numbers (SLLN. I particular, the SLLN asserts the followig: Let X be a real-valued radom variable with fiite mea X, ad let X, X 2,... be a sequece of i.i.d. copies of X. Deote X as the empirical average X := (X + + X. The, P( lim X = X =. I other words, the sample average of ay sequece of i.i.d. radom variables will coverge i probability to the mea of oe copy of the radom variable. To apply this theorem to our problem, we otice that with probability, the game will coverge to a game state 2

3 where h/ = p. A corollary of the SLLN is that o fiite itermediate state of the sequece X, X 2,... will cotradict the asymptotic covergece of the sample mea. Give the (potetially ifiite ature of Chow-Robbis, we ca cotiue playig from (h, util the proportio p is achieved. The SLLN gives the otio that we will reach this stoppig state with probability. This completes the proof. Usig the Lemma, we ca ow assert that for ay game state, V (h, max, p. (2 Moreover, we ca geeralize ( to cois with bias p: ( h V (h, = max, p V (h +, + + ( p V (h, + (3 Now, while (3 provides a useful recursive formula for solvig V (h,, the oly base cases have a ifiite horizo ( =. Sice we are essetially solvig usig backwards iductio, kowledge of the extreme values of V are required i order to solve explicitly. I the absece of this kowledge, we ca oly do so well as to boud V for large values of, ad solve backwards. 3 Geeralizig the Lower Boud Combiig equatios (3 ad (4, we offer a method to compute a lower boud V (0, 0. The boud will be tight oly up to the level of computatio permitted. I particular, we choose N sufficietly large, ad approximate V (h, N for h N usig (2. So, we assig the followig values: { p, for h p N V (h, N = h N, for h > p N Computig the V (h, values for < N is ow straightforward from (3. We use a dyamic programmig algorithm to implemet the recursio computatioally, ad report the values for V (0, 0, the lower boud o the expected value of the game uder optimal play. I additio, we report values for certai early game states. I particular, if V (h, > h/, we kow uequivocally that the optimal strategy i game state (h, is to cotiue. All results are summarized i sectio 5, ad i Figures, 2, ad 3. 4 Geeralizig the Upper Boud The tightest upper boud we preset i this paper is far less straightforward tha the lower boud, ad requires far more computatio to derive. The motivatio for the Theorem preseted i this sectio is also attributed to Häggström ad Wästlud i []. The extesio of the boud to biased cois is a origial result. Theorem 4.. V (h, max ( h ( p q h, p q + dq (4 max,p q 3

4 Before provig this theorem, we must itroduce some more otatio: Let P (h,, q := P(evetually achievig a proportio greater tha q from (h,. (5 Phrased differetly, P (h,, q is the iverse CDF (Cumulative Distributio Fuctio of the radom variable X (h,, where We ca ow rewrite P (h,, q as X (h, := maximum payoff possible from (h,. (6 P (h,, q = P(X (h, > q. (7 This gives that P (h,, q = ( P(X (h, q = ( CDF (X (h,. Usig the well kow fact from probability theory that E[X] = ( CDF (X dx, (8 we ca derive a explicit formula for V (h, : 4. Proof of Theorem 4. X V (h, = 0 P (h,, q dq. (9 Now, i derivig a upper boud for P (h,, q, cosider the followig procedure: give that the proportio q is evetually attaied from state (h,, say o flip m, the proportio of heads i flips +,..., m must be at least q. So, the coditioal probability of heads o each flip i +,..., m must be at least q. Let us ow cosider k to be the miimum possible m such that proportio q is reached from (h,. I other words, flippig k cosecutive heads from (h, will exceed proportio q. Usig this defiitio, we have the followig iequality: P(k cosecutive heads flipped proportio q attaied q k (0 From Bayes Rule, we also kow that P(k cosecutive heads flipped proportio q attaied P(k cosecutive heads flipped. P(proportio q attaied Rearragig the formula, ad substitutig P (h,, q for the deomiator o the RHS, we have P(k cosecutive heads flipped P (h,, q P(k cosecutive heads flipped proportio q attaied. The ucoditioal probability o the RHS is simply p k, so substitutig this quatity combied with the result of (0, we arrive at the upper boud ( p k. P (h,, q ( q 4

5 Before evaluatig our itegral formula, we first solve explicitly for k. Sice flippig k cosecutive heads from (h, gives a proportio strictly greater tha q, we kow that ad subsequetly, h + k + k q < h + k + k, k q h q. Sice we also kow P (h,, q = for q < max, p, we ca ow rewrite V (h, as ( p q h V (h, = P (h,, q dq max 0, p q + dq. (2 max,p q This completes the proof of Theorem 4., however we ow proceed to show a more rigorous formulatio of the upper boud of V (0, A More Rigorous Upper Boud Applyig the substitutio 2q = + t, the itegrad ad bouds of itegratio chage as follows, q h q dq = Usig the iequality we arrive at = 2 2 max,p ( p q max( 2h,2p max( 2h,2p exp exp max( 2h,2p max( 2h,2p log( + t t (+t 2h t t, ( 2p (+t 2h t dt + t 2. ( 2p dt + t 2 ( ( + t 2h (log(2p log( + t dt t ( ( + t 2h log(2p ( + tt + 2ht dt. t Next we set u = t ad obtai V (h, max, p + ( 2 exp log(2p + u 2h u 2 + 2h u du. max( 2h,(2p u (3 Whe we cosider the case of a fair coi (p = 2, we ca replace the bouds of itegratio ad achieve the form V (h, max, ( exp u 2 2h u du. (4

6 The itegral term i (4 ca further be bouded by each of the terms i the expoetial, which have closed form solutios, 2 exp( u 2 du = π 0 4 ( 2 2h exp u du = 2 2h 0 Combiig these solutios, the upper boud for the fair coi becomes V (h, max 2, ( π + mi 4,, (5 2 2h as derived i []. The upper boud of the biased coi has o kow closed form, so we istead preset results based o a computatioal approximatio of the upper boud for sufficietly large N. I particular, the computatio is easiest o the versio of the boud preseted i (2, before ay chage of variables is performed. The resultig V + (h, N values for our limitig horizo N are p + V + (h, N = p h N + h N ( p q qn h q dq, ( p q qn h q dq, for h p N for h > p N Agai, we use a dyamic programmig algorithm to compute all V (h, values for h, < N by (3. 5 Computatioal Results for V (0, 0 The results preseted i this paper are based o a = 0 4 horizo. Due to computatioal costraits, we were uable to reproduce the results produced i [], which were built o a = 0 7 horizo, ad had a error margi of < 0 5 for the V (0, 0 boud for a fair coi. Specifically, their results bouded V (0, 0 as whereas our results show < V (0, 0 < , < V (0, 0 < , givig bouds about 00 times as wide as []. The complete bouds for differet values of p are plotted i Figures ad 2. The error, or boud tightess, as a fuctio of the bias p is plotted i Figure 3. Iterestigly, Figures ad 2 seem to idicate a smooth fuctioal relatioship betwee V (0, 0 ad p. 6 Applicatio to the Stop Whe Ahead Problem Oe iterestig corollary to the results preseted i the previous sectio is the observatio that, for some p (.2,.25, we first obtai V (0, 0 >.5. This meas that, i 6

7 Figure : Upper boud o V (0, 0 Figure 2: Lower boud o V (0, 0 Figure 3: Boud Tightess 7

8 expectatio, the optimal stoppig rule for coi with bias p ecessarily yields more heads tha tails. Let us ow apply this observatio to a variatio of a classic gamblig problem. Cosider a game with the followig rules: You are give a (possibly biased coi. Flip the coi repeatedly, ad collect payoff equal to the ratio of heads to total flips mius the ratio of tails to total flips whe you decide to stop. Whe should you be willig to play this game? The results idicate that uder optimal play, for all p > p, the game has a positive expected value. This is eve more iterestig cosiderig that for p =.25, you should play the game. Uder this value of p, we already have a =.5625 probability of reachig state (0,2, so ituitio might lead us to believe the game is ot profitable. Our results show that we do, i fact, have sequece of optimal moves that will yield a profit i the game, o average. 7 Coclusio I this paper, we exted the work of Häggström ad Joha Wästlud i [] to study the case of a biased coi i the Chow-Robbis game. We use a similar methodology to work through upper ad lower bouds o the expected value of playig the game with a coi with particular bias p. Usig a computatioal approach with a horizo N = 0 4, ad solvig backwards usig the recursive formula i (3, we derive tight bouds o V (0, 0. The results are summarized i Figures ad 2. I each solutio, if the value of a give V (h, > h/, the the optimal move at game state (h, is ecessarily to cotiue. Likewise, if V + = h/, the the optimal move is to stop at (h,. With more computatioal resources, we should be able to approximate arbitrarily tight bouds for V (0, 0. Fially, Figures ad 2 seem to idicate the existece of a smooth fuctioal relatioship betwee V (0, 0 ad p. The search for such a fuctio, as well as a closed form for V (0, 0 remai ope problems. 8

9 Refereces [] Olle Häggström ad Joha Wästlud, Rigorous computer aalysis of the Chow Robbis game, The America Mathematical Mothly 20(0: (203. [2] Larry A. Shepp, Explicit solutios to some problems of optimal stoppig, The Aals of Mathematical Statistics, 40:993-00, 969. [3] Yua-Shih Chow ad Herbert Robbis, O optimal stoppig rules for s /, Ill. J. Math., 9: , 965. [4] Luis A. Media ad Doro Zeilberger, A Experimetal Mathematics Perspective o the Old, ad still Ope, Questio of Whe To Stop?, i Gems i Experimetal Mathematics, Cotemporary Mathematics series v. 57 (AMS, eds T. Amdeberha, L. Media, ad V. Moll, , also arxiv: v2 [math.pr]. 9