CS 70 Discrete Mthemtics d Proility Theory Sprig 206 Ro d Wlrd Lecture 7 Vrice We hve see i the previous ote tht if we toss coi times with is p, the the expected umer of heds is p. Wht this mes is tht if we repet the experimet multiple times, where i ech experimet we toss the coi times, the o verge we get p heds. But i y sigle experimet, the umer of heds oserved c e y vlue etwee 0 d. Wht c we sy out how fr off we re from the expected vlue? Tht is, wht is the typicl devitio of the umer of heds from p? Rdom Wlk Let us cosider simpler settig tht is equivlet to tossig fir coi times, ut is more mele to lysis. Suppose we hve prticle tht strts t positio 0 d performs rdom wlk. At ech time step, the prticle moves either oe step to the right or oe step to the left with equl proility, d the move t ech time step is idepedet of ll other moves. We thik of these rdom moves s tkig plce ccordig to whether fir coi comes up heds or tils. The expected positio of the prticle fter moves is ck t 0, ut how fr from 0 should we typiclly expect the prticle to ed up t? Deotig right-move y + d left-move y, we c descrie the proility spce here s the set of ll sequeces of legth over the lphet {±}, ech hvig equl proility 2. Let the r.v. X deote the positio of the prticle (reltive to our strtig poit 0) fter moves. Thus, we c write where X i = + if the i-th move is to the right d X i = otherwise. X = X + X 2 + + X, () Now oviously we hve E(X) = 0. The esiest wy to see this is to ote tht E(X i ) = ( 2 )+( 2 ( )) = 0, so y lierity of expecttio E(X) = E(X i) = 0. But of course this is ot very iformtive, d is due to the fct tht positive d egtive devitios from 0 ccel out. Wht we re relly skig is: Wht is the expected vlue of X, the distce of the prticle from 0? Rther th cosider the r.v. X, which is little difficult to work with due to the solute vlue opertor, we will isted look t the r.v. X 2. Notice tht this lso hs the effect of mkig ll devitios from 0 positive, so it should lso give good mesure of the distce from 0. However, ecuse it is the squred distce, we will eed to tke squre root t the ed. We will ow show tht the expected squre distce fter steps is equl to : Clim 7.. For the rdom vrile X defied i (), we hve. Proof. We use the expressio () d expd the squre: E((X + X 2 + + X ) 2 ) = E( X 2 i + X i X j ) = E(X 2 i ) + E(X i X j ) CS 70, Sprig 206, Lecture 7
I the lst lie we hve used lierity of expecttio. To proceed, we eed to compute E(Xi 2) d E(X ix j ) (for i j). Let s cosider first Xi 2. Sice X i c tke o oly vlues ±, clerly Xi 2 = lwys, so E(Xi 2) =. Wht out E(X i X j )? Well, X i X j = + whe X i = X j = + or X i = X j =, d otherwise X i X j =. Therefore, Pr[X i X j = ] = Pr[(X i = X j = +) (X i = X j = )] = Pr[X i = X j = +] + Pr[X i = X j = ] = Pr[X i = +] Pr[X j = +] + Pr[X i = ] Pr[X j = ] = 4 + 4 = 2, where i the ove clcultio we used the fct tht the evets X i = + d X j = + re idepedet, d similrly the evets X i = d X j = re idepedet. Thus Pr[X i X j = ] = 2 s well, d hece E(X i X j ) = 0. Pluggig these vlues ito the ove equtio gives s climed. + 0 =, So we see tht our expected squred distce from 0 is. Oe iterprettio of this is tht we might expect to e distce of out wy from 0 fter steps. However, we hve to e creful here: we cot simply rgue tht E( X ) =. (Why ot?) We will see lter i the lecture how to mke precise deductios out X from kowledge of E(X 2 ). For the momet, however, let s gree to view E(X 2 ) s ituitive mesure of spred of the r.v. X. I fct, for more geerl r.v. with expecttio E(X) = µ, wht we re relly iterested i is E((X µ) 2 ), the expected squred distce from the me. I our rdom wlk exmple, we hd µ = 0, so E((X µ) 2 ) just reduces to E(X 2 ). Defiitio 7. (Vrice). For r.v. X with expecttio E(X) = µ, the vrice of X is defied to e Vr(X) = E((X µ) 2 ). The squre root σ(x) := Vr(X) is clled the stdrd devitio of X. The poit of the stdrd devitio is merely to udo the squrig i the vrice. Thus the stdrd devitio is o the sme scle s the r.v. itself. Sice the vrice d stdrd devitio differ just y squre, it relly does t mtter which oe we choose to work with s we c lwys compute oe from the other immeditely. We shll usully use the vrice. For the rdom wlk exmple ove, we hve tht Vr(X) =, d the stdrd devitio of X, σ(x), is. The followig esy oservtio gives us slightly differet wy to compute the vrice tht is simpler i my cses. Theorem 7.. For r.v. X with expecttio E(X) = µ, we hve Vr(X) = E(X 2 ) µ 2. Proof. From the defiitio of vrice, we hve Vr(X) = E((X µ) 2 ) = E(X 2 2µX + µ 2 ) = E(X 2 ) 2µE(X) + µ 2 = E(X 2 ) µ 2. I the third step ove, we used lierity of expecttio. Moreover, ote tht µ = E(X) is costt, so E(µX) = µe(x) = µ 2 d E(µ 2 ) = µ 2. CS 70, Sprig 206, Lecture 7 2
Aother importt property tht will come i hdy is the followig: For y rdom vrile X d costt c, we hve Vr(cX) = c 2 Vr(X). The proof is simple d left s exercise. Exmples Let s see some exmples of vrice clcultios.. Fir die. Let X e the score o the roll of sigle fir die. Recll from the previous ote tht E(X) = 7 2. So we just eed to compute E(X 2 ), which is routie clcultio: 6 ( 2 + 2 2 + 3 2 + 4 2 + 5 2 + 6 2) = 9 6. Thus from Theorem 7., Vr(X) = E(X 2 ) (E(X)) 2 = 9 6 49 4 = 35 2. 2. Uiform distriutio. More geerlly, if X is uiform rdom vrile o the set {,...,}, so X tkes o vlues,..., with equl proility, the the me, vrice d stdrd devitio of X re give y: E(X) = + 2, Vr(X) = 2 2, σ(x) = 2 2. (2) You should verify these s exercise. 3. Numer of fixed poits. Let X e the umer of fixed poits i rdom permuttio of items (i.e., the umer of studets i clss of size who receive their ow homework fter shufflig). We sw i the previous ote tht E(X) =, regrdless of. To compute E(X 2 ), write X = X + X 2 + + X, where X i = if i is fixed poit, d X i = 0 otherwise. The s usul we hve E(X 2 i ) + E(X i X j ). (3) Sice X i is idictor r.v., we hve tht E(X 2 i ) = Pr[X i = ] =. Sice oth X i d X j re idictors, we c compute E(X i X j ) s follows: E(X i X j ) = Pr[X i X j = ] = Pr[X i = X j = ] = Pr[oth i d j re fixed poits] = Mke sure tht you uderstd the lst step here. Pluggig this ito equtio (3) we get + ( ) = ( ) + (( ) ( ) ) = + = 2. ( ). Thus Vr(X) = E(X 2 ) (E(X)) 2 = 2 =. Tht is, the vrice d the me re oth equl to. Like the me, the vrice is lso idepedet of. Ituitively t lest, this mes tht it is ulikely tht there will e more th smll umer of fixed poits eve whe the umer of items,, is very lrge. CS 70, Sprig 206, Lecture 7 3
Idepedet Rdom Vriles Idepedece for rdom vriles is defied i logous fshio to idepedece for evets: Defiitio 7.2 (Idepedet r.v. s). Rdom vriles X d Y o the sme proility spce re sid to e idepedet if the evets X = d Y = re idepedet for ll vlues,. Equivletly, the joit distriutio of idepedet r.v. s decomposes s Pr[X =,Y = ] = Pr[X = ]Pr[Y = ],. Mutul idepedece of more th two r.v. s is defied similrly. A very importt exmple of idepedet r.v. s is idictor r.v. s for idepedet evets. Thus, for exmple, if {X i } re idictor r.v. s for the i-th toss of coi eig Heds, the the X i re mutully idepedet r.v. s. Oe of the most importt d useful fcts out vrice is if rdom vrile X is the sum of idepedet rdom vriles X = X + + X, the its vrice is the sum of the vrices of the idividul r.v. s. I prticulr, if the idividul r.v. s X i re ideticlly distriuted (i.e., they hve the sme distriutio), the Vr(X) = i Vr(X i ) = Vr(X ). This mes tht the stdrd devitio is σ(x) = σ(x ). Note tht y cotrst, the expected vlue is E[X] = E[X ]. Ituitively this mes tht wheres the verge vlue of X grows proportiolly to, the spred of the distriutio grows proportiolly to, which is much smller th. I other words the distriutio of X teds to cocetrte roud its me. Let us ow formlize these ides. First, we hve the followig result which sttes tht the expected vlue of the product of two idepedet rdom vriles is equl to the product of their expected vlues. Theorem 7.2. For idepedet rdom vriles X,Y, we hve E(XY ) = E(X)E(Y ). Proof. We hve E(XY ) = = = ( Pr[X =,Y = ] Pr[X = ] Pr[Y = ] Pr[X = ] = E(X) E(Y ), ) ( Pr[Y = ] s required. I the secod lie here we mde crucil use of idepedece. We ow use the ove theorem to coclude the ice property tht the vrice of the sum of idepedet rdom vriles is equl to the sum of their vrices. Theorem 7.3. For idepedet rdom vriles X,Y, we hve Vr(X +Y ) = Vr(X) + Vr(Y ). Proof. From the ltertive formul for vrice i Theorem 7., we hve, usig lierity of expecttio extesively, Vr(X +Y ) = E((X +Y ) 2 ) E(X +Y ) 2 = E(X 2 ) + E(Y 2 ) + 2E(XY ) (E(X) + E(Y )) 2 = (E(X 2 ) E(X) 2 ) + (E(Y 2 ) E(Y ) 2 ) + 2(E(XY ) E(X)E(Y )) = Vr(X) + Vr(Y ) + 2(E(XY ) E(X)E(Y )). ) CS 70, Sprig 206, Lecture 7 4
Now ecuse X,Y re idepedet, y Theorem 7.2 the fil term i this expressio is zero. Hece we get our result. Note: The expressio E(XY ) E(X)E(Y ) pperig i the ove proof is clled the covrice of X d Y, d is mesure of the depedece etwee X,Y. It is zero whe X,Y re idepedet. It is very importt to rememer tht either of these two results is true i geerl, without the ssumptio tht X,Y re idepedet. As simple exmple, ote tht eve for 0- r.v. X with Pr[X = ] = p, p is ot equl to E(X) 2 = p 2 (ecuse of course X d X re ot idepedet!). This is i cotrst to the cse of the expecttio, where we sw tht the expecttio of sum of r.v. s is the sum of the expecttios of the idividul r.v. s lwys. Exmple Let s retur to our motivtig exmple of sequece of coi tosses. Let X the the umer of Heds i tosses of ised coi with Heds proility p (i.e., X hs the iomil distriutio with prmeters, p). As usul, we write X = X + X 2 + + X, where X i = if the i-th toss is Hed, d X i = 0 otherwise. We lredy kow E(X) = E(X i) = p. We c compute Vr(X i ) = E(X 2 i ) E(X i) 2 = p p 2 = p( p). Sice the X i s re idepedet, y Theorem 7.3 we get Vr(X) = Vr(X i) = p( p). As exmple, for fir coi (p = 2 ) the expected umer of Heds i tosses is 2, d the stdrd devitio is. Note tht sice the mximum umer of Heds is, the stdrd devitio is much 4 = 2 less th this mximum umer for lrge. This is i cotrst to the previous exmple of the uiformly distriuted rdom vrile (2), where the stdrd devitio σ(x) = 2 2 2 is of the sme order s the lrgest vlue. I this sese, the spred of iomilly distriuted r.v. is much smller th tht of uiformly distriuted r.v. CS 70, Sprig 206, Lecture 7 5