9.07 Introduction to Probability and Statistics for Brain and Cognitive Sciences Emery N. Brown

Transcription

1 9.07 Introducton to Probablty and Statstcs for Bran and Cogntve Scences Emery N. Brown Lecture 6: Epectaton and Varance, Covarance and Correlaton, And Movement Generatng Functons I. Objectves Understand how to compute epectatons and varances for lnear combnatons of random varables. Understand how to compute the covarance between lnear combnatons of random varables. Understand the concept of moments of a probablty densty. Understand how the moments of a probablty densty or probablty mass functon can be derved from the moment generatng functon. Understand the basc propertes of moment generatng functons and ther use n probablty calculatons. II. Epectatons and Covarances A. Epectaton In Lectures and 3, we defned the epected value of a dscrete and contnuous random varable respectvely as. Dscrete If then EX ( ) = p ( ) X : p () (6.) (6.) provded that p( ) <.. Contnuous If then E ( ) = f ( d ) X : f ( ) (6.3) (6.4) provded that f ( ) d<.

2 page : Lecture 6: Epectatons and Varances, Covarances and Correlaton, Moment Generatng Functons Remark 6.. The epected value has an mportant physcal nterpretaton as the center of mass or the frst moment. We also stated that f Y = g ( X ), we can obtan the epected value of Y drectly wthout frst computng py( y) or fy ( y ). That s, gven Y = g ( X ). Dscrete If X : p ()then provded that g ( ) p< ( ). EY ( ) = g ( ) p ( ) (6.5) If. Contnuous X : f ( )then EY ( ) = ( ) ( ) (6.5) g f d provded that g ( ) f( d ) <. Proof (Dscrete Case) EY ( ) = yp y ( y ). (6.6) Let A be the 's mapped nto by y by g. We have A f g ( ) = y.hence, p ( y )= p ( ) y A EY ( ) = y p ( ) A = yp ( ) (6.7) A = gp ( ) ( ) A = gp ( ) ( ). Remark 6.. A very mportant functon ( ) whose epected value we consdered s g g ( ) = ( µ ) whch s the varance of X. We know that ( ( ))= EX µ ) = EX X µ + µ ] = E µ. σ = E gx ( [ X

3 page 3: Lecture 6: Epectatons and Varances, Covarances and Correlaton, Moment Generatng Functons The standard devaton of X s σ = [EX ] µ. In general, f X,..., X n are jontly dstrbuted random varables and Y = g( X,..., X n ) then. Dscrete X, X,..., X n : p ( X, X,..., X n) then If EY ( ) = g (,..., n ) p (,..., n ) (6.8) provded that g (,..., n ) p (,..., n) <.. Contnuous X, X,..., X n : f (,..., n ) then If EY ( ) = g (,..., ) f (,..., ) d,..., d (6.9) n n n provded that E( Y ) <. Remark 6.3. If X and Y are ndependent and g and h are fed functons, then Eg [ ( X ) hy ( )]= E[ gx ( )] EhY [ ( )] provded that each of the epectatons on the rght s fnte. B. Epectatons of Lnear Combnatons of Random Varables In Lectures and 3 we stated that f Y = ax + b, then EY ( ) = ae ( X ) + b. That s, epectaton s a lnear operaton. We now state a very mportant generalzaton of ths result. Proposton 6.. If X,..., X n are jontly dstrbuted random varables wth epectatons EX ( ) and Y s a lnear functon of the X 's, n = + b E( X ). Y a (6.0) Smply stated, the epectaton of the lnear combnaton of a set of random varables s the lnear combnaton of the epectatons provded that each epectaton s fnte. Proof: (see Rce, p. 5). Eample. (contnued). The rat eecuted 40 trals of a learnng eperment wth a probablty p of a correct response on each tral. What s the epected number of correct responses? Each tral s a Bernoull eperment wth probablty p of the anmal eecutng t correctly. Let X be 40 the outcome on tral, then Y = X s the total number of correct responses on each tral. Hence,

4 page 4: Lecture 6: Epectatons and Varances, Covarances and Correlaton, Moment Generatng Functons 40 EY ( ) = E( X) 40 = E( X ) 40 = p = 40 p. (6.) In general, f there were n trals EY ( ) = np. We showed ths result before n Lecture usng the defnton of epectaton drectly and the bnomal theorem. Eample 5.4 (contnued). Suppose that the number of channel openngs n a 500 msec tme nterval at each of n stes at the frog neuromuscular juncton s a Posson random varable. Each ste has an epected number of openngs λ per mllsecond for =,..., n. What s the epected number of openngs n 500 msec across all n stes? We have X : P( λ ) E( X ) = λ (6.) and n 500 msec we have Z = 500X EZ ( ) = 500λ (6.3) and n Y = Z N (6.4) EY ( ) = E( Z) n = EZ ( ) = 500 λ. n Eample 5.4 (contnued). We assumed that the length of tme that each of two channels s open s an eponental random varable wth parameters λ and λ respectvely. What s the epected sum of the openng tmes at the two stes? We have X : E( λ ) =, and + and EZ EX ( )= λ. Hence, Z = X X ( )= E( X + X ) = λ + λ. Notce that when λ = λ = λ we have EZ ( ) = λ. Ths s completely consstent wth our observaton that when λ = λ = λ, Z : Γ(, λ), whch we showed n Lecture 5.

5 page 5: Lecture 6: Epectatons and Varances, Covarances and Correlaton, Moment Generatng Functons III. Covarance and Correlaton A. Covarance Eample 4.6 (contnued). In ths eample, we saw that postve values on one sensor were assocated wth postve values on the other sensor and negatve values on one sensor were assocated wth negatve values on the other sensor. The values seemed to be related. We formalze the characterzaton of ths relatonshp n terms of the concept of covarance and correlaton. The latter we have already dscussed n terms of the correlaton coeffcent. Defnton 6.. If X and Y are jontly dstrbuted random varables wth EX ( )= µ and EY ( )= µ y, then the covarance of X and Y s Cov( XY, ) = E( X µ )( Y µ y ) (6.5) = (X µ )( Y µ y ) f y (, y) dd y. Remark 6.4. The covarance of X and Y measures the strength of the assocaton between X and Y n the sense mentoned above n Eample 4.6. It s the frst cross moment of X and Y. It has unts of the random varables squared, assumng X and Y measure the same thng. Remark 6.5. If X = Y, we have Cov( XY, ) = E( X µ )( Y µ y ) ( µ ) (6.6) = E X =σ. Therefore, the covarance of X wth tself s smply the varance of X. Remark 6.6. Varance s the second central moment, and EX ( ) s the second moment. The varance tells us about how mass s dstrbuted away from the mean. We report the standard devaton because σ s on the same scale (unts) as X. Remark 6.7. We have on epandng the terms n Defnton 6. Cov( XY, ) = EX ( µ )( Y µ y ) = E ( XY µ Y µ y X + µ µ y ) = E ( XY ) µ E( Y ) µ y E ( X ) + µ µ y (6.7) = E ( XY ) µ µ y µ yµ + µ µ y = E( XY ) E( X ) E( Y ). Eample 6. (Rce p. 38). Suppose X and Y have the jont probablty densty f(, y ) = ( + y 4 y ) (6.8) where (0,) and y (0,).

6 page 6: Lecture 6: Epectatons and Varances, Covarances and Correlaton, Moment Generatng Functons Fnd Cov( XY)., Frst t s easy to show that EX ( ) = E( Y ) = E ( X ) = ( + y 4 y ) dyd = 0 0 E ( Y ) = y ( + y 4 y ) ddy = 0 0 (6.9) We need to compute EXY ( ) = y ( + y 4 yddy ) =. (6.0) Hence, Cov( XY, ) = EXY ( ) EXEY ( ) ( ) = ( ) =. (6.) 9 36 Here are 4 useful results. Result 6. Cov( a+ X, Y) Cov( a+ X, Y ) = E (( a+ X ) Y ) E ( a+ X ) E ( Y ) = E ( ay ) + E ( XY ) [ E ( a ) + E ( X )] E ( Y ) = ae ( Y ) + E ( XY ) E ( a ) E ( Y ) E ( X ) E ( Y ) = E ( XY ) E ( X ) E ( Y ) + ae ( Y ) ae ( Y ) = cov( XY, ) (6.) Result 6. Cov( ax, by ) Cov( ax, by ) = E ( axby ) E ( ax ) E ( by ) = abe ( XY ) abe ( X ) E ( Y ) = ab cov( X, Y ) (6.3) Result 6.3 Cov( XY, + Z) Cov( XY, + Z ) = EXY ( + Z ) EXEY ( ) ( + Z ) = EXY ( + XZ ) EX ( )[ EY ( ) + EZ ( )] = EXY ( ) + EXZ ( ) EXEY ( ) ( ) EXEZ ( ) ( ) = EXY ( ) EXEY ( ) ( ) + EXZ ( ) EXEZ ( ) ( ) = cov( XY, ) + cov( XZ, ) (6.4)

7 page 7: Lecture 6: Epectatons and Varances, Covarances and Correlaton, Moment Generatng Functons Result 6.4 Cov( aw + bx + cy + dz) Cov( aw + bx, cy + dz) = Cov( aw + bx, cy ) + Cov( aw + bx, dz ) = Cov( aw, cy ) + cov( bx, cy ) + Cov( aw, dz ) + Cov( bx, dz ) (6,5) We can state the followng general result = accov( W, Y ) + bccov( X, Y ) + ad Cov( WZ ) + bdcov( X, Z ). Proposton 6.. Let U = a 0 + a X V = b 0 + b j X j j= Then I J I J Cov( U, V) = ab j Cov( X, X j ). (6.6) j= We wll be more nterested n the followng specal cases Cov( X + Y, X + Y) = Cov( X, X ) + Cov( Y,Y) + Cov( X,Y) ) (6.7) = Var( X ) + Var( Y ) + Cov( X,Y) ) Cov( X Y, X Y ) = Var( X) + Var( Y) Cov( X,Y) (6.8) I ) If U = a + a X and X are ndependent then Var( U ) = Cov( U,U ) = Cov( a + a X, a + a X ) I j j I I j j j= = aa cov( X X ). I (6.9) If j Cov( X, X j ) = 0 because the X are ndependent. If = j Cov( X, X j ) = Var( X ). Thus, Var( ) I I U = a Var( X ) = a σ. (6.30) Ths result s to some etent the analog for varances of the result for epectatons establshed n Proposton 6.. We requred ndependence n the varance calculatons.

8 page 8: Lecture 6: Epectatons and Varances, Covarances and Correlaton, Moment Generatng Functons Eample. (contnued). Here X : B(, n p) where n = 40. What s the varance of X? We can wrte where Y : Bernoull wth probablty of a correct response p. 40 X = Y (6.3) If we assume (as we do) that the Y 's are ndependent, then by Proposton Var( X ) = Var( Y ) = p( p) = 40 p( p). (6.3) In general, for n we have Var( X ) = np ( p). We dd not show ths before because t s very tedous to do drectly from the defnton of the varance. Eample 6.. Integrate and Fre Model as a Random Walk + Drft As we dscussed n Lecture 3, an elementary stochastc model for the subthreshold membrane voltage of a neuron s the random walk wth drft model proposed by Gersten and Mandelbrot (964). In contnuous tme, t s t Vt ()= µt +σ d Wu ( ) (6.33) t 0 where W() t s a Wener process,.e. a contnuous tme Gaussan random varable wth mean 0 and varance. We assume EW ( ( sw ) ( t) ) = 0 for s t. To smulate ths on the computer we rewrte t n dscrete tme as Vt ( )= µ t +σ W (6.34) j=0 where W : N(0,). Fnd the varance of V( t ). Applyng Proposton 6. we get Var( Vt ( )) = Var( µt + σ W ) j=0 = σ Var( W ) (6.35) j=0 = σ ( + ). Ths s a restatement of the well-known classc result that the varance of a random walk s proportonal to the length of the walk. Uncertanty ncreases wth tme. Notce however, that

9 page 9: Lecture 6: Epectatons and Varances, Covarances and Correlaton, Moment Generatng Functons EV ( ( t ))= E(µt +σ W ) = E ( µt ) +σ EW ( ) = µt + 0 = µt. j=0 j=0 (6.36) On average the random walk wth drft wll be on the course of lnear ncrease n membrane voltage. (see Gersten GL, Mandelbrot M. Random walk models for the spkng actvty of a sngle neuron. Bophyscal Journal 4:4-68, 964). B. Correlaton Defnton 6.. If X and Y are jontly dstrbuted random varables and the varances and covarances of X and Y are non-zero, then the correlaton of X and Y s Cov( XY, ) Cov( X, Y) ρ = =. (6.37) [Var( X )Var( Y )] σσy Eample 6. (contnued). Fnd ρ. We have EX ( )= E( Y ) = and we computed Cov( X, Y) =. 36 Because X and Y are both unform on (0,) t s easy to show that Var( X ) = Var( Y ) =. Now, we have ρ = = =. (6.38) ( ) 3 We now establsh an mportant property of the correlaton coeffcent. Proposton 6.3. ρ. ρ =± f and only f Pr( Y = a+ b X ) = for some constants a and b. Proof: X Y 0 Var( + ) σ σ y Var( X ) Var( Y ) X Y = + + Cov(, ) σ σ y σ σ Var( X ) Var( Y ) Cov( X,Y) = + + (6.39) σ σ y σσ y + + ρ = ( + ρ) Or ( + ρ) 0 + ρ 0 or ρ. Also X X 0 Var( ) = ( ρ) (6.40) σ σ y

10 page 0: Lecture 6: Epectatons and Varances, Covarances and Correlaton, Moment Generatng Functons Or ( ρ) 0 ( ρ) 0 ρ. To prove the last statement of the proposton requres Chebyshev s nequalty whch we wll prove n Lecture 7. Remark 6.8. Correlaton provdes a normalzed (scale-free) measure of covarance. Eample 4.6 (contnued). We prevously stated n Lecture 4 that ρ was the correlaton coeffcent for a bvarate Gaussan probablty densty. Now we demonstrate ths by showng that cov( XY, ) = ρσ σ y. Cov( X Y, ) = ( µ )( y µ y ) f ( ydd., ) y (6.4) Make the change of varables u = ( µ )/ σ, v = (y µ y)/σ y and we have σσ y (u + v ρuv) dudv (6.4) ( ρ ) ( ρ ) = π uv ep completng the square σσ y v = vep u ep (u ρv) du du π ( ρ ) ( ρ ), (6.43) The nner ntegral s ( ) [ ( ρ ) = ρv[ π ( ρ Eu π ] )] ρσ σ y [ π ( ρ )] v = v ep dv π ( ρ ) y π = ρσ σ ( ) ( π ) Ev ( ) (6.44) π = ρσ σ y. IV. Moment Generatng Functons Learnng to use moment generatng functons n probablty calculatons s lke gong from usng a PC to usng a supercomputer. -- E.N. Brown As we wll see, the moment generatng functon smplfes many of the calculatons we have already done and wll enable many more, the most mportant of whch s the proof of the Central Lmt Theorem.

11 page : Lecture 6: Epectatons and Varances, Covarances and Correlaton, Moment Generatng Functons th Defnton 6.3. The r moment of a random varable X s r r EX ( ) = f ( d ) (6.45) provded the epectaton ests. The r th central moment of a random varable X s r provded that the epectaton ests. Remark 6.9. Some key factods about moments st EX ( ) = µ moment r E{[ X E( X )] } = ( E( )) f ( ) d (6.46) EX ( ) nd moment nd EX ( µ) = σ central moment (varance) EX ( µ) 3 3 rd central moment EX ( µ) 4 4 th central moment The skewness measures the asymmetry of the dstrbuton Fgure 6.A. Eamples of Negatve (left panel) and postve (rght) skew n a probablty densty. It s defned s

12 page : Lecture 6: Epectatons and Varances, Covarances and Correlaton, Moment Generatng Functons EX ( µ ) 3 σ 3 (6.47) The skewness of standard Gaussan dstrbuton s 0. The kurtoss measures how fat the tals of a dstrbuton s often reported as EX ( µ ) 4 σ 4 (6.48) or to make a comparson relatve to a standard Gaussan dstrbuton as EX ( µ ) 4 because for kurtoss for the standard Gaussan dstrbuton s 3. σ 4 3 (6.49) Defnton 6.4. The moment generatng functon (mgf) of a random varable X s Ee ( t ) whch s n the Dscrete Case and s n the t M () t = e p ( ) (6.50) Contnuous Case t M ()= t e f ( ) d (6.5) provded the epectatons est. Ths depends crtcally on how fast the tals n the probablty densty de out. Result 6.5. If M() t ests n an open nterval contanng zero t unquely determnes f ( ). Ths s an mportant, deep result whch allows us to perform computatons on M propertes of X and f ( ). For eample, f we dfferentate Mt ()we obtan () t to establsh and d t M '( t ) = e f ( ) d dt d t dt t = e f ( ) d (6.5) = e f ( ) d

13 page 3: Lecture 6: Epectatons and Varances, Covarances and Correlaton, Moment Generatng Functons M '(0) = f d ( Dfferentatng r tmes and evaluatng the dervatve at zero gves r ( ) = E X) (6.53) r M () (0) = E( X ). (6.54) Result 6.6. If Mt () ests n an open nterval contanng 0, then M () (0) = E( X ). In ths sense Mt ()can be used to generate the moments of X. r r Eample 6.3. Posson MGF tk k e λ λ M()= t e k! k=0 (λe t t ) λ k! k=0 = e (6.55) t k t t λ (λe ) λ λe λ(e ) = e = e e = e. k! k=0 Then sum averages for all t. Dfferentatng, we have M'( t) = λe e t λ(e ) t t t t λ(e ) t λ(e ) M''( t ) = λe e + λ e ( e ) M '(0) and λ E( k ) = λ (6.56) M ''(0) = λ+ λ and E( k ) = λ + λ or E( k ) [ E( k )] = λ + λ λ = λ. Eample 6.4 Gamma MGF e t β α α β M ()= t e d 0 Γ( α ) α β α (t β ) = e d (6.57) Γ( α ) 0 β α α Γ( α) β = = Γ( α) (β t) α (β t) α where we requre t < β for the ntegral to est

14 page 4: Lecture 6: Epectatons and Varances, Covarances and Correlaton, Moment Generatng Functons M '( t) = αβ (β t) α (α +) α M '(0) = E( X ) = β α (α +) αα ( +) M ''( t) = αα ( +) β (β t) M '(0) ' = E( X ) = β (6.58) αα ( +) α α and Var( X ) = E ( X ) [ E ( X )] = = β β β Eample 6.5. Standard Gaussan MGF. t = Mt () ( π ) e e d (6.59) Complete the square to rewrte and hence, t = ( + (6.60) t t ) t M () t = e t ( π ) ( t ). e d (6.6) Let u = t then d = du t e = ( π ) u e e du = ( π ) t ( π ) t. = e (6.6) Result 6.7 (Lnear Transformaton). If X has mgf M() t and Y = a + bx, then Y has mgf at My ()= t e M ( bt ). Proof: M Y ()= t E( e ty ) at +btx = E( e ) (6.63) at btx at btx at = Ee ( e ) = e Ee ( ) = e M ( bt ) t µt µt σ Eample 6.5 (contnued). Let Y = µ + σ X where X : N(0,) then MY () t = e M ( σt ) = e e. Result 6.8 (Convoluton of Independent Random Varables) If X and Y are ndependent wth mgf s M () t and M () t respectvely, and Z = X + Y, then y M t M t () on the common nterval where both mgf s est. ()= ()M t z y

15 page 5: Lecture 6: Epectatons and Varances, Covarances and Correlaton, Moment Generatng Functons by ndependence. M z ()= t E( e tz ) ( + ) = E( e t X Y ) tx ty = E( e e ) (6.64) tx ty = E( e ) Ee ( ) = M () t M y () t Eample 5.4 (contnued). We had X : Epon enta l( λ ) Y : Epone n t a l( λ ) and Z = X + Y. We showed that f λ = λ = λ and X and Y are ndependent, then Z : Γ(α =, β = λ). We now establsh the same result usng mgf s (Eample 6.4 and Result 6.8). Mz ()= t M ()M t y () t λ λ = ( )( ) (6.65) λ t λ t = ( λ ) λ t whch s the mgf of a gamma random varable wth α = and β = λ. Eample 6.6. X : N( µ, σ ) Y : N ( µ, σ ) and X and Y are ndependent. Fnd the dstrbuton y y of Z = X + Y. By Eample 6.5, Hence, Z N(µ + µ,σ +σ ). : y y Mz ()= t M ()M t y () t t µ σ t µ y y σ = e e e e (6.66) t + ) ( σ + σ y ) y (µ µ = e e Remark 6.0. There s an obvous generalzaton of Result 6.8, namely f X,..., X ndependent random varables wth mgf s M () t =,..,. n, and Z = X, then n Mz ( t) = M t The X 's do not have to come from the same dstrbuton. n n are ( ). (6.67) Remark 6.. In general, f two random varables follow the same dstrbuton the sum wll not necessarly follow the same dstrbuton. See Eample 5.4, 5.5 and consder the case of two ndependent gamma random varables wth β = β. Remark 6.. Result 6.8 s the specal case of the well-known mathematcal fact that the convoluton of two functons n ths case probablty densty functons n the probablty (tme)

16 page 6: Lecture 6: Epectatons and Varances, Covarances and Correlaton, Moment Generatng Functons doman corresponds to multplyng the transforms n the t (frequency) doman. Ths result s eploted a lot n lnear systems theory and Fourer analyss. Remark 6.3. There s a multvarate generalzaton of the mgf whch we wll not consder. Remark 6.4. The lmtaton that the mgf may not est s rectfed by workng nstead wth the characterstc functon φ () t = E( e tx ) t = e f ( ) d (6.68) where = ( ). The ntegral converges for all t because e t. The characterstc functon has the same propertes of the mgf wthout the estence lmtaton. Its use requres famlarty wth comple numbers and ther manpulatons. V. Summary We can now perform basc probablty calculatons on lnear combnatons of random varables. These results have now establshed the essental groundwork for provng the Central Lmt Theorem n Lecture 7. Acknowledgments I am grateful to Jule Scott for techncal assstance n preparng ths lecture. Reference Rce JA. Mathematcal Statstcs and Data Analyss, 3 rd edton. Boston, MA, 007.

17 MIT OpenCourseWare Statstcs for Bran and Cogntve Scence Fall 06 For nformaton about ctng these materals or our Terms of Use, vst: