Lecture 6 o BST 63: Statstcal Theory I Ku Zhag, /0/008 Revew for the prevous lecture Cocepts: t-dstrbuto, F-dstrbuto Theorems: Dstrbutos of sample mea ad sample varace, relatoshp betwee sample mea ad sample varace, relatoshp betwee ormal, ch-square, t, ad F dstrbuto Chapter 5 Propertes of a Radom Sample Secto 54 Order Statstcs Defto 54: The order statstcs of a radom sample,, are the sample values placed ascedg order They are deoted by (),, ( ), where () ( ) Defto 54: The otato { b }, whe appearg a subscrpt, s defed to be the umber b rouded to the earest teger the usual way More precsely, f s a teger ad 05 b 05, the { b} Some statstcs whch are fuctos of order statstcs: sample rage: R ( ) () sample meda: M (( ) / ),f s odd ( ( /) ( /) )/,f s eve
Lecture 6 o BST 63: Statstcal Theory I Ku Zhag, /0/008 3 (00 p) th sample percetle, s ({ p}) f /( ) p 05 ad ( {( ( p)}) f 05 p /( ), whch s the observato such that approxmately p of the observatos are less tha ths observato ad ( p) of the observatos are greater 4 Lower quartle, Q L s the 5 th percetle whle upper quartle, Q U s the 75 th percetle 5 Iterquartle rage, IQR QU QL Theorem 543: Let,, be a radom sample from a dscrete dstrbuto wth pmf f ( x) P( x) p, where x x x are the possble values of ascedg order Defe P0 0, P (),, ( ) deote the order statstcs from the sample The the cdf of ( j) s k p (,, ) Let k ad Proof: See book k k P ( ( j) x) P ( P) k j k, k k k k P ( ( j) x) [ P ( P) P ( P ) ] k j k Theorem 544: Let (),, ( ) deote the order statstcs of a radom sample,,,, from a cotuous populato wth cdf F ( x ) ad pdf f ( x ) The the pdf of ( j) s! f x f x F x F x ( j ) ( j)!( j)! j j ( ) ( )[ ( )] [ ( )]
Lecture 6 o BST 63: Statstcal Theory I Ku Zhag, /0/008 Example 545: (Uform order statstc pdf) The j th order statstc from uform(0,) sample has a j j ( j) beta( j, j ) Cosequetly, E( ( j) ) ad Var ( j) ( ) ( ) Theorem 546: Let (),, ( ) deote the order statstcs of a radom sample,,,, from a cotuous populato wth cdf F ( x ) ad pdf f ( x ) The the jot pdf of () ad ( j) j, s! f u v f u f v F u F v F u F v ( ) ( j) ( )!( j)!( j)! The jot pdf of (),, ( ) s gve by j j, (, ) ( ) ( )[ ( )] [ ( ) ( )] [ ( )]! f ( x) f( x), x x; f(),, ( x ( ),, x ) 0, otherwse for u v A Heurstc Dervato: For u v, the jot desty fucto of () ad ( j) ca be terpreted as the probablty that the th largest observato s close to u ad the j th kargest observato s close to v Defe fve classes of : Class : s that have values less tha u : Class : s that have values ear u : Class 3: s that have values greater tha u but less tha v : j Class 4: s that have values ear v : Class 5: s that have values greater tha v eed: j 3
Lecture 6 o BST 63: Statstcal Theory I Ku Zhag, /0/008 We ca use multomal dstrbuto to complete the heurstc argumet: Pu ( uduv, vdv) () ( j) P ( from class, from class, j- - from class 3, from class 4, - jfrom class 5)! j j P ( u) P( u udu) P P( u v) P( v vdv) P ( v) ( )!( j)!( j)!! j j F ( u) f ( u) du[ F( v) F( u)] f ( v) dv[ F( v)] ( )!( j)!( j)! Example: (Order statstcs from expoetal from Example 463) Let,, 3, 4 be a radom sample from expoetal() Fd the jot dstrbuto of the order statstcs (), (), (3), (4) Soluto: The jot pdf of (), (), (3), (4) s (from Theorem 546): 4!exp( x)exp( x)exp( x3)exp( x4),0 x x4 ; f(),, ( x (4),, x4) 0, otherwse Example 547: (Dstrbuto of the mdrage ad rage) Let,, be d uform(0, a ) ad let (),, ( ) deote ther order statstcs Fd the dstrbuto of the rage, R, ad the mdrage, V, where R ( ( ) () ) ad V ( () ( ) )/ Soluto: From Theorem 546, we have the jot pdf of () ad ( ) : ( ) x x ( )( x x) f (), ( x ( ), x ) ( ),0 x x a a a a a 4
Lecture 6 o BST 63: Statstcal Theory I Ku Zhag, /0/008 We also have () V R/, ( ) V R/, ad the Jacoba s - The support set of ( R, V ) s: {( rv, ) :0 rar, / va r/ } So the jot pdf of ( R, V ) s: The margal pdf of R s: The margal pdf of V s: Ad ( ) r frv, ( rv, ),0 rar, / va r/ a ar/ ( ) r ( ) r ( ar) R r / f () r dv,0 r a a a v ( ) r ( v) fv () v dr,0 va/, 0 a a ( av) ( ) r [( av)] fv () v dr, a/ va 0 a a Example: Let,, be cotuous, depedet radom varables ad let (),, ( ) deote ther order statstcs Assume that ~ f ( x ) What s the pdf of () ad ( )? Soluto: The cdf of () s: F ( x) P( x) P( x) P( x,, x) () () () [ F ( )] x P ( x) [ P ( x)] 5
Lecture 6 o BST 63: Statstcal Theory I Ku Zhag, /0/008 So the pdf of () s: The cdf of ( ) s: So the pdf of ( ) s: If f ( x) f( x), the ( ) d ( ) f x F x f ( x ) [ F ( x )] j () () dx F ( x) P( x) P( x,, x) ( ) ( ) P x F x ( ) ( ) d f ( x) F ( x) f ( x) F ( x) j dx ( ) ( ) j f ( x) f( x)[ F( x)] () ad f ( x) f( x)[ F( x)] ( ) j (same as theorem 544) Secto 55 Covergece Cocepts Defto 55: A sequece of radom varables,,,, coverges probablty to a radom varable f, for every 0, lm P( ) 0, or equvaletly, lm P( ) Theorem 55 (Weak Law of Large Numbers): Let,,, be d radom varables wth Var Defe Proof: Use Chebychev s Iequalty The for every 0, lm P( ) E ad 6
Lecture 6 o BST 63: Statstcal Theory I Ku Zhag, /0/008 Note: WLLN states that the sample mea s close to wth hgh probablty as gets larger Ths property of a estmator s kow as cosstecy I geeral, to show a estmator s cosstet for, oe eeds to prove that the estmator coverges probablty to Cosstecy of estmators wll be formally defed ad dscussed Chapter 0 Example 553 (Cosstecy of S ) Let,,, be d radom varables wth ad defe S ( ), ca we prove a WLLN for Ad thus, a suffcet codto that E ad Var S? Usg Chebychev s Iequalty, we have E( S ) Var( S) P( S ) S coverges probablty to s that Var( S ) 0 whe Note: If,, are d (, ), the Var( S ) 0 as, so that S s a cosstet estmator of Theorem 554: Suppose that,, coverges probablty to a radom varable ad that h s cotuous fucto The h ( ), h ( ), coverges probablty to h ( ) Example 555 (Cosstecy of S ) If stadard devato S 5) S s a cosstet estmator of, the by Theorem 554, the sample S s a cosstet estmator of Note that E( S ) s a based estmator of (Exercse 7
Lecture 6 o BST 63: Statstcal Theory I Ku Zhag, /0/008 Defto 556: A sequece of radom varables,,,, coverges almost surely to radom varable f, for every 0, P(lm ) Note: Covergece almost surely mples covergece probablty but the coverse s ot true Example 557 (Almost sure covergece) Let ( s) ss (0 s) ad defe P as the uform probablty dstrbuto o the sample space S [0,], the coverges almost surely to the radom varable () s s(0s ) Example 558 (Covergece probablty, ot almost surely) Let the sample space S be the closed terval [0,] wth the uform probablty dstrbuto Defe the sequece,, as follows: () s s I[0,] () s, () s s I[0,/] (), s 3 () s s I[/,] (), s 4 () s s I[0,/3] () s, 5 () s s I[/3,/3] (), s 6 () s s I[/3,] (), s Let ( s) s The t s straghtforward to prove that coverges probablty to but ot coverges almost surely to However, we ca fd a subsequece of that coverges almost surely to Theorem 559 (Strog Law of Large Numbers) Let,,, be d radom varables wth Var Defe The for every 0, P(lm ) E ad 8
Lecture 6 o BST 63: Statstcal Theory I Ku Zhag, /0/008 Defto 550: A sequece of radom varables,,,, coverges dstrbuto to a radom varable f lm F ( x) F( x) for all pots x where F ( x ) s cotuous Example 55 (Maxmum of Uforms) If,, are d uform (0,) ad ( ) max, the the radom varable ( ) coverges probablty to ad the radom varable ( ( ) ) coverges to a expoetal () radom varable Theorem 55: If the sequece of radom varables,,,, coverges probablty to, the the sequece coverges dstrbuto to Note: Covergece dstrbuto s also mpled by almost sure covergece Theorem 553: The sequece of radom varable,,,, coverges probablty to a costat f ad oly f the sequece also coverges dstrbuto to That s, the statemet lm P( ) 0 for every 0 s equvalet to 0, x u, lm P ( x), x u 9
Lecture 6 o BST 63: Statstcal Theory I Ku Zhag, /0/008 Theorem 554 (Cetral Lmt Theorem) Let,, be a sequece of d radom varables whose mgfs exts eghbor of 0 (that s, M () t exsts for t h, for some postve h ) Let E ad Var 0 (Both ad are fte sce the mgf exsts) Defe for ay x, Let G ( x ) deote the cdf of ( )/ The, x y lm G( x) exp( ) dy That s, ( )/ has a lmtg stadard ormal dstrbuto Proof: Use the theorem 3 Some Notes: assumptos: depedece, detcal dstrbuto ad mgf exsts fte varace s ecessary for covergece to ormalty (CLT wll ot apply to rvs from Cauchy dstrbuto) how good the approxmato s geeral depeds o the orgal dstrbuto Theorem 3: (Covergece of mgfs) Suppose {,,, } s a sequece of radom varables, each wth mgf M () t Furthermore, suppose that lm M ( t) M ( ) t for all t a eghborhood of 0, ad M () t s a mgf The there s uque cdf F whose momets are determed by M () t ad for all x where s cotuous, we have lm F ( x) F( x) That s, covergece, for t hof mgfs to a mgf mples covergece of cdfs 0
Lecture 6 o BST 63: Statstcal Theory I Ku Zhag, /0/008 Theorem 555 (Stroger form of the Cetral Lmt Theorem) Let,, be a sequece of d radom varables wth E ad 0 Var Defe ( )/ The, for ay x, x y lm G( x) exp( ) dy That s, ( )/ has a lmtg stadard ormal dstrbuto Let G ( x ) deote the cdf of Theorem 557 (Slutsky s Theorem) If a Y a dstrbuto b Y a dstrbuto dstrbuto ad Y a, a costat, probablty, the Example 558 (Normal approxmato wth estmated varace) Suppose that S ( ) (0,), but the value of s ukow We have see example 553 that f Var( S ) 0, the probablty By exercse 53, we have / probablty By theorem 557, we have ( ) ( ) (0,) S S S Notes (relatoshp betwee several covergeces)
Lecture 6 o BST 63: Statstcal Theory I Ku Zhag, /0/008 coverges almost surely coverges probablty coverges dstrbuto coverges probablty exsts a subsequece that coverges almost surely 3 coverges probablty to a costat coverges dstrbuto to a costat 4 Slutsky s Theorem Example 559 (Estmatg the odds) Suppose that,,, are d Beroull(( p ) radom varables The typcal parameter of terest s p, whch ca be estmated by We ca obta the dstrbuto of p, whch s Bomal( p), Sometmes we are terested the odds, p, whch may be estmated by The what are propertes of t? For example, how to calculate the varace of t? The exact calculato may be dffcult, but a approxmato ca be obtaed Defto 550: If a fucto gx ( ) has dervatves of order r, that s, costat a, the Tyalor polyomal of order r about a s () r g ( a) Tr ( x) ( xa) 0! Theorem 55 (Taylor) If r ( r d g ) ( a) g( x) r xa exsts, the dx r ( r d g ) ( x) g( x) exsts, the for ay r dx gx ( ) Tr ( x) lm xa 0 r ( x a)
Lecture 6 o BST 63: Statstcal Theory I Ku Zhag, /0/008 For the statstcal applcato of taylor s Theorem, we are most cocerted wth the frst-order Taylor seres Let T be a radom varable wth mea ad suppose that g s dfferetable fucto, the The we have Ad gt () g( ) g'( )( t ) E( gt ( )) Eg ( ( )) g'( ) ET ( ) g( ) ; Var g T E g T g E g T g Var T ( ( )) [ ( ) ( )] [ '( )( )] [ '( )] ( ) Example 55 (Cotuato of Example 559) Recall that we are terested the propertes of t gt (), E( ) p, the g'( t), thus t ( t) p Eg ( ( )) gp ( ) ; p ad p Var g g p Var p p ( p) ( p) ( p) ( ( )) [ '( )] ( ) ( ) 3 Let Example 553 (Approxmate mea ad varace) Suppose s a radom varable wth E( ) 0 If we wat to estmate the mea ad varace of the radom varable g( ), we have E( g( )) g( ) ad 3
Lecture 6 o BST 63: Statstcal Theory I Ku Zhag, /0/008 Var( g( )) [ g '( )] Var( ) Specfcally, for g( ) /, we have E(/ ) / ad 4 Var(/ ) (/ ) Var( ) Theorem 554 (Delta Method) Let Y be a sequece of radom varable that satsfes Y ( ) (0, ) dstrbuto For a gve g ad a specfc value of, suppose that g '( ) exsts ad s ot 0 The gy g g dstrbuto ' [ ( ) ( )] (0, [ ( )] ) Example 555 (Cotuato of Example 553) Suppose ow that we have the mea of a radom sample 4 the we have ( ) (0,(/ ) Var( )), 4