Mahemaical Saisics Iroducio o he maerials o be covered i his course. Uivariae & Mulivariae r.v s 2. Borl-Caelli Lemma Large Deviaios. e.g. X,, X are iid r.v s, P ( X + + X where I(A) is a umber depedig o A (eve). 3. Exreme value heory A) e I(A) X i iid N(, ) wih W = max i X i We will prove W log C a.s. P (W C + x) F (x) x 4. Mulivariae Normal Disribuio X. X k N(µ, Σ) 5. Expoeial Family Ee it X = exp[i T µ 2 T Σ] s p θ (x) = exp[ η i (θ)t i (x) B(θ)]h(x) i= 6. Sufficiecy & Facorizaio Thm p θ, θ Ω, T (x) is a saisic. If P (X A T = ) doe o deped o θ, he we say T is a sufficie saisic. If p θ (x) = g θ (T (x))h(x), he T is sufficie. 7. Rao-Blackwell Theorem Ubiasedess, uiqueess, Basu s hoerem, compleeess ad MVUE (i.e. miimum variace ubiased esimaors).
8. If ime permis, we will sudy some of he followig: Weak covergece of probabiliy measures. i.e. µ wih =, 2, 3, are measures, µ µ Empirical Processes Le X i ; i be iid radom variables wih disribuio N(, ). The i= δ x i F (x) where δ xi = { x A o.w. sup A A So δ xi (A) µ(a) i= If A is VC class, he garaeed o go o. X X Radom Marices.. X X If X ij s are radom, wha s he disribuio of he eigevalues? How abou as? 2 Probabiliy 2. Probabiliy measure ad probabiliy spaces 9. Ω is a se.. If F is a se of subses of Ω. (i) Ω F (ii) A c F if A F (iii) If A F, A 2 F, he i= A i F The F is called a σ-algebra or σ-field.. If F is a σ-algebra, he (i) φ F (ii) If A, A 2, are i F, he i= A i F 2
(iii) If B, B 2,, B are i F he i= A i F, i= A i F. 2. Eg. Ω =, 2,. F =all subses of Ω. The F is σ-algebra. I is geeraed by he se F =fiie subse of Ω. i.e. F is he smalles σ-algebra coaiig F. 3. Eg. Ω = R. F = (a, b), < a < b <. The smalles σ-algebra coaiig F, deoed by B(R ), is called he Borel σ-algebra of R. 4. For ay F F, assig a umber o F, call i P (F ) ad saisfies he followig properies (i) P (Ω = ) (ii) P (A c ) = P (A) for ay A F (iii) P ( i= A i) = i= P (A i) provided A is are disjoi, or muually exclusive, i.e. A i Aj = φ for ay i j. If (i),(ii) ad (iii) are rue, he P : F [, ] is called probabiliy. Righ ow, (Ω, F, P ) is a riple which is called a probabiliy space. 5. Eg. Ω =, 2,, F ={all subses of Ω}. Defie P (F ) = i F 2 i for F F. The P is a probabiliy. 6. Eg. Ω = [, ], B = σ algebra geeraed by {9a, b], [a, b), < a < b }. Oe ca verify ha B = [, ] B(R ). P (B) = he Lebesque measure of B for ay B B.(The r.v. geeraes his P is uiform.), defies a probabiliy over ([, ], B) 7. If X : Ω R, i.e. X(ω) is a real umber for ay ω Ω, saisifies ha X (B) = {w Ω, X(ω) B} F for ay B B(R ) he we say X is a radom variable. 8. Eg. For ay F F, defie F (ω) = { if ω F o.w. of F is a radom variable.. The F, he idicaor fucio 9. (Ω, F, P ) ad X is a r.v. Defie µ(b) = P (X B) = P (ω Ω; X(ω) B) for B B(Re ). Noe ha µ is a probabiliy o (R, B(R )), call µ he disribuio of X. 2. Verify: X is a r.v. iff {X x} F for ay x R. (HW) 3
2. Def. If r.v. X & Y geerae he same disribuio, he we say X & Y are ideically disribued, deoed by X d = Y, or L(X ) = L(Y) (law). 22. Eg. X U[, ], he X d = X. Acually, P ( X B) = P (X B) =Leb measure of (-B)=Leb measure of B. Problem. Le X be a radom variable ad p is a posiive umber such ha P ( X x) /x p for ay x >. Show ha E( X α ) p/(p α) for ay < α < p. Sep. 7, Wedesday, 23. Le X be a radom variable, he P ( X /r ) E X r P ( X > /r ), where r >. = = 2. I geeral, if g(x), X is a sricly icreasig coiuous fucio, le a = g(), =,, 2,..., he P (X a ) Eg (X) P (X > a ). = = Noice ha he firs saeme is jus a geeral case of he secod case, so we are goig o prove he secod oe. Proof: Le Φ(X) = g (X) ad Y = j j Φ(X)<j+, Z = (j + ) (j<φ(x) j+). j= j= Y j= = Φ(X) Φ(X) (j Φ(X)<j+) j= (j Φ(X)<j+) = Φ(X) ( Φ(X)) Φ(X) Z Φ(X) j= = Φ(X) Φ(X)> = Φ(X) (j<φ(x) j+) 4
Firs, P (X a ) = = = = = P (Φ(X) ) = = j= j= = P (j Φ(X) < j + ) j P (j Φ(X) < j + ) jp (j Φ(X) < j + ) j= = EY Similarly, = P (X > a ) = EZ. 3. E X p = p P ( X > ) p d = p P ( X ) p d, forp >. Proof: Noice ha X p = p = p = p Take expecaio o boh sides, we have: X o p d E X p = pe( usig Fubii s Theorem, = p p (< X ) d p ( X ) d X o p d) P ( X > ) p d 4. Cheroff s boud (simple example of large deviaios.) Le X, X 2,..., X be iid r.v. 5
wih mea µ. S = S i=, a > µ. The we like o sudy P ( P ( S a) P (S a), for all > = P (e S e a ) Thus, > a). e a Ee S, by Markov s Iequaliy = e a (Ee x ) = e (a logeex ) P ( S a) If (>)e (a logeex ) = e Sup (>)(a logm X ()) le M X () = Ee X, Thus, P ( S a) e I(a), where I(a) = Sup (>) (a logm X ()). Compare o Chevychev s iequaliy: 9/22 Cheroff s boud P ( S > a) P ( S µ a µ) V ar(x ) (a µ) 2 i.i.d Le X, X, X 2,..., X r.v s wih M() = Ee X <, <, R +, S = X i. The P ( S a) e A a > µ = EX Where P ( S b) e B b Defie: I(x) = sup R {x log M()}, he ()I(x) o [µ, + ] ad o(, µ) (2)I( ) is covex (3)I(x) = sup {x log M()} if x µ I(x) = sup {x log M()} if x µ < µ = EX A = I(a) = sup{a log M()} B = I(b) = sup{b log M()} 6
(4)I(x) = x = µ Proof: (4)Assume I(x ) =, log Ee Le, x lim X log M() x µ as, ad x µ So we ge x = µ. (3) x log M() > Lopiale = lim Ee X X Ee x M() = Ee X e µ Similarly, x log M() <, so x log M() (x µ) I(x) = sup{x log M()} If x > µ ad <, he x log M() < Bu I(x), so I(x) = sup {x log M()} Cheroff s boud: Geeral case Proof: P ( S A) 2e I(A) closed se A, where I(A) = if x A I(x) a, b s. b µ a, a A, b A ad A (, b] [a, + ) P ( S S A) P ( a) + P ( S b) e I(a) + e I(b) 2exp{ mi (I(a), I(b))} sup allsuch(a,b)pairs mi{i(a), I(b)} = if x A I(x) Comme: ()Lower boud: (2) lim if Wha s large deviaio? log P (S A) if I(x) A ope x A P ( S a) C e I(a) as where a µ for some C µ are probabiliy measures, if a fucio I(x) saisfies ()I(x) (2){x : I(x) l} is b.d.d. ad closed se l 7
(3) lim sup log µ (A) if x A I(x) A closed lim if log µ (B) if x B I(x) B ope The we say {µ } saisfies Large Deviaio Priciple (LDP) wih rae fucio I(x). Noe of Sa8, 9/24/Fall 23 Characerisic Fucio: X is a radom variable, he he characerisic fucio of X is ϕ X () = Ee ix = E(cos(x) + i si(x) where is a real umber, i =, e is = cos(s) + i si(s). Example. X Ber(p), P (X = ) = p, P (X = ) = q = p. ϕ X () = Ee ix = q + pe i Example 2. X Bi(, p). There are wo mehods o calculae he characerisic fucio. Mehod : P (X = k) = ( k) p k q k, so ϕ X () = Ee ix = ( e ik k = k= k= = (q + pe i ) ) p k q k ( ) (pe i ) k q k k Mehod 2: X = X + + X, X i s are i.i.d Ber(p), he Example 3. X N(µ, σ 2 ), he ϕ X () = Ee ix = E(e ix e ix2 e ix ) = Ee ix Ee ix2 Ee ix = (Ee ix ) = (q + pe i ) σ2 (iµ ϕ X () = e 2 2), R 8
Example 4. X Uif[, ], he ϕ X () = 2 Levy s Iversio Formular e ix dx = 2i (ei e i ) = (cos + i si (cos i si )) 2i = si Theorem: X is a radom variable wih characerisic fucio ϕ(). The, for ay a < b, Proof: Le Firs, [ c lim c + 2π e ia e ib i ] P (X = a) + P (X = b) ϕ()d = P (a < X < b) + 2 I(c) = e ia e ib 2π i = ( 2π E i = 2π E ( ϕ()d e ia e ib ) e ix d ) e i(x a) e i(x b) d i Similarly, The, e i(x a) i d = i = i = 2 = 2 = 2 [cos((x a)) + i si((x a))]d i si((x a)) d si((x a)) d si((x a)) d(x a) (x a) (x a) e i(x b) i I(c) = π E [ (x a) si d (x b) d = 2 si d si c(x b) d ] si d 9
(x b) si Le J c (x) = d c(x a) (i) Suppose x > a ad x < b, he (ii) Suppose x > b, he J c (x) si x x dx = π J c (x) as c + (iii) Suppose x < a, he (iv) Suppose x = a, he (v) Suppose x = b, he Le s summarize, J c (x) as c + J c (x) J c (x) si x x dx = π 2 si x x dx = π 2 Sice si x x Theorem: If lim J c(x) = π(a < x < b) + π ((x = a) + (x = b)) c 2 dx = π, we have sup J c (x) <. By domia covergece heorem, c lim I(c) = c π lim EJ c(x) = P (a < x < b) + c ϕ() d <, he he disribuio fucio of X has a bouded probabiliy desiy fucio give by. Levy s Iversio Formula P (x = a) + P (x = b) 2 f(y) = 2π e iy ϕ()d 9-26-3 Friday Suppose X has characerisic fucio φ(), he lim [ c 2π e ia e ib i φ()d] = P (a < X < b) + P (X = a) + P (X = b) 2 Give φ(), how do we ge disribuio of X? Noe: we ca o recover he radom variable X because differe radom variables could have he same disribuio. (e.g. X U[, ], he X U[, ]).
2. Discussio Give φ(), we recover F (x) = P (X < x). From he iversio formula, I kow ha P (a < X < b) for a ad b such ha P (X = a) =, P (X = b) =. Noe: D = a : P (X = a) is couable, i.e. discoious pois are couable. So I kow he value of F (b) F (a) for a, b R D. Le a R D, ad a, he I ge F (b) for ay b R D. Now for ay x D, choose b x, defie So I obai F (x) for ay x R 3. Proposiio F (x) = lim b x F (b ) If cdf F (x) ad G(x) are ideical o H, where he closure of H is R, he F (x) G(x), x R. Proof: x H, choose x H ad x x. Sice F (x ) = G(x ),. Now le, we he have F (x ) = G(x ), because F (x) ad G(x) are righ-coiuous. 4. Theroem If d <, he X does have ay discoiuous pois ad he desiy of X is give by f(x) = e ix φ()d 2π Also, f(x) is bouded. Noe: desiy of f(x) ca be ubouded. For example, f(x) = { x < 2 x < x < 5. Proof: Noe: e ia e ib i = b a e ix dx b a e ix dx = b a
Claim: No discoiuous pois. I fac, by he iversio formula, P (X = a) + P (X = b) 2 P (X = a) + P (X = b) + P (a < X < b) 2 c e ia e ib = lim φ()d c 2π i (b a) lim φ() d 2π c (b a) φ() d If P (X = a) >, he pick b = a + ɛ, ɛ > ad ɛ, ad b is a coiuous poi of F (x). Le, he P (X = a) P (X = a) = 6. So far he iversio formula becomes Now, Sice e ia e ib i b a Therorem, 2π f(a) = F (a) e ia e ib φ()d = F (b) F (a) i F (b) F (a) = lim b a b a = 2π lim e ia e ib φ()d b a i b a is bouded ad φ() d <, so by Domiaed Covergece F (a) = 2π = 2π i ( e ia ) φ()d e ia φ()d 2