556: MATHEMATICAL STATISTICS I

Transcription

1 556: MATHEMATICAL STATISTICS I INEQUALITIES 5.1 Concenraion and Tail Probabiliy Inequaliies Lemma (CHEBYCHEV S LEMMA) c > 0, If X is a random variable, hen for non-negaive funcion h, and P X [h(x) c] E f X [h(x)] c Proof (coninuous case) : Suppose ha X has densiy funcion f X which is posiive for x X. Le A = x X : h(x) c} X. Then, as h(x) c on A, E fx [h(x)] = h(x)f X (x) dx = h(x)f X (x) dx + h(x)f X (x) dx A A h(x)f X (x) dx and he resul follows. A A cf X (x) dx = c P X [X A] = c P X [h(x) c] SPECIAL CASE I - THE MARKOV INEQUALITY If h(x) = x r for r > 0, so P X [ X r c] E f X [ X r ]. c Alernaely saed (by Casella and Berger) as follows: If P [Y 0] = 1 and P [Y = 0] < 1, hen for any r > 0 P Y [Y r] E f X [Y ] r wih equaliy if and only if P Y [Y = r] = p = 1 P Y [Y = 0] for some 0 < p 1. SPECIAL CASE II - THE CHEBYCHEV INEQUALITY Suppose ha X is a random variable wih expecaion µ and variance σ 2. Then h(x) = (x µ) 2 and c = k 2 σ 2, for k > 0, [ P X (X µ) 2 k 2 σ 2] 1/k 2 or equivalenly Seing ɛ = kσ gives or equivalenly P X [ X µ kσ] 1/k 2. P X [ X µ ɛ] σ 2 /ɛ 2 P X [ X µ < ɛ] 1 σ 2 /ɛ 2. 1

2 Theorem (CHERNOFF BOUNDS) Suppose ha X 1,..., X n are independen binary rials (known as Poisson rials ) such ha 1 pi x = 0 P Xi [X i = x] = p i x = 1 and zero oherwise. Le X = (X X n ), so ha E fx [X] = n p i = µ, say. Then for d > 0 If 0 d 1, a simpler bound is e d } µ P X [X (1 + d)µ] (1 + d) (1+d). P X [X (1 + d)µ] exp µd 2 /3 }. Proof Le a > 0. Then, using he Chebychev Lemma wih h(x) = e ax, and c = e a(1+d)µ, we have Bu, by independence, Now for y 0, P X [X (1 + d)µ] = P X [expax} expa(1 + d)µ}] E f X [expax}] expa(1 + d)µ}. (1) E fx [expax}] = E fxi [expax i }] = [p i e a + (1 p i )] = [1 + p i (e a 1)] e y = 1 + y + y y so seing y i = p i (e a 1) and using his inequaliy erm by erm, we conclude from equaion (1) ha n } E fx [expax}] = [1 + p i (e a 1)] expp i (e a 1)} = exp p i (e a 1) = exp µ(e a 1)}. Hence and seing a = log(1 + d) yields For 0 d 1, we have ha P X [X (1 + d)µ] exp µ(ea 1)} expa(1 + d)µ} P X [X (1 + d)µ] To see his, consider aking logs, and he funcion e µd (1 + d) µ(1+d) = e d (1 + d) (1+d) e d } µ exp µd 2 /3}. (1 + d) (1+d) g(d) = d (1 + d) log(1 + d) + d 2 /3. We need o show ha g(d) is bounded above by zero for 0 d 1. Clearly g(0) = 0, and aking derivaives wice we have g (1) (d) = log(1 + d) + 2d/3 g (2) (d) = 1 (1 + d) + 2/3. Therefore g (1) (0) = 0, g (2) (0) = 1/3 < 0 and g (1) (1) = log 2 + 2/3 < 0, so g (1) (d) says negaive for all 0 < d 1 as here is no soluion of g (1) (d) = 0 in his inerval. Thus g(d) mus also be negaive for all d in his range. 2 } µ

3 Theorem (A CHERNOFF BOUND USING MGFS) If X is a random variable, wih mgf M X () defined on a neighbourhood ( h, h) of zero. Then P X [X a] e a M X () for 0 < < h Proof Using he Chebychev Lemma wih h(x) = e x and c = e a, for > 0, P X [X a] = P X [X a] = P X [expx} expa}] E f X [e X ] e a provided < h also. Using similar mehods, = M X() e a P X [X a] e a M X () for h < < 0 Theorem (TAIL BOUNDS FOR THE NORMAL DENSITY) If Z N (0, 1), hen for > /2 1 /2 π e 2 P Z [ Z ] π e 2 Proof By symmery, P Z [ Z ] = 2 P Z [Z ], so P Z [Z ] = ( ) 1 1/2 ( ) 1 1/2 ( ) e x2 /2 x 1 1/2 dx /2 e 2 /2 2π 2π e x2 dx =. 2π Similarly, for > 0, e x2 /2 dx [ x /2 x e x2 dx = 1 ] /2 1 x e x2 /2 x 2 e x2 dx 1 /2 e e x2 /2 dx afer wriing 1 = x/x, hen inegraing by pars, and hen noing ha, on (, ), x > 1/x 2 < 1/ 2, and ha he inegrand is non-negaive. Therefore, combining erms (1 + 1 ) 2 e x2 /2 dx 1 /2 e 2 and cross-muliplying by he posiive erm 2 /(1 + 2 ) yields e x2 /2 dx e 2 /2 P Z [ Z > ] 2 /2 π e 2. To see he qualiy of he approximaion, he able below shows he values of he bounding values for ranging from 1 o 5. Clearly he bounds improve as ges larger Lower 2.420e e e e e e e e e-07 True 3.173e e e e e e e e e-07 Upper 4.839e e e e e e e e e-07 3

4 5.2 Expecaion Inequaliies Lemma Le a, b > 0 and p, q > 1 saisfy Then wih equaliy if and only if a p = b q. Proof Fix b > 0. Le p 1 + q 1 = 1. (2) p 1 a p + q 1 b q ab g(a; b) = p 1 a p + q 1 b q ab. We require ha g(a; b) 0 for all a. Differeniaing wr a for fixed b yields g (1) (a; b) = a p 1 b, so ha g(a; b) is minimized (he second derivaive is sricly posiive a all a) when a p 1 = b, and a his value of a, he funcion akes he value p 1 a p + q 1 (a p 1 ) q a(a p 1 ) = p 1 a p + q 1 a p a p = 0 as, by equaion (2), 1/p + 1/q = 1 = (p 1)q = p. As he second derivaive is sricly posiive a all a, he minimum is aained a he unique value of a where a p 1 = b, where, raising boh sides o power q yields a p = b q. Theorem (HÖLDER S INEQUALITY) Suppose ha X and Y are wo random variables, and p, q > 1 saisfy 2. Then E fx,y [XY ] E fx,y [ XY ] E fx [ X p ]} 1/p E fy [ Y q ]} 1/q Proof (coninuous case) For he firs inequaliy, E fx,y [ XY ] = xy f X,Y (x, y) dx dy xyf X,Y (x, y) dx dy = E fx,y [XY ] and E fx,y [XY ] = xyf X,Y (x, y) dx dy xy f X,Y (x, y) dx dy = E fx,y [ XY ] so E fx,y [ XY ] E fx,y [XY ] E fx,y [ XY ] E fx,y [XY ] E fx,y [ XY ]. For he second inequaliy, se Then from he previous lemma p 1 a = X p E fx [ X p ] + q 1 X E fx [ X p ]} 1/p b = and aking expecaions yields, on he lef hand side, Y E fy [ Y q ]} 1/q. Y q E fy [ Y q ] XY E fx [ X p ]} 1/p E fy [ Y q ]} 1/q p 1 E f X [ X p ] E fx [ X p ] + q 1 E f Y [ Y q ] E fy [ Y q ] = p 1 + q 1 = 1 and on he righ hand side and he resul follows. E fx,y [ XY ] E fx [ X p ]} 1/p E fy [ Y q ]} 1/q 4

5 Theorem (CAUCHY-SCHWARZ INEQUALITY) Suppose ha X and Y are wo random variables. E fx,y [XY ] E fx,y [ XY ] E fx [ X 2 ] } 1/2 EfY [ Y 2 ] } 1/2 Proof Se p = q = 2 in he Hölder Inequaliy. Corollaries: (a) Le µ X and µ Y denoe he expecaions of X and Y respecively. Then, by he Cauchy-Schwarz inequaliy so ha and hence E fx,y [(X µ X )(Y µ Y )] E fx [(X µ X ) 2 ] } 1/2 EfY [(Y µ Y ) 2 ] } 1/2 E fx,y [(X µ X )(Y µ Y )] E fx [(X µ X ) 2 ]E fy [(Y µ Y ) 2 ] CovfX,Y [X, Y ] } 2 VarfX [X] Var fy [Y ]. (b) Lyapunov s Inequaliy: Define Y = 1 wih probabiliy one. Then, for 1 < p < Le 1 < r < p. Then and leing s = pr > r yields so ha for 1 < r < s <. E fx [ X ] E fx [ X p ]} 1/p. E fx [ X r ] E fx [ X pr ]} 1/p E fx [ X r ] E fx [ X s ]} r/s E fx [ X r ]} 1/r E fx [ X s ]} 1/s Theorem (MINKOWSKI S INEQUALITY) Suppose ha X and Y are wo random variables, and 1 p <. Then EfX,Y [ X + Y p ] } 1/p EfX [ X p ]} 1/p + E fy [ Y p ]} 1/p Proof Wrie E fx,y [ X + Y p ] = E fx,y [ X + Y X + Y p 1 ] E fx,y [ X X + Y p 1 ] + E fx,y [ Y X + Y p 1 ] by he riangle inequaliy x + y x + y. Using Hölder s Inequaliy on he erms on he righ hand side, for q seleced o saisfy 1/p + 1/q = 1, E fx,y [ X +Y p ] E fx [ X p ]} 1/p E fx,y [ X + Y q(p 1) ]} 1/q +EfY [ Y p ]} 1/p E fx,y [ X + Y q(p 1) ] and dividing hrough by E fx,y [ X + Y q(p 1) ] } 1/q yields E fx,y [ X + Y p ] EfX,Y [ X + Y q(p 1) ] } 1/q E f X [ X p ]} 1/p + E fy [ Y p ]} 1/p and he resul follows as q(p 1) = p, and 1 1/q = 1/p. } 1/q 5

6 5.3 Jensen s Inequaliy Jensen s Inequaliy gives a lower bound on expecaions of convex funcions. Recall ha a funcion g(x) is convex if, for 0 < λ < 1, g(λx + (1 λ)y) λg(x) + (1 λ)g(y) for all x and y. Alernaively, funcion g(x) is convex if Conversely, g(x) is concave if g(x) is convex. d 2 d 2 g()} =x = g(2) (x) 0. Theorem (JENSEN S INEQUALITY) Suppose ha X is a random variable wih expecaion µ, and funcion g is convex. Then E fx [g(x)] g(e fx [X]) wih equaliy if and only if, for every line a + bx ha is a angen o g a µ ha is, g(x) is linear. P X [g(x) = a + bx] = 1. Proof Le l(x) = a + bx be he equaion of he angen a x = µ. Then, for each x, g(x) a + bx as in he figure below. g(x) g(x) µ = 2 l(x) = a + bx x Figure 1: The funcion g(x) and is angen a x = µ. 6

7 Thus E fx [g(x)] E fx [a + bx] = a + be fx [X] = l(µ) = g(µ) = g(e fx [X]) as required. Also, if g(x) is linear, hen equaliy follows by properies of expecaions. Suppose ha E fx [g(x)] = g(e fx [X]) = g(µ) bu g(x) is convex, bu no linear. Le l(x) = a + bx be he angen o g a µ. Then by convexiy g(x) l(x) > 0 (g(x) l(x))f X (x) dx = g(x)f X (x) dx l(x)f X (x) dx > 0 and hence E fx [g(x)] > E fx [l(x)]. Bu l(x) is linear, so E fx [l(x)] = a + be fx [X] = g(µ), yielding he conradicion E fx [g(x)] > g(e fx [X]). and he resul follows. Corollary and examples: If g(x) is concave, hen g(x) = x 2 is convex, hus g(x) = log x is concave, hus E fx [g(x)] g(e fx [X]) E fx [ X 2 ] E fx [X]} 2 E fx [log X] log E fx [X]} Lemma Suppose ha X is a random variable, wih finie expecaion µ. Le g be a non-decreasing funcion. Then E fx [g(x)(x µ)] 0 Proof By definiion, E fx [g(x)(x µ)] = = µ g(x)(x µ)f X (x) dx g(x)(x µ)f X (x) dx + µ g(x)(x µ)f X (x) dx. Now µ g(x)(x µ)f X (x) dx µ g(µ)(x µ)f X (x) dx as, on (, µ), x < µ, so x µ is negaive, and hus as g is non-decreasing, on his range g(x)(x µ) g(µ)(x µ) 7

8 as he lef hand side is less negaive han he righ hand side. Similarly, µ g(x)(x µ)f X (x) dx µ g(µ)(x µ)f X (x) dx as, on (µ, ), x > µ, so x µ is posiive, and hus as g is non-decreasing, on his range g(x)(x µ) g(µ)(x µ) as he lef hand side is more posiive han he righ hand side. Hence E fx [g(x)(x µ)] µ g(µ)(x µ)f X (x) dx + µ g(µ)(x µ)f X (x) dx = g(µ) (x µ)f X (x) dx = 0 8