Boning Yang. March 8, 2018

Size: px

Start display at page:

Download "Boning Yang. March 8, 2018"

Hilary Randall
5 years ago
Views:

1 Concentraton Inequaltes by concentraton nequalty Introducton to Basc Concentraton Inequaltes by Florda State Unversty March 8, 2018

2 Framework Concentraton Inequaltes by 1. concentraton nequalty concentraton nequalty Introducton to Basc 2. Introducton to 3. Basc 4.

3 concentraton nequalty Concentraton Inequaltes by Value of sum of rvs s concentrated near ts expectaton. concentraton nequalty Introducton to Basc How about non-asymptotc and general functon case? The tal probablty of functon of r.v. s s bounded. 2 ncludes Markov case where no center exsts n event.

4 Mathematcal vew of Concentraton Inequaltes by measures unpredctablty of r.v. n terms of state. concentraton nequalty Introducton to Basc Varance measures that n terms of numercal value. It makes no sense to defne the expectaton of state! Unpredctablty s related to tal prob. to some extent.

5 Shannon s defnton from nformaton theory Concentraton Inequaltes by Let p j, I (p j ) be the prob. and nfo. of event E j. concentraton nequalty Introducton to Basc (1) I (p j ) 0, I (1) = 0, nfo. of any event s non-negatve! (2) I (p 1 p 2 ) = I (p 1 ) + I (p 2 ). Info. of events s addtve. I (p) = C log p where C < 0 s the target functon.

6 Example for I (p 1 p 2 ) = I (p 1 ) + I (p 2 ) Concentraton Inequaltes by Toss an unf. con twce. E 1 : head n 1st toss; E 2 : tal n 2nd. concentraton nequalty Introducton to Basc I (P (E 1 E 2 )) = I (P (E 1 )) + I (P (E 2 )) when E 1 E 2. Toss one con once. E 1 = E 2 : head n 1st toss. Not! I (P (E 1 E 2 )) = I (P (E 1 )) I (P (E 1 )) + I (P (E 2 )).

7 What do we learn from Shannon? Concentraton Inequaltes by May be hard to know the target functon n advance. concentraton nequalty Introducton to Basc But can lst the propertes target functon must satsfy. Is target functon unque? Add property 1 by 1 accordng to the relatve mportance f needed.

8 Addtonal comments Concentraton Inequaltes by Can buld confdence by checkng nduced propertes. concentraton nequalty Introducton to Basc I (p) = C log p wth C < 0. I (p) s non-ncreasng w.r.t p. I (0) = +.

9 : expected nformaton gan Concentraton Inequaltes by I (p) = C log p, C = 1 s used for conventon. concentraton nequalty Introducton to Do N experments, calculate the ave. nfo. Basc Expected ave. nfo. : H (X ) = x X Np(x) log p(x) N. : H (X ) = x X P (x) log P (x) = E log P (X ).

10 Unf. con e.g. dfference between var. and Concentraton Inequaltes by Y {head, tal}: the state result after con toss. concentraton nequalty Introducton to Basc X {1(when head), 1.01(when tal)} : numercal result... Var (X ) small whle H (X) = H (Y) large, sgn. dfference! /Var. tells the spread of categorcal/numercal r.v.

11 Condtonal Concentraton Inequaltes by {X = x} {Y = y X = x}, = {X = x, Y = y}. concentraton nequalty Introducton to Basc I (P (E 1 E 2 )) = I (P (E 1 )) + I (P (E 2 )) when E 1 E 2. I (P (X = x, Y = y)) = I (P (X = x)) + I (P (Y = y X = x)). Intutvely, nfo. of 2 events = Info. of A + nfo. of B A.

12 Condtonal (contnue 1) Concentraton Inequaltes by I (P (X = x, Y = y)) = I (P (X = x)) + I (P (Y = y X = x)). concentraton nequalty Introducton to Basc E Y X =x log (X = x, Y ) = log (P (X = x)) + E Y X =x (log (P (Y X = x))). Take exp. on X, E log P (X, Y ) = E log P (X ) + E log P (Y X ). H (X, Y ) = H (X ) + H (Y X ).

13 Condtonal (contnue 2) Concentraton Inequaltes by H (Y, X ) = H (X, Y ) = H (X ) + H (Y X ). concentraton nequalty Introducton to Basc Perm. nvar., exp. nfo. of 2 = that of X + extra of Y gven X. H (X ) H (X Y ) H (X Y, Z), proof omtted. Gven Y, X carres less ave. nfo., same for the 2nd.

14 Proof of Han s nequalty (nducton also works) Concentraton Inequaltes by H (X 1,..., X n) n=1 H(X () ) n 1, X () = (X 1,..., X 1, X +1,..., X n). concentraton nequalty Introducton to Basc Hnted by r.h.s, H (X 1,..., X n ) = H ( X ()) + H ( X X ()). To prove n =1 H ( X X ()) H (X 1,..., X n ), assume n = 3. Left H (X 1 ) + H (X 2 X 1 ) + H (X 3 X 2, X 1 ) = H (X 1, X 2, X 3 ).

15 Relatve (KL-dvergence) Concentraton Inequaltes by Let P, Q be the p.d.f. for r.v. X. concentraton nequalty Introducton to Basc D (Q P) = X =x Q(x) Q (x) log P(x), P (x) = 0 Q (x) = 0. Argu. for the def. from codng theory can be found (dubous). Amount of nfo. lost when P s used to approxmate Q.

16 Han s nequalty for relatve Concentraton Inequaltes by Let P = P 1 P n be p.d.f. for X n = (X 1,..., X n), P () ( x () ) = concentraton nequalty Introducton to Basc ) x P (x, x () = x P 1 (x 1) P n (x n), Q s defned smlarly. Then D (Q P) n =1 ( D (Q P) D ( Q () P ())), or equvalently, D (Q P) 1 n 1 n =1 D ( Q () P ()).

17 D (Q P) 1 n 1 n =1 D ( Q () P ()) (contnue 1) Concentraton Inequaltes by Hnted by Han s H (X n ) n =1 H(X () ) n 1, choose the 2nd form. concentraton nequalty Introducton to D (Q P) = x n Q (x n ) log Q (x n ) x n Q (x n ) log P (x n ). Basc Han s for : H (X n ) n =1 H(X () ) n 1 mples x Q (x n n ) log Q (x n ) =1[ n x () Q () (x () ) log Q () (x () )] n 1 ( - sgn).

18 Stll true when Q sn t prob. measure (contnue 2) Concentraton Inequaltes by Goal: x n Q (x n ) log P (x n ) = n =1[ x () Q () (x () ) log P () (x () )] n 1. concentraton nequalty Introducton to Basc x Q (x n n ) log P (x n ) = =1[ x n Q(x n )(log P () (x n() )+log P (x n ))] n n. x Q (x n n ) log P (x n ) = =1[ x n Q(x n )(log P () (x n() ))] n n 1. n =1[ x n Q(x n )(log P () (x n() ))] n n 1 = =1[ x () Q () (x () ) log P () (x () )] n 1.

19 D (Q P) n ( ( =1 D (Q P) D Q () P ())) Concentraton Inequaltes by ) ] φ (x) = x log x, Y = f (X n ) 0, E Y (X () = E [Y X () concentraton nequalty Introducton to Basc Eφ (Y ) φ (EY ) n =1 E [E φ (Y ) φ (E Y )] (Lemma 1). T for EY = 1 T for CY wth EY = 1. Assume T for EY = 0? Eφ (Y ) φ (EY ) = Eφ (Y ) = E [Y log Y ] = D (Q P) (Han s KL)?

20 Q (X n ) = f (X n ) P (X n ) (contnue 1) Concentraton Inequaltes by x n Q log Q P = D (Q P) = Eφ (Y ) = xn Pf log f. concentraton nequalty Introducton to Basc Han s for KL: D (Q P) n =1 ( D (Q P) D ( Q () P ())). Goal: D ( Q () P ()) = E [φ (E Y )] (take exp. on X () ). ) D (Q () P () = ) ( ) x (x () Q() () log Q() (x () ) P () (x () ), P() x () needed!

21 Goal: D ( Q () P ()) = E [φ (E Y )] (contnue 2) Concentraton Inequaltes by ) x (x () Q() () log Q() (x () ) P () (x () ) = ( ) x () P() x () Q () (x () ) P () (x () ) log Q () (x () ) P () (x () ). concentraton nequalty Introducton to Basc Need to prove E Y ( x ()) = E [ Y x ()] = Q() (x () ) P () (x () ). E [ Y x ()] = x n x () f (x n ) P ( x n x ()) = x n x () f (x n )P(x n ) P () (x () ). x n x () f (x n )P(x n ) P () (x () ) = x n x () Q(x n ) P () (x () ) = Q() (x () ) P () (x () ).

22 Some notatons and explanatons Concentraton Inequaltes by Let Z = g (X n ), Z = g ( X 1,..., X,..., X n ), ψ (x) = e x x 1. concentraton nequalty Introducton to Basc Z, Z dffers only by th component n X n, where X, X are..d. If n = 4, = 2, Z = g (3, 5, 7, 8), then Z 2 = g (3,?, 7, 8). Gven x () = (x 1,..., x 1, x +1,..., x n ), X X Z Z.

23 and Lemma 2 Concentraton Inequaltes by se [ Ze sz] E [ e sz] log E [ e sz] ( ( ))] n =1 [e E sz ψ s Z Z. concentraton nequalty Introducton to Basc Lemma 2 (proof omtted): for any postve r.v.s Y, Y > 0, E [Y log Y ] (EY ) log (EY ) E [ ( Y log Y Y log Y Y Y When provng L.S.I., Y = e sz s suggested by lemma 2! )].

24 Idea of the proof Concentraton Inequaltes by se [ Ze sz] E [ e sz] log E [ e sz] ( ( ))] n =1 [e E sz ψ s Z Z. concentraton nequalty Introducton to Basc Left = Eφ (Y ) φ (EY ) wth Y = e sz and φ (Y ) = Y log Y. Lemma 1: Eφ (Y ) φ (EY ) n =1 E [E φ (Y ) φ (E Y )]. [ ( Lemma 2: E φ (Y ) φ (E Y ) E Y log Y Y log Y Y Y )].

25 Idea of proof (contnue) Concentraton Inequaltes by ( ( ))] Rght of Lemma 2 = E [e sz ψ s Z Z wth Y = e sz. concentraton nequalty Introducton to Basc n =1 E [E φ (Y ) φ (E Y )] [ ( ( n =1 E E [e sz ψ s Eφ (Y ) φ (EY ) [ ( ( n =1 E E [e sz ψ s Z Z Z Z ))]]. se [ Ze sz] E [ e sz] log E [ e sz] ( ( ))] n =1 [e E sz ψ s Z Z. ))]].

26 Symmetrzed Verson of L.S.I. Concentraton Inequaltes by se [ Ze sz ] E [ e sz ] log E [ e sz ] ( ( n =1 [e E sz τ s Z Z )) ] 1 Z>Z, concentraton nequalty Introducton to Basc where τ (x) = x (e x 1) and the statement s true for all s > 0. Eφ (Y ) φ (EY ) [ ( ( n =1 E E [e sz ψ s ( ( E [e sz ψ s Z Z ))] [ ( ( E e sz τ s Z Z Z Z )) ))]]. 1 Z>Z ]?

27 E [ e sz ψ ( s ( Z Z ))] (symmetrzed verson!) Concentraton Inequaltes by ( ( = E [e sz ψ s Z Z )) ] [ ( ( 1 Z>Z + E e sz ψ s Z Z )) ] 1 Z<Z. concentraton nequalty Introducton to Basc ( ( E [e sz ψ s Z Z )) ] [ 1 Z<Z = E e sz ( ( ψ s ] ( E [e sz ( ψ s (Z Z )) 1 = E e sz e s Z Z ) ψ Z>Z ( ( E [e sz ψ s Z Z ( s ))] [ ( ( = E e sz τ s Z Z (Z Z )) Z Z )) ] 1 Z>Z. 1 Z>Z )). 1 Z>Z ]!

28 An example of applcaton Concentraton Inequaltes by n =1 (Z Z ) 2 C t > 0, P {Z > EZ + t} 2e t 2 /4C. concentraton nequalty Introducton to Basc x > 0, τ ( x) x 2 s > 0, se [ Ze sz ] E [ e sz ] log E [ e sz ] E [ e sz ( ) ] 2 n =1 s2 Z Z 1Z>Z s 2 CE [ e sz ]. Brefly, sm (s) M (s) log M (s) Cs 2 F (s), M (s) = E [ e sz ].

29 An example of applcaton (contnue) Concentraton Inequaltes by After math trck, M (s) e sez+s2c, Chernoff suggests: concentraton nequalty Introducton to Basc P (Z > EZ + t) = P ( e sz > e s(ez+t)) E[esZ ] e sez+st e s2 C st. e t2 /4C e s2 C st, s = t 2C gves the tghtest bound. Last word...

30 Take home message Concentraton Inequaltes by Mathematcal proof often comes after the correct ntuton. concentraton nequalty Introducton to Basc Rule 1: start wth smple case. Rule 2: replacng term by +, rather than,. Try to gve defntons n an axomatc way to convnce others.

3.1 Expectation of Functions of Several Random Variables. )' be a k-dimensional discrete or continuous random vector, with joint PMF p (, E X E X1 E X

3.1 Expectation of Functions of Several Random Variables. )' be a k-dimensional discrete or continuous random vector, with joint PMF p (, E X E X1 E X Statstcs 1: Probablty Theory II 37 3 EPECTATION OF SEVERAL RANDOM VARIABLES As n Probablty Theory I, the nterest n most stuatons les not on the actual dstrbuton of a random vector, but rather on a number