Ifomatio ad Codig Theoy Autum 207 Lectue: Madhu Tulsiai Lectue 6: Octobe 6, 207 The Method of Types Fo this lectue, we will take U to be a fiite uivese U, ad use x (x, x 2,..., x to deote a sequece of elemets fom U. Defiitio. The type P x of x, also called the empiical distibutio of x, is a distibutio ˆP o U, defied as ˆP(a : {i : x i a} a U. We use T to deote the set of all types comig fom sequeces of legth. We also use C P to deote the set of all seueces with the type P. C P is called the type class of P. C P : {x U P x P}. Execise.2 Check that T ( + ( +. Next, we boud the size of a give type class i tems of the etopy of that type. Popositio.3 Fo ay type P T, we have 2 H(P ( + C P 2 H(P. Poof: Fo each a i U, let P(a i k i /. The C P /(k!k 2!... k!. We pove the lowe boud by cosideig (k + k 2 + + k j + +j j!... j m! (kj... kj m ( + max j + +j j!... j! (kj... kj m, whee each tuple (j,..., j coespods to a distict type. We leave it as a execise to check that the maximum tem i the expessio above is whe (j,..., j (k,..., k.
Execise.4 Show that j!... j! (kj... kj k!... k! (kk... kk fo all (j,..., j such that j + + j. (Hit: if j s > k s fo some s, the j t < k t fo some t. Usig the above, we ca ow pove the lowe boud. ( + k!... k! (kk... kk ( + C P (k k... kk m m. We get C P ( + m k+k2+ +k ( + m ( + m k k i i... kk ( ki k i 2 k i log(/k i ( + m 2 H(P. The poof of the uppe boud is simila ad left as a execise. Next, we eed the obsevatio that the pobability of a sequece accodig to a poduct distibutio oly depeds o its type. Popositio.5 Let Q be ay distibutio o U ad let Q the poduct distibutio o U. Let x, y U be such that P x P y. The, Q (x Q (y. Poof: Let P P x P y. The we have: Q (x (Q(a {i:x i} (Q(a P(a Q (y. Now we give bouds o the pobability of a cetai type occuig, i tems of the KL divegece betwee the tue distibutio ad the empiical distibutio. Theoem.6 Fo ay poduct distibutio Q ad type P o U, we have 2 D(P Q ( + P x Q [P x P] 2 D(P Q. 2
Poof: Let x be of type P x P. Fo the lowe boud, we ote that Q (x P (x (Q(a P(a ( Q(a P(a (P(a P(a 2 P(a log P(a ( Q(a P(a We also kow fom the pevious popositio that fo ay x C P, we have P (x Fially, usig Popositio.3, we get x Q x P] (P(a P(a 2 H(P. x C P Q (x x C P 2 H(P 2 D(P Q C P 2 H(P 2 D(P Q 2 H(P ( + 2 H(P 2 D(P Q 2 D(P Q ( + 2 D(P Q The poof of the uppe boud is left as a execise. Note that It may be that Supp(Q Supp(P i.e., a U : Q(a 0, P(a 0. The the log(/q(a tem makes D(P Q udefied, so thikig of D(P Q as +, we get 2 D(P Q Pob Q (T P 0. 2 Cheoff bouds The above coutig ca be used to pove the Cheoff boud. Let U {0, }, ad let x (x,..., x be a sequece daw fom U accodig to Q, whee { 0 : with pobability /2 Q : with pobability /2. We expect thee to be aoud /2 occueces of i X; that is, E[i x i] /2. It is atual to ask how much the empiical distibutio is likely to deviate fom /2. If we set { 0 : with pobability /2 ε P : with pobability /2 + ε, the we have [ P X + + X ] Q 2 + ε P x Q [P x P] 2 D(P Q 2 c ε2, by Theoem.6, fo a costat c. This is sot of like Cheoff bouds, but we may wat to kow how likely we ae to see ay sufficietly lage deviatio, ad ot just the deviatio exactly equal to ε. 3
Theoem 2. (Cheoff boud Fo X (X,..., X Q U with Q the uifom distibutio o U {0, }, we have [ ] P X i Q 2 + ε i ( + 2 c ε2. Poof: Let U {0, } ad ote that that each type class coespods to a uique value of x + + x. Fom the above boud, we have that fo ay η > 0, [ P X + + X ] Q 2 + η 2 c η2. Goig ove all types fo all η ε, ad otig that the umbe of types is at most +, we get as claimed. [ ] P X i Q 2 + ε i ( + 2 c ε2, The above idea ca be geealized fo poduct distibutios ove abitay (fiite uiveses to pove a geeal lage deviatio esult kow as Saov s theoem. 3 Saov s theoem We geealize the Cheoff boud to udestad the pobability that P x Π fo a abitay set Π of distibutios ove U. Theoem 3. (Saov Let Π be a set of distibutios o U, ad U. The P Q [P x Π] ( + 2 δ, whee δ if P Π D(P Q. Moeove, if Π is the closue of a ope set ad P : ag mi D(P Q, P Π the ( log x Q x Π] D(P Q. 4
Poof: Fo ay P T, we have by Theoem.6 that P [x C P] 2 D(P Q. Q Let T δ {P T D(P Q δ}. The, we have P [D(P x Q δ] x Q P T δ 2 D(P Q ( + 2 δ. We ow use this to pove Saov s theoem. Take δ if P Π D(P Q, so fo all P Π we have D(P Q δ. The we get x Q x Π] x Π T Q ] P [D(P x Q δ] ( + 2 δ Q as desied. Now let s pove the othe diectio. Sice Π is the closue of a ope set ad P Π, thee is a 0 such that we ca fid a sequece {P ( } 0 satisfyig P ( P ad P ( P Π fo each. The we have x Q x Π] x Π] x Q x Π T x Q ] [P x P (] P x Q ( + 2 D(P( Q Thus we get D(P ( Q log( + ( log x Q x Π] D(P Q + log( + which gives as claimed. P Q [P x Π] D(P Q, Note that the uppe boud o the pobability i Saov s theoem holds fo ay Π. Howeve, fo the lowe boud we eed some coditios o Π. This is ecessay sice if (fo example Π is a set of distibutios such that all pobabilities i all the distibutios ae iatioal, the P Q [P x Π] 0. I paticula, we caot get ay lowe boud o this pobability fo such a Π. 5