Lecture 10: October 30, 2017

Transcription

1 Information an Coing Theory Autumn 2017 Lecturer: Mahur Tulsiani Lecture 10: October 30, I-Projections an applications In this lecture, we will talk more about fining the istribution in a set Π that minimizes D(P Q) for a fixe istribution Q. We encountere this when iscussing Sanov s theorem an will not iscuss its properties in some etail. When Q is the uniform istribution on U. Then we also have, D(P Q) = log U H(P) Hence, in this case P is a istribution that maximizes entropy. In general, when the given information oes not uniquely etermine a istribution, we choose P that maximizes entropy. This can be thought of as picking P in the set of istributions Π, subject to the least amount of aitional assumptions. This is sometimes calle the Maximum Entropy Principle. In this lecture we will charcterize the istributions obtaine by minimizing Klivergence (or maximizing entropy). For close convex set Π, such a P is calle the I-projection of Q onto Π. Definition 1.1 Let Π be a close convex set of istributions over U. In aition, assume that Supp(Q) = U. Then Proj Π (Q) := arg min D(P Q) = P P Π Note that the assumption Supp(Q) = U above is without loss of generality since D(P Q) = for any P such that Supp(P) Supp(Q). Use the (strict) convexity of KL-ivergence to check the following. Exercise 1.2 For a close, convex set Π, the projection P = Proj Π (Q) exists an is unique. It is immeiate from efinition that if P Π, then D(P Q) D(P Q). In fact, P tells us more. It also tells us how far P is away from Q in KL-ivergence measure. Theorem 1.3 Let P = Proj Π (Q). Then, for all P Π, Supp(P) Supp(P ) D(P Q) D(P P ) + D(P Q) 1

2 Proof: Define P t = tp + (1 t)p, where t [0, 1]. By minimality of P, it is clear that D(P t Q) D(P Q) 0. By the mean value theorem, we also have that 0 1 t (D(P t Q) D(P Q)) t D(P t Q) t=t [0,t] Since t 0 as t 0, we get We now compute t D(P t Q). lim t 0 t D(P t Q) 0. Note that t D(P t Q) = Using these facts, we have t p t(a) log p t(a) q(a) + p t (a) t (log p t(a) log q(a)) t p t(a) = p(a) p (a) t log p t(a) = 1 1 ln 2 p t (a) (p(a) p (a)) t D(P t Q) = (p(a) p (a)) log p t(a) q(a) + 1 ln 2 (p(a) p (a)) = (p(a) p (a)) log p t(a) q(a) Here, note that if ( a) such that p(a) > 0 an p (a) = 0, then lim t 0 t D(P t Q), which contraicts the fact that t D(P t Q) 0. Hence, if p(a) > 0, then p (a) > 0 an therefore, Supp(P) Supp(P ). This proves the first part of the theorem. Now we evaluate t D(P t Q) at t = 0. t D(P t Q) t=0 = p(a) log p (a) q(a) p (a) log p (a) q(a) = p(a) log p (a) p(a) q(a) p(a) D(P Q) Hence, D(P Q) D(P P ) + D(P Q). = p(a) log p(a) q(a) p(a) log p(a) p (a) D(P Q) = D(P Q) D(P P ) D(P Q) 0 2

3 Consier the following example, which shows that the inequality can in fact be strict. Exercise 1.4 Let U = 0, 1 an Π = P : P(1) 1/2. Let Q be efine as 1 with prob. 3/4 Q = 0 with prob. 1/4 1. Show that P = 1 with prob. 1/2 0 with prob. 1/2 2. Show that D(P Q) > D(P P ) + D(P Q) for the above example. Next, we show how to compute an characterize I-projections for some special sets of istributions. 1.1 Linear families an I-projections Definition 1.5 For any given functions f 1, f 2,..., f k on U an α 1, α 2,..., α k R, the set L = P is calle a linear family of istributions. P(a) f i (a) = E [ f i (a)] = α i, i [k] a P We show that for linear families, the inequality prove above, is in fact tight. Moreover, the projection P lies in the interior of the polytope efining L. Lemma 1.6 Let L be a linear family given by L = P : p(a) f i (a) = α i, i [k] an P L Supp(P) = U. Let P = Proj L (Q). Then, for all P L 1. There exists β > 0 such that for t [ β, 0], P t = tp + (1 t)p L. 2. D(P Q) = D(P P ) + D(P Q) Then the I-Projection P of Q onto L satisfies the Pythagorean ientity D(P Q) = D(P P ) + D(P Q) 3

4 Proof: Recall that Supp(P) Supp(P ) an p t (a) = t p(a) + (1 t) p (a). Since the conitions efining L are linear, we have that for all t R an all i [k] p t (a) f i (a) = t p(a) f i (a) + (1 t) p (a) f i (a) = α i However, we may not have p t (a) 0 for all t < 0. We fin a β > 0 such that for t [ β, 0] p t (a) 0 t(p(a) p (a)) p (a) Note that above inequality clearly hols if p(a) p (a) < 0. Now choose β such that p β = (a) min a:p(a) p (a)>0 p(a) p (a) Notice that β > 0 since Supp(P ) P L Supp(P). The above implies that t D(P t Q) t=0 = 0 by the minimality of P, which in turn implies the equality D(P Q) = D(P P ) + D(P Q). The above can also be use to show that the I-projection onto L is of a special form. To escribe this, we efine the following family of istributions. Definition 1.7 Let Q be a given istribution. For any given functions g 1, g 2,..., g k on U an λ 1, λ 2,..., λ k R, the set ( ) k E Q = P : p(a) = c Q(a) exp λ i g i (a) i=1 is calle an exponential family of istributions. We will show that P = Proj L (Q) E Q ( f 1,..., f k ). We prove this for a linear family efine by a single constraint. The proof for families with multiple constraints is ientical. Let f : U R an let L be efine as L = P P(a) f (a) = E [ f (a)] = α a P The projection P is the optimal solution to the convex program minimize subject to D(P Q) P(a) f (a) = α P(a) = 1 P(a) 0 a U. 4

5 For λ 0, λ 1 R, we write the Lagrangian as ( ) ( ) Λ(P; λ 0, λ 1 ) = D(P Q) + λ 0 P(a) 1 a + λ 1 P(a) f (a) α a The problem above can be written in terms of the Lagrangian as inf Λ(P; λ 0, λ 1 ). From the lemma above, we know that P lies in the relative interior of the polytope efining L. Then, strong uality hols for the above program an we can write. inf Λ(P; λ 0, λ 1 ) = inf Λ(P; λ 0, λ 1 ). We now characterize the form of the optimal solution by consiering the secon (ual) program. For a given value of λ 0, λ 1, we can fin the optimal solution P by setting the erivative of Λ(P; λ 0, λ 1 ) with respect to P(a) to zero, for every a U. This gives ( p ) (a) log + 1 q(a) ln 2 + λ 0 + λ 1 f (a) = 0 Thus, we have for all a U p (a) = q(a) 2 λ 0 λ 1 f (a). The proof for linear families efine by multiple constraints is ientical. The above also shows that maximum entropy istributions subject to linear constraints, always belong to an exponential family. 5