Lecture 8: September 26

Transcription

1 10-704: Information Processing and Learning Fall 2016 Lectrer: Aarti Singh Lectre 8: September 26 Note: These notes are based on scribed notes from Spring15 offering of this corse. LaTeX template cortesy of UC Berkeley EECS dept. Disclaimer: These notes have not been sbjected to the sal scrtiny reserved for formal pblications. They may be distribted otside this class only with the permission of the Instrctor. 8.1 Review Maximm Entropy and Information Projection Last time we discssed that the problem of finding the maximm entropy distribtion constrained to lie in a sbset Q P is essentially eqivalent to finding the information projection of the niform distribtion onto Q, i.e. the distribtion in Q that is closest to niform in KL sense 1 max H(p) = min D(p ) p Q p Q If the set of constraints in Q are linear in p, i.e. of the form E p [f j (X)] eqal to or bonded by some constant, then the maximm entropy distribtion belongs to the exponential family: p (x) = exp( j jf j (x)) Z where the Lagrange parameters = { j } are chosen so that p meets the constraints. The information projection can be defined more generally with respect to any given base distribtion p 0 (x) (instead of niform): min D(p p 0) p Q If the set of constraints in Q are linear in p, i.e. of the form E p [f j (X)] eqal to or bonded by some constant, then the information projection distribtion belongs to the Gibbs family: p (x) = p 0 (x) exp( j jf j (x)) Z where the normalizing constant is the partition fnction: Z = x p 0 (x)e j jfj(x). 1 Here the niform distribtion is defined sch that all distribtions in Q are absoltely continos with respect to it. 8-1

2 8-2 Lectre 8: September Maximm Entropy Dality with Maximm Likelihood Estimation So far, we haven t talked abot data in the discssion of maximm entropy. Often the constraints on the distribtion are actally specified sing the data. For example, when we seek Maximm likelihood model in the exponential (Gibbs) family then we are essentially seeking the Maximm Entropy distribtion (Information Projection) given empirical constraints based on data. We will show this connection next. Consider the maximm likelihood model given data X 1,..., X n p ML(x) = argmax p (X i ) p 1 = argmin log p p (X i ) [ ] 1 = argmin Eˆp log p p (X) [ = argmin Eˆp log ˆp(X) p p (X) = argmin p D(ˆp p ) + H(ˆp) = argmin p D(ˆp p ), ] + Eˆp [ log ] 1 ˆp(X) since the soltion is eqivalent withot H(ˆp). Note that the final soltion is not the same as the projection. The following theorem relates maximm likelihood estimation in exponential family with base distribtion p 0 to information projection of p 0 onto a set of distribtions with constraints specified by the empirical mean of the sfficient statistics: Theorem 8.1 Dality Theorem Let α j = Eˆp [f j (X)], then p ML(x) = argmin p D(ˆp p ) = argmin p P E p[f j(x)]=α i D(p p 0 ) = pip (x) The theorem states that the distribtion belonging to the exponential family (with sfficient statistics f j (x) and base distribtion p 0 (x)) whose parameters maximize the likelihood of data, is the same as the information projection of p 0 (x) on to a set of distribtions with linear eqality constraints (specified by f j (x)) that are given by data. Proof: Since we know the information projection lies in the exponential family, all we need to show is that the s in the maximm likelihood model satisfy the empirical linear constraints. So lets analyze the s that achieve the maximm likelihood of the data. Recall that Z = x = argmax = argmax p 0 (x) exp[ j j f j (x)] p (X i ) = argmax [log p 0 (X i ) + j and log p (X i ) j f j (X i ) log Z ].

3 Lectre 8: September Taking derivative with respect to 1,, m, of the log likelihood fnction, we get that log p (X i ) = f j (X i ) n log Z j j = f j (X i ) n Z Z j = f j (X i ) n p 0 (x)f j (x) exp[ k f k (x)] Z x k = f j (X i ) n [ p 0(x) exp[ k kf k (x)] ]f j (x) Z x = p (x)f j (x) f j (X i ) n x At the maximizing ML the derivative is eqal to 0, so we get: = p (x)f ML j(x) = 1 f j (X i ) n x = E p ML [f j (X)] = Eˆp [f j (X)] 8.3 Maximm Entropy Generalization and Dality with reglarized Maximm Likelihood We can consider a generalization of the maximm entropy (information projection) problem [DPS08] min D(p p 0) + U(E p [f]), p P where U(E p [f]) is a reglarizer and f = [f 1 (X)... f m (X)]. Here are three example reglarizers: Example 8.2 Standard Maximm entropy/information projection is obtained with U(E p [f]) = 1(E p [f] = Eˆp [f]) Notice that for the eqivalence to hold the indicator fnction is defined so that 1 A is 0 if A is tre and otherwise. This penalty reqires the tre constraints to match the empirical constraints exactly. Example 8.3 L1 Norm Reglarizer U(E p [f]) = 1( E p [f j ] Eˆp [f j ] β j ) j Here also we se the same definition of indicator fnction as above. This penalty reqires the tre constraints to match the empirical constraints in an l 1 sense.

4 8-4 Lectre 8: September 26 Example 8.4 L2 Norm Reglarizer U(E p [f]) = E p[f] Eˆp [f] 2 2α This penalty reqires the tre constraints to match the empirical constraints in l 2 sense. To find the soltion to the generalized MaxEnt problem, we cold consider taking the derivative of the reglarized objective with respect to p, however notice that some of the reglarizations are not differentiable. So far, we have mostly ignored sch isses, assming differentiability. Bt lets consider a more formal treatment via Fenchel dality (instead of Lagrangian dality) that allows s to handle convex bt nondifferentiable fnctions. First, lets define the convex conjgate or Fenchel dal of a fnction ψ(p) as ψ () = sp[ p ψ(p)]. p It is essentially the largest difference between a line throgh the origin with slope and the graph of the fnction. If the fnction is differentiable, the largest difference happens at a point p where the gradient of the fnction ψ (p ) =. See the image below, for example. For a convex fnction, the conjgate is jst a characterization of the fnction in terms of (intercept vales of) its spporting hyperplanes corresponding to different slopes. The following theorem relates a primal optimization problem of closed, proper and convex fnction(s) to the dal optimization problem specified in terms of convex conjgate of the fnction(s). Recall that a fnction is proper if it is not infinite everywhere. Definition 8.5 Fenchel s Dality Let ψ, ϕ be closed, proper, and convex, and A is any matrix. Fenchel s Dality states that inf p ψ(p) + ϕ(ap) = sp ψ (A T ) ϕ ( ) Retrning to the previos maximm entropy generalization problem. We can consider p(x) p x as a vector, which may be an infinite-dimensional object. Lets define a matrix F with entries F jx = f j (x). Then, F p = x f j(x)p(x) = E[f j (X)] and we have the primal min D(p p 0) + U(F p) p P

5 Lectre 8: September where U will be closed, convex, and proper. Let ψ(p) = D(p p 0 ) if p P and otherwise, which is closed, proper and convex in p. To apply Fenchel dality, we first derive the conjgate of ψ(p) as ψ () = ln( x p 0 (x)e x ). For closed, convex and proper fnctions, the conjgate of a conjgate is the fnction itself, hence we instead evalate ψ (p) = sp[ p ln( x p 0 (x)e x )] Taking derivative with respect to x and setting it eqal to 0, we get that optimal x satisfies p x = p 0(x)e x. Plgging this vale of we get: x p0(x)ex ψ (p) = p ln( x p 0 (x)e x ) = x p x ( x ln( x p 0 (x)e x )) = x p x ln e x x p 0(x)e x = x p x ln p x p 0 (x) = D(p p 0) = ψ(p) So, sing Fenchel dality we have the dal problem sp[ ψ (F ) U ( )] = sp[ ln p 0 (x)e (F ) x ] U ( ) = sp[ ln p 0 (x)e j jfj(x) U ( ) = sp[ ln Z U ( )] We will show that this dal problem is essentially finding the reglarized Maximm Likelihood model nder exponential family with base distribtion p 0 (x). Before we can do that, we need one more notion - that of a shifted reglarizer Shifted reglarization Define a shifted reglarizer with respect to any distribtion t as follows Then the dal of the shifted reglarizer is U t () = U(E t [f] ), Ut () = sp[ U t ()] = sp[ U(E t [f] )] = sp [ E t [f] U( )] = E t [f] + U ( )

6 8-6 Lectre 8: September Dal as Reglarized Maximm Likelihood Let Q() = ln Z U ( ), then Q() = ln Z U ( ) = ln Z U t () + E t [f] = E t [ln p 0 ] + E t [ln p 0 + f ln Z ] U t () = E t [ln p 0 ] + E t [ln p 0 exp( j jf j (x) ] Ut () = L t (0) L t () U t (), where L t () := E t [ln p ] is the loss of the exponential family model p (x) = p0 exp( j jfj(x) Z with respect to distribtion t. If t = ˆp, then this it jst the negative log likelihood of the data nder the model p. Therefore, the dal problem is sp Q() min L t () + Ut (). and when t = ˆp, this is jst reglarized Maximm likelihood estimation. We then look at some examples of how the reglarization on maximm entropy/information projection transforms to reglarization term in maximm likelihood soltion. Examples: 1. U(E p [f]) = I(E p [f] = E p [f]). Then, Z U p (E p [f]) = 1(E p [f] = 0). (8.1) U p () = sp[ 1( = 0)] = 0. (8.2) The last step follow since 1( = 0) is infinity everywhere except when = 0. The problem ths gets back to the basic maximm entropy dality with nreglarized maximm likelihood. 2. U(E p [f]) = I( E p [f j ] E p [f j ] β j, j). Then, U p (E p [f]) = 1( E p [f j ] β j, j). (8.3) U p () = sp[ 1( j β j )] = j β j j, (8.4) The last step follow since 1( j β j ) is infinity everywhere except when j β j. Ths the expression is maximized when j = sign( j )β j. This corresponds to maximm likelihood with l 1 reglarization. 3. U(E p [f]) = ( E p [f] E p [f] 2 2/2α. Then, U p (E p [f]) = E p [f] 2 2/2α. (8.5) U p () = sp[ /2α] = α 2 2/2, (8.6) The last step follows since the maximizing = α (in this case, penalty is differentiable - simply take derivative wrt and set to zero). This corresponds to maximm likelihood with l 2 2 reglarization.

7 Lectre 8: September References [DPS08] M. Ddik, S.J. Phillips and R. Schapire, Maximm Entropy Density Estimation with Generalized Reglarization and an Application to Species Distribtion Modeling, Jornal of Machine Learning Research 8, 2007, pp