Information Geometry and Natural Gradients

Transcription

1 Informaion Geomery and Naural Gradiens Nahan Raliff Nov 9, 203 Absrac This documen reviews some of he basic conceps behind naural gradiens. We sar by inroducing basic informaion heoreic conceps such as opimal codes, enropy, and he KL-divergence. We hen demonsrae how he choice of meric on perurbaions can significanly affec he performance of gradien descen algorihms. Wihin ha discussion, we review he Mehod of Lagrange Mulipliers for equaliy consrained opimizaion as well as inuiive inerpreaions of he acion of posiive definie marices and heir inverses and how ha relaes o heir role in generalized gradien descen updaes. Finally, we review how he secondorder Taylor expansion of he KL-divergence beween a disribuion and a slighly perurbed version of ha disribuion leads o he noion of he Fisher Informaion as a naural meric on he manifold of probabiliy disribuions. Informaion Theory and he KL-divergence Le p(x be some probabiliy disribuion over a discree se of objecs X. If we wan o encode hose objec and ransmi hem over a wire, hen here are rade-offs we need o consider. Firs, wha do we mean when we say we wan o encode he objecs? If I m on one side of a wire, and I m able o ransmi srings of bis (e.g across he line, hen I can ell he receiver in advance I m going o, for insance, use he sring 0 o represen objec x X, or he sring 0 o represen objec x 2 X, ec. If boh he sender and receiver agree on a paricular convenion, hen efficien communicaion is possible. Tha convenion for assigning srings o objecs, is wha we refer o as a code. Noe ha he se X may conain infiniely many objecs, so using srings of differen lenghs is imporan for boh feasibiliy of he code and, as we ll discuss below, for efficiency. On he oher side, once I ve sen a se of bi srings across he wire, if he receiver knows he code, she can hen decode he sring on he oher end. For insance, if she receives he sring 0, hen since we ve agreed in advance ha 0 represens objec x, hen she can confidenly say ha he firs objec

2 o come off he wire is x. A quesion cenral o Informaion Theory (Cover & Thomas, 2006 is, Wha code is mos efficien. I.e. how do we assign bi srings o objecs in a way ha minimizes he expeced lengh of he ransmied sring? For us o make sense of his quesion, we need o know somehing abou he disribuion of objecs p(x being sen. Formalizing ha disribuion gives he phrase he expeced lengh of he ransmied sring an exac meaning: E p [l c (x] = x p(xl c (x, where l c (x denoes he lengh of he bi sring assigned o x under code c. Since some bi srings are shorer han ohers, inuiively i behooves us o choose a code ha reserves he shores srings for frequenly sen objecs, and assigns longer srings o infrequenly sen objecs. I urns ou ha if we lierally do ha by ordering he srings by lengh from shores o longes and ordering he objecs by send-probabiliy from mos probable o leas probable, hen he code ha simply maches hose up one-o-one (wih he mos probable objec geing he shores sring, he second mos probable objec geing he second shores sring, ec. is he opimal code! Moreover, one can show ha he approximae number of bis assigned o objec x under his opimal code c is l c (x = log = log p(x. p(x The expeced number of bis under he opimal code is, herefore, E p [l c (x] = p(x log p(x. x }{{} H[p] This quaniy, which we denoe H[p], is known as he enropy of he disribuion. The enropy represens he degree uncerainy in he disribuion (in unis of bis. If we know exacly wih probabiliy ha a specific objec x will always be wha s sen across he wire, hen he opimal code is o send ha one objec wih he shores bi sring. The shores bi sring has no bis a all (excep a consan number of boundary bis ha we ignore in hese heoreical analyses since we don have o disinguish beween anyhing on he receiving end. Our mahemaical bi-lengh esimae reflecs ha inuiion since l c ( x = log p( x = 0 when p( x =. An enropy of 0 is he smalles i can be when we consider only discree disribuions. On he oher hand, when we really don know wha he user s going o send over he wire (i.e. p(x is flaer, hen here s nohing we can do bu assign some relaively long bi srings o objecs ha may acually be raher frequen. The expeced number of bis for even he opimal code (given by he enropy H(x, in his case, will be a much larger number, significanly away 2

3 from zero. In general, he more uncerain he disribuion, he more bis i akes o opimally encode wha we wan o ransmi. H(x is herefore a measure of uncerainy in he disribuion. Now, suppose we misakenly use he wrong disribuion q(x o creae he code for a se of objecs X ha acually will have disribuion p(x when i comes ime o ransmi i. By ha we mean we assign objecs ha have high probabiliy under q he shores bi srings raher han hose ha have high probabiliy under p. Since he ranking of objec is increasingly wrong he more p and q differ, he more subopimal our code is going o be. We can explicily measure his subopimaliy by calculaing he expeced difference beween he number of bis we ge from he opimal code and he number of bis we end up wih under he subopimal code. Essenially we ask, How many bis did we wase by encoding he objecs using disribuion q raher han he correc disribuion p? Mahemaically, for p, each objec x is assigned a sring wih log p(x bis, where as for q he objec is assigned a sring wih log q(x bis. The expeced difference beween hose wo a ransmission ime is [ E p log q(x log ] = p(x x p(x log p(x q(x }{{} KL(p q Tha las expression is wha we ypically erm he KL-divergence beween p and q. I represens he degree of subopimaliy in erroneously encoding a se of objecs using disribuion q when you should have used disribuion p. Since we can never do beer han encoding he disribuion in erms of p, which would assign a sring of lengh l c (x = log p(x bis o each objec x, his divergence measure is never negaive. And more han ha, i s sricly posiive for all q p and exacly zero for q = p. The KL-divergence is used hroughou Informaion Theory and Machine Learning (Bishop, 2007, and is he saring poin for discussions abou Informaion Geomery as we discuss in Secion 3. 2 Gradien descen updae rules This secion elaboraes more on he derivaion of gradien descen updae rules under varying merics. Secion 2. gives an inuiive and more easily undersood derivaion, and Secion 2.2 grounds he resul using he Theory of Lagrange Mulipliers. The firs secion is more imporan; he second is here mainly for compleeness for hose who are ineresed. 2. A basic derivaion Suppose we re opimizing a funcion f(θ. Then we can derive a generalized gradien descen rule for updaing a poin θ by wriing down a consrained. 3

4 opimizaion problem ha says we wan o minimize a firs-order approximaion o our objecive around θ subjec o he consrain ha our sep δθ = θ θ, wha I ll someimes call he perurbaion, shouldn be oo big: θ + =arg min θ f(θ + f(θ T (θ θ ( s.. 2 θ θ 2 A = ɛ 2. The generalized A-weighed norm is defined as x 2 A = xt Ax; we discuss is properies in relaion o how i affecs he derived gradien updae below. Technically, he consrain should really be ha he perurbaion δθ = θ θ is less han or equal o ɛ (since we jus wan he perurbaion o be small, no necessarily an exac fixed size ɛ, bu for our purposes i s sufficien o deal sricly wih equaliy. We can solve his problem by using Lagrange mulipliers. If you ve never seen Lagrange mulipliers before, suffice i o say ha he problem can equivalenly be rewrien as an unconsrained problem of he form θ + =arg min θ f(θ + f(θ T (θ θ + λ 2 θ θ 2 A, (2 where here is some relaionship beween λ and ɛ ha we don really care abou because we ypically need o une hose anyway for a given problem. Secion 2.2 deails he Mehod of Lagrange Mulipliers for hose who are ineresed. Solving Equaion 2 by seing is gradien o zero gives θ + = θ λ A f(θ. (3 In oher words, ha opimizaion enirely characerizes gradien descen! The weigh marix A ha defines he norm on perurbaions o θ plays a significan role in defining he search direcion. A mus be posiive definie o define a valid norm, and as such we can hink of he operaion of such a marix as a sreching or squishing along paricular dimensions of some orhogonal basis in he space. 2 Inuiively, if A, hrough i s ineracion wih he norm δθ 2 A = δθt Aδθ, penalizes δθ srongly along cerain direcions and less along If we have inequaliy consrains here, he opimal poin is eiher going o be on he surface of ha consrain (i.e. i ll hold a equaliy, or he consrain is irrelevan because he minimizer of he linearizaion is already in he inerior. If i s he laer case, since he approximaion is linear, i mus be ha f(θ = 0, in which case, we ve already arrived a he minimum (or a leas a fixed poin and he overall opimizaion problem is solved (or, in he case of a fixed poin, furher gradien seps won help. In all relevan cases, he gradien is never exacly zero, so we can safely assume ha he sep will be exacly of lengh ɛ in his derivaion. 2 Mahemaically, his is represened explicily in he diagonalizaion of he marix A = UDU T, where U is an orhogonal marix whose columns consis of he n (muually orhonormal Eigenvecors of A, and D is a diagonal marix whose diagonal enries are he corresponding Eigenvalues. This diagonalizaion always exiss wih sricly posiive Eigenvalues because he marix is posiive definie. Anoher way of wriing ha, which makes he above inerpreaion explici is A = n i= λ ie i e T i, where λ i and e i are he ih Eigenvalue and 4

5 oher direcions, he acion of A on he gradien in Equaion 3 does exacly he opposie. I shrinks componens along he direcions of heavy penalizaion so ha he updae has lile effec in hose direcion. Or if A doesn penalize he perurbaion much along a paricular dimension, hen A will eiher keep hose dimensions of he gradien unouched or poenially even amplify hose direcions o emphasize heir effec during he updae. Secion 3 below shows ha we can use he KL-divergence beween a disribuion and a perurbed version of ha disribuion in he above framework by aking is second-order Taylor expansion. 2.2 Full derivaion in erms of Lagrange Mulipliers This secion briefly reviews how one migh solve Equaion direcly using he Mehod of Lagrange Mulipliers. I won go ino deail abou why his procedure works, bu a leas I ll lay ou he procedure iself which is generally applicable in a number of seings where analyic equaliy consrained opimizaion is needed. The procedure for handling equaliy consrains in an opimizaion, such he consrain 2 θ θ 2 A = ɛ2 in Equaion, is o place hem up ino he objecive wih a paricular unknown weigh λ o form wha s called a Lagrangian. In his case, we ge ( L(x, λ = f(θ + f(θ T (θ θ + λ 2 θ θ 2 A ɛ 2. (4 The Theory of Lagrange Mulipliers ells us ha he soluion o he consrained opimizaion problem is necessarily described by he condiions { x L(x, λ = 0 λ L(x, λ = 0. A full discussion of hese opimaliy condiions is beyond he scope of hese noes, bu a reasonable reference for a firs pass is given in he Wikipedia page on Lagrange Mulipliers. 3 In his case, using he Lagrangian in Equaion 4 we ge he following sysem of wo equaions { f(θ + λ(θ θ = 0 θ θ 2 A ɛ2 = 0. The second equaion simply resaes he original consrain, bu he wo equaions in combinaion allow us o firs explicily solve for λ in erms of ɛ λ = ɛ f(θ A Eigenvecor, respecively. Each marix e i e T i is a projecion operaor ha projecs a vecor x ono he Eigenvecor e i (is acion is (e i e T i x = (et i xe i. Scaling ha erm by λ i, hen sreches he componen of x in he direcion of e i by a facor of λ i. The marix A herefore sreches or squishes a vecor along he orhogonal direcions e i by he facors λ i. 3 hp://en.wikipedia.org/wiki/lagrange muliplier 5

6 and hen explicily solve he problem by solving he firs equaion for θ in erms of λ and hen plugging in our expression for λ in erms of ɛ: θ = θ ( λ A f(θ = θ ɛa f(θ f(θ A = θ ɛa f(θ. Relaing his back o he above expression in Equaion 3, and noing ha for a given λ, he condiion x L(x, λ = 0 compleely characerizes he soluion o he penalized problem we described in Equaion 2, we can say ha he consrained problem of Equaion and he unconsrained (penalized problem of Equaion 2 are equivalen under he relaionship λ = f(θ /ɛ. Inuiively, choosing a ball consrain enforces ha he lengh of he sep is precisely ɛ, whereas choosing a paricular weigh λ in Equaion 2 defines a sep ha depends on he lengh of he gradien f(θ (unless, of course, we choose λ o be λ = ɛ f(θ A. 3 KL-divergence beween perurbed disribuions Le p(x; θ be some family of probabiliy disribuions over x parameerized by a vecor of real numbers θ. We re ineresed in knowing how much he disribuion changes when we perurb he parameer vecor from a fixed θ o some new value θ + δθ. As a measure of change in probabiliy disribuion, we can use he KL-divergence measure we inroduced in Secion. Specifically, we wan o measure KL( p(x; θ p(x; θ + δθ, bu we wan o wrie i in a form amenable o he gradien-based updae formulaion presened in Secion 2. We can do his by aking i s second-order Taylor expansion around θ. During he derivaion, we ll find ha a lo of erms in he expansion disappear leaving us wih a very simple expression ha s perfec for our purposes. Looking firs a he full KL-divergence, we see ha he erm we wan o expand using a second-order Taylor approximaion is log p(x; θ + δθ: KL( p(x; θ p(x; θ + δθ ( p(x; θ = p(x; θ log dx p(x; θ + δθ = p(x; θ log p(x; θ dx p(x; θ log p(x; θ + δθdx. The second-order Taylor series expansion generically is f(θ f(θ + f(θ T δθ + 2 δθt ( 2 f(θ δθ, where θ = θ + δθ, or equivalenly δθ = θ θ. Applying ha expansion o he perinen erm in he KL-divergence expression, we ge ( T p(x; θ log p(x; θ + δθ log p(x; θ + δθ + ( p(x; θ 2 δθt 2 log p(x; θ δθ. 6

7 Plugging his second-order Taylor expansion back ino he above expression for he KL-divergence gives KL( p(x; θ p(x; θ + δθ (5 p(x; θ log p(x; θ dx ( ( T p(x; θ p(x; θ log p(x; θ + δθ + ( p(x; θ 2 δθt 2 log p(x; θ δθ dx = p(x; θ log p(x; θ ( T p(x; θ dx p(x; θ dx δθ }{{}}{{} =0 =0 ( 2 δθt p(x; θ 2 log p(x; θ δθ. As indicaed, he firs erm, since i s he KL-divergence beween a disribuion and iself, is zero, and he second erm, because of he relaion p(x; θ dx = p(x; θ dx = = 0 is also zero. Probably he mos challenging par of he derivaion, depending on your fluency in vecor calculus, is simply calculaing he Hessian of log p(x; θ. I s someimes easies o wrie ou complicaed vecor calculus compuaions ou in erms of heir individual parial derivaives firs and hen only laer recognize how he resuling expression can be more compacly wrien in marix form wihou direcly resoring o vecor calculus formulas ha can be easy o mess up. We ll use ha echnique here o derive he Hessian: [ ] 2 θ (i θ (j log p(x; θ = θ (i p(x; θ p(x; θ θ (j p(x; θ 2 θ = (i = p(x; θ θ (i θ (j 2 θ (j p(x; θ θ (i p(x; θ 2 p(x; θ p(x; θ θ (i θ (j p(x; θ p(x; θ p(x; θ p(x; θ. p(x; θ θ (j The firs erm is an elemen of he Hessian 2 p(x; θ weighed by a facor p(x;θ, and he second erm is an elemen of he ouer produc beween log p(x; θ and iself. So in marix form, his becomes 2 log p(x; θ = p(x; θ 2 p(x; θ log p(x; θ log p(x; θ T. 7

8 Finally, plugging his expression for he Hessian back ino our laes KL-divergence expression in Equaion 5 we ge KL( p(x; θ p(x; θ + δθ ( 2 δθt p(x; θ 2 log p(x; θ dx δθ = ( 2 δθt 2 p(x; θ dx δθ }{{} =0 + ( [ ] 2 δθt p(x; θ log p(x; θ log p(x; θ T dx δθ. }{{} G(θ As indicaed, he firs erm in his expression is zero for he same reason as before: 2 p(x; θ dx = 2 p(x; θ dx = 2 = 0. The cenral marix here G(θ is known as he Fisher Informaion marix and can has been horoughly sudied wihin he field of Informaion Geomery (Amari & Nagaoka, 2000 as he naural Riemannian srucure on a manifold of probabiliy disribuions. As such i defines a naural norm on perurbaions o probabiliy disribuions, which was our original moivaion for examining he second-order Taylor expansion of he KL-divergence in he firs place. Given his analysis, seing A = G(θ in he above generalized gradien updae expression of Equaion 3 gives an updae rule ha explicily measures he size of perurbaions o he parameers based on he exen o which hey change he probabiliy disribuion as measured by KL-divergence. In full, he naural gradien updae for opimizing a funcion f(θ is Here we use η in place of λ change a each ieraion. References θ + = θ η G(θ f(θ. o emphasize ha he sep size parameer may Amari, S. and Nagaoka, H. Mehods of Informaion Geomery, volume 9 of Translaions of Mahemaical monographs. Oxford Universiy Press, Bishop, Chrisopher M. Paern Recogniion and Machine Learning. Springer, Cover, Thomas M. and Thomas, Joy A. Elemens of informaion heory. Wiley- Inerscience, 2nd ediion ediion,