Outline. Expectation Propagation in Practice. EP in a nutshell. Extensions to EP. EP algorithm Examples: Tom Minka CMU Statistics

Size: px

Start display at page:

Download "Outline. Expectation Propagation in Practice. EP in a nutshell. Extensions to EP. EP algorithm Examples: Tom Minka CMU Statistics"

Deirdre Richard
5 years ago
Views:

1 Eecion Progion in Prcice Tom Mink CMU Sisics Join work wih Yun Qi nd John Lffery EP lgorihm Emles: Ouline Trcking dynmic sysem Signl deecion in fding chnnels Documen modeling Bolzmnn mchines Eensions o EP Alernives o momen-mching Fcors rised o owers Skiing fcors EP in nushell Aroime funcion by simler one: ( = f ( q ( = ( Where ech f ( lives in rmeric, eonenil fmily (e.g. Gussin Fcors f ( cn be condiionl disribuions in Byesin nework f

2 EP lgorihm Iere he fied-oin equions: \ f ( = rg min D( f ( q ( f ( q \ where \ q ( = f ( b \ q ( secifies where he roimion needs o be good Coordined locl roimions f b ( (Looy Belief rogion Secilize o fcorized roimions: f ( = fi ( i messges i Minimize KL-divergence = mch \ mrginls of ( ( (rilly fcorized nd f q ( \ f q ( (fully fcorized send messges EP versus BP EP roimion cn be in resriced fmily, e.g. Gussin EP roimion does no hve o be fcorized EP lies o mny more roblems e.g. miure of discree/coninuous vribles EP versus Mone Crlo Mone Crlo is generl bu eensive A sledgehmmer EP elois underlying simliciy of he roblem (if i eiss Mone Crlo is sill needed for comle roblems (e.g. lrge isoled eks Trick is o know wh roblem you hve

3 Emle: Trcking Byesin nework Guess he osiion of n objec given noisy mesuremens y 2 2 y y2 y y 3 4 y y y 4 e.g. + = (rndom wlk Objec y = + noise wn disribuion of s given y s Terminology Filering: oserior for ls se only Smoohing: oserior for middle ses On-line: old d is discrded (fied memory Off-line: old d is re-used (unbounded memory Klmn filering / Belief rogion Predicion: y Smoohing: d ( < = ( ( < Mesuremen: ( y, y ( y ( y < < ( y, y> ( y ( + ( + y +, y> + d + y

4 (, y = ( Aroimion y ( ( ( > y Aroimion q( = ( o ( ( + = (forwrd msg(observion(bckwrd msg q( = ( o ( ( ( ( > Fcorized nd Gussin in o EP equions re ecly he redicion, mesuremen, nd smoohing equions for he Klmn filer - bu only reserve firs nd second momens Consider cse of liner dynmics EP in dynmic sysems Loo =,, T (filering Predicion se Aroime mesuremen se Loo = T,, (smoohing Smoohing se Divide ou he roime mesuremen Re-roime he mesuremen Loo =,, T (re-filering Predicion nd mesuremen using revious ro Generlizion Insed of mching momens, cn use ny mehod for roime filering E.g. Eended Klmn filer, sisicl linerizion, unscened filer, ec. All cn be inerreed s finding liner/gussin ro o originl erms

5 Inerreing EP Afer more informion is vilble, reroime individul erms for beer resuls Oiml filering is no longer on-line y Emle: Poisson rcking is n ineger vlued Poisson vrie wih men e( Poisson rcking model ( N(0,00 ( N(,0.0 ( y = e( y e / y! Aroime mesuremen se ( y ( y < is no Gussin Momens of no nlyic Two roches: Guss-Hermie qudrure for momens Sisicl linerizion insed of momenmching Boh work well

6 Poserior for he ls se

7 EP for signl deecion Wireless communicion roblem Trnsmied signl = sin( ω + φ (, φ vry o encode ech symbol In comle numbers: e iφ Im Binry symbols, Gussin noise Symbols re nd (in comle lne Received signl = sin( ω +φ + noise Recovered ˆ ˆ φ φ e = e + noise = Oiml deecion is esy y φ y Re 0 s s Fding chnnel Chnnel sysemiclly chnges mliude nd hse: y = s + noise chnges over ime Differenil deecion Use ls mesuremen o esime se Binry symbols only No smoohing of se = noisy y s y y 0 s y

Byesin nework 2 3 4 On-line imlemenion Iere over he ls δ mesuremens Previous mesuremens c s rior y y2 y y 3 4 s s2 s3 s4 Resuls comrble o ricle filering,

8 Byesin nework On-line imlemenion Iere over he ls δ mesuremens Previous mesuremens c s rior y y2 y y 3 4 s s2 s3 s4 Resuls comrble o ricle filering, bu much fser Symbols cn lso be correled (e.g. error-correcing code Dynmics re lerned from rining d (ll s Documen modeling Wn o clssify documens by semnic conen Word order generlly found o be irrelevn Word choice is wh mers Model ech documen s bg of words Reduces o modeling correlions beween word robbiliies

9 Generive sec model (Hofmnn 999; Blei, Ng, & Jordn 200 Ech documen mies secs in differen roorions Generive sec model Asec Asec 2 Asec Asec 2 λ λ ( word w λ λ λ2 λ 2 λ λ 3 3 Documen Mulinomil smling λ Dirichle( α,..., α ( J Inference: Two sks Given secs nd documen i, wh is (oserior for? Lerning: λ i Given some documens, wh re (mimum likelihood secs? Aroimion Likelihood is comosed of erms of form nw nw ( λ = ( w = ( λ ( w w Wn Dirichle roimion: (λ w = λ β w n w

10 EP wih owers These erms seem oo comliced for EP Cn mch momens if n w =, bu no for lrge n w Soluion: mch momens of one occurrnce ime Redefine wh re he erms Momen mch: w ( λ q EP wih owers \ w ( λ ( λ q Cone funcion: ll bu one occurrence \ w nw nw' q ( λ = ( λ ( λ w' w' w Fied oin equions for w w β \ w ( λ EP wih skiing Cone fcn migh no be roer densiy Soluion: ski his erm (kee old roimion In ler ierions, cone becomes roer Anoher roblem Minimizing KL-divergence of Dirichle is eensive Requires ierion Mch (men,vrince insed Closed-form

11 w One erm ( λ = ( λ0.4 + ( λ0.3 Ten word documen VB = Vriionl Byes (Blei e l Generl behvior For long documens, VB recovers correc men, bu no correc vrince of Dissrous for lerning No Occm fcor Ges worse wih more documens No symoic slvion EP ges correc vrince, lerns roerly λ Lerning in robbiliy simle 00 docs, Lengh 0

12 Lerning in robbiliy simle Lerning in robbiliy simle 0 docs, Lengh 0 0 docs, Lengh 0 Lerning in robbiliy simle Bolzmnn mchines 0 docs, Lengh Join disribuion is roduc of ir oenils: ( = f ( q ( = f ( Wn o roime by simler disribuion

13 Aroimions Aroiming n edge by ree Ech oenil in is rojeced ono he ree-srucure of q f (, 2 f 4 (, 4 f 24 ( 2, f ( 4 f 34 (, 3 4 BP EP Correlions re no los, bu rojeced ono he ree Fied-oin equions 5-node comlee grhs, 0 rils Mch single nd irwise mrginls of Mehod FLOPS Error Ec nd Reduces o ec inference on single loos Use cuse condiioning TreeEP BP/double-loo GBP 3, , ,

14 88 grids, 0 rils TreeEP versus BP Mehod Ec TreeEP FLOPS 30, ,000 Error TreeEP lwys more ccure hn BP, ofen fser GBP slower hn BP, no lwys more ccure BP/double-loo 5,500, TreeEP converges more ofen hn BP nd GBP GBP 7,500, Conclusions End EP lgorihms eceed se-of-r in severl domins Mny more ooruniies ou here EP is sensiive o choice of roimion does no give guidnce in choosing i (e.g. ree srucure error bound? Eonenil fmily consrin cn be limiing miures?

Particle Filtering. CSE 473: Artificial Intelligence Particle Filters. Representation: Particles. Particle Filtering: Elapse Time

Particle Filtering. CSE 473: Artificial Intelligence Particle Filters. Representation: Particles. Particle Filtering: Elapse Time CSE 473: Arificil Inelligence Pricle Filers Dieer Fo Universiy of Wshingon [Mos slides were creed by Dn Klein nd Pieer Abbeel for CS88 Inro o AI UC Berkeley. All CS88 merils re vilble h://i.berkeley.ed.]