The need for complex dynamic models

Transcription

1 School of Compuer Sciece Probabilisic Graphical Models Complex Graphical Models Eric ig Lecure 5, Ocober 28, 2009 Readig: Eric CMU, The eed for complex dyamic models Complex dyamic sysems: No-lieariy No-Gaussiaiy Muli-modaliy Limiaio of LDS xˆ ˆ + + = x+ + K+ y+ Cxˆ + P + + = P + KCP + defies oly lieariy evolvig, uimodal, ad Gaussia belief saes Kalma filer will predic he locaio of he bird usig a sigle Gaussia ceered o he obsacle. more realisic model allows for he bird s evasive acio, predicig ha i will fly o oe side or he oher. Eric CMU,

2 Represeig complex dyamic processes The problem wih HMMs Suppose we wa o rack he sae e.g., he posiio of D objecs i a image sequece. Le each objec be i K possible saes. The =,, D ca have K D possible values. Iferece akes ime ad space. P - eed parameers o specify. Eric CMU, Dyamic Bayesia Nework DBN represes he sae of he world a ime usig a se of radom variables,,, D facored/ disribued represeaio. DBN represes P - i a compac way usig a parameerized graph. DBN may have expoeially fewer parameers ha is correspodig HMM. Iferece i a DBN may be expoeially faser ha i he correspodig HMM. Eric CMU,

3 DBNs are a kid of graphical model I a graphical model, odes represe radom variables, ad lack of arcs represes codiioal idepedecies. DBNs are Bayes es for dyamic processes. Iformally, a arc from i o + j meas i "causes" j. Ca "resolve" cycles i a "saic" BN Eric CMU, road map o complex dyamic models discree Y discree Y coiuous Y discree coiuous coiuous Mixure model e.g., mixure of muliomials Mixure model e.g., mixure of Gaussias Facor aalysis y y 2 y 3 y N y y 2 y 3 y N y y 2 y 3 y N x x 2 x 3 x N x x 2 x 3 x N x x 2 x 3 HMM HMM for discree sequeial daa, e.g., ex for coiuous sequeial daa, e.g., speech sigal x N Sae space model S S 2 S 3 S N S S 2 S 3 S N y y 2 y 3 y N y y 2 y 3 y N Facorial HMM y k y k2 y k3 y kn Swichig SSM y k y k2 y k3 y kn x x 2 x 3 x N x x 2 x 3 x N Eric CMU,

4 HMM varias represeed as DBNs The same code sadard forward-backward, vierbi, ad Baum-Welsh ca do iferece ad learig i all of hese models. Eric CMU, Facorial HMM The belief sae a each ime is = {, k Q, K Q } ad i he mos geeral case has a sae space Od k for k d-ary chais The commo observed child Y couples all he pares explaiig away. Bu he parameerizaio cos for fhmm is Okd 2 for k chai-specific i i rasiio models p Q Q, raher ha Od 2k for p Eric CMU,

5 Facorial HMMs vs HMMs Le us compare a facorial HMM wih D chais, each wih K values, o is equivale HMM. Num. parameers o specify HMM: p fhmm: Compuaioal complexiy of exac iferece: HMM fhmm: Eric CMU, Triagulaig fhmm Is he followig riagulaio correc? S S 2 S 3 S N y y k x y 2 y 3 Here is a riagulaio y k2 y k3 x 2 x 3 y N y kn x N? We have creaed cliques of size k+, ad here are OkT of hem. The jucio ree algorihm is o efficie for facorial HMMs. Eric CMU,

6 Special case: swichig HMM The clipper S S 2 S 3 S N Perso y y 2 y 3 y N Perso k y k y k2 y k3 y kn The scee x x 2 x 3 x N Differe chais have differe sae space ad differe semaics The exac calculaio is iracable ad we mus use approximae iferece mehods Eric CMU, Hidde Markov decisio rees Jorda,Ghahramai,&Saul,97 combiaio of decisio rees wih facorial HMMs This gives a "commad srucure" o he facorial represeaio ppropriae for muli-resoluio ime series gai, he exac calculaio is iracable ad we mus use approximae iferece mehods Eric CMU,

7 Recall Sae Space Models SSMs lso kow as liear dyamical sysem, dyamic liear model, Kalma filer model, ec. The Kalma ler ca compue P i Omi{M 3 ;D 2 Y } : operaios per ime sep. Eric CMU, Facored liear-gaussia models produce sparse marices Direced arc from i - o j iff i,j >0 udireced arc bewee i o j iff Σ i,j >0 e.g., cosider a 2-chai facorial SSM wih i i i i i i P = N ;, Q P, 2 2, = Eric CMU,

8 8 Eric CMU, Discree-sae models Facored discree-sae models do NOT produce sparse rasiio marices e.g., cosider a 2-chai facorial HMM,, = P P P Eric CMU, Problems wih SSMs lieariy Gaussiaiy Ui-modaliy

9 Swichig SMM Possible world: muliple moio sae: Task: Trajecory predicio Model: Combiaio of HMM ad SSM p = x = x, S = i = N x ; x p Y = y = x = N ; Cx, R p S = j S = i = M i, j, Q Belief sae has Ok Gaussia modes: i i Eric CMU, Daa associaio correspodece problem Opimal belief sae has Ok modes. Commo o use eares eighbor approximaio. For each ime slice, ca eforce ha a mos oe source causes each observaio Correspodece problem also arises i shape machig ad sereo visio. Eric CMU,

10 Kids of iferece for DBNs Eric CMU, Complexiy of iferece i DBN Eve wih local coeciviy, everyhig becomes correlaed due o shared commo iflueces i he pas. E.g. coupled HMM chmm Eve hough CHMMs are sparse, all odes eveually become correlaed, so P y : has size O2 N. Eric CMU,

11 Jucio ree for coupled HMMs Cliques form a froier ha sakes from o. Eric CMU, pproximae Filerig May possible represeaios for belief sae α P Y : : Discree disribuio hisogram Gaussia Mixure of Gaussias Se of samples paricles Eric CMU,

12 Belief sae = discree disribuio Discree disribuio is o-parameric flexible, bu iracable. Oly cosider k mos probable values --- Beam search. pproximae joi as produc of facors DF/BK approximaio α ~ α = C i= P i Y : Eric CMU, Example: ssumed desiy filerig DF DF forces he belief sae o live i some resriced family F, e.g., produc of hisograms, Gaussia. Give a prior ~ α F, do oe sep of exac Bayesia updaig o ge αˆ F. The do a projecio sep o fid he closes approximaio i he family: ~ α arg mi KL q α q F The Boye-Koller BK algorihm is DF applied o a DBN e.g., le F be a produc of sigleo margials: This is also a variaioal mehod, ad he updaig sep ca sill be iracable Eric CMU,

13 pproximae smoohig off-lie Two-ler smoohig Loopy belief propagaio Variaioal mehods Gibbs samplig Ca combie exac ad approximae mehods Used as a subrouie for learig Eric CMU, NLP ad Daa Miig We wa: Semaic-based search ifer opics ad caegorize documes Mulimedia iferece uomaic raslaio Predic how opics evolve Research opics Eric CMU,

14 The Vecor Space Model Represe each docume by a high-dimesioal vecor i he space of words Eric CMU, Lae Semaic Idexig Docume Term = * * m x T m x k r w= K k= r d k λ T k Λ k x k k D T k x LS does o defie a properly ormalized probabiliy disribuio of observed ad lae eiies Does o suppor probabilisic reasoig uder uceraiy ad daa fusio Eric CMU,

15 moey bak bak loa river2 sream2 bak moey river2 bak moey bak loa moey sream2 bak moey bak bak loa river2 sream2 bak moey river2 bak moey bak loa bak moey sream2 moey bak bak loa river2 sream2 bak moey river2 bak moey bak loa moey sream2 bak moey bak bak loa river2 sream2 bak moey river2 bak moey bak loa bak moey sream2 moey bak bak loa river2 sream2 bak moey river2 bak moey bak loa moey sream2 bak moey bak bak loa river2 sream2 bak moey river2 bak moey bak loa bak moey sream2 moey bak bak loa river2 sream2 bak moey river2 bak moey bak loa moey sream2 bak moey bak bak loa river2 sream2 bak moey river2 bak moey bak loa bak moey sream2 Lae Semaic Srucure Lae Srucure l Disribuio over words P w = P w,l l Words w Iferrig lae srucure Pw l P l P l w = Pw Predicio P w = w + Eric CMU, dmixure Models Objecs are bags of elemes Mixures are disribuios over elemes Objecs have mixig vecor θ Represes each mixures coribuios Objec is geeraed as follows: Pick a mixure compoe from θ Pick a eleme from ha compoe Eric CMU,

16 Topic Models =dmixure Models Geeraig a docume Draw θ from he prior For each word - Draw - Draw z from muliomia l w z, θ { β } from muliomia l β : k z Prior θ z Which prior o use? β K w N d N Eric CMU, Choice of Prior Dirichle LD Blei e al Cojugae prior meas efficie iferece Ca oly capure variaios i each opic s iesiy idepedely Logisic Normal CTM=LoNTM Blei & Laffery 2005, hmed & ig 2006 Capure he iuiio ha some opics are highly correlaed ad ca rise up i iesiy ogeher No a cojugae prior implies hard iferece Eric CMU,

17 Logisic Normal Vs. Dirichle Dirichle Eric CMU, Logisic Normal Vs. Dirichle Logisic Normal Eric CMU,

18 Mixed Membership Model M 3 Mixure versus admixure Bayesia mixure model Bayesia admixure model: Mixed membership model Eric CMU, Lae Dirichle llocaio: M 3 i ex miig docume is a bag of words each geeraed from a radomly seleced opic Eric CMU,

19 9 Eric CMU, Populaio admixure: M 3 i geeics The geeic maerials of each moder idividual are iheried from muliple acesral populaios, each DN locus may have a differe geeric origi cesral labels may have e.g., Markovia depedecies Eric CMU, Iferece i Mixed Membership Models Mixure versus admixure Iferece is very hard i M 3, all hidde variables are coupled ad o facorizable! = } {,,, m z N m m z m d d d G p p z p x p D p φ π π φ α π π φ L L } {,,, ~ m z i m m z m d d G p p z p x p D p φ π φ α π π φ π

20 pproaches o iferece Exac iferece algorihms The elimiaio algorihm The jucio ree algorihms pproximae iferece echiques Moe Carlo algorihms: Sochasic simulaio / samplig mehods Markov chai Moe Carlo mehods Variaioal algorihms: Belief propagaio Variaioal iferece Eric CMU,