Sool o oputr Sin roilisti rpil Mols xt Inrn: Vril liintion g H ri Xing Ltur 4 Jnury 27 2014 Ring: K-p 9 ri Xing @ MU 2005-2014 1
Rp: n: is prt p -p or istriution i II. ri Xing @ MU 2005-2014 2
Qustion: Is tr N tt is prt p or givn MN? T "ion" MN ri Xing @ MU 2005-2014 3
Qustion: Is tr N tt is prt p or givn MN? {} {} {} {} Tis MN os not v prt I-p s N! ri Xing @ MU 2005-2014 4
Qustion: Is tr n MN tt is prt I-p to givn N? V-strutur xpl ri Xing @ MU 2005-2014 5
Qustion: Is tr n MN tt is prt I-p to givn N? V-strutur s no quivlnt in MNs! ri Xing @ MU 2005-2014 6
rtilly irt yli rps lso ll in grps Nos n isjointly prtition into svrl in oponnts n g witin t s in oponnt ust unirt n g twn two nos in irnt in oponnts ust irt in oponnts: {} {} {}{}{H} {I} ri Xing @ MU 2005-2014 7
Sury Invstigt t rltionsip twn Ns n MNs Ty rprsnt irnt ilis o inpnn ssuptions Not ntion: in ntworks suprst o ot Ns n MNs Wy w r out tis: N n MN or irnt sntis or signr to ptur or xprssion onitionl inpnns ong vrils Unr rtin onition N n rprsnt s n MN n vi vrs In t utur or rtin oprtion i.. inrn w will using singl rprsnttion s t t strutur or wi n lgorit n oprt on. Tis ks lgorit sign n nlysis o t lgorits siplr ri Xing @ MU 2005-2014 8
roilisti Inrn n Lrning W now v opt rprsnttions o proility istriutions: rpil Mols M M sris uniqu proility istriution Typil tsks: Tsk 1: How o w nswr quris out M.g. M XY? W us inrn s n or t pross o oputing nswrs to su quris Tsk 2: How o w stit plusil ol M ro t? i. W us lrning s n or t pross o otining point stit o M. ii. ut or ysin ty sk pm wi is tully n inrn prol. iii. Wn not ll vrils r osrvl vn oputing point stit o M n to o inrn to iput t issing t. ri Xing @ MU 2005-2014 9
Qury 1: Liklioo Most o t quris on y sk involv vin vin is n ssignnt o vlus to st vrils in t oin Witout loss o gnrlity = { X k+1 X n } Siplst qury: oput proility o vin x x tis is otn rrr to s oputing t liklioo o x 1 x k 1 k ri Xing @ MU 2005-2014 10
Qury 2: onitionl roility Otn w r intrst in t onitionl proility istriution o vril givn t vin X X x X X x tis is t postriori li in X givn vin W usully qury sust Y o ll oin vrils X={YZ} n "on't r" out t rining Z: Y Y Z z t pross o suing out t "on't r" vrils z is ll rginliztion n t rsulting y is ll rginl pro. z ri Xing @ MU 2005-2014 11
pplitions o postriori li rition: wt is t proility o n outo givn t strting onition? t qury no is snnt o t vin ignosis: wt is t proility o iss/ult givn syptos t qury no n nstor o t vin Lrning unr prtil osrvtion? ill in t unosrv vlus unr n "M" stting or ltr T irtionlity o inortion low twn vrils is not rstrit y t irtionlity o t gs in M proilisti inrn n oin vin or ll prts o t ntwork ri Xing @ MU 2005-2014 12
xpl: p li Ntwork p li Ntwork N [Hinton t l. 2006] nrtiv ol wit ultipl in lyrs Sussul pplitions Rognizing nwrittn igits Lrning otion ptur t ollortiv iltring W 3 W 2 H 3 H 2 W 1 visil nos t H 1 V ri Xing @ MU 2005-2014 13
Qury 3: Most rol ssignnt In tis qury w wnt to in t ost prol joint ssignnt M or so vrils o intrst Su rsoning is usully pror unr so givn vin n ignoring t vlus o otr vrils z : M Y rg x yy y rg x yy z y z tis is t xiu postriori onigurtion o y. ri Xing @ MU 2005-2014 14
pplitions o M lssiition in ost likly ll givn t vin xplntion wt is t ost likly snrio givn t vin utionry not: T M o vril pns on its "ontxt"---t st o vrils n jointly quri xpl: M o Y 1? M o Y 1 Y 2? y 1 y 2 y 1 y 2 0 0 0.35 0 1 0.05 1 0 0.3 1 1 0.3 ri Xing @ MU 2005-2014 15
oplxity o Inrn T: oputing X = x in M is N-r Hrnss os not n w nnot solv inrn It iplis tt w nnot in gnrl prour tt works iintly or ritrry Ms or prtiulr ilis o Ms w n v provly iint prours ri Xing @ MU 2005-2014 16
ppros to inrn xt inrn lgorits T liintion lgorit Mssg-pssing lgorit su-prout li propgtion T juntion tr lgorits pproxit inrn tniqus Stosti siultion / spling tos Mrkov in Mont rlo tos Vritionl lgorits ri Xing @ MU 2005-2014 17
Mrginliztion n liintion signl trnsution ptwy: Wt is t liklioo tt protin is tiv? Qury: nïv sution ns to nurt ovr n xponntil nur o trs y in oposition w gt ri Xing @ MU 2005-2014 18
liintion on ins Rrrnging trs... ri Xing @ MU 2005-2014 19
Now w n pror innrost sution Tis sution "liints" on vril ro our sution rgunt t "lol ost". X p liintion on ins ri Xing @ MU 2005-2014 20
p p p X X liintion in ins Rrrnging n tn suing gin w gt ri Xing @ MU 2005-2014 21
liintion in ins X X X X liint nos on y on ll t wy to t n w gt oplxity: p stp osts OVlX i *VlX i+1 oprtions: Okn 2 opr to nïv vlution tt sus ovr joint vlus o n-1 vrils On k ri Xing @ MU 2005-2014 22
Hin Mrkov Mol y 1 y 2 y 3... y T px y = px 1 x T y 1 y T = py 1 px 1 y 1 py 2 y 1 px 2 y 2 py T y T-1 px T y T x 1 x 2 x 3... x T onitionl proility: ri Xing @ MU 2005-2014 23
Hin Mrkov Mol onitionl proility: y 1 x 1 y 2 y 3 x 2 x 3...... y T x T ri Xing @ MU 2005-2014 24
Rrrnging trs... Unirt ins Z Z 1 1 ri Xing @ MU 2005-2014 25
onitionl Rno ils Y 1 Y 2 Y 5 X 1 X n ri Xing @ MU 2005-2014 26
T Su-rout Oprtion In gnrl w n viw t tsk t n s tt o oputing t vlu o n xprssion o t or: z wr is st o tors W ll tis tsk t su-prout inrn tsk. ri Xing @ MU 2005-2014 27
Inrn on nrl M vi Vril liintion nrl i: Writ qury in t or tis suggsts n "liintion orr" o ltnt vrils to rginliz Itrtivly Mov ll irrlvnt trs outsi o innrost su ror innrost su gtting nw tr Insrt t nw tr into t prout wrp-up X 1 X 1 x n x x i 3 2 X 1 X x 1 x i p i 1 ri Xing @ MU 2005-2014 28
Outo o liintion Lt X so st o vrils lt st o tors su tt or Sop[ ] X lt Y X st o qury vrils n lt Z = X Y t vril to liint T rsult o liinting t vril Z is tor Y z Tis tor os not nssrily orrspon to ny proility or onitionl proility in tis ntwork. xpl ortoing ri Xing @ MU 2005-2014 29
ling wit vin onitioning s Su-rout Oprtion T vin potntil: Totl vin potntil: Introuing vin --- rstrit tors: i i i i i i i 0 i 1 z Y I i i i ri Xing @ MU 2005-2014 30
T liintion lgorit rour liintion // t M // vin Z // St o vrils to liint X // qury vrils 1. Initiliz 2. vin 3. Su-rout-liintion Z 4. Norliztion ri Xing @ MU 2005-2014 31
T liintion lgorit rour Initiliz Z 1. Lt Z 1... Z k n orring o Z su tt Z i Z j i i < j 2. Initiliz wit t ull t st o tors rour vin 1. or i = i i rour Su-rout-Vril- liintion Z 1. or i = 1... k Su-rout-liint-Vr Z i 2. 3. rturn 4. Norliztion ri Xing @ MU 2005-2014 32
T liintion lgorit rour Initiliz Z 1. Lt Z 1... Z k n orring o Z su tt Z i Z j i i < j 2. Initiliz wit t ull t st o tors rour vin 1. or i = i i rour Su-rout-Vril- liintion Z 1. or i = 1... k Su-rout-liint-Vr Z i 2. 3. rturn 4. Norliztion rour Norliztion 1. X= X/ x X rour Su-rout-liint-Vr // St o tors Z // Vril to liint 1. { : Z Sop[]} 2. Z 5. rturn {} ri Xing @ MU 2005-2014 33
or oplx ntwork oo w H Wt is t proility tt wks r lving givn tt t grss onition is poor? ri Xing @ MU 2005-2014 34
Qury: N to liint: H Initil tors: oos n liintion orr: H Stp 1: onitioning ix t vin no i.. on its osrv vlu i.. : Tis stp is isoorpi to rginliztion stp: H g ~ p ~ p ~ xpl: Vril liintion ri Xing @ MU 2005-2014 35
Qury: N to liint: Initil tors: Stp 2: liint oput H g g 1 g g g p g xpl: Vril liintion ri Xing @ MU 2005-2014 36
Qury: N to liint: Initil tors: Stp 3: liint oput H xpl: Vril liintion g g p ri Xing @ MU 2005-2014 37
Qury: N to liint: Initil tors: Stp 4: liint oput H xpl: Vril liintion g g p ri Xing @ MU 2005-2014 38
Qury: N to liint: Initil tors: Stp 5: liint oput H xpl: Vril liintion g g p ri Xing @ MU 2005-2014 39
Qury: N to liint: Initil tors: Stp 6: liint oput H xpl: Vril liintion p g g ri Xing @ MU 2005-2014 40
Qury: N to liint: Initil tors: Stp 7: liint oput H xpl: Vril liintion g g p ri Xing @ MU 2005-2014 41
Qury: N to liint: Initil tors: Stp 8: Wrp-up H xpl: Vril liintion g g ~ p p p p ~ p p ~ ri Xing @ MU 2005-2014 42
oplxity o vril liintion Suppos in on liintion stp w oput Tis rquirs x y1 yk ' x x y1 yk k Vl X Vl Y ultiplitions i or vlu or x y 1 y k w o k ultiplitions i ' x x Vl X Vl Y itions i y x k 1 yk i x y i i1 i or vlu o y 1 y k w o VlX itions oplxity is xponntil in nur o vrils in t intrit tor ri Xing @ MU 2005-2014 43
grp liintion lgorit orliztion H H grp liintion Unrstning Vril liintion ri Xing @ MU 2005-2014 44
rp liintion gin wit t unirt M or orliz N rp V n liintion orring I liint nxt no in t orring I Roving t no ro t grp onnting t rining nigors o t nos T ronstitut grp 'V ' Rtin t gs tt wr rt uring t liintion prour T grp-torti proprty: t tors rsult uring vril liintion r ptur y roring t liintion liqu ri Xing @ MU 2005-2014 45
grp liintion lgorit Intrit trs orrspon to t liqus rsult ro liintion orliztion H H grp liintion Unrstning Vril liintion H ri Xing @ MU 2005-2014 46
liintion liqus H H H g ri Xing @ MU 2005-2014 47
rp liintion n rginliztion Inu pnny uring rginliztion vs. liintion liqu Sution <-> liintion Intrit tr <-> liintion liqu H g g ri Xing @ MU 2005-2014 48
liqu tr H g g p ri Xing @ MU 2005-2014 49
oplxity T ovrll oplxity is trin y t nur o t lrgst liintion liqu Wt is t lrgst liintion liqu? pur grp torti qustion Tr-wit k: on lss tn t sllst ivl vlu o t rinlity o t lrgst liintion liqu rnging ovr ll possil liintion orring goo liintion orrings l to sll liqus n n ru oplxity wt will ppn i w liint "" irst in t ov grp? in t st liintion orring o grp --- N-r Inrn is N-r ut tr otn xist "ovious" optil or nr-opt liintion orring ri Xing @ MU 2005-2014 50
xpls Str Tr ri Xing @ MU 2005-2014 51
Mor xpl: Ising ol ri Xing @ MU 2005-2014 52
Liittion o rour liintion Liittion H H ri Xing @ MU 2005-2014 53
Our lgorit so r nswrs only on qury.g. on on no o w n to o oplt liintion or vry su qury? liintion ssg pssing on liqu tr Mssgs n rus H g H H ro liintion to Mssg ssing g p ri Xing @ MU 2005-2014 54
ro liintion to Mssg ssing Our lgorit so r nswrs only on qury.g. on on no o w n to o oplt liintion or vry su qury? liintion ssg pssing on liqu tr notr qury... g Mssgs n r rus otrs n to roput ri Xing @ MU 2005-2014 55 H
Sury T sipl liint lgorit pturs t ky lgoriti Oprtion unrlying proilisti inrn: --- Tt o tking su ovr prout o potntil untions Wt n w sy out t ovrll oputtionl oplxity o t lgorit? In prtiulr ow n w ontrol t "siz" o t suns tt ppr in t squn o sution oprtion. T oputtionl oplxity o t liint lgorit n ru to purly grp-torti onsirtions. Tis grp intrprttion will lso provi ints out ow to sign iprov inrn lgorit tt ovro t liittion o liint. ri Xing @ MU 2005-2014 56