Physical Fluctuomatics Applied Stochastic Process 9th Belief propagation

Transcription

1 Physcal luctuomatcs ppled Stochastc Process 9th elef propagaton Kazuyuk Tanaka Graduate School of Informaton Scences Tohoku Unversty Stochastc Process (Tohoku Unversty)

2 Tetbooks Kazuyuk Tanaka: Introducton of Image Processng by Probablstc Models Morkta Publshng o. Ltd. 006 (n Japanese) hapter 8. Kazuyuk Tanaka: Mathematcs of Statstcal Inference by ayesan Network orona Publshng o. Ltd. October 009 (n Japanese) hapters 6-9. Physcal uctuomatcs (Tohoku Unversty)

3 What s an mportant pont n computatonal complety? How should we treat the calculaton of the summaton over N confguraton? T T T N f If t takes second n the case of N=0 t takes 7 mnutes n N=0 days n N=30 and 34 years n N=40. Markov han Monte arlo Method elef Propagaton Method N a 0; for( for( } for( } } a T or ){ T or ){ T or ){ Physcal uctuomatcs (Tohoku Unversty) 3 N a f N fold loops Prevous Talk L Ths Talk ;

4 Probablstc Model and elef Propagaton Probablstc Informaton Processng ayes ormulas ayesan Networks Probablstc Models elef Propagaton J. Pearl: Probablstc Reasonng n Intellgent Systems: Networks of Plausble Inference (Morgan Kaufmann 988).. errou and. Glaveu: Near optmum error correctng codng and decodng: Turbo-codes I Trans. omm. 44 (996). Stochastc Process (Tohoku Unversty) 4

5 Mathematcal ormulaton of elef Propagaton Smlarty of Mathematcal Structures between Mean eld Theory and epef Propagaton Y. Kabashma and. Saad elef propagaton vs. TP for decodng corrupted messages urophys. Lett. 44 (998). M. Opper and. Saad (eds) dvanced Mean eld Methods ---Theory and Practce (MIT Press 00). Generalzaton of elef Propagaton S. Yedda W. T. reeman and Y. Wess: onstructng free-energy appromatons and generalzed belef propagaton algorthms I Transactons on Informaton Theory 5 (005). Interpretatons of elef Propagaton based on Informaton Geometry S. Ikeda T. Tanaka and S. mar: Stochastc reasonng free energy and nformaton geometry Neural omputaton 6 (004). Stochastc Process (Tohoku Unversty) 5

6 Generalzed tensons of elef Propagaton based on luster Varaton Method Generalzed elef Propagaton J. S. Yedda W. T. reeman and Y. Wess: onstructng freeenergy appromatons and generalzed belef propagaton algorthms I Transactons on Informaton Theory 5 (005). Key Technology s the cluster varaton method n Statstcal Physcs R. Kkuch: theory of cooperatve phenomena Phys. Rev. 8 (95). T. Morta: luster varaton method of cooperatve phenomena and ts generalzaton I J. Phys. Soc. Jpn (957). Stochastc Process (Tohoku Unversty) 6

7 elef Propagaton n Statstcal Physcs In graphcal models wth tree graphcal structures ethe appromaton s equvalent to Transfer Matr Method n Statstcal Physcs and gve us eact results for computatons of statstcal quanttes. In Graphcal Models wth ycles elef Propagaton s equvalent to ethe appromaton or luster Varaton Method. Trandfer Matr Method (Tree Structures) elef Propagaton ethe ppromaton luster Varaton Method (Kkuch ppromaton) Generalzed elef Propagaton Stochastc Process (Tohoku Unversty) 7

8 pplcatons of elef Propagatons Image Processng K. Tanaka: Statstcal-mechancal approach to mage processng (Topcal Revew) J. Phys. 35 (00).. S. Wllsky: Multresoluton Markov Models for Sgnal and Image Processng Proceedngs of I 90 (00). Low ensty Party heck odes Y. Kabashma and. Saad: Statstcal mechancs of low-densty party-check codes (Topcal Revew) J. Phys. 37 (004). S. Ikeda T. Tanaka and S. mar: Informaton geometry of turbo and low-densty party-check codes I Transactons on Informaton Theory 50 (004). M Multuser etecton lgorthm Y. Kabashma: M multuser detecton algorthm on the bass of belef propagaton J. Phys. 36 (003). T. Tanaka and M. Okada: ppromate elef propagaton densty evoluton and statstcal neurodynamcs for M multuser detecton I Transactons on Informaton Theory 5 (005). Satsfablty Problem O.. Martn R. Monasson R. Zecchna: Statstcal mechancs methods and phase transtons n optmzaton problems Theoretcal omputer Scence 65 (00). M. Mezard G. Pars R. Zecchna: nalytc and algorthmc soluton of random satsfablty problems Scence 97 (00). Stochastc Process (Tohoku Unversty) 8

9 Strategy of ppromate lgorthm n Probablstc Informaton Processng P P It s very hard to compute margnal probabltes eactly ecept some tractable cases. T T T 3 What s the tractable cases n whch margnal probabltes can be computed eactly? Is t possble to use such algorthms for tractable cases to compute margnal probabltes n ntractable cases? L L Stochastc Process (Tohoku Unversty) 9

10 Graphcal Representatons of Tractable Probablstc Models W ( ) W ( ) W ( ) W ( ) = X X X W ( ) W ( ) W ( ) ( ) W = W ( ) W ( ) W ( ) ( ) W Stochastc Process (Tohoku Unversty) 0

11 Graphcal Representatons of Tractable Probablstc Models X Stochastc Process (Tohoku Unversty)

12 Graphcal Representatons of Tractable Probablstc Models X Stochastc Process (Tohoku Unversty)

13 Graphcal Representatons of Tractable Probablstc Models X Stochastc Process (Tohoku Unversty) 3

14 Graphcal Representatons of Tractable Probablstc Models X Stochastc Process (Tohoku Unversty) 4

15 Graphcal Representatons of Tractable Probablstc Models Stochastc Process (Tohoku Unversty) 5

16 Graphcal Representatons of Tractable Probablstc Models X Stochastc Process (Tohoku Unversty) 6

17 Graphcal Representatons of Tractable Probablstc Models X X Stochastc Process (Tohoku Unversty) 7

18 Graphcal Representatons of Tractable Probablstc Models X X Stochastc Process (Tohoku Unversty) 8

19 Graphcal Representatons of Tractable Probablstc Models X X Stochastc Process (Tohoku Unversty) 9

20 Graphcal Representatons of Tractable Probablstc Models Stochastc Process (Tohoku Unversty) 0

21 Graphcal Representatons of Tractable Probablstc Models Stochastc Process (Tohoku Unversty)

22 Graphcal Representatons of Tractable Probablstc Models Stochastc Process (Tohoku Unversty)

23 Graphcal Representatons of Tractable Probablstc Models Stochastc Process (Tohoku Unversty) 3

24 Graphcal Representatons of Tractable Probablstc Models Stochastc Process (Tohoku Unversty) 4

25 Graphcal Representatons of Tractable Probablstc Models W ( ) W ( ) W ( ) W ( ) W ( ) = X X X X W ( ) W ( ) W ( ) ( ) W W ( ) = Stochastc Process (Tohoku Unversty) 5

26 Graphcal Representatons of Tractable Probablstc Models Stochastc Process (Tohoku Unversty) 6

27 Graphcal Representatons of Tractable Probablstc Models Stochastc Process (Tohoku Unversty) 7

28 Graphcal Representatons of Tractable Probablstc Models Stochastc Process (Tohoku Unversty) 8

29 Graphcal Representatons of Tractable Probablstc Models Stochastc Process (Tohoku Unversty) 9

30 Graphcal Representatons of Tractable Probablstc Models Stochastc Process (Tohoku Unversty) 30

31 Graphcal Representatons of Tractable Probablstc Models Stochastc Process (Tohoku Unversty) 3

32 Graphcal Representatons of Tractable Probablstc Models Graphcal Representaton of Margnal Probablty n terms of Messages Pr{} Stochastc Process (Tohoku Unversty) 3

33 Graphcal Representatons of Tractable Probablstc Models Graphcal Representaton of Margnal Probablty n terms of Messages Pr{} = Stochastc Process (Tohoku Unversty) 33

34 Graphcal Representatons of Tractable Probablstc Models Graphcal Representaton of Margnal Probablty n terms of Messages Pr{} = = Stochastc Process (Tohoku Unversty) 34

35 Graphcal Representatons of Tractable Probablstc Models Graphcal Representaton of Margnal Probablty n terms of Messages Pr{} = = = Stochastc Process (Tohoku Unversty) 35

36 Stochastc Process (Tohoku Unversty) 36 Graphcal Representatons of Tractable Probablstc Models Graphcal Representaton of Margnal Probablty n terms of Messages } Pr{ = = =

37 Graphcal Representatons of Tractable Probablstc Models Graphcal Representaton of Margnal Probablty n terms of Messages Pr{} = Pr{ } = Pr{ } Pr{ } Stochastc Process (Tohoku Unversty) 37 Recurson ormulas for Messages

38 Graphcal Representatons of Tractable Probablstc Models Graphcal Representaton of Margnal Probablty n terms of Messages Step Step Step 3 Stochastc Process (Tohoku Unversty) 38

39 Graphcal Representatons of Tractable Probablstc Models Graphcal Representaton of Margnal Probablty n terms of Messages Step Step Step 3 Pr{} Pr{} Pr{ } Pr{ } = = = = Stochastc Process (Tohoku Unversty) 39

40 elef Propagaton Probablstc Models wth no ycles P W a b c d a W b W W c W d {} 6 d a c b 3 5 a c 4 6 b d Stochastc Process (Tohoku Unversty) 40

41 Stochastc Process (Tohoku Unversty) 4 elef Propagaton } { W 4 b W b a 3 W a d 6 W c c 5 W d {} W W W W W P d c b a d c b a a b c d Probablstc Model on Tree Graph

42 Probablstc Model on Tree Graph P {} M 6 Pa b c d 3 d M W M M a abcd c 4 b {} 6 5 M 3 Wa M W a 4 b b M 5 W c W d c M 6 d Stochastc Process (Tohoku Unversty) 4

43 elef Propagaton Probablstc Model on Tree Graph M 6 d a W W a W b c b a b W {} M M {} 3 4 Stochastc Process (Tohoku Unversty) 43 W M 3 a a W M 4 b b

44 elef Propagaton for Probablstc Model on Tree Graph Pr Z X W { j} j { j} X X X 3 X k 3 X k No ycles!! X k X k X k Stochastc Process (Tohoku Unversty) 44

45 elef Propagaton for Probablstc Model on Square Grd Graph P P L { j} j { j} : Set of all the lnks Stochastc Process (Tohoku Unversty) 45

46 elef Propagaton for Probablstc Model on Square Grd Graph Stochastc Process (Tohoku Unversty) 46

47 elef Propagaton for Probablstc Model on Square Grd Graph Stochastc Process (Tohoku Unversty) 47

48 P Margnal Probablty P 3 4 N 3 4 N Stochastc Process (Tohoku Unversty) 48

49 P Margnal Probablty P 3 4 N 3 4 N 3 4 N Stochastc Process (Tohoku Unversty) 49

50 P Margnal Probablty P 3 4 N 3 4 N 3 4 N Stochastc Process (Tohoku Unversty) 50

51 P {} Margnal Probablty P N N Stochastc Process (Tohoku Unversty) 5

52 P {} Margnal Probablty P N N 3 4 N Stochastc Process (Tohoku Unversty) 5

53 P {} Margnal Probablty P N N 3 4 N Stochastc Process (Tohoku Unversty) 53

54 elef Propagaton for Probablstc Model on Square Grd Graph P P {} Stochastc Process (Tohoku Unversty) 54

55 elef Propagaton for Probablstc Model on Square Grd Graph P P {} Stochastc Process (Tohoku Unversty) 55

56 elef Propagaton for Probablstc Model on Square Grd Graph P P {} Stochastc Process (Tohoku Unversty) 56

57 elef Propagaton for Probablstc Model on Square Grd Graph 3 P P {} 8 3 Message Update Rule Stochastc Process (Tohoku Unversty) 57

58 elef Propagaton for Probablstc Model on Square Grd Graph M M M 3 3 z z z W W 5 M 4 M 5 4 z M z M z M z 3 z z M z M z M z M M ed Pont quatons for Messages Stochastc Process (Tohoku Unversty) 58

59 ed Pont quaton and Iteratve Method ed Pont quaton * * M M Stochastc Process (Tohoku Unversty) 59

60 ed Pont quaton and Iteratve Method ed Pont quaton * * M M Iteratve Method y y 0 M * y () Stochastc Process (Tohoku Unversty) 60

61 ed Pont quaton and Iteratve Method ed Pont quaton * * M M Iteratve Method y y y () 0 M * M 0 Stochastc Process (Tohoku Unversty) 6

62 ed Pont quaton and Iteratve Method ed Pont quaton * * M M Iteratve Method M M 0 y y M y () 0 M * M 0 Stochastc Process (Tohoku Unversty) 6

63 ed Pont quaton and Iteratve Method ed Pont quaton * * M M Iteratve Method M M M M 0 y M y y () 0 M * M M 0 Stochastc Process (Tohoku Unversty) 63

64 ed Pont quaton and Iteratve Method ed Pont quaton * * M M Iteratve Method M M M M 0 y M M 0 M * M y M 0 y () Stochastc Process (Tohoku Unversty) 64

65 ed Pont quaton and Iteratve Method ed Pont quaton M * M * Iteratve Method M M M 3 M M M 0 y M M 0 M * M y M 0 y () Stochastc Process (Tohoku Unversty) 65

66 elef Propagaton for Probablstc Model on Square Grd Graph our Knds of Update Rule wth Three Inputs and One Output Stochastc Process (Tohoku Unversty) 66

67 Stochastc Process (Tohoku Unversty) 67 Interpretaton of elef Propagaton based on Informaton Theory 0 )ln ( P P ) ( 0 Z Z W P j j j ln ] [ ln )ln ( ln ) ( ] [ ] [ } { 0 P P Z P ln ] [ ] [ mn j j j W Z } { j j j V W Z P } { ree nergy Kullback-Lebler vergence

68 Stochastc Process (Tohoku Unversty) 68 Interpretaton of elef Propagaton based on Informaton Theory Z P ln W W W )ln ( ln )ln ( ln ) ( )ln ( ln ) ( \ 0 )ln ( P P ree nergy KL vergence W Z P \ ) ( ) (

69 Interpretaton of elef Propagaton based on Informaton Theory KL vergence ree nergy P lnz P Z W lnw { j} { j} V { j} ( )ln j j lnw ln j j j ln ln ln j ( Stochastc Process (Tohoku Unversty) 69 ( ) ( ) ) () \ \ j ethe ree nergy j

70 ethe Interpretaton of elef Propagaton based on Informaton Theory { j} P ln Z ethe j j j lnwj ln { j} j arg mn P arg mn ln ln ln j V j j arg mn P arg mn j ethe j j j Stochastc Process (Tohoku Unversty) 70

71 Stochastc Process (Tohoku Unversty) 7 Interpretaton of elef Propagaton based on Informaton Theory j j j V V j j j j j L } { ethe ethe mn arg ethe j j j j Lagrange Multplers to ensure the constrants

72 Stochastc Process (Tohoku Unversty) 7 Interpretaton of elef Propagaton based on Informaton Theory j j j V V j j j j j j j j V j j j j j j V V j j j j j W L ln ln ln ln ln } { } { } { ethe ethe 0 ethe j L tremum ondton 0 ethe j j j L

73 Interpretaton of elef Propagaton based on Informaton Theory L ethe j 0 j j L ethe 0 j tremum ondton M 3 3 M M 8 M 4 4 M 5 5 M 4 M 4 M 5 5 W M 6 Stochastc Process (Tohoku Unversty) 73 6 M 7 M M M M M Z M 4 M 5 3 Z W M M M

74 Interpretaton of elef Propagaton based on Informaton Theory M 3 M 4 4 M M 4 M 3 M 4 M W M 6 Stochastc Process (Tohoku Unversty) M 8 M 7 M M M M M Z M 4 M 5 3 Z W M M M 3 6 M 4 W M M 4 7 M Message Update Rule 8

75 Interpretaton of elef Propagaton based on Informaton Theory M W W M M M Message Passng Rule of elef Propagaton M 3 M M 5 5 M 3 M M M = a 7 Stochastc Process (Tohoku Unversty) 75 5

76 Summary elef Propagaton Transfer Matr Method ethe ppromaton and luster Varaton Method Iteratve lgorthm uture Talks 0th Probablstc mage processng by means of physcal models th ayesan network and belef propagaton n statstcal nference Stochastc Process (Tohoku Unversty) 76

77 Practce 9- We consder a probablty dstrbuton P(abcdy) defned by P abcd yw a W b W yw c yw d y Show that margnal Probablty XY s epressed by P y Pabcd y P a b c d M W a M W b X a X M Y y W c y M y W d y c XY y M M W ym ym y XY X X XY Y Stochastc Process (Tohoku Unversty) 77 b d

78 Stochastc Process (Tohoku Unversty) 78 Practce M M M M Z M M M W M M M Z M M M W Z Z M y substtutng to derve the followng equaton.

79 Practce 9-3 Make a program to solve the nonlnear equaton =tanh() for varous values of. Obtan the solutons for = numercally. scuss how the teratve procedures converge to the fed ponts of the equatons n the cases of = by drawng the graphs of y=tanh() and y=. 3 tanh tanh tanh 0 Stochastc Process (Tohoku Unversty) 79