Probabilistic image processing and Bayesian network

Transcription

1 robabilistic imae processin and Bayesian network Kazuyuki Tanaka Graduate School o Inormation Sciences Tohoku University kazu@smapip.is.tohoku.ac.jp Reerences K. Tanaka: Statistical-mechanical approach to imae processin (Topical Review) J. hys. A vol.35 pp.r8-r50 (00). K. Tanaka H. Shouno M. Okada and D. M. Titterinton: Accuracy o the Bethe approimation or hyperparameter estimation in probabilistic imae processin J. hys. A vol.37 pp (004). 8 November 005 CISJ005

2 Bayesian Network and Belie ropaation robabilistic Inormation rocessin Bayes Formula Bayesian Network robabilistic Model Belie ropaation J. earl: robabilistic Reasonin in Intellient Systems: Networks o lausible Inerence (Moran Kaumann 988). C. Berrou and A. Glavieu: Near optimum error correctin codin and decodin: Turbo-codes IEEE Trans. Comm. 44 (996). 8 November 005 CISJ005

3 Contents. Introduction. Belie ropaation 3. Bayesian Imae Analysis and Gaussian Graphical Model 4. Concludin Remarks 8 November 005 CISJ005 3

4 Belie ropaation How should we treat the calculation o the summation over N coniuration? L W ( L ) = 0 = 0 = 0 N It is very hard to calculate eactly ecept some special cases. N Formulation or approimate alorithm Accuracy o the approimate alorithm 8 November 005 CISJ005 4

5 8 November 005 CISJ005 5 Tractable Model Factorizable Not Factorizable robabilistic models with no loop are tractable. robabilistic models with loop are not tractable. ( ) = d c b a d c b a d c b a ) ( ) ( ) ( ) ( ) ( ) ( ) ( D C B A D C B A a b c d a b c d a b c d ( ) a b c d d c b a W

6 8 November 005 CISJ005 6 robabilistic model on a raph with no loop ( ) ( ) ( ) ) ( ) ( ) ( y W y W y W W W y D C B A d c b a d c b a ( ) ( ) a b c d d c b a y y a b c d Marinal probability o the node

7 robabilistic model on a raph with no loop 6 d ( y) = M ( y) M ( y) M ( y) 5 6 M y = W y M 3 M 4 M = y a c ( ) ( ) ( ) ( ) ( ) W ( ) ( ) ( ) y M 5 y M 6 y b Marinal probability can be epressed in terms o the product o messaes rom all the neihbourin nodes o node. Messae rom the node to the node can be epressed in terms o the product o messae rom all the neihbourin nodes o the node ecept one rom the node. 8 November 005 CISJ005 7

8 robabilistic Model on a Graph with Loops Z ( ) ( ) = W L L ij i j ij N Ω Marinal robability Z W ij N ij ( ) i j ( ) = ( L ) \{ } L ( ) = ( L ) { } L \ 8 November 005 CISJ005 8

9 Q Belie ropaation M 3 M 4 M 7 4 M 5 ( ξ ) ( ξ ζ ) Q = Q ς 3 5 M 4 M 3 M 4 M 5 M 6 8 November 005 CISJ W 8 6 M 8 M 7 ( ) = M ( ) M ( ) Q ( ) = M ( ) M ( ) M ( ) Z M 4 ( ) M ( ) 5 3 Z W ( ) M ( ) M ( ) M ( ) 3 6 M 4 W ς M ( ξ ) ( ς ξ ) M ( ς ) 4 ( ς ) M ( ς ) Messae Update Rule 8

10 Messae assin Rule o Belie ropaation M ( ξ ) = ξ ς ς W W ( ς ξ ) M ( ς ) M ( ς ) M ( ς ) 3 ( ς ξ ) M ( ς ) M ( ς ) M ( ς ) M 3 M M 5 5 M Fied oint Equations or Massae r M r = Φ r ( ) M 8 November 005 CISJ005 0

11 Fied oint Equation and Iterative Method r ( ) Fied oint Equation * * M = Φ M r r Iterative Method r M r M r M 3 Φ Φ Φ M r ( M ) 0 ( r M ) ( r M ) y M 0 M * M y = M 0 y = Φ() 8 November 005 CISJ005

12 Contents. Introduction. Belie ropaation 3. Bayesian Imae Analysis and Gaussian Graphical Model 4. Concludin Remarks 8 November 005 CISJ005

13 Bayesian Imae Analysis Noise Transmission Oriinal Imae Deraded Imae { Oriinal Imae Deraded Imae} r = A osteriori robability Deradation rocess A riori robability r { Deraded Imae Oriinal Imae} r{ Oriinal Imae} { Deraded Imae} r Marinal Likelihood 8 November 005 CISJ005 3

14 Bayesian Imae Analysis ( ) Deradation rocess i + n i Ω = n i i + i ( 0 ) ~ N i ( ) = ep ( ) i i i Ω π Additive White Gaussian Noise Transmission Oriinal Imae Deraded Imae 8 November 005 CISJ005 4

15 Bayesian Imae Analysis A riori robability ( + ) i ( ) = Z ( ) j ep R ij N ( ) i j Generate Similar? Standard Imaes 8 November 005 CISJ005 5

16 Bayesian Imae Analysis A osteriori robability i j ( + ) ( ) = = Z ( ) ( ) ( ) OS ep ( ) ( ) ( ) i i i j i Ω ij N Gaussian Graphical Model 8 November 005 CISJ005 6

17 Bayesian Imae Analysis ( ) ( ) y A riori robability Ω Oriinal Imae = { i Ω} = { i Ω} i Deraded Imae Deraded Imae i iels A osteriori robability ( ) = ( ) ( ) ( ) ˆ i i ( ) d = ( ) i i d i = + 8 November 005 CISJ005 7

18 ( ˆ ˆ ) Hyperparameter Determination by Maimization o Marinal Likelihood = ar ma ( ) ( ) ˆ i ( ˆ ˆ ) = d i y ( ) = ( ) d = ( ) ( ) d ( ) = ( ) ( ) Ω Oriinal Imae { Ω} = i i ( ) ( ) ( ) Marinal Likelihood Marinalization Deraded Imae { Ω} = i i 8 November 005 CISJ005 8

19 8 November 005 CISJ005 9 Maimization o Marinal Likelihood by EM (Epectation Maimization) Alorithm ( ) ( ) ( ) d = Marinal Likelihood { } Ω = i i Incomplete Data Ω y ( ) ( ) ( ) d Q = ' ' ln ' ' ( ) ( ) 0 ' ' ' 0 ' ' ' ' ' ' ' = = = = = = Q Q ( ) ( ) 0 0 = = Equivalent Q-Function

20 Maimization o Marinal Likelihood by EM (Epectation Maimization) Alorithm Marinal Likelihood Q-Function Q ( ) = ( ) ( ) d ( ' ' ) = ( ) ln ( ' ') d EM Alorithm Iterate the ollowin EM-steps until converence: E -Step : Q M -Step : ( ' ' ( t) ( t) ) ( t) ( t) ln ( ( t + ) ( t + ) ) ar ma Q ' ' () t () t ( ) ( ' ') ( ' β ') ( ). A.. Dempster N. M. Laird and D. B. Rubin Maimum likelihood rom incomplete data via the EM alorithm J. Roy. Stat. Soc. B 39 (977). y d. Ω 8 November 005 CISJ005 0

21 One-Dimensional Sinal Oriinal Sinal 00 i 00 EM Alorithm 0 Deraded Sinal 00 i = Estimated Sinal 00 i i ˆi i 8 November 005 CISJ005

22 Imae Restoration by Gaussian Graphical Model Oriinal Imae Deraded Imae EM Alorithm with Belie ropaation MSE: 5 MSE: 59 8 November 005 CISJ005

23 8 November 005 CISJ005 3 Eact Results o Gaussian Graphical Model ( ) ( ) ( ) ( ) = == Ω C C d Z N ij j i i i i T T OS ep ep ep ( ) ( ) ( ) ( ) + + = Ω C I C C I C T ep det det π ( ) C I ˆ + = Multi-dimensional Gauss interal ormula Ω y ( ) ( ) ( ) ma ar ˆ ˆ =

24 MSE = Ω Comparison o Belie ropaation with Eact Results in Gaussian Graphical Model = 40 = 40 i Ω ( ˆ ) i i Belie ropaation Eact Belie ropaation Eact ( ˆ ˆ ) = ar MSE ˆ ˆ ln ( ˆ ˆ ) ( ) ( ) MSE ˆ ˆ ln ( ˆ ˆ ) November 005 CISJ005 4 ma

25 Oriinal Imae Imae Restoration by Gaussian Graphical Model Deraded Imae Belie ropaation Eact Lowpass Filter MSE: 5 Wiener Filter MSE: 35 Median Filter MSE:35 MSE: 4 MSE: 545 MSE: 447 ( ) 8 November MSE = ˆ 005 i i CISJ005 5 Ω i Ω

26 Oriinal Imae Imae Restoration by Gaussian Graphical Model Deraded Imae Belie ropaation Eact Lowpass Filter MSE: 59 Wiener Filter MSE: 60 Median Filter MSE36 MSE: 4 MSE: 37 MSE: 44 ( ) 8 November MSE = ˆ 005 i i CISJ005 6 Ω i Ω

27 Etension o Belie ropaation Generalized Belie ropaation J. S. Yedidia W. T. Freeman and Y. Weiss: Constructin ree-enery enery approimations and eneralized belie propaation alorithms IEEE Transactions on Inormation Theory 5 (005). Generalized belie propaation is equivalent to the cluster variation method in statistical mechanics R. Kikuchi: A theory o cooperative phenomena hys. Rev. 8 (95). T. Morita: Cluster variation method o cooperative phenomena and its eneralization I J. hys. Soc. Jpn (957). 8 November 005 CISJ005 7

28 Imae Restoration by Gaussian Graphical Model ( ˆ ˆ ) = 40 = 40 = ar ma ( ) MSE = Ω i Ω ( ) ( ˆ ) i i Belie ropaation Generalized Belie ropaation Eact Belie ropaation Generalized Belie ropaation Eact MSE ˆ ˆ ln ( ˆ ˆ ) MSE ˆ ˆ ln ( ˆ ˆ ) November 005 CISJ005 8

29 Imae Restoration by Gaussian Graphical Model and Conventional Filters = 40 MSE MSE Belie ropaation 37 Lowpass Filter (33) (55) Generalized Belie ropaation 35 Median Filter (33) (55) GB Eact 35 Wiener Filter (33) (55) MSE = Ω ( y) ( ˆ ) y Ω y (33) Lowpass (55) Median (55) Wiener 8 November 005 CISJ005 9

30 Imae Restoration by Gaussian Graphical Model and Conventional Filters = 40 MSE MSE Belie ropaation 60 Lowpass Filter (33) (55) 4 4 GB Generalized Belie ropaation Eact Median Filter Wiener Filter (33) (55) (33) (55) MSE = Ω ( y) ( ˆ ) y Ω y (55) Lowpass (55) Median (55) Wiener 8 November 005 CISJ005 30

31 Contents. Introduction. Belie ropaation 3. Bayesian Imae Analysis and Gaussian Graphical Model 4. Concludin Remarks 8 November 005 CISJ005 3

32 Summary Formulation o belie propaation Accuracy o belie propaation in Bayesian imae analysis by means o Gaussian raphical model (Comparison between the belie propaation and eact calculation) 8 November 005 CISJ005 3