Probabilistic image processing and Bayesian network

Transcription

1 Computatonal Intellgence Semnar (8 November, 2005, Waseda Unversty, Tokyo, Japan) Probablstc mage processng and Bayesan network Kazuyuk Tanaka 1 Graduate School of Informaton Scences, Tohoku Unversty, Senda , Japan Abstract The basc frameworks and practcal schemes of the Bayesan network and the belef propagaton to the probablstc mage processng are revewed. The probablstc mage processng s formulated by means of Bayesan statstcs and arkov random felds. The system s regarded as one of Bayesan networks. In general, the Bayesan network has serous computatonal complexty because the probablstc models nclude a number of random varables for nodes or pxels. Recently, many researchers n the ntermedate regon of the mathematcal scences and the computer scences are nterested n the belef propagatons, whch s one of powerful approxmate methods for probablstc nference. In the present tutoral lecture note, we brefly explan the formulaton of the probablstc mage processng and the theoretcal structure of the belef propagaton. As an examples of the applcatons of the probablstc mage processng, we revew a nose reducton by means of Gaussan graphcal model as arkov random felds. 1 Introducton Probablstc nformaton processng based on statstcal scences and statstcal mechancs s applcable to computer scences. Conversely, many problems n probablstc nformaton processng have created research deas n statstcal scence and statstcal mechancs. any schemes n probablstc nformaton processng s based on the Bayesan statstcs and are reduced to probablstc models. The probablstc models can be represented by graphs ncludng nodes and drected lne segments and are sometmes called graphcal models. The system whch conssts of such a graphcal model s called Bayesan network. Probablstc mage processng based on the Bayes formula s one of the useful applcatons n the Bayesan network. By usng the Bayes formula and assumng an a pror probablty for the orgnal mage, one creates probablstc mage processng n the form of an a posteror probablty for the orgnal mage when the observed mage s gven[1]. Image processng by means of probablstc models and the Bayes statstcs are usually based on arkov random felds. In the arkov random felds, the state of a pxel s dependent only on the states of ts neghbourng pxels. any knds of arkov random feld models whch are applcable to the practcal mage processng, mage restoratons, segmentatons, edge detecton, mage compressons and moton detectons, were proposed[2, 3, 4]. Comparng the conventonal technques n the mage processng, the probablstc mage processng s expected to gve good performance even for large nose and to construct the robust systems for varous data. As for most of famlar arkov random felds as well as Bayesan networks, t s hard to acheve the crtera to obtan the estmated mage n the statstcal frameworks exactly except some specal cases[5]. In the artfcal ntellgence, the belef propagaton was proposed n the mddle of 1980s[6], 1 E-mal: kazu@smapp.s.tohoku.ac.p, Webpage: kazu/ 1

2 as an effcent algorthm of statstcal nference, whch ams at mplementng probablty-based artfcal ntellgence capable of handlng uncertantes. The belef propagaton s proved to gve exact results n the specal cases where the graphcal representaton of the underlyng probablty model has no loops n ts graphcal representaton. Although the belef propagaton s not guaranteed to gve exact results for general graphcal models wth loops, t works reasonably well n varous cases, and furthermore, n some specfc cases t turns out to work excellently. In response to such emprcally demonstrated good performance n the belef propagaton, a number of researchers n the mathematcal scence have attempt to acheve the theoretcal understandng of the performance and the frameworks n the algorthm[7, 8]. It s worthy of notce that the belef propagaton has a lnk wth statstcal mechancs[9, 10, 11] and has been equvalent to a Bethe approxmaton and a cluster varaton method, whch are ones of the advanced mean-feld methods n statstcal mechancs. oreover, some nterpretatons for the belef propagaton have been gven also from the nformaton geometrcal pont of vew[12, 13]. The belef propagaton has been appled to many practcal problems n the computer vson[14, 15, 16, 17]. Theoretcal study of the applcaton of the belef propagaton to mage processng has been done n the statstcal mechancal pont of vew[18, 19]. In ths paper, the basc frameworks and ts applcatons of the Bayesan network and the belef propagaton n the probablstc mage processng are revewed. As examples of the probablstc mage processng, we ntroduce a nose reducton and a segmentaton and extend the segmentaton to a moton detecton. Secton 2 descrbes the formulaton of the probablstc mage processng as Bayesan networks. In Secton 3, we explan the belef propagaton whch s reduced to the numercal teratve calculatons of the smultaneous fxed pont equatons wth respect to messages. In Secton 4, we explan the probablstc mage restoratons and nose reductons by usng the Gaussan graphcal model. concludng remarks. Secton 5 provdes 2 Formulaton of mage processng by Bayesan network We consder an mage on a square lattce Ω {1, 2,,L}. The states at pxel ( Ω) n the orgnal mage and the observed mage are regarded as random varables denoted by A and D, respectvely. In the nose reducton, the observed mage corresponds to a degraded mage and the orgnal mage corresponds to an natural mage before degraded by nose. In the segmentaton, a gven natural mage s regarded as the observed mage and a segmented mage wth a few knds of regons s regarded as the orgnal mage. Then the random felds of states n the orgnal mage and the observed mage are represented by A =(A 1,A 2,,A L )andd =(D 1,D 2,,D L ). The actual orgnal mage and the observed mage are denoted by a =(a 1,a 2,,a L )andd =(d 1,d 2,,d L ), respectvely. The probablty that the orgnal mage s a, P( A = a α), s called the a pror probablty of the mage. Here, α s not a set of random varables but a set of hyperparameters, whch specfy the functon to represent the a pror probablty. In the Bayes formula, the a posteror probablty P( A = a D = d), that the orgnal mage s a when 2

3 the gven observed mage s d, s expressed as P( A = a D = d, α, β)= P( D = d ) A = a, β P( A = a, α) P( D = d A = z, β)p( A = z, α), (1) z where the summaton z = z 1 z 2 z L s taken over all possble confguratons of mages z = (z 1,z 2,,z L ). If z takes any real number n the nterval (, + ), where the summaton z [ ] should be replaced by the ntegral d z + [ ]dz 1dz 2 dz L and the probablty should be regarded as a probablty densty functon. The probablty P ( ) D = d A = a, β s the condtonal probablty that the observed mage s d when the orgnal mage s a and descrbes the generatng process of data. Here β s a set of hyperparameters n the condtonal probablty. In the Bayesan statstcs, the estmate â of the state at each pxel n the orgnal mage are determned so as to maxmze the posteror margnal probablty: P(A = a D = d, α, β)= z δ a,z P( A = z D = d, α, β). (2) In the present framework, we have to estmate not only the estmate (â 1, â 2,, â L ) but also the hyperparameters α and β. We apply the maxmum lkelhood estmaton to the estmaton of hyperparameters from an observed mage, whch corresponds to data, n the statstcal scences as follows: ( α, β)=max P( D = d α, β) (3) ( α, β) P( D = d α, β)= z P( A = z, D = d α, β)= z P( D = d A = z, β)p( A = z, α). (4) In ths framework, the probablty P( D = d α, β) s gven by margnalzng the ont probablty P( A = a, D = d α, β) over all the possble mages a and can be regarded as a lkelhood for α and β when the observed mage d s gven n statstcs. P( D = d α, β) s referred to as evdence or margnal lkelhood[21, 22]. In order to maxmze the margnal lkelhood, the expectaton maxmzaton (E) algorthm s often employed. In the E algorthm, we ntroduce a Q-functon defned by Q( α, β α, β, d) z P( A = a D = d, α, β)lnp( A = a, D = d α, β ). (5) The extremum condtons of margnal lkelhood α P( D = d α, β)= 0, are equvalent to the followng equaltes: [ α Q( α, β α, β, d) ] = 0, α = α, β = β β P( D = d α, β)= 0, (6) [ β P( D = d α, ] β) = 0, (7) α = α, β = β respectvely. As an algorthm to calculate the set of estmates, ( α, β), whch satsfes the extremum condtons (6), we terate the followng E-steps untl convergence: E-Step: Q( α, β α(t), β(t), g) P( A = z D = d, α(t), β(t))lnp( A = z, D = d α, β). z 3

4 -Step: ( α(t +1), β(t +1)) argmaxq( α, β α(t), β(t), g). ( α, β) Though the above scheme can gve us the soluton for the extremum condtons (6), we have to remark that t does not necessarly provde the global maxmum for the margnal lkelhood P( D = d α, β). We denote a set of lnks between some pars of pxels by N and assgn a functon Φ (a,a )ateach par of lnks belongng to the set N. The a posteror probabltes P( A = a D = d, α, β)andthe a pror probabltes, P( A = a α) n the Bayesan mage analyss, are often expressed n terms of the followng functon P ( a): Φ (a,a ) N P ( a) = Φ (z,z ). (8) z N In the case, the random feld A s the set of random varables n whch the state of each pxel depends only on the confguraton of all the pxels lnked to the pxel, N { N } and s called the arkov random feld[1]. 3 Belef propagaton Except some specal cases, t s hard to calculate the margnalzaton n Eqs.(2) and (4) exactly, and we have to employ an approxmate algorthm. In the present secton, we use the belef propagaton as one of approxmate algorthms to calculate approxmate values of statstcal quanttes numercally. For a probablty gven n Eq.(8), the approxmate form of margnal probabltes n the belef propagatonsassumedtobep (a D = d) to the posteror margnal probablty: P (a ) k (a ) k N δ z,a P ( a), (9) z k (z ) k N z P (a,a ) z ( ) k (a ) Φ (a,a ) ( ) l (a ) k N l N δ z,a δ z,a P ( a) ( ) k (z ) Φ (z,z ) ( ). (10) l (z ) k N l N z z Here the notaton N { N } represents the set of all the nearest-neghbor pars of pxel. Thequantty (a ) n Eqs.(9) and (10) s called a message propagated from to Both quanttes (a )and (a ) are assgned at each lnk and are determned so as to satsfy the followng smultaneous fxed pont equatons: Φ z (a )= ) k N \{} ( Ω N ). (11) (z,a k (z ) z Φ (z,z ) k (z ) z k N \{} 4

5 2 (a) (b) (c) Fgure 1: Graphcal representatons of the belef propagaton. (a) Eq.(9). (b) Eq.(10). (c) Eq.(11). The graphcal representatons of Eqs.(9)-(11) are shown n Fgure 1. The smultaneous fxed pont equatons (11) are derved by substtutng Eqs.(9) and (10) to the reducblty condtons: P (a )= P (a,z ) ( Ω, N ). (12) z We can calculate the numercal solutons of the smultaneous fxed pont equatons (11), numercally, by usng the teraton method. By applyng the numercal solutons to Eqs.(9) and (10), we obtan the margnal probablty P (a )andp (a,a ). The approxmate expressons of margnal probabltes n Eqs.(9) and (10) can be obtaned by means of Bethe approxmaton n the statstcal mechancs. In Bethe approxmaton, we ntroduce an approxmate free energy for every probablstc model n Eq.(8) whch s defned by F Bethe [{ρ,ρ Ω, N }] F [ρ ]+ ( ) F [ρ ] F [ρ ] F [ρ ], (13) Ω N where F [ρ ] ρ (ζ)ln ( ρ (z ) ), F [ρ ] ( ρ (ζ,ζ ρ (z,z ) ) )ln. (14) Φ z z (z,z ) Here F Bethe [{ρ,ρ Ω, B}] s often referred to as Bethe free energy and s regarded as a good approxmaton for the true free energy of probablstc model n Eq.(8) whch s defned by F[ρ] = ) ln( z N Φ (z,z ) n the statstcal mechancs. The smultaneous fxed-pont equatons n Eqs.(9)- (11) are equvalent to the extremum condtons of F Bethe [{ρ,ρ Ω, B}] wth respect to the margnal probablty dstrbutons {ρ,ρ Ω, B} under the normalzaton condtons, z ρ (z )=1( Ω) and z z ρ (z,z )=1( N ), and the reducblty condtons (12). However, t s known that the Bethe free energy F Bethe [{ρ,ρ Ω, B}] does not provde any bounds for the true free energy F[ρ], whle a mean-feld free energy s a bound for the true free energy[10]. Furthermore, n some cases the soluton of the smultaneous fxed pont equatons (9)-(11) corresponds not to a local mnmum but to a saddle pont of the Bethe free energy[20]. 4 Nose Reducton by Gaussan Graphcal odel As an example for the applcatons of mage processng, we show the nose reducton from the degraded z 5

6 mage whch s corrupted by the addtve whte Gaussan nose wth the mean 0 and the varaton σ 2.In ths case, the random felds of the orgnal mage and of the degraded mage are denoted by A and D, respectvely. The degradaton process s expressed n terms of the followng condtonal probablty: P( D = d A = a, β = σ 2 )= 1 ( exp 1 Ω 2πσ 2σ 2 (a d ) 2), (15) P( A = a α) = N N ( ) exp 12 α(a a ) 2 ( ) exp 12 α(a a ) 2 d z (α >0). (16) By substtutng Eqs.(15) and (16) to the Bayes formula, the a posteror probablty s expressed as follows: P( A = a D = a, α, β = σ 2 )= ( exp ( 1 2σ (a 2 d ) 2))( exp ( 1 2 α(a a ) 2)) Ω N ( exp ( 1 2σ (z 2 d ) 2))( ( )). (17) exp 12 α(z z ) 2 d z Ω N By settng ( Φ (a,a ) exp 1 8σ 2 (a d ) 2 1 8σ 2 (a d ) α(a a ) 2), (18) Eq.(17) can be reduced to P( A = a D = a, α, β = σ 2 )= Φ (a,a ) N ( N ). (19) Φ (z,z ) d z By applyng the belef propagaton to these expressons of the probabltes, we can acheve the maxmzaton of the margnal lkelhood P( D = d α, β = σ 2 ) P( D = d A = z,β = σ 2 )P( A = z α)d z wth respect to the hyperparameters (α, σ), and can calculate the posteror margnal probablty P(A = a D = d, α, β = σ 2 )[19]. Now we assume that (ξ) can be approxmately expressed as (ξ) λ 2π exp ( λ 2 (ξ µ ) 2). (20) The smultaneous fxed-pont equaton (11) can be reduced to the followng fxed pont equatons for {λ,λ,µ,µ N, Ω}: βg + µ k λ k 1 = 1 λ α + 1 β + k N \, µ = λ k β + ( Ω N ). (21) λ k k N \ k N \ If we substtute Eq. (20) nto Eqs. (9) and (10), the one- and two-body margnal probablty denstes ρ (a )andρ (a,a ) are approxmately obtaned. The smultaneous fxed-pont equatons (21) are solved by the followng teratve algorthm: 6

7 Algorthm LBP[ g,α,β] Step 1: Set r 0 asanntalvalue. Step 2: Update r r +1and ( ) 1 1 a (r) α + 1 ( Ω, N ). (22) β + a k (r 1) k N \ b (r) βg + b k (r 1)a k (r 1) k N \ β + k N \ a k (r 1) ( Ω, N ). (23) Step 3: Update R r, λ a (r) andµ b (r) ( Ω). Go to Step 4 f, for prespecfed ε, ( ) a (r) a (r 1) + b (r) b (r 1) <ε, (24) Ω N and go to Step 2 otherwse. Step 4: Substtute {λ,µ N, Ω} nto Eq. (20), and calculate ρ (ξ g, α, β)andρ (ξ,ξ g,α,β)byusng Eqs.(9) and (10). Stop after ρ (ξ g, α, β) andρ (ξ,ξ g,α,β) are set as outputs n the present algorthm. Agan, t s usually adequate to set ε =10 6. In the denomnators of Eqs.(22) and (23), the summatons a k (r 1) and b k (r 1)a k (r 1) can be evaluated n O(1) tme per par of pxels k N \ k N \ and, because the number of elements n the set N \ s equal to 3 per par of pxels. Hence the teratve algorthm for solvng the smultaneous fxed-pont equatons (21) requres a total of O( Ω ) computatons per update. For fxed values of α and σ, the extremum condtons of Q(α,σ α, σ, g) wth respect to σ and α are reduced to the followng equatons: Ω (ξ g ) 2 ρ (ξ g, α, σ 2 )dξ = Ω σ 2, (25) (ξ ξ ) 2 ρ (ξ,ξ g, α, σ 2 )dξdξ = N N (ξ ξ ) 2 ρ (ξ,ξ g, α, 0)dξdξ, (26) respectvely. The margnal probablty densty functon ρ (ξ g, α, σ 2 ), ρ (ξ,ξ g, α, σ 2 )andρ (ξ,ξ g, α, 0) are approxmately obtaned as outputs of the loopy belef propagaton algorthms LBP[ g, α, σ 2 ]and LBP[ g, α, 0]. Therefore, we can gve the E algorthm for the maxmzaton of margnal lkelhood n Eq.(3) by usng the loopy belef propagaton as follows: E Algorthm n Loopy Belef Propagaton Step 1: Set ( α(0),σ(0) ) and t 0. 7

8 Step 2: Run the algorthm LBP[ g,α(t),σ(t) 2 ] and update σ(t) anda as follows: σ(t +1) 1 Ω Ω (ξ g ) 2 ρ (ξ g, α(t),σ(t) 2 )dξ, (27) A N (ξ ξ ) 2 ρ (ξ,ξ g,α(t),σ(t) 2 )dξdξ. (28) Step 3: Run the algorthm LBP[ g,α,0] for varous postve values of α and set α(t +1)tothevalueof α whch satsfes the follows equaton: and t t +1. N (ξ ξ ) 2 ρ (ξ,ξ g, α, 0)dξdξ = A, (29) Step 4: Update ( α, σ ) ( α(t),σ(t) ). (30) Stop f the values of α and σ converge, and return to Step 2 otherwse. Because both probabltes n Eqs.(16) and (17) are mult-dmensonal Gaussan dstrbuton, we can calculate the some statstcal quanttes and the margnal lkelhood, exactly, by means of the multdmensonal Gaussan ntegral formula and the dscrete Fourer transformaton[22]. Hence we can check the accuracy of the belef propagaton for the probablstc mage processng based on the Bayesan statstcs and the maxmum lkelhood estmaton. We show n Fgure 2 and Table 1 results of numercal experments of nose reducton from the degraded mage whch s generated by the addtve whte Gaussan nose wth the mean 0 and the varaton The estmates of α and σ gven n Table 1 are obtaned by usng the E algorthm. The process for the convergence of (α(t),σ(t)) (σ(t) β(t)) n the E algorthm s shown n Fgure 3. The open and the sold crcles are correspondng to the results n the E algorthm for the loopy belef propagaton and the exact calculaton, respectvely. (a) (b) (c) (d) (e) Fgure 2: Nose reducton by usng the Gaussan graphcal model and the belef propagaton. (a) Orgnal mage. (b) Degraded mage by the addtve whte Gaussan nose wth the mean 0 and the varance (c) Restored mage n the loopy belef propagaton. (d) Restored mage n the generalzed belef propagaton. (e) Restored mage n the exact calculaton by means of mult-dmensonal Gaussan ntegral formula and the dscrete Fourer transformaton. The loopy belef propagaton can be extended to a generalzed belef propagaton by means of the cluster varaton method whch s one of famlar advanced mean-feld methods n the statstcal mechancs[11]. The estmates of hyperparameters, α and σ, and the restored mage obtaned by replacng the loopy belef 8

9 Table 1: Values of estmates ( α, σ), mean-squareerrorbetweenthe orgnalmage a andthe restoredmage a, SE( a, a) 1 L a a 2 and the mprovement of sgnal to nose rato, SNR 10log 10 ( a d 2 a a 2 ) (db), n the nose reductons by means of the Gaussan graphcal model gven n Fgure 2. α σ SE( a, a) SNR (db) Fgure 2(c) Fgure 2(d) Fgure 2(e) α(t) Exact Loopy BP σ(t) Fgure 3: Convergence of (α(t),σ(t)) (σ(t) β(t)) n the E algorthm. The open crcle and the sold crcle are correspondng to the results n the E algorthm for the loopy belef propagaton and the exact calculaton, respectvely. propagaton by the generalzed belef propagaton n the scheme of the present secton are also shown n Table 1 and Fgure 2. The results are more close to the exact results than the loopy belef propagaton. The detals have been gven n Ref.[27]. 5 Concludng Remarks In the present tutoral lecture note, we have brefly explaned the formulaton of the probablstc mage processng based on the arkov random felds, the Bayesan networks and the loopy belef propagaton. As an examples of the applcatons of the probablstc mage processng, we have revewed the nose reducton by means of the Gaussan graphcal model as arkov random felds. The accuracy of the loopy belef propagaton has been nvestgated for the Gaussan graphcal model. As s one of mportant results n the nvestgaton of the loopy belef propagaton, t s well known that the loopy belef propagaton can gve us the exact results for the average n the Gaussan graphcal model, though the varance and the co-varance mght be approxmate ones[7]. Ths fact s vald not only n the loopy belef propagaton but also the generalzed belef propagaton for the Gaussan graphcal model[27]. References [1] D. Geman, Random Felds and Inverse Problems n Imagng, Lecture Notes n athematcs, no.1427, pp , Sprnger-Verlag,

10 [2] R. Chellappa and A. Jan (eds), arkov Random Felds: Theory and Applcatons, Academc Press, New York, [3] S. Z. L, arkov Random Feld odelng n Computer Vson, Sprnger-Verlag, Tokyo, [4] A. S. Wllsky, ultresoluton arkov odels for Sgnal and Image Processng, Proceedngs of IEEE, vol.90, no.8, pp , [5] D.. Chckerng, D. Heckermana and C. eek, Large-Sample Learnng of Bayesan Networks s NP-Hard, Journal of achne Learnng Research vol.5, pp , [6] J. Pearl: Probablstc Reasonng n Intellgent Systems: Networks of Plausble Inference, organ Kaufmann, [7] Y. Wess and W. T. Freeman, Correctness of belef propagaton n Gaussan graphcal models of arbtrary topology, Neural Computaton, vol.13, no.10, pp , [8] F. R. Kschschang, B. J. Frey and H.-A. Loelger, Factor graphs and the sum-product algorthm, IEEE Trans. Inform. Theory, vol.47, no.2, pp , February [9] Y. Kabashma and D. Saad: Belef propagaton vs. TAP for decodng corrupted messages, Europhyscs Letters, vol.44, no.5, pp , [10]. Opper and D. Saad (eds): Advanced ean Feld ethods Theory and Practce, IT Press, [11] J. S. Yedda, W. T. Freeman and Y. Wess: Constructng Free-Energy Approxmatons and Generalzed Belef Propagaton Algorthms, IEEE Transactons on Informaton Theory, vol.51, no.7, pp , [12] S. Ikeda, T. Tanaka and S. Amar: Stochastc reasonng, free energy, and nformaton geometry, Neural Computaton, vol.16, no.9, pp , [13] S. Ikeda, T. Tanaka and S. Amar: Informaton geometry of turbo and low-densty party-check codes, IEEE Transactons on Informaton Theory, vol.50, no.6, pp , [14] W. T. Freeman, E. C. Pasztor, O. T. Carmchael, Learnng Low-Level Vson, Internatonal Journal of Computer Vson, vol.40, no.1, pp.25-47, [15] W. T. Freeman, T. R. Jones and E. C. Pasztor, Example-based super-resoluton, IEEE Computer Graphcs and Applcatons, vol.22, no.2, pp.56-65, [16] J. Sun, N.-N. Zheng and H.-Y. Shum, Stereo matchng usng belef propagaton, IEEE Transactons on Patten Analyss and achne Intellgence, vol.25, no.7, pp , [17] B. J. Frey and N. Joc: A comparson of algorthms for nference and learnng n probablstc graphcal models, IEEE Transactons on Pattern Analyss and achne Intellgence, vol.27, no.9, pp , 2005 [18] K. Tanaka, Statstcal-mechancal approach to mage processng (Topcal Revew), Journal of Physcs A: athematcal and General, vol.35, no.37, pp.r81-r150, [19] K. Tanaka, H. Shouno,. Okada and D.. Ttterngton, Accuracy of the Bethe approxmaton for hyperparameter estmaton n probablstc mage processng, Journal of Physcs A: athematcal and General, vol.37, no.36, pp , [20] T. Heskes, On the unqueness of loopy belef propagaton fxed ponts, Neural Computaton, vol.16, pp , [21] D. J. ackay, Bayesan nterpolaton, Neural Computaton, vol.4, pp , [22] K. Tanaka and J. Inoue, axmum lkelhood hyperparameter estmaton for solvable arkov random feld model n mage restoraton, IEICE Transactons on Informaton and Systems, vol.e85-d, no.3, pp , arch [23] K. Tanaka, Statstcal-mechancal Iteratve Algorthm by means of cluster varaton method n compound Gauss-arkov random feld model, Transactons of Japanese Socety for Artfcal Intellgence, vol.16, no.2, pp , [24] K. Tanaka, J. Inoue and D.. Ttterngton, Probablstc mage processng by means of Bethe approxmaton for Q-Isng model, Journal of Physcs A: athematcal and General, vol.36, no.43, pp , [25] H. Nshmor, Statstcal Physcs of Spn Glasses and Informaton Processng: An Introducton, Oxford Unversty Press, Oxford, [26] J. Inoue and K. Tanaka, Dynamcs of the maxmum lkelhood hyper-parameter estmaton n mage restoraton: Gradent descent versus expectaton and maxmzaton algorthm, Physcal Revew E, vol.65, no.1, Artcle No , pp.1-11, [27] K. Tanaka, Generalzed belef propagaton formula n probablstc nformaton processng based on Gaussan graphcal odel, IEICE Transactons (D-II), vol.j88-d-ii, 2005, n press. 10