A quantum-statistical-mechanical extension of Gaussian mixture model

Transcription

1 A quantum-statstcal-mechancal extenson of Gaussan mxture model Kazuyuk Tanaka, and Koj Tsuda 2 Graduate School of Informaton Scences, Tohoku Unversty, Aramak-aza-aoba, Aoba-ku, Senda , Japan 2 Max Planck Insttute for Bologcal Cybernetcs, Spemannstrasse 38, Tübngen, Germany E-mal: kazu@smappstohokuacjp Abstract We propose an extenson of Gaussan mxture models n the statstcal-mechancal pont of vew The conventonal Gaussan mxture models are formulated to dvde all ponts n gven data to some knds of classes We ntroduce some quantum states constructed by superposng conventonal classes n lnear combnatons Our extenson can provde a new algorthm n classfcatons of data by means of lnear response formulas n the statstcal mechancs Introducton Statstcal approaches are appled to data mnng n computer scences Data n real worlds nclude some fluctuatons and t s expected to employee some statstcal methods to do systematcal approaches for the treatments of such data Many statstcal approaches for nferences n data mnng are based on Bayesan formulas and maxmum lkelhood[] One of fundamental approaches s a classfcaton by means of Gaussan mxture models In Gaussan mxture models, each data pont yelds from one of Gaussan dstrbutons wth averages and varances If we hope to dvde a gven data to three knds of classes, we have to prepare three knds of Gaussan dstrbutons For the classfcaton of gven data by means of Gaussan mxture models, a temperature has been ntroduced n order to search the maxmal pont of hyperparameters for the margnal lkelhood by usng an teratve procedure[2] The procedure s proposed wth beng based on an dea of smulated annealng methods n the Markov random felds As one of the other physcal approaches to fnd optmal solutons for massve probablstc nferences, we have quantum annealng method Quantum annealng method s formulated by replacng thermal effect by quantum effect n the smulated annealng[3, 4, 5, 6, 7] Quantum annealng has been ntroduced n the probablstc mage processng as well as optmzaton problems[8] It has been proved that the quantum annealng can fnd an optmal soluton more quckly than the smulated annealng[9] Moreover, some quantum error correctng codes are proposed and some performance lmts have been gven by usng a gauge theory n the physcs[0,, 2] On the other hand, an edge state n mage processng has been extended to a quantum state[3] Thus, many physcsts are nterested n applcatons of quantum fluctuatons to computer scences c 2008 Ltd

2 In applyng quantum effects to computer scences, we have manly two dfferent vewponts One of them s an annealng and s appled to fnd an optmal soluton n massve computatonal problems The other s to adopt a quantum state tself as a state n computatonal models or probablstc models Our data often nclude a pont to whch t s dffcult to assgn one label n the statstcal classfcaton It s nterestng to ntroduce quantum states as label states n the classfcatons n both statstcal and physcal vewponts In ths paper, we extend conventonal Gaussan mxture models to quantum Gaussan mxture models and gve an algorthm to determne the estmates of hyperparameters n our proposed models In secton 2, we summarze conventonal Gaussan mxture models and the scheme of determnaton of hyperparameters n the statstcal framework In secton 3, we propose quantum Gaussan mxture models The probablty densty functons of conventonal Gaussan mxture models are expressed n terms of densty matrx representatons Densty matrx representatons of conventonal Gaussan mxture models are gven by dagonal matrces We ntroduce offdagonal elements n the reformulated densty matrx representaton of Gaussan mxture model In order to derve the extremum condton of the margnal lkelhood n quantum Gaussan mxture model wth respect to hyperparameters, we adopt lnear response formulas for densty matrces In sectons 4 and 5, we gve some numercal experments and concludng remarks, respectvely 2 Conventonal Gaussan Mxture Model In the present secton, we explan a classfcaton of gven data by usng the conventonal Gaussan mxture model The framework s based on maxmum lkelhood estmaton and Bayesan formula n the statstcs We ntroduce a set of data consstng of N real number y 0,y,,y The set of data s expressed n terms of an N-dmensonal column vector y =(y 0,y,,y ) t We consder to classfy the gven data to K knds of classes A label assgned to y s expressed by x andthe set of labels s expressed n terms of an N-dmensonal column vector x =(x 0,x,,x ) t The varable y can take any real value n the nterval (, + ) An ntegers of 0,, 2,,K s assgned to the label varable x Now we consder to nfer how a gven data s classfed to K knds of classes by means of Bayes formula and maxmum lkelhood estmaton x and y can be regarded as sets of random varables for parameters and data, respectvely We assume that a set of parameters x s generated by accordng to the followng pror probablty: P(x α) = α k δ x,k, () K k=0 where the set α {α 0,α,,α K } satsfes the condton K k=0 α k =andδ a,b s Kronecker s delta Gven data y are generated from parameters x by accordng to the followng condtonal probablty: P(y x, μ, σ) = g x (y μ, σ), (2) g k (y μ, σ) exp( 2πσk 2σ 2 (y μ k ) 2 ) (3) k In the statstcs, the set μ {μ 0,μ,,μ K }, σ {σ 0,σ,,σ K } as well as α {α 0,α,,α K } are referred to as hyperparameters Equatons () and (2) gve us the followng 2

3 jont probablty of x and y: P(x, y α, μ, σ) =P(y x, μ, σ)p(x α) = α x g x (y σ, α) (4) In the statstcal pont of vew, the estmates of α, μ and σ, α { α 0, α,, α K }, μ { μ 0, μ,, μ K } and σ { σ 0, σ,, σ K }, can be determned from gven data y so as to maxmzng the margnal lkelhood defned by P(y α, μ, σ) = K K x 0 =0x 2 =0 K x =0 P(x, y α, μ, σ) = α x g x (y σ, α) (5) K x =0 The extremum condton of P(y α, μ, σ) wth respect to α, μ, σ can be gven as μ k = y Ψ (k μ, σ, α) Ψ (k μ, σ, α), (6) α k = Ψ (k μ, σ, α), (7) N σ k 2 = (y μ k ) 2 Ψ (k μ, σ, α) Ψ, (8) (k μ, σ, α) Ψ (k μ, σ, α) α k g k (y μ, σ) K k=0 α kg k (y μ, σ) (9) By solvng equatons (6)-(8) n the teraton method numercally, we obtan the estmates μ, σ and α for gven data y By usng Bayesan formula, we derve a posteror probablty of the set of parameter x for gven data y as follows: P(x y, μ, σ, α) = P(x, y μ, σ, α) P(y μ, σ, α) = Ψ (x μ, σ, α) (0) The estmates of x, x ( x, x 2,, x ) t can be determned so as to maxmzng the posteror probablty P(x y, μ, σ, α) wth respect to x 3 Quantum Gaussan Mxture Model In ths secton, we extend conventonal Gaussan mxture models to quantum Gaussan mxture models Our extenson s based on the constructon of quantum states by superposng conventonal classes n the lnear combnatons We gve also the determnaton of hyperparameters by means of the maxmzaton of lkelhood for the quantum Gaussan mxture models For any matrx A, the exponental functon e A and the logarthm functon ln(a) are defned by e A + n=0 n! An, lna + n= n (I A)n () 3

4 We ntroduce the followng two K K real dagonal matrces F and G(y ): lnα lnα 0 F 0 0 lnα K, (2) g 0 (y ) g (y ) 0 G(y ) 0 0 g K (y ) (3) The margnal lkelhood P(y μ, σ, α) n equaton (5) s replaced by the followng densty matrx: where P(y μ, σ, α) = tr e H(y ), (4) tr e F H(y ) F lng(y ) ln(α 0 g 0 (y )) ln(α g (y )) 0 = 0 0 ln(α K g K (y )) (5) Now we extend the dagonal matrx F to any K K real symmetrc matrx as follows: lnα 0 γ γ γ lnα γ F = γ γ lnα K (6) The N N matrx H(y ) can be rewrtten as K K H(y ) B () kk X kk k=0 k =0 ln(α 0 g 0 (y )) γ γ γ ln(α g (y )) γ = γ γ ln(α K g K (y )), (7) where X k,k defnes by s a K K matrx whose (l, l )-component l X kk l (l, l =0,,,K ) are l X kk l δ k,l δ k,l, (8) 4

5 and the coeffcents B k,k (k, k =0,,,K ) are defned by B () kk ln(α kg k (y )), B () kk γ (k k ) (9) The functon P(y μ, σ, α) defned by equatons (4) and (7) does not always satsfy the normalzaton condton P(y μ, σ, α)dy 0dy dy =, so that t s not regarded as a probablty densty functon of data set y In the present paper, we regard the functon P(y μ, σ, α) as a margnal lkelhood approxmately and formulate a quantummechancal extenson of Gaussan mxture model n the present paper By usng the lnear response theory, we have B () kk ln(tr e H(y) )= tr e H(y ) tr 0 e ( λ)h(y) X kk e λh(y) dλ = tr X kk e H(y ) tr e H(y (20) ) The extremum condtons for the margnal lkelhood P(y μ, σ, α) wth respect to μ, σ and α can be derved as μ k = y ( trx kke H (y ) ) tr e H (y ) ( trx kke H (y ) ), (2) tr e H (y ) σ k 2 = (y μ k ) 2 ( trx kke H (y ) ( trx kke H (y ) ) tr e H (y ) tr e H (y ) ), (22) ( α k =exp tr X k,k ln( e H(y ) ) N tr e H(y ) ) (23) For gven data y, we obtan estmates of μ, σ and α by means of the teraton method numercally We remark that equatons (2)-(23) can be reduced to equatons (6)-(8) by settng γ =0 When we ntroduce the energy matrx H(y ), the posteror dstrbuton P(x y, μ, σ, α) can be also replaced by P (y, μ, σ, α) = e E (24) tr e E E H(y 0 ) I I I + I H(y ) I I +I I H(y 2 ) I + + I I I H(y ) (25) The estmates of label for y can be gven as an K-dmensonal egenvector x = ( x (0), x (),, x (K ) ) t whch corresponds to the mnmal egenvalue of K K local energy matrx H(y ) Here all egenvectors should be normalzed n ther absolute values The egenvector whch corresponds to the mnmal egenvalue of KN KN global energy matrx E s gven by a KN-dmensonal vector x = x 0 x x 2 x (26) 5

6 4 Numercal Experments In ths secton we gve some numercal experments for estmatng the hyperparameters α, μ and σ when the data y s set to a standard mage The obtaned set of label can be regarded as a segmented mage for a gven standard mage By usng equatons (2)-(23), we do some numercal experments for a monochrome mage gven n fgure (a) In table, we gve estmates of hyperparameters, α, μ and σ and the values of logarthm of margnal lkelhood, L(y, α, μ, σ) N lnp(y α, μ, σ) fork =3 Forthe estmates, α = α, μ = μ and σ = σ, egenvectors x =( x (0), x (), x (2) ) t whch corresponds to the mnmal egenvalue of 3 3 matrxh(y ) are drawn n fgure (b)-(c) In fgure (b)-(c), x =(, 0, 0) t, x =(0,, 0) t and x =(0, 0, ) t correspond to red, green and blue, respectvely (a) (b) (c) (d) Fgure Image segmentaton based on the quantum Gaussan mxture model for K = 3 (a) Orgnal mage (256 grades, N = ) (b) Segmented mage x for γ = 0 (c) Segmented mage x for γ =02 (d) Segmented mage x for γ =04 The probablty densty functon P(y α, μ, σ) tr e H(y) /tr e F and hstogram for the gven mage y are shown n fgure 2 In the curves of fgure 2, the values of γ are set to 0, 02, and 04, respectvely In γ =02, we can fnd a very soft peak at y = 72 around The soft peak nclude some regons whch are segmented as red areas for γ =02 and are segmented as green areas 5 Concludng Remarks In ths paper, we have gven an extenson of the probablty densty functon for conventonal Gaussan mxture model to a densty matrx The formulaton s based on Bayesan statstcs and the maxmum lkelhood method In the estmaton of hyperparameters, we have to derve 6

7 Table Estmates of hyperparameters, α, μ and σ and values of L(y, α, μ, σ) N lnp(y α, μ, σ) γ L(y, α, μ, σ) α k μ k σ k k = k = k = k = k = k = k = k = k = the extremum condtons and we employee the lnear response formulas for quantum-statstcal models We have constructed the estmaton algorthm as an teratve procedure and have gven some numercal experments The functon P(y μ, σ, α) defned by equatons (4) and (7) does not always satsfy the normalzaton condton so that t s not regarded as a probablty densty functon of data set y In order to guarantee the normalzaton condton, we have to defne P(y μ, σ, α) = tr e H(y ) + tr, (27) e H(z) dz nstead of equaton (4) We should formulate our quantum-mechancal extenson for equaton (27) However, t may be dffcult to calculate the ntegrals + tr e H(z) dz analytcally It remansasafutureproblem Densty matrces nclude some quantum effects and are based on states constructed by superposng some knds of classcal states For example, when we denote three possble classcal states n terms of three-dmensonal vectors (, 0, 0) t,(0,, 0) t and (0, 0, ) t, all possble quantum states are expressed n terms of lnear combnatons of the three vectors n every extenson to densty matrx In some capactes, quantum statstcal models are expected to be beyond the conventonal statstcal models For example, quantum statstcal models may gve us new optmal solutons n statstcal nferences In fact, our numercal experments yelds some nce segmentaton results by ntroducng quantum states n the Gaussan mxture model Quantum Gaussan mxture model has succeeded n splttng some regons from the background, though the conventonal Gaussan mxture model cannot splt the correspondng regons from the background In the present problems n ths paper, we assume not to nclude nteractons between any pars of components y and y j n gven data y = (y 0,y,,y ) t Moreover we do not consder nteractons between any pars of elements n the set of labels Thus the densty matrx n equaton (4) has been factorzed wth respect to each Though the energy matrx H n equaton (25) seems to be an NK NK matrx, the rank s just K Such problems can be regarded as one-body problems n the statstcal mechancs If we consder nteractons between some pars of components, n the gven data or n the set of label to estmate, t s hard to express the densty matrx n terms of a factorzable form as shown n equaton (4) When 7

8 002 P(y α,μ,σ) ^ ^ ^ 00 γ = P(y α,μ,σ) ^ ^ ^ γ = 02 y 002 P(y α,μ,σ) ^ ^ ^ y γ = y Fgure 2 Curves of the probablty densty functon P(y α, μ, σ) tr e H(y) /tr e F and hstogram for the gven mage y n fgure (a) The curves of red lne show the values of P(y α, μ, σ) for varous values of y n the nterval [0, 255] The hyperparameter γ s set to 0, 02 and04 The hstogram of the gven mage y n fgure s expressed n terms of green area n each graph the energy matrx H sgvennsuchaway,therankofh s not K any more and we have to dagonalze the NK NK matrx H Many authors have nvestgated belef propagaton and the other advanced mean-feld methods to statstcal nferences[5, 6, 7, 8, 9] It s nterestng to apply quantum belef propagaton and the other advanced quantum mean-feld methods to such cases as an approxmate algorthm Suzuk et al nvestgated some quantum annealng algorthms by means of a quantum statstcal verson of Bethe approxmaton[20] One of quantum statstcal-mechancal extensons of belef propagatons corresponds to a quantum 8

9 cluster varaton method[2] It s nterestng to apply a quantum cluster varaton method to statstcal nferences wth quantum states and some nteractons n the computer scences Ths s one of future problems Acknowledgements Ths work was partly supported by the Grants-In-Ad (No , No and No ) for Scentfc Research from the Mnstry of Educaton, Culture, Sports, Scence and Technology of Japan References [] Bshop C M 2006 Pattern Recognton and Machne Learnng (New York: Sprnger) [2] Ueda N and Nakano R 998 Neural Networks 27 [3] Kadowak T and Nshmor H 998 Phys Rev E [4] Brooke J, Btko D, Rosenbaum T F and Aeppl G 999 Scence [5] Santoro G E, Martoňák R, Tosatt E and Car R 2002 Scence [6] Santoro G E and Tossat E 2006 J Phys A: Math Gen 39 R393 [7] Suzuk S and Okada M 2005 J Phys Soc Jpn [8] Tanaka K and Horguch T 997 IEICE Transactons (A), J80-A, 27 (n Japanese); translated n Electroncs and Communcatons n Japan, Part 3: Fundamental Electronc Scence, 83, 84 [9] Morta S and Nshmor H 2006 J Phys A: Math Gen [0] Denns E, Ktaev A, Landahl A and Preskll J 2002 J Phys A: Math Gen [] Nshmor H and Sollch P 2004 J Phys Soc Jpn [2] Takeda K and Nshmor H 2004 Nucl Phys B [3] Tanaka K 2002 J Phys A: Math Gen 35 R8 [4] Nshmor H 200 Statstcal Physcs of Spn Glasses and Informaton Processng, An Introducton (New York: Oxford Unversty Press) [5] Opper M and Saad D (eds) 200 Advanced Mean Feld Methods Theory and Practce (Cambrdge: MIT Press) [6] Tanaka K 2003 IEICE Transactons on Informaton and Systems E86-D 228 [7] Ikeda S, Tanaka T and Amar S 2004 Neural Computaton [8] Yedda J S, Freeman W T and Wess Y 2005 IEEE Transactons on Informaton Theory [9] Pelzzola A 2005 J Phys A: Math Gen 38 R309 [20] Suzuk S, Nshmor H and Suzuk M 2007 Phys Rev E [2] Morta T 957, J Phys Soc Jpn