2010 Inernaonal Conference on Paern Recognon Foreground Segmenaon va Bacground Modelng on Remannan Manfolds Ru Casero, João F. Henrques and Jorge Basa Insue of Sysems and Robocs, DEEC-FCTUC, Unversy of Combra, Porugal Absrac Sascal modelng n color space s a wdely used approach for bacground modelng o foreground segmenaon. Neverheless, somemes compung such sascs drecly on mage values s no enough o acheve a good dscrmnaon. Thus he mage may be convered no a more nformaon rch form, such as a ensor feld, n whch can be encoded color and gradens. In hs paper, we explo he heorecally well-founded dfferenal geomercal properes of he Remannan manfold where ensors le. We propose a novel and effcen approach for foreground segmenaon on ensor feld based on daa modelng by means of Gaussans mxures (GMM) drecly n he ensor doman. We nroduced a Expecaon Maxmzaon (EM) algorhm o esmae he mxure parameers, and are proposed wo algorhms based on an onlne K-means approxmaon of EM, n order o speed up he process. Theorec analyss and expermenal evaluaons demonsrae he promse and effecveness of he proposed framewor. 1 Inroducon Foreground segmenaon can be descrbed as he process ha subdvdes an mage no regons of neres and bacground. Ths as usually reles on he exracon of suable feaures from he mage ha are hghly dscrmnave. Mos of he foreground segmenaon echnques are based on nensy or color feaures. However here are suaons, where hese feaures may no be dsnc enough o acheve a good dscrmnaon (e.g. dynamc scenes). Thus he mage may be convered no a more nformaon rch form, such as a ensor feld, o yeld laen dscrmnang feaures (e.g. gradens, flers responses). Texure s one of he mos mporan feaures, herefore s consderaon can grealy mprove mage analyss. The srucure ensor [2] [7] has been nroduced for such exure analyss provdng a measure of he presence of edges and her orenaon. Smple aemps a ensors sascal analyss are based on a Gaussan model of he lnear ensor coeffcens. However he space of ensors does no form a vecor space, hus sandard lnear sascal echnques do no apply. We propose o accoun for he Remannan geomery of he ensor manfold when compung he probably dsrbuons used Ths wor was funded by BRISA, Auo-esradas de Porugal, S.A. n segmenaon, preservng he naural properes of he ensors. In order o explo he nformaon presen n all he componens of he srucure ensor, a bacground modelng mehod for ensor-value daa s presened based on he defnon of mxure of Gaussans over ensor felds. 2 Srucure Tensor The combnaon of color and exure feaures can mprove segmenaon performance. In order o exrac suable nformaon from an mage, we focus on he combnaon of hese wo feaures, encoded by means of a srucure ensor [7]. Wh regard o he exracon of he exure nformaon, we use a graden-based mehod. For vecor-valued mages (color mages, n our wor RGB) he srucure ensor s defned as [T = K ρ (vv T )] wh v = [I x, I y, I r, I g, I b ], where I s a vecor-valued mage, K ρ s a Gaussan ernel wh sandard devaon ρ, (I x, I y ) are he paral dervaes of he gray mage, and (I r, I g, I b ) are he color componens n RGB space. The ensor feld obaned drecly from he mages s nosy and needs o be regularzed before beng furher analyzed. A nave bu smple and ofen effcen regularzaon mehod s smoohng wh a Gaussan ernel. 3 Remannan Manfolds Henceforh, S + d denoes a (d d) symmerc posve defne marx, S d s a symmerc marx and N d s a Gaussan dsrbuon of dmenson d wh zero mean. I well-nown ha S + d (ensors) do no conform o Eucldean geomery, because he space of S + d s no a vecor space, e.g., he space s no closed under mulplcaon wh negave scalers. Insead, S + d form (les on) a conneced Remannan manfold (dfferenable manfold equpped wh a Remannan merc). A manfold s a opologcal space whch s locally smlar o an Eucldean space. Le M be a opologcal n-manfold (see Fgure 1). A coordnae char on M s a par (U, ϕ), where U s an open subse of M and ϕ: U Ũ s a homeomorphsm from U o an open subse Ũ = ϕ(u) Rn. Gven a char (U, ϕ) he se U s called a coordnae doman. The map ϕ s denomnaed as (local) coordnae map, and he componen funcons of ϕ are called local coordnaes on U,.e., for any pon p M, ϕ(p) = x = (x 1,..., x n ) T s he local coordnae represenaon of p. See more deals n [3] [10] [16] [15] [11]. 1051-4651/10 $26.00 2010 IEEE DOI 10.1109/ICPR.2010.871 3558 3574 3570
Fgure 1: The geodesc γ() defned by he sarng pon p and he nal velocy γ(0). The endpon q = γ(1) s compued by applyng he exponenal map, such ha q = exp p [ γ(0)], see [16]. A Remannan manfold (M, G) s a dfferenable manfold M endowed wh Remannan merc G. A Remannan merc s a collecon of nner producs <.,. > p, defned for every pon p of M, on he angen space (T p M) of M a p. I can be represened as a (n n) symmerc, blnear and posve-defne form G, ha assocaes o each pon p M, a dfferenable varyng nner produc <.,. > p on he angen space (T p M) a p. The angen space T p M s smply he vecor space, aached o p, whch conans he angen vecors o all curves on M passng hrough p,.e., he se of all angen vecors a p. More precsely, le γ(): I = [a, b] R M denoe a curve on he manfold M passng hrough γ(a) = p M. The angen vecor a p s represenaed by γ(a) = dγ(a)/d. The dervaves of all possble curves compose he T p M. Gven a coordnae char (U, ϕ), wh ϕ(p) = (x 1,..., x n ) T, s possble o deermne a bass n he angen space T p M denoed by ( / x 1,..., / x n ), and also wren ( 1,..., n ). So ha any elemen of he T p M can be expressed n he form x () / x. I follows ha, G p, called he local represenaon of he Remannan merc a p can be defned hrough he nner producs G p = gp j =< / x, / x j > p. Therefore he nner produc of wo angen vecors u, v T p M s hen expressed as < u, v > p = u T G p v, nducng a norm for he angen vecors n he angen space such ha u 2 p =< u, u > p. Dsances on manfolds are defned n erms of mnmum lengh curves beween pons. The geodesc beween wo pons γ(a) and γ(b) on a Remannan manfold s locally defned as he mnmum lengh curve γ(): I = [a, b] R M over all possble smooh curves on he manfold connecng hese pons. Ths mnmum lengh s called geodesc/nrnsc dsance. The angen vecor γ() defnes he nsananeous velocy of he curve and s norm γ() = γ(), γ() 1/2 γ() s he nsananeous speed. The geodesc dsance can be calculaed negrang γ() along γ. Tang I = [0, 1] for smplcy, and le γ(0) = p, gven a angen vecor γ(0) T p M here exss a unque geodesc γ() sarng a p wh nal velocy γ(0). Therefore he geodesc γ() s unquely defned by s sarng pon p and s nal velocy γ(0). The endpon γ(1) can be compued by applyng he exponenal map a p, such ha γ(1) = exp p ( γ(0)). Two maps are defned for mappng pons beween he manfold M and a angen plane T p M. The exponenal map exp p : T p M M, defned on he whole T p M, s a mappng beween he T p M and he correspondng manfold M. I maps he angen vecor γ(0) a pon p = γ(0) o he pon of he manfold q = γ(1) ha s reached by he geodesc a me sep one. The nverse of he exponenal map s gven by he logarhm map and denoed by log p : M T p M. I maps any pon q M o he unque angen vecor γ(0) a p = γ(0) ha s he nal velocy of he unque geodesc γ() from p = γ(0) o q = γ(1). In oher words, for wo pons p and q on he manfold M he angen vecor o he geodesc curve from p o q s defned as γ(0) = log p (γ(1)). I follows ha, he geodesc dsance D(p, q) s gven by D(γ(0), γ(1)) = γ(0), γ(0) 1/2 γ(0) (1) The velocy γ(0) can be compued [11] from he graden of he squared geodesc dsance wh respec o γ(0) 4 Tensor Sascs γ(0) = γ(0) D 2 (γ(0), γ(1)) (2) We defne an mage-ensor T as T: Ω R 3 S + d, where Ω s he orgnal color mage (3 rd dmenson represen he color channels), T(x, y) s a pxel-ensor n mage poson (x, y) and S + d denoes he space of (d d) symmerc posve defne marces (d = 5 n our case). Snce (d d) symmerc marces have only n = (d/2)(d+1) ndependen componens, applyng a local coordnae char ϕ: S + d Rn s possble o assocae o each ensor T(x, y) S + d s n ndependen componens, such ha S + d s somorphc o Rn (n = 15 n our case). A ensor can be undersood as he parameers (covarance marx) of a d-dmensonal normal dsrbuon. Therefore our goal s o defne sascs beween mulvarae normal dsrbuons and apply o ensor daa. In order o acheve hs goal, we need frs o defne, he mean and covarance marx over a se of ensors, and he respecve probably densy funcon. See more deals abou hs secon n, [11] [4] [5] [13] [12]. Fgure 2: Depcon of he velocy feld β a he emprcal mean p. The blac doed lnes represen he geodescs beween each par (p, p), he red arrows represen nal angen vecors β, see [13]. As defned by Fréche n [9] he emprcal mean ensor T, over a se of N random ensors {T }, s defned as he mnmzer T = T of he expecaon E[D 2 (T, T )] and he emprcal covarance marx Λ, wh respec o he mean en- 3559 3575 3571
sor T, s esmaed as E[D 2 1 N (T, T )] = D 2 (T, T ) N =1 Λ = 1 (3) N ϕ(β )ϕ(β ) T N =1 where β = TD 2 ( T, T ). The Gaussan law on he ensor manfold s defned as follows «p(t T, 1 Λ) = p (2π)n Λ exp ϕ(β)t Λ 1 ϕ(β ) (4) 2 Ths characerzaon of S + d hrough s sascal parameers allow us o derve sascs on ensors based on dfferen mercs. Nex, we wll apply hese conceps o he Remannan merc suded n hs paper (Affne-Invaran). Namely, we wll descrbe he geomery of S + d equpped wh a merc derved from he Fsher nformaon marx [17], from whch can be nduced a geodesc dsance D g. 4.1 Geodesc Merc : Affne-Invaran Usng he fac ha he manfold N d can be denfed wh he manfold of S + d marces, a Remannan merc on S+ d can be nroduced n erms of he Fsher nformaon marx [17]. Thus s possble o nduce several properes of S + d and derve a Gaussan law on ha manfold. An Affne- Invaran Remannan merc [5] for he space of mulvarae normal dsrbuons wh zero mean X S + d s gven by g j X =, j X = 1 2 r(x 1 X 1 j ) (5) Recallng ha ( / x 1,..., / x n ) = ( 1,..., n ) defne a bass n he angen space T X M. For any angen vecors u, v S d, n angen space T X M, her nner produc relave o pon X s gven by u, v X = 1 2 r(x 1 ux 1 v) (6) Le γ : [0, 1] R M be a curve n S + d, wh endpons γ(0) = X and γ(1) = Y, X, Y S + d. The geodesc defned by he nal pon γ(0) = X and he angen vecor γ(0) can be expressed [14] as ] γ() = exp X [ γ(0)] = X 1 2 exp [()X 1 2 γ(0)x 1 2 X 1 2 (7) whch n case of = 1 correspond o he exponenal map exp X : T X M M wh γ(1) = exp X ( γ(0)). The respecve logarhm map log X : M T X M s defned as γ(0) = log X (Y) = X log(y 1 X) (8) Noce ha hese operaors are pon dependen where he dependence s made explc wh he subscrp. The geodesc dsance D g (X, Y) beween wo pons X, Y S + d, nduced by he Affne-Invaran Remannan merc, derved from Fsher nformaon marx was proved (Theorem: S.T.Jensen, 1976, see n [1]) o be gven as 1 D g (X, Y) = 2 r(log2 (X 1 2 YX 1 2 )) (9) Ths merc exhbs all he properes necessary o be a rue merc such ha, posvy, symmery, rangle nequaly and s also affne nvaran and nvaran under nverson. Wh regard o he graden of he squared geodesc dsance s equal o he negave of nal velocy γ(0) ha defne he geodesc [14] and s wren as X D 2 g(x, Y) = X log(y 1 X) (10) Usng hs merc as soon as N > 2, a closed-form expresson for he emprcal mean T g of a se of N ensors {T } S + d canno be obaned. The mean s only mplcly defned based n he fac ha he Remannan barycener exss and s unque for he manfold S + d. In he leraure [13], hs problem s solved eravely, for nsance usng a Gauss- Newon mehod (graden descen algorhm) gven by T +1 g = exp T g (V) (11) Usng he exponenal map equaon (7) proposed we oban T +1 g = ( T [ g) 1 2 exp ( T ] g) 1 2 V( T g ) 1 2 ( T g) 1 2 (12) where V s a angen vecor ( = 1), see equaon 8, gven by he graden of he varance. V = 1 NX T N g D 2 g( T g, T ) = 1 N T NX g log(t 1 T g) =1 =1 (13) Ths opmzaon mehod has he advanage of havng a fas convergence speed, le all Newon mehods (converge very fas : 5-10 eraons). The covarance marx Λ g can be esmaed pluggng β = Tg D 2 g( T g, T ) = T g log(t 1 T g ) no equaon (3). 5 Bacground Modelng In order o model he bacground we use a mxure of K Gaussans on ensor doman as proposed n [6] for DT-MRI segmenaon. Based on he defnon of a Gaussan law on ensor space, we can defne a GMM as follows p(t Θ) = KX exp ` (1/2)ϕ(β, ) T Λ 1 ω p (2π)n Λ =1 ϕ(β,) (14) where each gaussan densy N (T T, Λ ) s a componen of he mxure. Each componen s characerzed by, a mxng coefcen ω (pror), a mean ensor T and a covarance marx Λ. Θ denoes he vecor conanng all he parameers 3560 3576 3572
of he gven mxure. The marx β, = T D 2 ( T, T ) depends on he chosen merc. The cluserng of daa lyng on he ensor manfold s posed as a maxmum lelhood esmaon problem, whch requres he defnon of a complee procedure o derve he mxure parameers. The Expecaon-Maxmzaon (EM) algorhm [8] s a general echnque for fndng maxmum lelhood esmaors n laen varable models, n cases where analyc soluons are dffcul or mpossble. I has been wdely used o esmae he mxure parameers due o s smplcy and numercal sably. Neverheless, an exac EM algorhm mplemenaon as proposed n [6] for DT- MRI segmenaon can be a cosly procedure. In order o reduce he processng me we propose wo algorhms based on an onlne K-means approxmaon of EM, adaped from he verson presened by Sauffer n [19]. The foreground deecon s performed n he same way as n [19]. 5.1 GMM - Kmeans Onlne Algorhm 1 The frs algorhm proposed s smlar o he Sauffer s algorhm [19] excep for he fac ha he pxel s modeled usng ensors nsead of vecors. The new mxure parameers combne he pror nformaon wh he observed sample. The basc dea s o aemp o mach he curren ensor observaon (T ) a a gven pxel wh one of he Gaussans for ha pxel-ensor. If a mach s found, he parameers of ha dsrbuon are updaed usng he curren observaon. If no mach s found, he Gaussan wh lower confdence s replaced wh a new one ha represens he curren observaon. The model parameers are updaed usng an exponenal decay scheme wh learnng raes (α and ρ). The mxure weghs are updaed usng ω = (1 α)ω 1 + (α)(m ), where M s 1 for he model whch mached and 0 for he remanng models. The parameers of he dsrbuon whch maches he new observaon (T ) are updaed as follows T 1 = (1 ρ) T + ρt (15) Λ = (1 ρ)λ 1 + ρϕ(β,) T ϕ(β,) (16) ρ = αn (T 1 T, Λ 1 ) β, = T D 2 ( T, T ) (17) The ensor mean ( T ) updae equaon presened (15) can only be drecly appled n he case of he convenonal Eucldean merc. As menoned prevously we need o ae no accoun he Remannan geomery of he ensor manfold o apply he geodesc merc (Affne-Invaran). We propose an approxmaon mehod o updae he ensor mean, based on he concep of nerpolaon beween wo ensors. The ensors nerpolaon can be seen as a wal along he geodesc jonng he wo ensors. A closed-form expresson s gven by he geodesc equaon (exponenal map), see equaon (7). In order o smplfy we change he noaon as follows ˆZ = T ˆX = T 1 = γ(0) [Y = T = γ(1)] (18) Le γ(): [0, 1] R M be he geodesc wh γ(0) = X and γ(1) = Y, defned by he nal pon γ(0) = X and he angen vecor γ(0). The pon Z on he ensor manfold s he nerpolaon beween X and Y a = ρ Y = exp X ( γ(0)) Z = exp X (ρ γ(0)) (19) γ(0) = log X (Y) = X log(y 1 X) (20) Pluggng eq. (20) no (7), he pon Z on he manfold ha s reached by he geodesc γ() a me = ρ s esmaed as h(ρ)x 1 2 ˆ X log(y 1 X) X 1 2 X 1 2 (21) In order o speed up he process we can smplfy h( ρ)x 1 2 log(y 1 X)X 1 2 X 1 2 (22) usng he fac ha log(a 1 BA) = A 1 log(b)a h ( ρ) log(x 1 2 Y 1 X 1 2 ) X 1 2 (23) and ha exp(a) = (exp(a)), where s a scalar Z = X 1 2 h h exp log(x 1 2 Y 1 X 1 ρ 1 2 ) X 2 (24) fnally, seeng ha exp(log(a)) = A, and afer some mahemacal smplfcaon urns no Z = X ( 1 p 2 ) Y ρ X ( 1 p 2 ) (25) 5.2 GMM - Kmeans Onlne Algorhm 2 Regardng o he second proposed algorhm, he mehod s somewha dfferen. Usng a fxed number of K dsrbuons/clusers, we assocae o each one of he clusers a buffer, where are sored he samples ha mach wh ha cluser. Represenng as N he number of samples sored n cluser a me, (S ) he sum of samples n all clusers, and s fxed a maxmum value for S gven by (Υ). The updae of he parameer (ω ) s performed n he same way as he frs algorhm. Wh regard o he parameers ( T ) and (Λ ) he new values are esmaed usng all he samples conaned n some cluser a some me. These parameers are calculaed usng he dervaons presened n secon 4 for empral mean and covarance. If a mach s found and (S = Υ), he cluser ha conans he oldes sample s esmaed. Ths sample s removed, and he parameers ( T ) and (Λ ) of he cluser are updaed. 6 Resuls In order o analyze he effecveness of he proposed mehods, we conduc several expermens on wo mage sequences presened n prevous leraure. The frs sequence (Sequence1) s he HghWayI sequence from ATON projec (hp://cvrr.ucsd.edu/aon/shadow/). The second scene (Sequence2) s he movng camera sequence from [18]. The 3561 3577 3573
Sequence 1 Sequence 2 Mehods T P R F P R T P R F P R GMM[Ir, Ig, Ib] 59 10 63 14 GMM[Tensor] EM 91 2.6 89 2.1 GMM[Tensor] KM1 86 4.4 84 4.2 GMM[Tensor] KM2 88 4.1 85 3.8 Table 1: True posve rao (TPR), False posve rao (FPR) groundruh foreground was obaned by manual segmenaon. In hs secon several resuls of applyng he proposed mehods o hese sequences are presened. The wdely-used vecor space mehod, namely GMM [19], s employed o compare wh he proposed ensor framewor. I s sressed ha no morphologcal operaors were used. Tradonal bacground modelng mehods assume ha he scenes are of sac srucures wh lmed perurbaon. Ther performance wll noably deerorae n he presence of dynamc bacgrounds. In dynamc scenes, alhough some pxels sgnfcanly changes over me, hey should be consdered as bacground. As shown n Fg.1, he radonal vecor mehod [19] can no accuraely deec movng objecs n dynamc scenes. I labels large numbers of movng bacground pxels as foreground and also oupu a huge amoun of false negaves on he nner areas of he movng objec. However, he proposed framewor can accuraely dsngush movng bacground pxels and rue movng objecs. The vecor GMM mehod a he begnnng of he sequences whch do no nclude foreground objecs performs poorly and deeced as foreground a lo of, bacground pxels. The reason for hs, s because hs mehod explo only sngle pxels color, and so need o ae longer me o ran he bacground models han he proposed mehods n order o accuraely deec he foreground pxels. Our mehod handles small dynamc bacground moons mmedaely and acheves accurae deecon, snce he proposed procedure uses feaures whch effecvely models he spaal correlaons of neghborng pxels. The spaal correlaons provde he subsanal evdence for labelng he cener pxel and hey are exploed o susan hgh levels of deecon accuracy. Fgure 3: Row op o boom : Orgnal frames ; GMM [Ir,Ig,Ib] ; GMM [Tensor] - KM1 7 Conclusons In hs paper, we proposed a new mehod o perform bacground modelng o foreground segmenaon, usng he concep of ensor. The ensor was used o combne color and exure nformaon. We revew he geomercal properes of he space of mulvarae normal dsrbuons wh zero mean and focus on he characerzaon of he mean, covarance marx and generalzed normal law on ha manfold. Tang no accoun he naural Remannan srucure of he ensor manfold, Gaussan mxures on ensor felds have been nroduced o approxmae he probably dsrbuon of ensor daa and hs probablsc modelng was employed o formulae segmenaon of ensor felds. We nroduced a EM algorhm o esmae he mxure parameers, and were proposed wo algorms based on a onlne K-means approxmaon of EM, n order o speed up he process. References [1] C. Anson and A. Mchell. Rao dsance measure. Sanhya: The Indan Journal of Sascs, 43(3):345 365, 1981. [2] J. Bgun and J. Wlund. Muldmensonal orenaon esmaon applcaons exure analyss opcal flow. IEEE Trans. on Paern Analyss Machne Inellgence, 13(8):775 790, 1991. [3] M. P. Carmo. Remannan Geomery. Brhauser, 1992. [4] C. A. Casaño-Moraga, C. Lengle, and R. Derche. A fas rgorous ansoropc smoohng mehod DT-MRI. ISBI, 2006. [5] C. A. Casaño-Moraga, C. Lengle, and R. Derche. A Remannan approach o ansoropc flerng of ensor felds. Sgnal Processng, 87(2):263 276, 2007. [6] R. de Lus Garca and C. Alberola-Lopez. Mxures of gaussans on ensor felds DT-MRI segmenaon. MICCAI, 2007. [7] R. de Lus Garca, M. Rousson, and R. Derche. Tensor processng for exure and colour segmenaon. Scandnavan Conference on Image Analyss, 2005. [8] A. P. Dempser, N. M. Lard, and D. B. Rubn. Maxmum lelhood from ncomplee daa va he EM algorhm. Journal of he Royal Sascal Socey, Seres B, 39(1):1 38, 1977. [9] M. Fréche. Les élémens aléaores naure quelconque dans espace dsancé. Ann. Ins. H. Poncaré, 10:215 310, 1948. [10] J. M. Lee. Inroducon o Smooh Manfolds. Sprnger, 2003. [11] C. Lengle, M. Rousson, and R. Derche. A sascal framewor for DTI segmenaon. IEEE ISBI, 2006. [12] C. Lengle, M. Rousson, R. Derche, and O. Faugeras. Sascs Mulvarae Normal Dsrbuons:Geomerc Approach Applcaon Dffuson Tensor MRI. INRIA, (RR-5242), 2004. [13] C. Lengle, M. Rousson, R. Derche, and O. Faugeras. Sascs Manfold Mulvarae Normal Dsrbuons Theory Applcaon Dffuson Tensor MRI. JMIV, 25(3):423 444, 2006. [14] M. Moaher. A Dfferenal Geomerc Approach o Geomerc Mean Symmerc Posve-Defne Marces. SIAM Journal on Marx Analyss Applcaons, 26(3):735 747, 2005. [15] X. Pennec. Inrnsc Sascs on Remannan Manfolds: Basc Tools for Geomerc Measuremens. Journal Mahemacal Imagng Vson, 25(1):127 154, 2006. [16] X. Pennec, P. Fllard, and N. Ayache. A Remannan Framewor for Tensor Compung. IJCV, 66(1):41 66, 2006. [17] C. Rao. Informaon accuracy aanable esmaon sascal parameers. Bullen Calcua Mahemacal, 37:81 91, 1945. [18] Y. Sheh and M. Shah. Bayesan modelng dynamc scenes for objec deecon. IEEE TPAMI, 27(11):1778 1792, 2005. [19] C. Sauffer and W. Grmson. Adapve bacground mxure models for real-me racng. IEEE CVPR, 1999. 3562 3578 3574