Motion Compensated Color Video Classification Using Markov Random Fields

Transcription

1 Moion Compenaed Color Video Claificaion Uing Markov Random Field Zolan Kao, Ting-Chuen Pong, John Chung-Mong Lee Hong Kong Univeriy of Science and Technology, Compuer Science Dep., Clear Waer Bay, Kowloon, Hong Kong, Tel: Fax: , Abrac. Thi paper deal wih he claificaion of color video equence uing Markov Random Field (MRF) aking ino accoun moion informaion. The heoreical framework relie on Bayeian eimaion aociaed wih MRF modelizaion and combinaorial opimizaion (Simulaed Annealing). In he MRF model, we ue he CIE-luv color meric becaue i i cloe o human percepion when compuing color difference. In addiion, ineniy and chroma informaion i eparaed in hi pace. The equence i regarded a a ack of frame and boh inra- and iner-frame clique are defined in he label field. Wihou moion compenaion, an iner-frame clique would conain he correponding pixel in he previou and nex frame. In he moion compenaed model, we add a diplacemen field and i i aken ino accoun in iner-frame ineracion. The diplacemen field i alo a MRF bu here are no iner-frame clique. The Maximum A Poeriori (MAP) eimae of he label and diplacemen field i obained hrough Simulaed Annealing. Parameer eimaion i alo conidered in he paper and reul are hown on color video equence uing boh he imple and moion compenaed model. 1 Inroducion Image claificaion i an imporan early viion ak where pixel wih imilar feaure are grouped ino homogeneou region. Many high level proceing ak (urface decripion, objec recogniion, for example) are baed on uch a preproceed image. Uing color informaion can coniderably improve capabiliie of image claificaion algorihm compared o purely ineniy-baed approache. However, we need a good color pace in order o ue color informaion in he ame way a human perceive color difference. There are everal meric propoed for compuer viion [5]. We ue he CIE-luv [5] color pace here becaue i eparae luminance and chroma informaion and i i eay o compue color difference in hi meric. The viual moion derived from a equence of ime-varying image [11, 4] i alo a valuable ource of informaion. Baically, i can be ued o deec moion in he cene bu i i alo poible o derive more deailed informaion uch a Thi reearch wa uppored by Hong Kong Reearch Gran Council under gran HKUST616/94E

2 poiion, orienaion of a viible urface or 3D reconrucion of a cene. Herein, we are inereed in compuing diplacemen vecor [11, 13, 4] in order o build a moion compenaed Markov Random Field (MRF) image claificaion model. When we have a equence of color image, ill image MRF model [2, 9, 12, 7] can be eaily exended o ake ino accoun he informaion in he previou and nex frame [13, 11] (ee Secion 2.1). Inead of a 2D neighborhood yem, we can ue a 3D one wih iner-frame clique. If he camera or he objec in he cene are no moving hen hi model yield good egmenaion. In he cae of moving objec, however, hi aic model can fail. To overcome he problem caued by moving objec, we inroduce a diplacemen field (DF) [13] in Secion 2.2 in order o ake ino accoun moion informaion in he label field. For impliciy, he DF i defined over he ame laice a he label field by placing a new laice beween wo neighboring frame. DF i a vecor-valued MRF giving he diplacemen vecor a each ie beween wo frame in he equence. The eimaion of he DF i done in parallel wih he label field and no exernal algorihm or iniializaion i needed. The energy funcion of he o-defined yem i minimized by he Meropoli algorihm [10]. The reul i he claificaion of he inpu frame and he diplacemen vecor beween frame. Uually, MRF-baed egmenaion mehod uffer from a lack of parameer eimaion. The majoriy of he propoed mehod are upervied, which limi heir pracical ue becaue a human inervenion i needed o compue he model parameer. Herein, we are inereed in compleely daa driven algorihm ince in real-life applicaion, hee parameer are uually unknown and one ha o eimae hem wihou human inervenion. In Secion 2.3, we conider parameer eimaion of he propoed model. Finally, ome reul are preened in Secion 3. 2 MRF model In hi ecion, we decribe a paio-emporal MRF model for color video claificaion. Fir, we define a model which ue only color informaion and hen we exend our model o ake ino accoun moion informaion. 2.1 Color video equence claificaion (Label Field) Le u uppoe ha he oberved image coni of hree pecral componen value (luv) a each pixel denoed by he vecor f, where S i he paial index and T i he emporal index. We are looking for he labeling ˆω, which maximize he a poeriori probabiliy P (ω F), ha i he maximum a poeriori (MAP) eimae. Baye heorem ell u ha: P (ω F) = 1 P (F ω)p (ω). (1) P (F)

3 Acually P (F) doe no depend on he labeling ω and we make he aumpion ha: P (F ω) = P (f ω). (2) T S I i hen eay o ee ha he global labeling, which we are rying o find, i given by: ˆω = arg max P (f ω) exp( V C (ω C )) exp( V C (ω C )), (3) ω Ω T S C C S C C T where C S i he e of paial (or inra-frame) clique and C T i he e of emporal (or iner-frame) clique. I i obviou from hi expreion ha he a poeriori probabiliy alo derive from a MRF. The energie of clique of order 1 direcly reflec he probabiliic modeling of label wihou conex, which could be ued for labeling he pixel independenly. Thi iem ie he reuling egmenaion o he original inpu. A naural aumpion i ha P (f ω ) i Gauian, he clae λ Λ = {0, 1,..., L 1} are repreened by he mean vecor µ λ and he covariance marice Σ λ. I i hen clear ha P (f ω ) = 1 (2π) 3 Σ ω ( exp 1 ) 2 (f µ ω )Σ 1 ω (f µ ω ) T. (4) We ge he following energy funcion: U(ω, F) = U 1 (ω, F) + U 2 (ω) + U 3 (ω), (5) U 1 (ω, F) = ( ln( (2π) 3 Σ ω ) + 1 ) 2 (f µ ω )Σ 1 ω (f µ ω ) T (6) T S U 2 (ω) = V 2 (ω C ) (7) C C S { where V 2 (ω C ) = V {,r} (ω, ωr) 0 if ω = = ωr β if ω ωr U 3 (ω) = V 3 (ω C ) (9) C C T { where V 3 (ω C ) = V {,+1} (ω, ω +1 0 if ω ) = = ω +1 γ if ω ω +1 (10) where β > 0 and γ > 0 are model parameer conrolling he homogeneiy of he region and he imporance of paial and emporal ineracion. A hey increae, he reuling region become more homogeneou. (8)

4 2.2 Uing moion informaion (Diplacemen Field) To furher elaborae our model, we inroduce a new field called he diplacemen field (DF). In hi way, we can ake ino accoun moion informaion when doing claificaion of a video equence. The DF could be defined over a differen laice han he label field (one could ue a lower reoluion, for inance) bu, for impliciy, we define i over he ame laice, placing a new laice beween each neighboring frame (, + 1). DF i a vecor-valued field, φ Φ denoe he diplacemen vecor a ie beween frame and + 1. The energy funcion of he DF i defined a follow: U DF = U1 DF (ω, φ) + U2 DF (φ) U DF 1 (ω, φ) = T where V DF 1 (ω, φ ) = S V DF 1 (ω, φ ) (11) { 0 if ω = ω +1 +φ α if ω ω +1 +φ (12) U DF 2 (φ) = C C DF V DF 2 (φ C ) (13) where V DF 2 (φ C ) = r C φ, φ r 2. (14) Unlike convenional approache, herein we ue he label field ω inead of he color value in he fir order poenial (Equaion (12)). The econd order poenial (Equaion (14)) i a moohing conrain favoring imilar diplacemen vecor in neighboring ie. Noe ha we have only inra-frame clique here. In our e, we have ued a fir order neighborhood yem. For moion compenaed claificaion, we have o ake ino accoun he DF in he energy funcion of he label field. For hi purpoe, we will redefine V 3 (ω C ) (ee Equaion (10)) in he following way: V 3(ω C ) = V {,+1} (ω, ω +1 ) = { 0 if ω = ω +1 +φ γ if ω ω +1 +φ (15) The energy funcion of he moion compenaed model i hen given by he following equaion: U(ω, φ, F) = U 1 (ω, F) + U 2 (ω) + U 3(ω) + U DF 1 (ω, φ) + U DF 2 (φ) (16) where U 3(ω) i he moion compenaed energy funcion of iner-level clique (ee Equaion (10) and Equaion (15)). The MAP eimae of he label and diplacemen field i obained rough he minimizaion of U(ω, φ, F): (ˆω, ˆφ) = min U(ω, φ, F). (17) ω,φ Since he energy funcion ha many local minima, we ue he Meropoli algorihm [10] o find he global minima. A each ieraion, he label field i updaed fir followed by he DF.

5 2.3 Parameer Eimaion Our goal i o propoe a compleely daa-driven, unupervied claificaion algorihm. Thu, we have o eimae he mean vecor µ λ and he covariance marix Σ λ for each cla, and he hyper-parameer α, β and γ. The mean vecor and covariance marice can be obained from he fir frame uing an unupervied claificaion algorihm (for more deail, ee [7]). The hyper-parameer are le eniive. We have found in pracice ha α = 15.0, β = 2.5 and γ = 2.0 give good reul. Of coure, one could alo ue an eimaion algorihm (ee [3, 8, 6]) o obain he righ value depending on he inpu video equence. However, we found ha hee algorihm need a huge compuing power and he obained value were very cloe o our ad hoc eimae. The mean vecor and covariance marice could alo be re-eimaed during he claificaion uing an adapive claificaion algorihm imilar o [7] bu experimen how ha he one frame eimae are good enough o obain a reaonably good claificaion. 3 Experimen The propoed algorihm ha been eed on a variey of color video equence. Herein, we preen a few of our reul obained on a variey of color video equence and alo compare he moion compenaed and aic model. In all cae, he opimizaion algorihm ha been opped when he number of changed ie wa le han 0.01% of he ie. In Table 1, we give he compuing ime for he preened video equence. One can ee, ha moion compenaed claificaion need more ieraion and more compuing ime becaue of he addiional diplacemen field. However, he qualiy of hee reul i alo beer. The compuing ime depend alo on he opimizaion mehod. ICM [1] i a deerminiic algorihm which converge in a few ieraion bu i find only a local minima. Thi may no be a good a he one given by a ochaic mehod, like he Meropoli algorihm [10]. In Figure 1 and Figure 2, we compare he reul obained by he aic and moion compenaed model on wo color video equence. The reul in he econd (rep. hird) column ha been obained by he aic (rep. moion compenaed) model. One can ee ha he reul are beer in he cae of moion compenaion. The propoed model can be eaily applied o gray-level image, only he fir order clique-poenial have o be changed in Equaion (6): Inead of a 3-variae Gauian diribuion, we ue here a univariae one. In Figure 3, we how he reul obained on he enni equence uing only gray-value. The reul clearly how ha color informaion can improve coniderably he final reul. The compuing ime i only lighly lower han in he cae of color image (ee Table 1). In Figure 4, we give he claificaion and diplacemen obained on he enni equence uing he Meropoli algorihm. The diplacemen are diplayed over block. The diplacemen field i noiy inide homogeneou

6 region bu i i reaonably good over region boundarie. Thi i good enough for he purpoe of moion compenaed claificaion. More accurae diplacemen field could be obained hrough a more elaboraed homogeneiy conrain in Equaion (14). 4 Concluion We have propoed an unupervied, moion compenaed color video claificaion algorihm. The claificaion model i defined in a Markovian framework and ue a fir order poenial derived from a hree-variae Gauian diribuion in order o ie he final claificaion o he oberved image. The label field ha paio-emporal clique and he diplacemen vecor are aken ino accoun by iner-frame (or emporal) clique. In he DF energy funcion we ue he label field inead of compuing he color difference of correponding pixel in order o reduce compuing ime. The energy funcion i minimized hrough a Meropoli algorihm [10] and we obain he claificaion of he frame and he diplacemen vecor a he ame ime. The algorihm i unupervied; only he number of clae i upplied by he uer. The mehod ha been eed on a variey of color video equence and he reul are encouraging. Reference 1. J. Beag. On he aiical analyi of diry picure. Jl. Roy. Sai. Soc. B., M. J. Daily. Color Image Segmenaion Uing Markov Random Field. In Proc. DARPA Image Underanding, D. Geman. Bayeian Image Analyi by Adapive Annealing. In Proc. IGARSS 85, page , Amher, USA, Oc F. Heiz and P. Bouhemy. Mulimodal Eimaion of Diconinuou Opical Flow Uing Markov Random Field. IEEE-PAMI, 15(12): , Dec A. K. Jain. Fundamenal of Digial Image Proceing. Prenice Hall, Z. Kao, M. Berhod, J. Zerubia, and W. Pieczynki. Unupervied Adapive Image Segmenaion. In ICASSP 95, Deroi, USA, May Z. Kao, T. C. Pong, and J. C. M. Lee. Moion Compenaed Color Image Claificaion and Parameer Eimaion in a Markovian Framework. Technical Repor HKUST-CS97-04, The Hong Kong Univeriy of Science and Technology, July S. Lakhmanan and H. Derin. Simulaneou Parameer Eimaion and Segmenaion of Gibb Random Field Uing Simulaed Annealing. IEEE PAMI, 11(8): , Aug J. Liu and Y. H. Yang. Mulireoluion Color Image Segmenaion. IEEE-PAMI, 16(7): , July N. Meropoli, A. Roenbluh, M. Roenbluh, A. Teller, and E. Teller. Equaion of ae calculaion by fa compuing machine. J. of Chem. Phyic, Vol. 21, pp , D. W. Murray and B. F. Buxon. Scene Segmenaion from Viual Moion Uing Global Opimiziaion. IEEE-PAMI, 9(2): , Mar

7 12. D. K. Panjwani and G. Healey. Markov Random Field Model for Unupervied Segmenaion of Texured Color Image. IEEE - PAMI, 17(10): , Oc M. I. Sezan and R. L. Lagendijk, edior. Moion Analyi and Image Sequence Proceing, chaper E. Duboi and J. Konrad: Eimaion of 2-D Moion Field from Image Sequence wih Applicaion o Moion-Compenaed Proceing, page Kluwer Academic Publiher, Model Mehod Num. of ieraion CPU ime color enni equence (23 frame, 6 clae) Saic ICM hour Saic Meropoli hour Moion compenaed Meropoli hour gray-level enni equence (23 frame, 6 clae) Moion compenaed Meropoli color car equence (21 frame, 4 clae) hour Saic ICM hour Saic Meropoli hour Moion compenaed Meropoli hour Table 1. Compuing ime on a SPARC aion Original frame Saic Moion compenaed Fig. 1. Reul obained by he aic and moion compenaed model (21 frame, 4 clae) uing he Meropoli algorihm.

8 Original frame Saic Moion compenaed Fig. 2. Reul obained by he aic and moion compenaed model uing he Meropoli algorihm on he enni equence (23 frame, 6 clae). Original gray-level frame Ineniy-baed Color-baed Fig. 3. Comparion of ineniy- and color-baed claificaion reul on he enni equence (23 frame, 6 clae) uing he moion compenaed model wih he Meropoli algorihm. Fig. 4. Claificaion and diplacemen obained on he enni equence (23 frame, 6 clae) uing he Meropoli algorihm.