A Novel Objec Deecon Mehod Usng Gaussan Mxure Codebook Model of RGB-D Informaon Lujang LIU 1, Gaopeng ZHAO *,1, Yumng BO 1 1 School of Auomaon, Nanjng Unversy of Scence and Technology, Nanjng, Jangsu 10094, Chna Absrac RGB-D sensors are beng wdely used n he compuer vson world. Deph nformaon s parcularly aracve and suable for objec deecon applcaon as s complemenary wh RGB nformaon n solvng many classc ssues such as shadow nerference, llumnaon change mpacs and nose. In hs paper a novel objec deecon mehod s proposed based on Gaussan mxure codebook model. Frsly, we esablsh he Gaussan mxure codebook model by combnng he Gaussan model wh codebook model by usng he color daa and he deph daa. A four-dmensonal Gaussan model based on R, G, B and D componens s bul n codewords. Then he objec deecon resul s obaned based on he proposed model and he model updang mehod gven. We hen evaluaed he mehod usng publcly avalable daases. The qualave and quanave expermenal resuls show ha he proposed mehod s more effecve han he compared mehods n complex scenes. Keywords - objec deecon; gaussan mxure codebook model; RGB-D nformaon I. INTRODUCTION In recen years, he accurae movng objec deecon n vdeo sequences plays an mporan role n he applcaons of compuer vson such as vdeo survellance, human compuer neracon and so on [1]. The wdely used approach of movng objec deecon s based on RGB color nformaon. Researchers have proposed many effecve deecon algorhms, such as Bayesan decson rules [], Mxure of Gaussans [3-4], and Kernel densy esmaon [5]. However, he color-based algorhms face many challengng problems ncludng he followng: vulnerable o llumnaon changes; shadows cas by movng objecs; camouflage.e., smlar color beween movng objecs and he background [6] e al. Wh he rapd developmen of deph daa acquson echnology, movng objec deecon based on deph nformaon s able o compensae for drawbacks n RGB daa. Bu he deph daa are usually nosy and unrelable. There are nvald resuls n he deecon algorhm based on deph daa alone [7]. On he base of he above consderaons, he opmal algorhm, whch fuses he RGB nformaon and deph nformaon, s aracve. So ha he nrnsc lmaons of sngle nformaon can be counerbalanced and mproved deecon resuls can be obaned [8]. Recenly, some researchers have proposed some mehods based on RGB-D nformaon o deec movng objec. A logcal operaon or s used o combne he dfferen foregrounds ha respecvely come from grayscale mage and dsance mage [9]. The mehod can successfully cope wh problems ha he objec has smlar color wh he background and s dsance s close o he dsance of he background, bu fals o overcome he edge nose. The auhors exrac foreground objecs from deph mage based on he mehod of regon growng and use RGB nformaon o refne he foreground objec [10]. I no only solved he lmaon of color camouflage, bu also decreased he deph nose. However, s less effecve n complex scenes. In hs paper, we presen a novel background subracon algorhm, whch amed o solve he dffculy of adjusng he parameers and overcome he dsadvanages only based on RGB nformaon or deph nformaon. In hs algorhm, we combne he codebook model (CB) [11] wh Gaussan model ogeher. And a four-dmensonal Gaussan model s bul whch based on R, G, B and D componens n codewords. In hs way, here s he characersc of mxure of Gaussan model n codebook algorhm. The proposed mehod s more effecve n complex scenes han he compared background subracon algorhm. The resuls show a quanave and qualave mprovemen n he movng objec deecon applcaon. II. PROPOSED METHOD Deph-based algorhm has srong robusness on sudden lghng changes, hghlghed regons and shadows, whch s dffcul o he color-based algorhm. However, when he objecs are closed o he background, deph-based codebook algorhm can classfy he pxels o he background. An example s depced n Fgure 1, Fgure 1 (a) s he orgnal color mage, Fgure 1(b) s he correspondng deph mage, Fgure 1(c) s he resul of codebook algorhm n [11], whch only uses he color nformaon; Fgure 1(d) s he resul of codebook algorhm n [11], whch only uses he deph nformaon. We can see ha foreground objecs are msakenly deeced n Fgure 1(c) and Fgure 1(d), due o here are shadows n Fgure 1(a) and he close range beween he book and he wall n Fgure 1(b). Consequenly, he deecon resul s no sasfed when he color or deph nformaon s used ndvdually. In hs paper, a Gaussan mxure codebook model, named as GMCB, s presened hrough combnng he Gaussan model and codebook model based on RGB-D nformaon. The proposed background subracon mehod based on he GMCB ncludes background model DOI 10.5013/IJSSST.a.17.45.40 40.1 ISSN: 1473-804x onlne, 1473-8031 prn
consrucon and foreground deecon. The background model consrucon mehod s llusraed n secon II.A; and he foreground deecon and model updang mehod s llusraed n secon II.B. (a) Orgnal color mage (b) Orgnal deph mage (c) Resul of codebook algorhm wh color daa The Gaussan mxure codebook model s he use of quanave echnques n he long-erm observaon sequence o buld background model. I bulds a codebook model for each pxel. In he process of nalzaon algorhm ranng, X, whch s he pxel value of a sngle pxel n a ranng sequence, s conssed of N RGB-D vecors: X { x1, x,..., x N }. Le { c1, c,..., c L } represen he codebook ha s composed of L codewords. For each pxel, he number of codewords may be no he same depend on s sample varaon. Each codeword c ( 1... L) consss of a welve-uple as formula (1). c,,,, R, G, B, D, R, G, B, D, f,, p, q In equaon (1), every symbol s expressed as follows: R,, G,, B,, D, --- mean value of R, G, B, D for each pxel; R,, G,, B,, D, --- mean square devaon of R, G, B, D for each pxel; f --- he frequency wh whch codeword c has occurred; --- he maxmum negave run-lengh, defned as he longes nerval durng he ranng perod ha he codeword has no recurred; p, q--- he frs and las access mes, respecvely, ha he codeword c has occurred; The condon, whch an ncomng pxel x R, G, B, D s mached successfully wh he codeword c, s defned as formula (). bx (, c) (d) Resul of codebook algorhm wh deph daa Fgure 1. Deecon resuls by codebook algorhm [11] A. Background Modelng Based on he assumpon ha he pxel value of he same poson n he vdeo sequence can be modeled o Gaussan dsrbuon, we propose a novel Gaussan mxure codebook model usng Gaussan model o mprove codebook model based on RGB-D nformaon. The whole codebook has he characerscs of he mxure of Gaussan as he Gaussan model s bul n he R, G, B, D channel separaely. We defne 1 for mached correcly and 0 for mached ncorrecly. The machng condon s defned as formula (3). bx, c 1, ( Z ) a 0, oherwse z, z, m. Accordng o he sascs, he probably of a random varable, whch obeys Gaussan dsrbuon and s n (.58,.58 ), s 99.7%. Therefore, pror parameer a s.58. Mean and varance updae mehods are defned as he formula (4) and formula (5). Where Z R, G, B, D, z R, G, B, D DOI 10.5013/IJSSST.a.17.45.40 40. ISSN: 1473-804x onlne, 1473-8031 prn
1 1 (1 ) Z z, z, f 1 f 1 1 1 z, (1 ) z, ( Z z, ) f 1 f 1 The dealed GMCB background model consrucon mehod s shown n Table I. TABLE I. Seps THE GMCB BACKGROUND MODEL CONSTRUCTION 1 L 0, ζ Φ(empy se) for =1 N do GMCB background model consrucon { c, c,..., c L }, fndng he codeword ⑴ In 1 mach o x sasfy he followng condons: bx (, c) 1. ⑵ If ζ=φ or here s no mached codeword, c ha B. Foreground Deecon The foreground deecon can be depced as a classfcaon queson and ge he deecon resul. An ncomng pxel s classfed no foreground or background accordng o he formula () and (3) n secon II.A. The deecon process s gven n formula (6). FG BGS( x) BG f ζ=φ or here s no mached codeword, oherwse Where FG represens he foreground, BG represens he background, BGS x represens he deecon resul. C. Model Updang When he pxel s classfed o foreground, he model s no need o updae. Bu f he pxel s classfed o background, he model updang s execued o resolve he effec of he background change. The model updang process s gven n formula (7). The x s used o updae he mached codeword c. The formula (7) s he same as he sep (3) n Table I. 3 4 5 hen L L +1, a new codeword cl s creaed and added o ζ. cl R, G, B, D,0,0,0,0,1, 1,, ⑶ If no, updae he mached codeword c, fr, R fg, G c f 1 f 1 fb, B fd, D f 1 f 1 fr, R R, fg, G G, f 1 f 1 fb, B B, fd, D D, f 1 f 1 f 1, max, q, p,. For each codeword c ( 1,..., L) updae by, max,( N q p 1). For each codeword c ( 1,..., L), Delee he c whch N /, L L 1. Fnally, he fnal model s C c 1 L N/. c fr, R fg, G f 1 f 1 fb, B fd, D f 1 f 1 f 1 f 1 fb, B B, fd, D D, f 1 f 1 f 1, max, q, p,. fr, R R, fg, G G, III. EXPERIMENTS AND ANALYSIS A. Tes Seup We es our mehod on he publcly daase. Three complex scenes are evaluaed such as he scene 1 ha objec s close o he background and gradually keeps away from he background, he scene ha color of objec s he same as he background and he scene 3 of sudden llumnaon changes. The daase comes from he webse (hp://acproyecos.ugr.es/mvson/). We also compare he proposed GMCB mehod wh hree dfferen mehods. These mehods are he followng ones: he orgnal color-based Codebook (CB Color ) mehod, he Codebook mehod based only on deph daa (CB Deph ), he 4D Codebook (CB4D) mehod whch he deph nformaon s only as he fourh channel n codeword. The codes are execued n VS010 envronmen, whle he OpenCV lbrary DOI 10.5013/IJSSST.a.17.45.40 40.3 ISSN: 1473-804x onlne, 1473-8031 prn
s adoped o asss mage processng. We provde he vsual comparson of he above mehods and he quanave resuls are gven by usng he hand-segmened ground ruh. The frs 50 frames n each sequence are used o background model consrucon and he oher frames are used o deecon evaluaon. B. Qualave Analyss 1) Scene 1: Targe and Background Smlar Dsance In he scene 1, a person hands a book keeps away from he wall. We selec he 78 h frame, he 135 h frame and he 14 h frame for vsual comparson as shown n Fgure from column 1 o column 3, among hem he 78 h frame s neares from he background and he 14 h s furhes from he background. As can be seen n Fgure, here are large deecon error n he resul of he CB Color mehod because of he shadow of he book. The CB Deph mehod can solve hs shadow problem, bu objec canno be deeced n he 78 h frame because of he smlar dsance beween objec and background. The CB4D mehod shows beer effec by usng he color and deph nformaon. Bu here s sll some msakes. Alhough he resul of he GMCB mehod n he 78 h frame s no sasfed, he GMCB mehod ges he beer resul han he ohers. (a) Color mage (b) Deph mage (c) Ground ruh (d) Resul of CBColor (e) Resul of CBDeph DOI 10.5013/IJSSST.a.17.45.40 40.4 ISSN: 1473-804x onlne, 1473-8031 prn
(f) Resul of CB4D (g) Resul of GMCB Fgure. Vsual comparson of he par of scene1 resuls.(from lef o rgh: he 78 h frame, he 135 h frame and he 14 h frame) ) Scene : Targe and Background Smlar Color As shown n Fgure 3, a man s holdng a whe box and wo blue books and goes hrough he scenes. The color of box s smlar o he wall and he color of books s smlar o he rash. The 90 h frame s seleced for vsual comparson as shown n Fgure 3(a) and Fgure 3(b). Fgure 3(d)-(g) are he deecon resuls of he 90 h frame wh he dfferen mehods. Compared wh he ground ruh n Fgure 3(c), we can fnd here are many arfacs n he resul of he CB Color mehod n Fgure 3(d). The deecon resul s beer n Fgure 3(e), whch s he resul of he CB Deph mehod, bu he whe box and books are only parally deeced. The GMCB mehod deecs he whe box effecvely n Fgure 3(g). The effec of he proposed GMCB mehod s obvously effecve and s robus o he dffcul suaons. (a) 90 h color mage (b) 90 h deph mage (c) Ground Truh (d) Resul of CBColor (e) Resul of CBDeph (f) Resul of CB4D (g) Resul of GMCB Fgure 3. Vsual comparson of he par of scene resuls. 3) Scene 3: Sudden Illumnaon Changes There are sudden llumnaon changes n scene 3. The 368 h frame s seleced for vsual comparson as shown n Fgure 4. We can fnd ha CB Color mehod and CB Deph mehod canno adap o sudden changes n llumnaon. There are a lo of noses n CB4D mehod. The GMCB mehod ges he beer deecon resul han he ohers. DOI 10.5013/IJSSST.a.17.45.40 40.5 ISSN: 1473-804x onlne, 1473-8031 prn
(a) 368 h color mage (b) 368 h deph mage (c) Ground Truh (d) Resul of CBColor (e) Resul of CBDeph (f) Resul of CB4D (g) Resul of GMCB Fgure 4. Vsual comparson of he par of scene 3 resuls. C. Qualave Analyss precson ( P ), recall ( R ) and F -measure ( F ) are he common evaluaon crera of objec deecon. recall s he rue posve; precson s he rao beween he number of correcly deeced pxels and he oal number pxels marked as foreground; F - measure s a successful combnaon of P and R o comprehensvely evaluae he performance of he algorhm. The values of P, R, F can be compued as P TP, TP R TP, P F R. The FP TP FN P R TP (True Posve) value s he number of pxels as he movng objec s correcly deeced. FP (False Posve) s msdenfed as he background pxels. FN (False Negave) s he number of pxels o be msaken for he movng objec. Ths F -measure no only offers a rade-off beween he ably of an algorhm o deec foreground and background pxels, bu also provdes a general evaluaon of robusness of he algorhm. In general, he value of F s hgher, he beer he performance. TABLE II. Mehod COMPARISON OF F VALUE F value Scene 1 Scene Scene 3 CBColor 0.600 0.6438 0.4 CBDeph 0.7863 0.798 0.7331 CB4D 0.7705 0.7965 0.6586 GMCB 0.810 0.89 0.817 For each scene, by compung he P value, R value, and F value for every deecon resuls obaned by above mehod, he fnal F values are gven by averagng he correspondng values. The resuls are shown n Table II. I can be observed n Table II ha he deecon performance based solely on RGB or deph nformaon s poorer and even appear falure. The deecon performance of CB4D s also he case. However, he proposed GMCB mehod can keep a hgh deecon performance n all he es complex suaons. IV. CONCLUSIONS The problem of objec deecon s a well-known problem, bu sll far from beng solved. In hs work we proposed he Gaussan mxure codebook model of RGB-D nformaon. Deph nformaon s as he fourh channel n codeword, and he Gaussan model s bul n he RGBD channels separaely. We evaluaed he mehod on he publcly daase, whch ncludes he complex scenes such as smlar dsance, smlar color and sudden llumnaon changes. Expermenal resuls show ha he proposed mehod obaned a sasfyng deecon resuls on accuracy and robusness. ACKNOWLEDGMENT Ths work was suppored by he Naonal Naural Scence Foundaon of Chna (Gran No. 610366). REFERENCES [1] Z.G. Sh. Research on movng objec deecon and rackng n vdeo sequences [D], Xdan Unversy, -13, 014. [] Y.Y. L, B. Zeng, K.F. Xu, e al. Foreground objec deecon n complex background based on Bayes-oal probably jon esmaon [J], Journal of Elecroncs &Informaon Technology, 34(): 388-39, 01. [3] R. Sngh, B.C. Pal, and R.A. Jabr. Sascal represenaon of dsrbuon sysem loads usng Gaussan mxure model [J], IEEE Trans on Power Sysems, 5(1): 9-37, 010. [4] J.Y. Zhou, X.P. Wu, C. Zhang, e al. A movng objec deecon mehod based on sldng wndow Gaussan mxure model [J], DOI 10.5013/IJSSST.a.17.45.40 40.6 ISSN: 1473-804x onlne, 1473-8031 prn
Journal of Elecroncs & Informaon Technology, 35(7): 1650-1656, 013. [5] J. Lee and M. Park. An adapve background subracon mehod based on kernel densy esmaon [J], Sensors, 1: 179-1300, 01. [6] L.M. Hu, L.L. Duan, X.D. Zhang, e al. Movng objec deecon based on he fuson of color and deph nformaon [J], Journal of Elecroncs & Informaon Technology, 36(9): 047-05, 014. [7] R. Grabb, C. Tracey, A. Purank. Real-me foreground segmenaon va range and color magng [C], Proceedngs of he CVPR, 1-5, 008. [8] S. Oonell, P. Spagnolo, P.L. Mazzeo, e al. Improved vdeo segmenaon wh color and deph usng a sereo camera [C], IEEE Inernaonal Conference on Indusral Technology, 1134-1139, 013. [9] J. Leens, S. Perard, O. Barnch, e al. Combnng color, deph, and moon for vdeo segmenaon [J], LNCS, 5815: 104-113, 009. [10] E. Mrane, M. Georgev, A. Gochey. A fas mage segmenaon algorhm usng color and deph map [C], IEEE 3DTV-Conference on he True Vson-Capure, Transmsson and Dsplay of 3D Vdeo, 1-4, 011. [11] K. Km, T.H. Chaldabhongse, D. Harwood, e al. Real-me foreground background segmenaon usng codebook model [J], Real-Tme Imagng, 11(3): 17-185, 015. DOI 10.5013/IJSSST.a.17.45.40 40.7 ISSN: 1473-804x onlne, 1473-8031 prn