A emporal fuson algorhm for mulsensor rackng n wde areas O. Wallar, C. Moamed, M. Benjelloun Unversé du Loral Côe d'opale Laboraore ASL : 95 av P.L.Kng 62228 Calas, FRANCE Olver.Wallar@laslgw.unvloral.fr Absrac Ths arcle presens a dsrbued vson sysem for rackng of moble objecs over wde areas. A emporal daa fuson s used n order o mprove he decson makng a he daa assocaon sage. The emporal fuson s performed wh a possblsc M.H.T. (Mulple Hypohess Trackng). We have decded o use he Possbly Theory whch handle effcenly unceranes. The orgnaly of hs M.H.T. s he conrol of s developmen accordng o he onlne qualy esmaon of he daa assocaon based on a Necessy measuremen. Keywords: Dsrbued sensors, Temporal daa fuson, Mulple Hypohess Trackng, Possbly heory Inroducon The am of our research concerns he survellance of wde areas by a mulsensor approach. Works are focused on he rackng of moble objecs wh a Dsrbued Vson Sysem (DVS). Applcaons are mulple and varous, as monorng of sgnfcan ses (nuclear hermal power), conrol and esmaon of flows (arpor, por, moorway), n our case concerns ndvdual car rackng on hghways or n urban area. Our DVS s based on a cooperaon of geographcally dsrbued fxed sensors whch observe he scene. Due o economc and compuaonal consrans, a lmed number of sensors s used and make mpossble o fully cover he scene. So sensors are separaed by blnd areas.e., are as whch we do no allow any observaon. The man problem for he nerpreaon sysem s o mach objecs from a sensor wh anoher one. Moreover several dffcules may occur. Lack of nformaon resulng from blnd areas We are parcularly concerned by movng objecs for whch we canno defne an exac behavoral model as human acves. A hrd dffculy les n he fac ha n he ransporaon applcaons area, here are numerous cases where objecs are vsually close o each oher hus creang mporan ambgues a he daa assocaon sage. In hs suaon, he global moon nerpreaon becomes naurally a dffcul ask because has o suppor he ncompleeness and uncerany of he avalable nformaon Our approach frs res o generae coarse emporal consrans from conexual knowledge, reducng he complexy of he observed dynamc scene. Then a emporal mulsensor fuson approach s used o help he decson makng a he daa assocaon sage. A MHT algorhm s proposed. Is orgnaly s he conrol of s developmen. For us he defnon of daa fuson consss n he combnaon of mulple complemenary and redundan ses of nformaon n order o mprove he decson makng. For a dsrbued mulsensor archecure wh no common feld of vew, he daa fuson has o work manly wh he complemenary of observaons over me and over feld of sensors. Is objecve s o solve ambgues a he daa assocaon sage. From a daa fuson pon of vew we have decded o use he Possbly Theory. Ths formal approach nroduced by Zadeh [] s based on he noon of fuzzy ses. Inuvely, n hs formalsm when a fac s enrely possble here s no surprse lnk o s occurrence. There s a heursc lnk beween probably and he possbly, snce an nformaon s mpossble, s lkely o be mprobable, however a hgh degree of possbly does no mply a hgh degree of probably. The possblsc approach s parcularly convenen when he sysem has o deal wh coarse model and ncomplee nformaon because can effcenly represen s gnorance. In hs heory each fac s assocaed wh a degree of possbly and a degree of necessy. The necessy expresses he cerany of a fac wh akng no accoun all evenuales. A hgh degree of possbly means ha he fac s fully possble. A low degree of necessy wh a hgh degree of possbly, means ha he sysem knows nohng abou he fac [2][3]. In addon n framework of he possbly heory, combnaon operaors are very large and sar from an opmsc o pessmsc behavor [4]. The user can effcenly choose s operaor a each decson level.
In hs paper we presen frs our emporal fuson approach, he DVS. And hen, we explan how he emporal fuson s conrolled. Fnally, we presen some resuls over a real world sequence. 2 Temporal fuson The emporal fuson consss o combne nformaon acqured a dfferen nsans and hen o decde. I mples ha he sysem mus be able o predc objecs sae a each nsan (see Fgure ). sensor sensor 2 I I2 I : nformaon I3 I4 In our mulsensor sysem, when a sensor perceves an observaon, res o make a decson. For hs, focuses on objecs whch have he possbly o appear a hs momen. Ths sage s based on he moon predcons of moble objecs. If he decson qualy s suffcen hen he fuson s sopped and he decson s made (see fgure2). The qualy s esmaed by a necessy measuremen. In our case, he machng decson s realzed when he necessy measuremen of an hypohess s hgh. Ths measure akes no accoun he possbly measuremen of all he hypoheses. The necessy of an hypohess s defned by : N H ) = P( H ) () ( Wh Ω = all hypoheses H = Ω { H P ( H ) max P ( H ) = j } H j H percepon decson predcon decson Fgure : An example of emporal fuson Among works based on emporal fuson approaches, we can quoe hose of M. Rombau [5] whch are assocaed whn he framework of he european projec Promeheus. Two sensors are placed on a moble vehcle, he frs sensor s locaed n fron of he vehcle and he oher one n he back. Blnd zones exs a lef and rgh sdes. The sysem res o esmae he close envronmen. In parcular, o fnd objecs deeced by he frs sensor on he level of he second one. A sgnfcan aspec of hs fuson concerns he mechansms of predcon, makng possble o brng ogeher nformaon ssued from he sensors. One can also quoe works of A.Nfle [6] whch res o classfy mssles by analyzng her dynamc behavor. The recognon s based on he observaon of known daed evens. Our emporal fuson breaks up no 2 levels. A he frs level each sensor res o decde wh s own nformaon. A he second level he decson s obaned by combnng nformaon acqured by several sensors. The crossng of he s level o he 2nd one s based on he qualy of he s level decsons. The esmaon of hs qualy akes no accoun he whole of nformaon whch has been combned. 2. Frs level predcons (DOPs) sensor "" 2.2 Second level Fgure 2 : s level of fuson If ambgues occur,.e. ha he qualy of he curren decsons s consdered o be nsuffcen, hen he sensor whch has deeced he ambgues, collec complemenary nformaon from close sensors. Ths fuson s carred ou n me as long as he qualy of he decsons s low and as long as he sysem s lkely o oban useful nformaon. The conrol n me of hs fuson wll be presened more n deal n secon 4. We now descrbe our DVS as well as he coarse emporal consrans from conexual knowledge whch are essenal o reduce he complexy of he rackng process. 3 Presenaon of he DVS 3. The archecure Use of mulple sensors gves rse o one problem: how o connec all he sensors ogeher (organzaon and archecure).
percepon wh ones, whch are lkely o be perceved, called awaed objec ". predcons (DOPs)! sensor ""... decson percepon predcons (DOPs) sensor "j" Fgure 4. Tracks managemen useful sensors Fgure 3 : 2nd level of fuson We have decded o use a decenralsed archecure. Ths laer has no cenral processng facly, no cenralsed communcaons medum. The srucure of hs archecure s equvalen o a nework of nellgen sensor nodes. Each sensor node s auonomous; has s own processng un and s own communcaons facles. Communcaon can ake place beween any wo conneced sensor nodes. Each node can assmlae and receve nformaon ndependenly of oher nodes. Ths ype of archecure has many advanages [7]. Among he prncpal ones, we can quoe he fac ha s compleely modular, ha ensures he maxmum benef derved from he use of mulple sensors. In parcular, s robus o he loss of sensors. I can use dfferen varees of sensors workng smulaneously. Tracks assocaed wh a moble objec (nalsaon, manenance and ermnaon) are managed by he sensor ha has nalsed hem. When a moble objec s perceved (fgure 4a) by a sensor, emporary racks assocaed wh possble rajecores of he objec n blnd zones are nalsed (fgure 4b). If a close sensor recognses he objec (fgure 4c), hen he sensor whch has nalsed emporary racks s nformed (fgure 4d) n order o valdae he curren rack connecng he wo sensors and o remove he ohers (fgure 4e). Ths managemen mode has been movaed n order o acheve good racks ermnaon. To realse he rack managemen, s essenal ha sensors are able frsly, o predc moon of moble objecs n blnd zones and secondly, o mach perceved objecs 3.2 Compably measuremens When a sensor perceves an observaon, measures he compably of hs observaon agans awaed objecs by a se of measuremens. The machng decson wll be based on hese measuremens. The frs operaon carred ou by a sensor percevng an observaon s o exrac s prmves. The choce of prmves s mporan because objec machng s manly based on hem. These prmves mus be me nvaran. A prmve exraced by a sensor mus be logcally found by anoher one. Afer he exracon of he prmves, compably measuremens are compued for each awaed objec. They are he resuls of he combnaon of her vsual prmves and emporal compables. Each degree akes a value rankng beween 0 and ; value mean a full compably. We have esed several possblsc operaors. Our choce was dreced owards a ype of operaor supporng compably measuremens favourng some exreme suaons,.e. very srong compables or ncompables (" X². Y² "). 3.3 Temporal knowledge We have decded o use fuzzy emporal curves of evens descrbed by Dubos and Prade [8][9] (DOP: Doman Occurrence Possbly) n order o predc objec moon n blnd zones. Ths choce s movaed by he fac ha we work n wde oudoor scenes ncludng blnd zones. Under such condons, we mus be able o manage unceranes relaed o he sysem. I s hus necessary o buld rough models olerang ranges of varaon for he numercal parameers.
A DOP(O,Sj,Sk) s he predcon generaed by he sensor Sj, explanng he emporal appearance possbly of a recognsed objec O n he feld of a specfc sensor Sk (fgure 5). possbly measuremen _mn Fgure 5 : A DOP As soon as an awaed objec s deeced by a sensor, hs laer predcs all he fuure moon of he moble objec by generang some DOPs. The number of predcons depends of he number of neghbour sensors lkely o perceve he moble objec. The generaon done, he DOPs are ransmed o hese sensors (fgure 6). We have developed a mehod allowng auomac generaon of DOP accordng o hs knowledge. The creaon of a DOP depends on he conex and dynamc characerscs of he moble objec. For our dsrbued nerpreaon sysem, he conex s based upon four knowledge whch are spaal and workng confguraon of he scene, class nformaon for movng objecs, mage acquson nformaon and dynamc envronmen [9]. C0 DOP(,0,) C DOP(,0,2) Fgure 6 : DOPs generaon 4 The conrol of he fuson 4. Inroducon In order o realze daa assocaon handlng he ambgues, we have developed a emporal mulsensor fuson algorhm based on a mulple hypohess approach : a possblsc MHT algorhm. The orgnaly of hs MHT s he conrol of s developmen accordng o he onlne qualy esmaon of he daa assocaon based on a Necessy measuremen. C2 The underlyng prncple s ha he daa assocaon decson s delayed as long as he confdence level n a hypohess s no sgnfcan enough comparavely o he oher ones. Ths confdence s an ndcaor of he qualy of he daa assocaon. Ths adapave approach has been chosen because one of he man problems regardng MHT s o know when should be sopped n order o realse an effcen daa assocaon. In [0], Cox presens a MHT algorhm o rack vsual prmves. He assumes ha hree observaons are suffcen n s applcaon. Accordng o our conex, a fxed number seems no o be convenen. Indeed, n some suaons, many moble objecs may appear smulaneously n fron of a sensor. Ths ype of suaon nduces many ambgues and s necessary o acqure addonal nformaon n order o be able o make he rgh machng decson. Ths could be obaned by a hgher developmen of he MHT. Thus, we have developed a necessy measuremen expressng he qualy of he daa assocaon a each new observaon. Wh hs measuremen, we can conrol he naon (low necessy) and he ermnaon (hgh necessy) of he MHT. Whn he conex of mulple hypoheses, he wo man algorhms are he JPDA (Jon Probablsc Daa Assocaon) fler [] and he MHT (Mulple Hypohess Trackng) [2]. These wo las ouperform sngle hypohess echnques bu when he problem sze ncreases, he requred compuaon ncreases exponenally. The JPDA, n rackng of closely spaced arges, has poor performance because of he perssen nerference from neghbourng arges. The MHT mehod s a mulple scan mehod. All nformaon concernng he assocaons beween moble objecs and he observaons s sored n order o solve ambguous suaons. The assocaons are defned as hypoheses. Ths mehod manans n parallel a se of hypoheses ha represens compeve worlds. In order o reduce he compuaonal complexy of he MHT, several opmsaon mehods have been developed. Among he prncpal ones we can menon hose whch reduce he number of hypoheses by mananng only he kbes ones. Cox has developed a MHT deermnng he kbes assgnmens n a polynomal me due o he use of he Mury algorhm. [0]. The choce of a Mulple Hypohess echnque was movaed by he fac ha we are n our case confroned wh many ambgues 4.2 Ambgues These ambgues are classfed no hree caegores, as lsed below. * Ambgues of assocaon Our sysem mus be able o solve ambgues relaed o daa assocaon. They occur when several moble objecs are emporally and vsually compable wh an
observaon. Such ambgues occur naurally n cluered envronmens. * Moonrelaed ambgues Moonrelaed ambgues are manly due o he lack of nformaon assocaed wh blnd zones and he absence of accurae behavoural model of movng objecs. Indeed, as no observaons are perceved whn he blnd zones, s very dffcul o defne precsely whch sensor wll perceve a moble objec, whch jus has been deeced. * Ambguous suaons due o new objecs The appearance of a new objec vsually close o an awaed objec, may generae an ambguy. 4.3 The MHT descrpon We remnd ha when an awaed objec s deeced by a sensor, hs laer predcs all he fuure dsplacemens of he moble objec by generang some DOPs. A he begnnng of each daa assocaon ambguy, a ree descrbng all he feasble hypoheses resulng from he observaons s generaed, wh he consran ha each observaon ares from a sngle (exsng or new) arge Each MHT ree examnes all he plausble combnaons, wh he consran ha each observaon arses from eher a sngle (exsng or new) arge. In he ree, each leaf node represens a feasble hypohess. For each branch, a possbly measuremen s assgned. Ths laer s obaned by a combnaon of he compably measuremens of he hypoheses ncluded n he branch by a mean operaor. The Tnorms and Tconorms combnaon operaors are no suable because he combnaon has no o be oo ndulgen or cauous. The machng decson s performed, as n he frs level fuson sep (secon2.), when he necessy measuremen (c.f. formula ) of a branch s hgh. Ths measure akes no accoun he possbly measuremen of all he branches. Such necessy measuremen s movaed by he fac ha an hypohess wll be seleced f s possbly measuremen s hgh and f hs value s dscrmnang wh respec o oher hypoheses. If an awaed objec does no appear n fron of a sensor feld afer a gven perod, hen hanks o he DOP assocaed wh hs objec, whch has s own lfespan, he rack wll be ermnaed. In hs suaon, curren MHTs remove naurally hese hypoheses. 4.4 Machng compably of new objecs The esmaon of machng compably (C j ) of a new objec (H ) wh an observaon j akes no accoun he compably measuremens of hs observaon wh he possble moble objecs (2). If several objecs are emporally and vsually compable wh an observaon, seems naural o hnk ha he observaon could be assocaed o one of hese objecs. Consequenly, he machng compably (MC) for a new objec s low, oherwse we ncrease he compably of new objec. O Φ, C ( H ) = max C j ( O ) j O Φ Φ:moble objecs lkely o be assocaed (2) wh he observaon j 4.5 Implemenaon In our envsaged applcaons workng wh dsan sensors, he majory of objecs assocaons are realsed before hey appear n fron of nex sensors. Oherwse, an ambguy persss, he developmen of he ree s sopped. In hs case, he algorhm valdae he mos plausble hypohess even he necessy measuremen s low. In complemen of our MHT, we use a hgh level exper. Is am s o correc some errors ha he PMHT s unable o manage. The reason s due o he fac ha he PMHT consder ha he predcons and he feaures exracon are rgh. The wo ypes of errors are he false alarms and he predcon errors. The false alarms may occur n nosed envronmens. The predcon errors are generaed when an unexpeced even slow down or speed up an objec moon. The hgh level exper, focus s aenon o new objecs ha he MHT has decded, f possble o ry o assocae o objecs for whch her racks have been jus ermnaed. 4.6 An example An example on fgure 7 shows hree objecs (O,,) and hree observaons (Obs,Obs2,Obs3). The purpose of hs example s frsly o show ha he problem of daa assocaon s no necessarly easy, and secondly o show how nformaon fuson can help o ease he decsonmakng process. The DVS has o mach on sensors C and C2 he hree objecs, whch have been deeced by he sensor C0. In hs example, as he hree objecs are vsually close o each oher, he machng decson s manly based on he emporal compably measuremens. In he realy, he compably measuremens are based on a combnaon of emporal and vsual compably measuremens. On fgure 8, The DOPs assocaed o he moble objecs and he appearance daes of he observaons are represened. The resulng MHT s shown on fgure 9. The hypohess corresponds o a poenal new objec permng a rack naon. The frs level of he ree represens all he feasble assocaons of he observaon Obs. There are hree branches because wo awaed objecs (O and ) are compable and a new objec s also possble. As he necessy measuremen s low because of wo hypoheses are possble, he MHT was he appearance of a new observaon n order o oban more nformaon o mprove a beer machngdecson. In hs example, he decson s realsed afer he
appearance of Obs3. Twenywo hypoheses have been generaed. The frs branch s seleced because s necessy measuremen s hgh (). C C2 {O,,} C0 5 Applcaon {Obs,Obs3} C {Obs2} C2 Fgure 7 : he sensors nework O O Obs Obs2 Obs3 Fgure 8 : Observaons appearances Ths s an applcaon of wde area survellance usng dsrbued sensors n a hghway envronmen ( fgure 0). The specfcy of hs applcaon s ha he spaal and workng confguraons of he observed scene are relavely well defned. The confguraon of each sensor s uned n order o make he objec recognon ask easer. Insde a closed segmen of he hghway, an objec seen by a camera has o appear n he feld of he nex sensor. The cameras are 3km dsan. Ths choce was aken accordng o he conex. A hgher dsance (nvolvng more naccuracy n he predcon) would enlarge he suppor of he DOPs and also decrease he qualy of he oal nformaon. In our applcaon, we used four classes of objecs, whch are rucks, vans, cars and moor cycles. Each local vson un has o exrac cnemac and nvaran feaures from each objec deeced. The deecon s based on a dfference beween he curren mage and an adapave reference mage whou movng objecs as Vannoorenberghe [3]. The nvaran feaures chosen for hs applcaon are normalsed color hsograms and calbraed 2D surface of objecs. The parameers used o classfy he objecs are surface and he cnemac behavor of objecs. Color compably beween objecs s based on he χ² dsance of her hsograms [3]. Obs Obs2 Obs3 0.7 O 0.8 O 0.4 0.2 0.8 0.4 O 0.8 possbly measuremen Fgure 9 : resulng PMHT necessy measuremen We have esed our mehod n hs real envronmen wh wo onehour sequences (able ). The performance of he possblsc MHT s compared wh a sngle hypohess ( neares neghbour : NN) approach and evaluaed n erm of daa assocaon errors esmaed by a human operaor (able2). The NN algorhm assocaes he observaon wh he objec havng he sronges possbly
6 Concluson C C2 no sequence sequence Fgure 0 : Hgh way mulsensor rackng Observaons perceved by he sensor C O O4 O5 Dops : Temporal predcons of objecs O Fgure : DOPs generaed by sensor C number of vehcles 52 sequence 2 324 number of ambguous vehcles 3 75 number of predcon errors 0 Table : sequences and 2 ( daa observed by a human operaor) algorhm N.N. M.H.T. M.H.T. wh hgh level exper Observaons perceved by he sensor C2 sequence 9 (2%) 7 (%) 0 (6%) Table 2 : machng errors 3 sequence 2 70 (22%) 45 (4%) 35 (%) Expermenal resuls show ha he NN approach and he MHT have an equvalen number of good assocaons n an uncluered envronmen (sequence ). The hgh level exper perms o mprove daa assocaon by solvng a par of predcon errors. In hs arcle, we have proposed a dsrbued vson sysem for rackng over wde area ncludng blnd zones. Our approach s based on he uncerany managemen of he global percepon sysem. A a frs level, concerns fuzzy emporal consran generaon expressng he naccuracy mprecson of he avalable behavoral model. Then a possblsc mehodology perms o model he sysem gnorance by akng no accoun all possble evenuales. And fnally a daa fuson approach based on a MHT s proposed o help machng decson. I works wh he complemenary of observaons over me and over feld of sensors. We have also proposed a soluon o conrol hs fuson algorhm over me. References [] L.A. Zadeh, Fuzzy ses as a bass for a heory of possbly, Fuzzy ses and sysems, (978), 328. [2] D. Dubos, H. Prade. Possbly heory, Applcaons o nformaon represenaon n nformac (n french). Masson, Pars, (988). [3] O. Wallar, C. Moamed, M. Benjelloun. A possblsc approach of hgh level rackng n a wde area, Proc Second The 2nd nernaonal conference on nformaon fuson fuson 99, Sunny Vale, USA, (999), 47477. [4] I.Bloch, Informaon combnaon operaors for daa fuson : a comparave revew wh classfcaon, IEEE ransacons on sysems, man and cybernecs 2 () (996) 5267. [5] M. Rombau & D. Mezel. Dynamc daa emporal mulsensor fuson n he Promeheus Prolab2 demonsraor. IEEE Inernaonal Conference on Robocs and Auomaon. San Dego,.May 83, (994). 3676 [6] A. Nfle, R. Reynaud. Behavor classfcaon based on evens occurrence n possbly heory (n french). Traemen du sgnal vol spécal 4, n 5 (997), 523434. [7] B.S.Y. Rao, H.F. DurranWhye & J.A. Sheen, A fully decenralzed mulsensor sysem for rackng and survellance, he nernaonal journal of robocs research,vol 2, n, (993), 2044. [8] D. Dubos And H. Prade,, Processng fuzzy emporal knowledge, IEEE ransacons on sysems man and cybernecs 9, (4), (989),729744. [9] O. Wallar, C. Moamed & M. Benjelloun, Temporal knowledge for Cooperave Dsrbued Vson, sevenh IEE Conference on Image Processng and s applcaons, Mancheser,July (999) [0] I.J. Cox and S.L. Hngoran, An effcen mplemenaon of Red s mulple hypohess rackng algorhm and s evaluaon for he purpose of vsual rackng, IEEE ransacons on paern analyss and machne nellgence 8 (2), february 996,3850.
[] Y. BarShalom And T. Formann, 988, Trackng and daa assocaon, Academc press, London. [2] D.B. Red, An algorhm for rackng mulple arges, IEEE ransacons on auomac conrol AC24 (6), december (979),843854. [3] P. Vannoorenberghe, C. Moamed and J.G. Posare, Updang a reference mage for deecng moon n urban scenes, Revue de raemen du sgnal 5 (2) (998)39 48.