Recursive neural networks for object detection

Transcription

1 Rursiv nurl ntworks for ojt ttion M. inhini, M. Mggini, L. Srti,. Srslli iprtimnto i Inggnri ll Informzion Univrsità gli Stui i Sin Vi Rom, Sin (Itly) -mil: {moni,mggini,srti,frno}@ii.unisi.it strt In this ppr, nw rursiv nurl ntwork mol, l to pross irt yli grphs with ll gs, is introu, in orr to rss th prolm of ojt ttion in imgs. In ft, th ttion is prliminry stp in ny ojt rognition systm. Th propos mtho ssums grph s rprsnttion of imgs, tht omins oth sptil n visul fturs. In prtiulr, ftr sgmnttion, n g twn two nos stns for th jny rltionship of two homognous rgions, th g ll ollts informtion on thir rltiv positions, whrs no lls ontin visul n gomtri informtion on h rgion (r, olor, txtur, t.). Suh grphs r thn pross y th rursiv mol in orr to trmin th vntul prsn n th position of ojts insi th img. Som xprimnts on f ttion, rri out on sns quir y n inoor mr, r rport, showing vry promising rsults. Th propos thniqu is gnrl n n ppli in iffrnt ojt ttion systms, sin it os not inlu ny priori knowlg on th prtiulr prolm. I. INTROUTION Rntly, thr hs n growth of intrst in ojt ttion mols, whih must inhrntly roust to th wi vritions tht r osrv in nturl imgs. In ft, th gnrl prolm of ojt ttion is vry hllnging tsk, sin th ojt ttion systm is rquir to istinguish prtiulr lss of ojts from ll othr ojts rprsnt in th imgs. Th iffiulty of this tsk lis in th finition of gnrl mol of th ojt lss to tt, whih hs high intr lss n low intr lss vriility. n ojt ttion systm shoul l to loliz ojts whih n vry thir pprn w.r.t. light onitions, orinttion, n imnsion. urthrmor, th ojts n prtilly olu or n lnt in with th kgroun. Ojt ttion mthos n lssifi in four min tgoris [1]: Knowlg s; tur invrint; Tmplt mthing; pprn s. Knowlg s mthos xploit th humn knowlg on th srh ojts n us som ruls in orr to sri th ojt mols. Thos ruls r thn us to tt n loliz ojts tht mth th prfin mols. Inst, th im of ftur invrint pprohs [2], [3] is to fin st of fturs tht r invrint w.r.t. ojt orinttion, light onitions, imnsion, t. Tmplt mthing mthos stor svrl pttrns of ojts n sri h pttrn y visul n gomtril fturs. Th orrltion mong n input img n th stor pttrns is omput for tting ojts [4]. inlly, in pprn s mthos, mhin lrning thniqus r xploit to lrn tmplts y xmpls [5], [6], [7]. In this ppr w prsnt nw pprn s mtho tht uss Rursiv Nurl Ntworks (RNNs). Th novlty of th pproh onsists in rprsnting imgs y grphs with ll gs. Suh grphs r thn trnsform into forsts of trs n pross y nw RNN mol l to l with struturs with ll gs. Th propos mtho os not us ny priori or huristi informtion on th ojt mols n n usful to tt ojts unr ny illumintion, orinttion, n position. Th ppr is orgniz s follows. In th nxt stion, som nottion is introu. Th nw RNN mol is prsnt in Stion III, whil Stion IV sris th grph s rprsnttion of imgs. In Stion V som xprimntl rsults on f ttion tsk r rport, n, finlly, Stion VI ollts som onlusions. II. NOTTION Lt G = (V,, L) irt grph, whr V is th st of nos, V V rprsnts th st of rs, n L : V L v is lling funtion, ing L v R m finit st of lls. Givn ny no v V, p[v] is th st of th prnts of v, whil h[v] rprsnts th st of its hilrn. Th outgr of v, o[v], is th rinlity of h[v], n o = mx v o[v] is th mximum outgr. h no stors st of omin vrils into ll. Th prsn of n g (v, w) in ll grph stns for th xistn of usl link twn th lls of v n w. Morovr, for rursiv prossing, G shoul hv suprsour, i.. no s V with no inoming gs, n from whih ny othr no in V n rh. Th suprsour s my vntully following th lgorithm in [8]. In this ppr, w onsir th lss of irt yli Grphs (Gs), whr prtil orring n fin on, suh tht v w if v is onnt to w y irt pth. irt Positionl yli Grphs (PGs), for whih rursiv ntworks wr originlly fin, r sulss of Gs, whr n injtiv funtion o v : h[v] {1,..., o} ssigns position o v () to h hil of no v. Thrfor, PG is rprsnt y th tupl (V,, L, O), whr O =

2 {o 1,..., o V } is th st of funtions fining th position of th hilrn for h no. inlly, th finition of th lss of Gs with Ll gs (Gs L) rquirs th introution of n itionl g lling funtion, suh tht G = (V,, L, ), whr : L, n L is finit sust of R k. Th prsn of n g ll introus som smntil ontnts into th link twn two nos. III. RURSIV NURL NTWORKS OR PROSSING GS L Rursiv nurl ntworks wr originlly propos to pross PGs [9], [10], [8]. In this s, th stt trnsition funtion f, omput y n RNN, pns on th orr of th hilrn of h no, sin th stt of h hil oupis prtiulr position in th list of th rgumnts of f. In orr to ovrom suh limittion, in [11], wight shring pproh ws sri, l to rlx th orr onstrint, n to vis nurl ntwork rhittur suit for Gs with oun outgr. In ft, th wight shring thniqu nnot ppli to Gs with lrg outgr o, u to th ftoril growth in th ntwork prmtrs w.r.t. o. vn if th mximum outgr n oun, for instn y pruning thos onntions tht r huristilly lssifi s lss informtiv, nvrthlss som importnt informtion my isr in suh prprossing phs. On th othr hn, oth th orring onstrint n th oun on th mximum outgr n rmov whn onsiring Gs L [12]. In ft, for Gs L, stt trnsition funtion f n fin whih hs not prfin numr of rgumnts n tht os not pn on thir orr. Th iffrnt ontriution of h hil pns on th ll tth to th orrsponing g. t h no v, th totl ontriution X(h[v]) R p of th stt of its hilrn is omput s X(h[v]) = 1 h[v] k j L (j) h[v] (v,h i[v]) X hi[v] j=1 (1) whr L (v,hi[v]) R k is th ll tth to th g (v, h i [v]), n R p,n,k is th wight mtrix. In prtiulr, j R p,n is th j th lyr of mtrix n L (j) (v,h i[v]) is th j th omponnt of th g ll. inlly, th stt t no v is omput y two lyr prptron with linr outputs, s X v = f(x(h[v]), U v, θ f ) = V σ(x(h[v])+u v +)+ (2) whr θ f ollts R q,p, R q,m, R q, R n, n V R n,q, ing q th numr of hin units. t th suprsour, lso n output funtion is vlut y fforwr ntwork, Y s = g(x s, θ s ) = W σ(x s + ) + G with R q,n, R q, G R r, n W R r,q. Strting from qs. (1) n (2), n MLP implmnttion of th rursiv ntwork n riv, simply rwriting q. (1) Rursiv Ntwork for Gs L g f W 1 V 2 G k r nurons q nurons n nurons q nurons p nurons ig. 1. n MLP implmnttion of th rursiv ntwork. Gry lyrs r sigmoil, whrs whit lyrs r linr in oth th fforwr ntworks. s X(h[v]) = 1 h[v] k j=1 j h[v] L (j) (v,h X i[v]) h i[v] (3) Thrfor, q. (3) suggsts tht th ontriution of th hilrn of h no to its stt n omput using thr lyr prptron with k inputs h[v] L (j) (v,h X i[v]) h i[v], j = 1,..., k, n with j s input to hin wight mtris. Th son hin lyr n th output lyr rmin unhng n thir ontriution to th lultion is sri y q. (2) (s ig. 1). Rmrk: In th mor gnrl s, X(h[v]) my omput using nonlinr funtion φ : R (n+k) R p, pning on st of prmtrs θ φ : X(h[v]) = 1 h[v] h[v] φ(x hi[v], L (v,hi[v]), θ φ ) Morovr, it is worth noting tht, vn if th g lls inrs th smntil ontnt tth to th links, thy n lso us to oify th orr rltionship. In ft, h PG n rprsnt y G L. In prtiulr, in [13], it ws prov tht for ny PG G n ny stnr rursiv nurl ntwork RNN with trnsition funtion f, thr xists G L L n n RNN L, with trnsition funtion f suh tht f(g) = f(l). s irt onsqun, th novl rursiv rhittur mintins th univrsl pproximtion pilitis on th st of PGs. inlly, sin ny G n rprsnt with PG, y ssigning n ritrry position to h hil, th ov pproximtion proprty n lso xtn to Gs. IV. T GRP S RPRSNTTION O IMGS Our ojt ttion mtho ssums grph-s rprsnttion of imgs. Thus, in orr to sri our pproh, w n to introu th prprossing phs tht llow us to rprsnt h img y G L.

3 irst of ll, h img is sgmnt in orr to otin st of istint rgions. h rgion hs homognous ontnt w.r.t. som visul fturs. Th sgmnttion mtho w us is s on olor informtion. Th fftivnss of olor s sgmnttion pprohs ws lry shown in [14], [15]. W ssum tht imgs r rprsnt in th RG (R, Grn, lu) olor sp. Our sgmnttion lgorithm prforms K mns lustring in this sp on th pixls longing to th imgs, s on th ulin istn. t th n of th lustring, rgion growing prour is rri out to ru th numr of rgions. suitl hoi of th K initil pixls whih orrspon to th ntrois of th initil lustrs, n n pproprit rgion growing poliy, llow to otin n invrint st of rgions w.r.t. oth rottion n trnsltion. Th sgmnttion pross yils st of rgions. h rgion n sri y vtor of rl vlu fturs whih ollt gomtril n visul informtion. On th othr hn, th struturl informtion rlt to th jny rltionship mong rgions n o y n unirt grph with ll gs. Stritly spking, th Rgion jny Grph with Ll gs (RG L) is xtrt from th sgmnt img y: () () () () ig. 2. Th originl img (), th sgmnt img (), n th xtrt RG L (,). 1) ssoiting no to h rgion. Th rl vtor of fturs rprsnts th no ll; 2) Linking th nos ssoit to jnt rgions with unirt gs; 3) tthing rl vtor of fturs to h g of th grph. Th vtor sris th mutul position of th two jnt rgions. In orr to st up lrning nvironmnt, trgt qul to 1 is tth to h no of RGs L tht orrspon to prt of th ojt in whih w r intrst, whrs trgt qul to 0 is tth othrwis. In ig. 2, th RG L xtrtion is summriz. In this xmpl, w wnt to loliz th r toy r. Th lk nos orrspon to prts of th toy r n hv trgt 1, whil whit nos orrspon to prts of othr ojts n hv trgt 0. Our ojt ttion softwr provis visul intrf to tg th rgions tht orrspon to th ojt or not (s ig. 3). Sin th RNN mol sri in Stion III n pross only Gs L, h RG L must trnsform into irt yli grph. Th trnsformtion prour tks RG L R, long with slt no n, s input, n prous tr T hving n s its root. Th mtho must rpt for h no of th RG L, or, mor prtilly, for rnom st of nos. It n prov tht th forst of trs uilt from R is rursiv quivlnt to R, tht is th RNN hvior is th sm ithr if th ntwork prosss R or if it prosss th forst of trs [16], [17]. Th first stp of th prour is prprossing phs tht trnsforms R into irt RG L G, y ssuming tht pir of irt rs rpls h unirt g. h r in th pir is ssign th sm ll s th originl unirt g. G is unfol into T y th following lgorithm: ig. 3. Th visul intrf us to rt th lrning nvironmnt. 1) Insrt opy of n in T ; 2) Visit G, strting from n, using rth first strtgy; for h visit no v, insrt opy of v into T, link v to its prnt no prsrving th informtion tth to h r; 3) Rpt stp 2 until prfin stop ritrion is stisfi, n, howvr, until ll rs hv n visit t lst on. 4) tth th trgt ssoit to n to th root no of t. If th rth first visit is hlt whn ll th rs hv n visit on, th miniml rursiv quivlnt tr is otin (Miniml Unfoling s ig 4()). owvr, iffrnt stop ritri n hosn. or xmpl, h r n visit

4 () Miniml Unfoling RG L () Proilisti Unfoling irt RG L () Rnom Unfoling ig. 4. Th trnsformtion from RG L to rursiv quivlnt tr. Th imnsion of th rursiv quivlnt tr pns on th stop ritrion hosn uring th unfoling of th irt RG L. on, thn stp 2 is rpt until stohsti vril x oms tru (Proilisti Unfoling s ig. 4()). Othrwis, th rth first visit n rpl with rnom visit (Rnom Unfoling s ig. 4()). nywy, h r must visit t lst on, in orr to gurnt th rursiv quivln twn R n T. V. XPRIMNTL RSULTS s s stuy, th xprimnttion prform to vlut th fftivnss of our pproh hs n fous on tting fs in imgs. owvr, Th propos mtho os not xploit ny priori informtion out th prtiulr ojt mol n, thrfor, is inpnnt of th prolm t hn. Th xprimntl tst ontins 500 imgs n 384 fs (h img ontins t most on f) n ws quir y n inoor mr, whih ws pl in front of oor. prson t tim wnt in through th oor n wlk until h/sh ws out of th mr y. h img orrspons to frm of th quir sn. W r intrst only in tting th f position, whrs no trking of th fs ws prform n no informtion riv y th movmnt of th ojt ws xploit. Th fs ppr in iffrnt orinttions n imnsions (s ig. V). Th imgs wr ivi in thr sts: trining st, ross vlition st, n tst st. oth th trining n th ross vlition sts ontin 100 imgs, whrs 300 imgs (199 fs) onstitut th tst st. h img ws sgmnt, prouing RG L. h no in th RG L stors ll tht sris som visul fturs (vrg olor, ouning ox oorints, ryntr ig. 5. Vriility of th f pprn in th tst. s vry w.r.t. imnsion n pos n n prtilly olu. Th imgs us to prform th xprimnttion wr provi y LSG S.p..; ll th imgs wr us stritly for rsrh purpos n r pulish unr lins of th rprou prsons. oorints, r, primtr, n momntum). Th mutul sptil position of two jnt rgions is rprsnt y th ll tth to th orrsponing g. Givn pir of jnt rgions i n j, th ll of th g (i, j) is rprsnt y th vtor [,,, ] (s ig. 6), whr: rprsnts th istn twn th two ryntrs; msurs th ngl twn th two prinipl inrtil xs; is th ngl twn th intrstion of th prinipl inrtil xis of i n th lin onnting th ryntrs; is th ngl twn th intrstion of th prinipl inrtil xis of j n th lin onnting th ryntrs. Susquntly, h RG L ws unfol using th Proilisti Unfoling Strtgy sri in Stion IV, whih ws mpirilly shown in th pst to th mor promising on [18]. Th irt RGs L wr otin omposing th lls tth to h g in th following wy: th ll [,, ] ws tth to th g from i to j; th ll [,, ] ws tth to th g from j to i. inlly, th RNN ws trin to prit whthr h no in th trs longs to f or not. Thn, using th trin RNN, fs wr loliz in givn img, prforming th following stps: 1) Th img is sgmnt n th orrsponing RG L is uilt; 2) Th RG L is unfol, prouing forst of trs; 3) h tr is pross y th trin RNN. Th ntwork prits whthr th root no of h tr is prt of f or not; 4) jnt rgions prit s prt of f r mrg togthr in orr to omput th minimum ouning oxs tht ontin th fs. 5) f is rogniz if th prit ouning ox quls th tru ouning ox. Noti tht, in th gnrl s of ojt ttion, if th numr of istint ojts whih must tt is qul to n, thn n RNNs must trin. h RNN is trin to loliz prts of singl ojt.

5 ig. 6. Rgion j Rgion i turs stor into th g ll. RNN RNN Rll Prision rhittur ury Rt on lyr 5 stt nurons 77.29% 85.22% 78.23% on lyr 10 stt nurons 83.62% 90.88% 85.33% on lyr 15 stt nurons 73.22% 79.85% 72.18% TL I RSULTS OTIN Y T PROPOS RSULT, VRYING T RNN RITTUR uring th xprimnttion, svrl rursiv nurl ntworks wr trin with th im of trmining th st rhittur. Th prformn hiv y som trin nurl ntworks r rport in Tl V. Th ury rt is th numr of rgions orrtly lssifi (s prt of f or not) ivi y th whol numr of rgions in th tst st, whil th rll n th prision rt r omput onsiring tt fs. f is onsir tt if, givn th prit n th orrsponing orrt ouning ox, th rtio omput iviing th intrstion of th ouning oxs y thir union is smllr thn prfin thrshol. In our xprimnts, this thrshol ws st to 90%. Gnrlly, th rll n th prision rt r grtr thn th ury rt. In ft, f n orrtly loliz vn if som prts of it r not orrtly lssifi. on lyr RNN with 10 stt nurons yils th st prformns. Th otin rsults n furthrly improv onsiring tht som fls fs orrspon to vry smll ouning oxs. ssuming tht f, or in gnrl n ojt, n not smllr thn prfin thrshol, this kin of rronous ttion n oun. owvr, this kin of post prossing ws not introu, in orr to vlut th gnrlity of our pproh. VI. ONLUSIONS In this ppr, w propos nw pprn s mtho for tting ojts in imgs, whih r rprsnt y Gs L. This pproh is invrint unr img trnsltions n rottions, u to th invrin of th grphil rprsnttion us. Th mtho lolizs th prts tht onstitut th ojts, using th lrning pility of nw rursiv nurl ntwork mol, l to l with Gs L. urthrmor, th ttion is prform without ny huristis or priori informtion on th spifi ojt mol. Th xprimntl rsults vlit th fftivnss of th propos pproh. KNOWLGMNT This rsrh ws rri out in ollortion with lsg S.p.. Gno Itly. RRNS [1] M.-. Yng, J. Krigmn, n N. huj, tting fs in imgs: survy, I Trnstions on Pttrn nlysis n Mhin Intllign, vol. 24, no. 1, pp , Jnury [2] S. MKnn, Y. Ry, n S. Gong, Trking olour ojts using ptiv mixtur mol, Img n Vision omputing, vol. 17, no. 3/4, pp , [3] T.K. Lung, M.. url, n P. Pron, Proilisti ffin invrints for rognition, in Proings I onfrn on omputr Vision n Pttrn Rognition, 1998, pp [4] P. Shin, Prossing n Rognizing 3 forms, Ph.. thsis, Msshustts Institut of Thnology, [5]. Shnirmn n T. Kn, sttistil mtho for 3 ojt ttion ppli to fs n rs, in Proings of I onfrn on omputr Vision n Pttrn Rognition, 2000, pp [6]. Ppgorgiou, M. Orn, n T. Poggio, gnrl frmwork for ojt ttion, in Proings of th 6th I Intrntionl onfrn on omputr Vision, 1998, pp [7]. Moghm n. Pntln, Proilisti visul lrning for ojt rognition, I Trnstion on Pttrn nlysis n Mhin Intllign, vol. 19, no. 7, pp , July [8]. Spruti n. Strit, Suprvis nurl ntworks for th lssifition of struturs, I Trnstions on Nurl Ntworks, vol. 8, pp , [9] P. rsoni, M. Gori, n. Spruti, gnrl frmwork for ptiv prossing of t struturs, I Trnstions on Nurl Ntworks, vol. 9, no. 5, pp , Sptmr [10]. Kühlr n. Gollr, Inutiv lrning in symoli omins using strutur rivn rurrnt nurl ntworks, in vns in rtifiil Intllign, G. Görz n S. öllolr, s., pp Springr, rlin, [11] M. inhini, M. Gori, n. Srslli, Prossing irt yli grphs with rursiv nurl ntworks, I Trnstions on Nurl Ntworks, vol. 12, no. 6, pp , [12] M. Gori, M. Mggini, n L. Srti, rursiv nurl ntwork mol for prossing irt yli grphs with ll gs, in Proings of th Intrntionl Joint onfrn on Nurl Ntworks, 2003, pp [13] M. inhini, M. Mggini, L. Srti, n. Srslli, Rursiv nurl ntworks for prossing grphs with lll gs, in Proings of SNN 2004, rugs (lgium), pril 2004, pp [14]. Wu, Q. hn, n M. Yhi, n pplition of fuzzy thory: ttion, in Intrntionl Workshop on utomti n Gstur Rognition, Zurih (Switzrln), Jun , pp [15] Y. i n Y. Nkno, xtrtion of fil imgs from omplx kgroun using olor informtion n sgl mtris, in Intrntionl Workshop on utomti n Gstur Rognition, Zurih (Switzrln), Jun , pp [16] M. inhini, M. Gori, n. Srslli, Rursiv prossing of yli grphs, I Intrntionl Joint onfrn on Nurl Ntworks, pp , [17] M. inhini, M. Gori, L. Srti, n. Srslli, kpropgtion through yli struturs, in LNI - I*I 2003: vn in rtifiil Intllign,. pplli n. Turini, s., Pis (Itly), Sptmr 2003, pp , LNS Springr. [18] M. inhini, P. Mzzoni, L. Srti, n. Srslli, spotting in olor imgs using rursiv nurl ntworks, in IPR - T3 Intrntionl Workshop on rtifiil Nurl Ntworks in Pttrn Rognition, M. Gori n S. Mrini, s., lorn (Itly), Sptmr 2003.