Use of Fourier and Karhunen-Loeve Decomposition for Fast Pattern Matching With a Large Set of Templates

Transcription

1 IEEE RANSACINS N PAERN ANAYSIS AND ACHINE INEIGENCE, V. 9, N. 8, AUGUS Use of Fourer and arhunen-oeve Decomposton for Fast Pattern atchng Wth a arge Set of emplates chhro Uenohara, ember, IEEE, and akeo anade, Fellow, IEEE Abstract We present a fast pattern matchng algorthm wth a large set of templates. he algorthm s based on the typcal template matchng speeded up by the dual decomposton; the Fourer transform and the arhunen-oeve transform. he proposed algorthm s approprate for the search of an obect wth unknown dstorton wthn a short perod. Patterns wth dfferent dstorton dffer slghtly from each other and are hghly correlated. he mage vector subspace requred for effectve representaton can be defned by a small number of egenvectors derved by the arhunen-oeve transform. A vector subspace spanned by the egenvectors s generated, and any mage vector n the subspace s consdered as a pattern to be recognzed. he pattern matchng of obects wth unknown dstorton s formulated as the process to extract the porton of the nput mage, fnd the pattern most smlar to the extracted porton n the subspace, compute normalzed correlaton between them at each locaton n the nput mage, and fnd the locaton wth the best score. Searchng for obects wth unknown dstorton requres vast computaton. he formulaton above makes t possble to decompose hghly correlated reference mages nto egenvectors, as well as to decompose mages n frequency doman, and to speed up the process sgnfcantly. Index erms emplate matchng, pattern matchng, arhunen-oeve transform, Fourer transform, egenvector. INRDUCIN EPAE matchng has been a useful and famlar tool to detect an obect n an mage []. emplate matchng fnds a pattern n the mage that s smlar to a gven reference mage usng correlaton or normalzed correlaton as the measurement of smlarty. he drawback of template matchng s ts hgh computaton cost. It s not robust for rotaton and other dstorton of obects, ether. Dependng on the partcular pont of vew, the obect can appear as a number of dfferent-lookng mages. emplate matchng can be appled to obects wth unknown dstorton by dong matchng wth many reference mages from a number of ponts of vew. he dffculty les n ts hgh computaton cost. he more templates used for detectng the obect from a wde range of vews precsely, the hgher the computaton cost becomes. A multresoluton mage structure can reduce the search area, and, therefore, the computaton cost. In the coarse-tofne strategy, mages and templates at dfferent levels of resoluton are generated, and the templates are searched at the lower resoluton frst, and the best-match locaton s found. he neghborhood of the best-match locaton s searched n an mage usng ncreasngly hgher resolutons, up to the orgnal resoluton mage. he coarse-to-fne. Uenohara s wth oshba R&D Center, 4-, Ukshma-cho, awasakku, awasak,, Japan. E-mal: mue@eml.rdc.toshba.co.p.. anade s wth the Robotcs Insttute, Carnege ellon Unversty, Smth Hall, Pttsburgh, PA E-mal: tk@cs.cmu.edu. anuscrpt receved Apr Recommended for acceptance by H.R. eshavan. For nformaton on obtanng reprnts of ths artcle, please send e-mal to: transpam@computer.org, and reference IEEECS og Number 54. strategy works well for an obect wth sgnfcantly low spatal frequency components whch are retaned n a low resoluton mage. However, t does not work well for cluttered scenes and obects whose detals need to be checked n order to dstngush one from another. If an obect s mssed n a lower resoluton mage, t cannot be recovered at later stages. o mplement the coarse-to-fne strategy, t s also necessary to decde how many levels of resoluton are to be used. It depends on the obects and scene. here s a trade-off between reducng computaton and ncreasng the rsk of mssng obects. Recently, fast template matchng for multple rotated templates has been proposed [], [3]. he arhunen-oeve transform s frst appled to a set of rotated mages, and egenvectors are extracted from them. Each template n ths set s approxmated by a lnear combnaton of these egenvectors. Snce rotated templates dffer slghtly from each other and are hghly correlated, templates can be approxmated reasonably well by a smaller number of egenvectors. Normalzed correlaton between rotated templates and the nput mage s effcently computed by substtutng the approxmatons for the templates when computng the normalzed correlaton. he computaton cost for detectng targets from the whole mage s stll hgh. ultresoluton mages are used to reduce computaton cost, whch leads to the same dffculty for cluttered scenes, as descrbed before. We wll present a fast pattern matchng algorthm wth a large set of templates wthout multresoluton mages. he algorthm s based on the typcal template matchng, whch s the search for the gven pattern n the 6-888/97/$. 997 IEEE

2 89 IEEE RANSACINS N PAERN ANAYSIS AND ACHINE INEIGENCE, V. 9, N. 8, AUGUS 997 mage, speeded up by the dual decomposton, the Fourer transform, and the arhunen-oeve transform. he matchng crteron s normalzed correlaton. Patterns wth dfferent dstorton dffer slghtly from each other and are hghly correlated. he mage vector subspace requred for effectve representaton can be defned by a small number of egenvectors derved by the arhunen-oeve transform. Instead of approxmatng each template as a lnear combnaton of the egenvectors, a vector subspace spanned by the egenvectors s generated. he vector subspace nvolves not only the dscrete reference mages wth dfferent dstorton, but also ther nterpolaton. Any mage vector n the vector subspace s consdered to be a pattern to be recognzed. he pattern matchng of obects wth unknown dstorton s now formulated as the process to extract a porton of the nput mage, fnd the pattern most smlar to the extracted porton n the vector subspace, compute the normalzed correlaton at each locaton n the whole nput mage, and fnd the locaton wth the hghest score. It s well known that the computaton of correlaton can be reduced greatly by usng the Fourer transform, especally when the mage sze s large. he formulaton above makes t possble to apply the Fourer transform n an effcent way and speeds up the process sgnfcantly. It should be noted that the normalzed correlaton to multple reference mages represented as lnear combnatons of the egenvectors s not speeded up by the Fourer transform. he number of Fourer transforms and nverse Fourer transforms s reduced by representng the reference mages as the lnear combnatons of the egenvectors; however, the computaton for the lnear combnaton of the Fourer transform of the egenvectors s not neglgble, and the whole computaton cost s not reduced greatly. he alternatve process s to generate the vector subspace, then fnd the most smlar pattern n the vector subspace, and to compute the normalzed correlaton between them. hs elmnates the computaton for the lnear combnaton. he computaton cost of the proposed pattern matchng n an mage of sze N N s ea + 5fN log N, whle the computaton cost of the normalzed correlaton wth the orgnal P reference mages s eap +fn when the sze of the reference mages are. s the number of egenvectors used for the vector subspace. N s assumed to be a power of two. Its reducton rate s elog N P, whch s the product of the reducton by the arhunen- oeve transform and that by the Fourer transform. he paper s organzed as follows. he vector subspace and normalzed correlaton n the vector subspace s explaned n Secton. In Secton 3, the proposed pattern matchng algorthm, whch s the normalzed correlaton n the vector subspace usng the Fourer transform, s presented, and expermental results are shown n Secton 4. NRAIZED CRREAIN IN HE VECR SUBSPACE. he arhunen-oeve ransform he arhunen-oeve (-) transform s a famlar technque for proectng a large amount of data onto a small dmensonal subspace n pattern recognton and mage compresson [4], [5]. he - transform gves the orthogonal bass functons as the egenvectors of the covarance matrx. hs transform s optmal n that t s a canoncal transform mnmzng the mean square error between a truncated representaton and the actual data. et the set of nput data be x, =,,, P; vectors of dmenson, representng square mages. he covarance matrx of the nput data s P = x - c x - c A P = c hc h () where c s the average mage vector. he vectors e and scalars l are the egenvectors and egenvalues of the covarance matrx A, respectvely. We obtan the optmal approxmaton of the nput data by selectng egenvectors n decreasng order of magntude of the egenvalues and representng each datum by a lnear combnaton of maor egenvectors as = x ª c + p e where p = e, e,, e, x - c error s () c h and mean approxmaton e af P- = l = + he cumulatve proporton m af s useful for decdng the number of egenvectors af + P P- = = N N m l l. Vector Subspace When we search an obect wth unknown dstorton, the straghtforward way s to do template matchng wth a large set of templates. Each template s the ntensty pattern at a dfferent pont of vew. Reference mages wth dfferent ponts of vew dffer slghtly from each other and are hghly correlated. he mage vector subspace requred for effectve representaton can be defned by a small number of egenvectors derved by the - transform. (See Fg..) he maor egenvectors, n addton to the average mage vector c, span a ( + )-dmensonal subspace of all possble mages, and a set of mages n the subspace s consdered as a template to be recognzed [6]. A set of reference mages n the vector subspace s therefore expressed n terms of a lnear combnaton of a fnte set of orthonormal bass:. (3)

3 UENHARA AND ANADE: USE F FURIER AND ARHUNEN-EVE DECPSIIN FR FAS PAERN ACHING 893 where F HG x = p e (a) (b) Fg.. (a) Dstrbuton of reference mages. (b) Example of reference mages. = I c c e = c -pe c - e J p = = s the normalzed average mage vector from whch the proecton nto the subspace spanned by the egenvectors c s subtracted, and p = e e e c..3 Normalzed Correlaton n the Vector Subspace he nput mage s evaluated at each locaton as to how t fts the template by extractng the regon and fndng the pattern most smlar n the vector subspace and computng normalzed correlaton between them. When we extract a porton of mage y, we normalze t so that the average ntensty of the whole pxels s zero,.e., ~ y = =. (4) Fg.. emplate space and the proecton of nput mage. s the number of pxels n the reference mages. hs normalzaton makes matchng nsenstve to the varaton of the background ntensty. he most smlar pattern n the vector subspace s the proecton of the normalzed extracted regon vector y ~ nto the vector subspace (Fg. ). Its normalzed correlaton to the vector y ~ s the largest. he normalzed correlaton between the vector y ~ and a reference vector x s gven by xy C x, y ~ ~ b g =. Replacng the reference mage vector x wth the proecton yelds b ~ x= e ye ~ g = C y, ~ ~ x = he coeffcent vector e = e e (5) ~ p for the proecton ~ x s ~ ~ p = e y,e y, ~,e. he normalzed correlaton score above s the measure of smlarty consderng not only the prestored dscrete P reference mages but also ther nterpolaton. hs makes the system robust aganst the varaton of llumnaton. he computaton cost s greatly reduced compared to the normalzed correlaton to the orgnal P reference mages. he orgnal normalzed correlaton requres ap +f operatons for P reference mages, whle (5) requres only + a f operatons, where can be much smaller than P. 3 NRAIZED CRREAIN USING HE FURIER RANSFR It s well known that the Fourer transform of the correlaton of the two functons s the product of the Fourer transform of the one functon and the complex conugate of another functon [7]. b g = FCx, y FxF y * (6)

4 894 IEEE RANSACINS N PAERN ANAYSIS AND ACHINE INEIGENCE, V. 9, N. 8, AUGUS 997 he nverse Fourer transform of the product above gves the values of the correlaton at dfferent lags. he computaton usng the Fourer transform s much more computatonally effcent than the correlaton n spatal doman, especally n the case of large sze mages. 3. Normalzed Correlaton n the Vector Subspace Usng the Fourer ransform Normalzed correlaton n the vector subspace s computed as below: e C y, ~ ~ = b xg = (7) We compute egenvectors e l from normalzed reference mages x l =. he summaton of all pxels n each egenvector s, therefore, = zero. e l = = he correlaton between the egenvectors e l and the normalzed porton of the nput mage ~ y s the same as the correlaton between the egenvectors e l and the porton of the nput mage y e y ~ = e y - y = e y l c = l y s the average of the porton of the nput mage y. he correlaton above can be computed effcently usng the Fourer transform as: FF the two data sets e l and y, multply the one resultng transform by the complex conugate of the other, and nversely transform the product. he norm of the normalzed porton of the mage y ~ at locaton (, ) can also be computed usng the Fourer transform: N c h F H ~y = l - = G l= l= l= l y y yl y J - e h l I - * (8) (9) () y = I ƒ y = F F I F y () - * y = yl = I ƒ y = F FI F y () N l= where I s the matrx of the sze wth all the elements of unty. We obtan the normalzed correlaton (7) by computng the summaton of the correlaton between the egenvectors e l and the porton of the nput mage y, computng the square root of t, and then dvdng t by the norm of the normalzed porton of the nput mage (). 3. ff-ne and n-ne Processng of the Proposed Pattern atchng We wll show the fast pattern matchng algorthm wth a large set of templates to detect the locaton of the obect and the best matched template whch ndcates the geometrc dstorton parameters of the obect. It nvolves off-lne processng and on-lne processng. In off-lne processng, we frst gather or generate reference mages of the obect wth dfferent dstorton parameters. x Æ x, x,, x p (3) We compute the average ntensty of each reference mage and subtract them from the reference mages, and normalze the reference mages to unt energy: x = = (4) xx= (5) We compute the average mage vector c and the frst egenvectors by the arhunen-oeve transform. We select egenvectors whose correspondng egenvalues are the largest. We use a cumulatve proporton af = P P- = = N N m l l for decdng the number of egenvectors. hs measurement shows well how many egenvectors contrbute to approxmate reference mages. We subtract the proecton of the average mage vector c nto the subspace spanned by the egenvectors from the average mage vector c, and normalze t to unt energy, and call t the No. egenvector e. We then calculate coeffcents by proectng reference mages onto the vector subspace spanned by these a + f egenvectors. he coeffcent vector p s the representaton of reference mages n the vector subspace correspondng to dstorton parameters. p = e,e,, e x (6) We compute the Fourer transform of the a + f egenvectors: F e, F e,, F e (7) We generate the matrx of the sze wth all the elements of unty and compute the Fourer transform of t: F I = F NN (8) In on-lne processng, we obtan mage y, and compute the squared mage y y = (9) We then compute the complex conugate of the Fourer transform of the nput mage y and the squared mage y F * * y, F y () We have the Fourer transform of the egenvectors and the matrx I that are precomputed off-lne. We calculate the

5 UENHARA AND ANADE: USE F FURIER AND ARHUNEN-EVE DECPSIIN FR FAS PAERN ACHING 895 (a) Fg. 3. (a) Prnted crcut board, (b) rotated reference mages ( of are shown). (b) ABE RIENAIN F HE SA PAR orentaton(deg) detected orentaton (deg) correlaton between the egenvectors and nput mage, the matrx I and the nput mage, and the matrx I and the squared nput mage. e ƒ y = F F e F y - * I ƒ y = F F I F y - * () () I ƒ y = F F I F y (3) - * We obtan the correlaton between the nput mage and the pattern most smlar n the vector subspace at each locaton n the mage by b g = ƒ = C, ~ x e y (4) he norm of the normalzed porton of the nput mage ~ y at each locaton s computed: F H ~y = I ƒ y - I ƒ y he normalzed correlaton at each locaton s then obtaned: b g C ~ x, b I (5) C ~ x, = (6) We obtan the locaton of the obect by searchng the locaton wth the maxmum score. he vector ey,e ~ y, ~,e that s already computed n () represents the coeffcent vectors ~ p for the proecton of the vector y ~ nto the vector subspace. ~ x s represented as g ~ x = ~ p e = whle each reference mage s represented as x = ~ pe = (7) (8) he dstance e between the coeffcent vectors ~ p and the coeffcent vectors of the reference mage vectors p s computed as: e = ~ p - ~ p = e (9) We fnd the reference mage x wth the mnmum dstance and obtan the dstorton parameter of the obect. 4 EXPERIENA RESUS We have conducted experments to verfy the accuracy and computatonal effcency of the proposed algorthm. he target obect n the experments s a small part on the prnted crcut board shown n Fg. 3a. he sze of the mage s Fg. 3b shows the rotated reference mages obtaned by rotatng the orgnal reference mage at the upperleft corner n Fg.3b. he sze of the reference mages s 5 5, and the number of reference mages s (5 degrees to +5 degrees, every one degree). hey are normalzed so that the average of pxel ntenstes s zero and the energy s unt. ur frst step s to compress the reference mage sets nto the low-dmensonal subspace that captures the most appearance characterstcs of obects by the arhunen-oeve transform. We use = n the experments. he cumulatve proporton of t s.85.

6 896 IEEE RANSACINS N PAERN ANAYSIS AND ACHINE INEIGENCE, V. 9, N. 8, AUGUS 997 Fg. 4. Normalzed correlaton at each locaton. Fg. 5. (a) ower resoluton mages. (b) Normalzed correlaton at each locaton. he Fourer transform of the egenvectors, as well as the Fourer transform of the matrx I, are computed and stored n the fle. At the begnnng of the on-lne operaton, those data are loaded from the fle and used for computaton. 4. Detecton of the Rotated Prnted Crcut In the frst experment, we generate rotated mages of the prnted crcut board synthetcally and use them for nput. We detect the locaton and orentaton of the small part by the proposed algorthm and compare wth the locaton and orentaton of the small part n the mages we generate. able shows the rotaton angle of the mage (whch s the rotaton angle of the small part also) and the rotaton angle of the small part detected by the program. he program computes the coeffcent vector ~ p and the normalzed correlaton score at each locaton n the mage. he program then searches the locaton wth the hghest score, whch s the locaton of the small part, and fnds the coeffcent parameter of the reference mage wth the mnmum dstance from the coeffcent parameter ~ p at the locaton wth the hghest score. In the experment, we use the reference mages rotated every one degree. he resoluton of the detected rotaton angle s one degree. We could obtan subdegree accuracy by applyng a quadratc fttng functon to the correlaton score.

7 UENHARA AND ANADE: USE F FURIER AND ARHUNEN-EVE DECPSIIN FR FAS PAERN ACHING 897 able shows that the proposed algorthm detects the orentaton of the small part reasonably well, consderng that the sze of the reference mages n the experment s 5 5, and one degree rotaton moves each pxel n the reference mages less than one pxel locaton. he program works well even f the small part s located to the orentaton between reference mages. 4. Comparson wth the Coarse-to-Fne Strategy he second experment shows that the proposed algorthm works well for an obect wth hgh spatal frequency components whch cause troubles when we use the coarse-to-fne strategy. Fg. 4 shows the normalzed correlaton score at each locaton detected by the proposed algorthm. he reference mages are the same as n the frst experment and the Fourer transform of the egenvectors and the matrx I are generated. he nput mage s the prnted crcut board shown n Fg. 3a. Fg. 4 shows that the small part s well dstngushed from others. he same type of chps are located horzontally. he cue to dstngush one locaton from another s the characters on the chps and the pattern of the prnted crcut that are hgh spatal frequency components. Fg. 5 shows that the coarse-to-fne search does not work well for the obects lke ths. Fg. 5a shows the low resoluton mages of the prnted board. We generate half resoluton mages and reference mages ust by averagng four neghborhood pxels. Fg. 5b shows the normalzed correlaton results between the prnted crcut board mage and the reference mage at each resoluton. he small part s detected successfully n the half resoluton mage. However, t s mssed n the one-fourth resoluton mage. he computaton tme for the proposed algorthm n the experments s 3.9 seconds on the Sparc /3. he number of reference mages P =, the number of the egenvectors =, the sze of the nput mage s 56 36, and the sze of the reference mage s 5 5. he nput mage s extended to by paddng the last rows wth zeros. We use Fourer transform to compute correlaton n the proposed algorthm so that the sze of the reference mage does not affect the computaton tme. he computaton tme for the normalzed correlaton between the prnted crcut board mage shown n Fg. 3a and the reference mage at the upper-left corner n Fg. 3b s 8.3 seconds. We need P = tmes computaton for computng normalzed correlaton between reference mages, whch leads to a computaton tme of,48.3 seconds; 9.5 tmes the computaton tme n the case of usng the proposed algorthm. he search n the one-fourth resoluton mages requres /56 computaton. he computaton tme n the experment shows that the proposed pattern matchng algorthm reduces computaton more than the search n the onefourth resoluton mage and stll leads to the correct result. 5 CNCUSIN We have presented a novel pattern matchng technque wth a large set of templates. he obect to be recognzed s gven as multple ntensty patterns wth dfferent dstorton parameters such as rotaton angle, scalng factor. he proposed algorthm decomposes the gven pattern by the arhunen-oeve transform, and generates the vector subspace spanned by the maor egenvectors. he algorthm then decomposes the nput mage and the derved egenvectors by the Fourer transform, fnds the pattern most smlar n the vector subspace, and computes normalzed correlaton between the most smlar pattern and the nput mage. he domnant part of the proposed algorthm n computaton s the Fourer transform. he computaton cost of the proposed pattern matchng n the mage of the sze N N s ea + 5fN log N, whle the computaton cost of the normalzed correlaton wth the orgnal P reference mages used for generatng the vector subspace s eap +fn. s the number of egenvectors. he sze of the reference mages s. he arhunen-oeve transform speeds up the process by /P, and the Fourer transform speeds up the process by elog N. In the experment, P =, =, = 5, N = 56, and the proposed algorthm speeds up the process by almost, tmes. REFERENCES [] A. Rosenfeld and A. a, Dgtal Pcture Processng. New York: Academc Press, 98. [] S. Yoshmura and. anade, Fast emplate atchng Based on the Normalzed Correlaton by Usng ultresoluton Egenmages, Proc. IRS '94, unch, Germany, 994. [3]. Amd, Y. esak,. anade, and. Uenohara, Research on an Autonomous Vson-Guded Helcopter, Proc. Ffth RI/SE World Conf. Robotcs Research, Cambrdge, ass., 994. [4]. Fukunaga, Introducton to Statstcal Pattern Recognton. Boston: Academc Press, 99. [5] H. urase and S. Nayar, Vsual earnng and Recognton of 3-D bects from Appearance, Int l J. Computer Vson, vol. 4, no., pp. 5-4, 995. [6]. Uenohara and. anade, Vson-Based bect Regstraton for Real-me Image verlay, Proc. 995 Conf. Computer Vson, Vrtual Realty, and Robotcs n edcne, Nce, France, 995. [7] W. Press, S. eukolsky, W. Vetterlng, and B. Flannery, Numercal Recpes n C. Cambrdge Unv. Press, 99. [8]. Uenohara and. anade, Vson-Based bect Recognton for Real-me Image verlay, Int l J. Computers n Bology and edcne, vol. 5, no., pp. 49-6, 995. [9]. urk and A. Pentland, Egenfaces for Recognton, J. Cogntve Neuroscence, vol. 3, no., 99. chhro Uenohara receved the BS degree n electrcal engneerng from the Unversty of okyo, okyo, Japan, n 985. He oned oshba R&D Center, awasak, Japan, n 985, and currently s a research scentst. He was a vstng research scentst at Carnege ellon Unversty, Pttsburgh, Pennsylvana, from 993 to 995. Hs research areas of nterest are computer vson and robotcs. He s a member of the IEEE.

8 898 IEEE RANSACINS N PAERN ANAYSIS AND ACHINE INEIGENCE, V. 9, N. 8, AUGUS 997 akeo anade receved hs PhD degree n electrcal engneerng from yoto Unversty, Japan, n 974. After holdng a faculty poston at the Department of Informaton Scence, yoto Unversty, he oned Carnege ellon Unversty n 98, where he s currently the U.A. Helen Whtaker Professor of Computer Scence and drector of the Robotcs Insttute. He was a foundng charperson of CU s robotcs PhD program, probably the frst of ts knd. Dr. anade has worked n multple areas of robotcs: vson, manpulators, autonomous moble robots, and sensors. He has wrtten more than techncal papers and reports n these areas. He has been the prncpal nvestgator of several maor vson and robotcs proects at Carnege ellon. Dr. anade s a fellow of the IEEE, a foundng fellow of the Amercan Assocaton of Artfcal Intellgence, and the foundng edtor of the Internatonal Journal of Computer Vson. Dr. anade has been elected to the Natonal Academy of Engneerng as a foregn assocate.