562 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 20, NO. 5, MAY Here, d i 1 min. and the vector a j.

Transcription

1 562 IEEE TRNSCTIONS ON PTTERN NLYSIS ND MCHINE INTELLIGENCE, VOL. 20, NO. 5, MY 1998 Fast Desgn of Reduced-Complexty Nearest-Neghbor Classfers Usng Trangular Inequalty Eel-Wan Lee and Soo-Ik Chae bstract In ths paper, we propose a method of desgnng a reduced complexty nearest-neghbor (RCNN) classfer wth near-mnmal computatonal complexty from a gven nearest-neghbor classfer that has hgh nput dmensonalty and a large number of class vectors. We appled our method to the classfcaton problem of handwrtten numerals n the NIST database. If the complexty of the RCNN classfer s normalzed to that of the gven classfer, the complexty of the derved classfer s 62 percent, 2 percent hgher than that of the optmal classfer. Ths was found usng the exhaustve search. Index Terms Nearest-neghbor classfer, trangular nequalty, computatonal complexty, NIST database, fast desgn. F ²²²²²²²²²²²²²²²² The authors are wth the School of Electrcal Engneerng, Seoul Natonal Unversty, San 56-1, Shnlm-dong, Kwanak-gu, Seoul, Korea. E-mal: {wan; chae}@belle.snu.ac.kr. Manuscrpt receved 15 Oct. 1996; revsed 5 Mar Recommended for acceptance by I. Seth. For nformaton on obtanng reprnts of ths artcle, please send e-mal to: tpam@computer.org, and reference IEEECS Log Number INTRODUCTION THE nearest-neghbor (NN) classfer wth representaton vectors <, c for a gven nput x, calculates the dstance from c to x for all and produces a representaton vector, c mn such that ts dstance to x s the smallest [1], [2]. Ths NN classfer wth multple representaton vectors per class, s smple and powerful, but t suffers from huge computatonal complexty f the number of classes s large. Therefore, much work on reducng ts computatonal complexty as much as possble has been reported [2]. One method of dong ths s to reorganze the search procedure by preprocessng the representaton vectors. Ths was frst explored n the paper of Fsher and Patrck [3] and n many other related papers [2]. These fast search algorthms reduce the computatonal complexty wthout reducng the number of representaton vectors so as not to ncrease the msclassfcaton rate of the NN classfer. nother method that can be ncorporated before applyng the frst one s to reduce the number of representaton vectors so long as the ncrease n the msclassfcaton rate remans acceptable. Ths reducton can be obtaned wth a proper selecton or tranng procedure, called edtng [2]. The effectveness of the fast search algorthms usng these two methods s relatvely reduced f the nput space dmenson D s ncreased or the number of representaton vector s reduced. It s reported n [4] that the expected number of dstance calculaton n NN classfer s O(N 1-1/D ). Most fast search algorthms [2] have outlned the asymptotc behavor of ther performances for relatvely low nput space dmenson (D = 2 ~ 10) and a large number of representaton vectors (N = 1,000 ~ 10,000). In the VQ of mages, D s typcally 16 and N s reduced to 256, or 512 after optmzaton of the representaton vectors. There has also been much work on reducng large computatonal complexty n the encodng procedure of the VQ [5]. In partcular, several fast search algorthms usng trangular nequalty elmnaton (TIE) [6], [7], [8] were recently reported to reduce the computatonal complexty of encodng procedure further n comparson wth the algorthms usng PDE (Partal Dstance Elmnaton), K-d tree and other nequaltes especally n the mage codng applcatons. In these fast search algorthms, nequalty (1) s used as a necessary condton for the calculaton of the dstance d(x, c ),.e., the calculaton of the dstance between the current representaton vector, c, and an nput vector, x, s performed only f the condton s satsfed. dxa (, j) dc (, aj) < d 1 mn. (1) Here, d 1 mn s the current mnmum dstance, whch s MIN dxc (, ), j j= 1, L, 1 and the vector a j s called an anchor vector. It s reported that three or four anchor vectors are enough n the VQ codng of mages [6]. Consequently, most prevous work on the fast encodng procedure for the mage VQ dd not focus on a systematc method of determnng the number of anchor vectors by fully explotng the nformaton wthn the tranng data, such as the nput space dmenson, the number of representaton vectors, and ts varance. The classfcaton procedure n the NN classfer s smlar to the encodng procedure of the VQ. Therefore, we focus on the desgn problem of a classfer wth reduced computatonal complexty after proper selecton of representaton vectors. We wll denote these properly edted representaton vectors as class vectors. We propose a reduced-complexty nearest-neghbor (RCNN) classfer usng the TIE, n whch a parameter to be optmzed s the number of anchor vectors. In the RCNN classfer wth class vector {c }, an anchor vector a j s defned as a class vector such that a j {c j } and ts dstance to the nput s always calculated. Note that ths defnton s slghtly dfferent from that n [6] because the anchor vectors defned here are selected only from the class vectors. We also propose a fast algorthm of selectng an anchor vector set that mnmzes the computatonal complexty of the RCNN classfer. Ths paper s organzed lke the followng: In Secton 2, we ntroduce the RCNN classfer wth a parameter M, the number of anchor vectors. We also propose a desgn procedure of the classfers for each parameter M {1,..., N} n Secton 3. We then ntroduce the computatonal complexty curve that plots the computatonal complexty versus the parameter M n Secton 4. Wth examples, we show that the optmal value of M that mnmzes the computatonal complexty depends on the nose of the data as well as other propertes of the data. We explan the fast desgn procedure of the near-optmal RCNN classfer n Secton 5 and apply t to the classfer desgn for the handwrtten numerals n the NIST data base HWDB-1 [9] n Secton 6. Then we conclude the paper n Secton 7. 2 REDUCED COMPUTTIONL COMPLEXITY NN CLSSIFIER For a gven classfer wth a class vector set U = {c = 1,..., N}, = {a = 1,..., M} and B = {b b U, b } denote the anchor vector set and the nonanchor vector set of an RCNN classfer, respectvely. The RCNN classfer outputs a class vector c mn U for an nput x f ts dstance to the nput s the smallest among the class vectors. We wll call c mn the NN (nearest neghbor) vector. The classfer frst calculates the dstance d(x, a ) to each anchor vector a. The dstance to a nonanchor vector b s calculated only f ts greatest lower bound defned n (2) s less than the current mnmum of the nput dstances to the class vectors that have been calculated so far. d ([E, ) = d( [D, ) d( E, D), (2) LOW MX D j j j /98/$ IEEE

2 IEEE TRNSCTIONS ON PTTERN NLYSIS ND MCHINE INTELLIGENCE, VOL. 20, NO. 5, MY Fg. 1. Search phases n the reordered ndex for N = 10. In ths example, the wnner vector s b 3 and the ntal nonanchor search vector s b 5. The search sequence of phase II s represented wth sold lnes whle that of phase III s represented wth dashed lnes. 2 k7 2 7 c U dlow ([E, ) < dmn (2B) Note that f d xb, = MIN d xc,, dlow ([E, k) d( [E, k) < d( [E, ) for all b n B. Therefore, the dstance d(x, b k ) s always calculated because ts greatest lower bound s always less than the current mnmum dstance. Consequently, the performance of the RCNN classfer usng nequalty (2B) s equal to that of the gven NN classfer. We need (N M) M memory locatons to store the precalculated dstance between each par (b, a j ) for b B and a j. To get the mnmum dstance as early as possble, n many fast search algorthms the search lst s rearranged by usng a proper dstance estmate such as the dstance from the orgn [6], [7], [8]. Smlarly, we rearrange the nonanchor vectors n the ascendng order of ther dstances to an arbtrarly chosen vector b f so that ths orderng can be used n the zg-zag scan as shown n Fg. 1 [10]. fter calculatng the dstances to M anchor classes, we fnd a mnmum among them. Then, we calculate the greatest lower bound of all the nput dstances to each nonanchor vector, and determne a nonanchor vector b 0 whose greatest lower bound s the mnmum. Startng from the vector b 0, we test the condton (2B) for nonanchor vectors n the zg-zag order as shown n Fg. 1. The flowchart of the search procedure of the RCNN classfer s llustrated n Fg. 2. The dstance calculatons n the RCNN classfer can be decomposed nto three phases. The dstance calculatons for M anchor vectors are n phase I. If the NN vector s one of nonanchor vectors, the dstance calculatons for the nonanchor vectors untl the NN vector, are n phase II. If the NN vector s one of the anchor vectors, no calculaton s undertaken n phase II. ll the dstance calculatons undertaken after fndng the NN vector, are n phase III. In Fg. 1, for example, the dstance calculatons for {a 1, a 2, a 3 } are n phase I, those for {b 3, b 4, b 5, b 6 } are n phase II, and those for {b 1, b 2, b 7 } are n phase III. Each phase s dfferent from the others n many respects, especally n the varaton of the computatonal complexty whch depends on the nose n the data. The number of dstance calculatons n phase I s constant and equal to M. The number of dstance calculatons n phase III depends only on the nose n the data. The number of dstance calculatons n phase II depends on both the tghtness of the greatest lower bounds and the nose n the data. Note that the dstance from the NN vector to the nput s not zero because of the nose n data. Therefore, the Fg. 2. Search procedure of the RCNN classfer usng TIE. computatonal complexty s dependent on the nose n data. The tghtness of the greatest lower bound, whch affects the computatonal complexty n phase II, s a functon of the parameter M, as well as of the nose n the data. 3 SELECTION OF THE NCHOR VECTORS We should select an anchor vector set that mnmzes the computatonal complexty of the RCNN classfer from N C M canddates for a gven number M by explotng the mutual dstance relatons among the class vectors. Frst, we construct a sequence of anchor vector sets for = 0,..., N startng wth the empty set 0 = φ by applyng a greedy algorthm. Smlarly, we construct a sequence of nonanchor vector set B for = 0,..., N. dscrmnatve measure, P e (. ), s requred to determne whch class vector n B outperforms all the other nonanchor vectors n dstngushng all the remanng nonanchor vectors. We select a class vector a as an anchor vector and add t to to form +1 f a = arg max P( r ) (3) rk B To defne the dscrmnatve measure, we defne the nearest vector n j of b j, for each b j, such that n j {b j }, n j b j and d( Qj, EM ) d( El, Ej ) for all b l {b j }. To smplfy the error model, we neglect the probablty of class b j beng msclassfed to classes whch are not n j because ther dstances to the nput are larger than that of the class n j. Therefore, the dscrmnatve measure ncreased after the addton of r k for the constructon of +1 from s defned as follows: e k

3 564 IEEE TRNSCTIONS ON PTTERN NLYSIS ND MCHINE INTELLIGENCE, VOL. 20, NO. 5, MY 1998 Fg. 3. Computaton of the proposed dscrmnatve measure. k Pe ( Uk ) = Jd { U } LOW ( Ej, Qj) dlow( Ej, Qj) L. (4) Ej B { Uk } Each term n the summaton corresponds to the nonnegatve change of the greatest lower bound for a nonanchor vector b j resultng from the addton of an anchor vector, r k. By applyng the proposed algorthm to each M {1,..., N} sequentally, we obtan a sequence of the class vectors (a 1, a 2,..., a N ) before determnng the optmal number M mn of anchor vectors that mnmzes the computatonal complexty of the RCNN classfer. set that contans frst M elements of the sequence s defned as the anchor vector set of the RCNN classfer wth a specfed M. 4 EXMPLES OF THE COMPUTTIONL COMPLEXITY CURVE We obtaned the computatonal complexty curves of the proposed algorthm for the problems wth real-numbered nput spaces wth dmenson D {4, 8, 12, 16} and 128 class vectors, whch are plotted n Fg. 4. Here, the x-axs represents the number of anchor vectors and the y-axs represents the expected search tme K(M). The unt of expected search tme s the tme requred for the dstance calculaton per class vector. Ths represents the computatonal complexty of the RCNN classfer wth a real-valued number between zero and N. The 128 class vectors and 1,000 test vectors were selected randomly and unformly n the R D space. s D ncreases, the effectveness of the fast search algorthm decreases rapdly due to the curse of dmensonalty as shown n Fg. 4b. Selectng data wth unform dstrbuton results n very nosy data n the classfcaton problem. If we select the test vectors near the class vectors, thus reducng the nose n the data, we obtan a dfferent computatonal complexty curve, as shown n Fg. 4c, where the expected search tme s much smaller than that for the nosy data. Ths s manly because the number of dstance calculatons n phase III s reduced. Therefore, the expected search tme, K(M), of the classfer can be decomposed nto two terms: a lnear term and a nonlnear term: K( M) = M + f σ ( M). (5) The lnear term corresponds to the number of dstance calculatons n phase I, whch s constant for all nput vectors. The nonlnear term corresponds to the number of dstance calculatons n phases II and III, whch depends on the propertes of the nput Fg. 4. Computatonal complexty curves of the RCNN classfers. (a) Top left: Two components of the searches. (b) Top rght: Computatonal complexty curve for noseless data. (c) Bottom left: Computatonal complexty curve for nosy data. (d) Bottom rght: M mn as a functon D and nose.

4 IEEE TRNSCTIONS ON PTTERN NLYSIS ND MCHINE INTELLIGENCE, VOL. 20, NO. 5, MY vector. We do not requre any detaled knowledge about the nonlnear component f ( σ M). It s conjectured, however, to be nondecreasng functon of M. Note that the number of anchor vectors, M mn, that mnmzes the search tme s located near the M value of the ntersecton pont of the two curves. s shown n Fg. 4d, M mn decreases f the nose n the data s reduced snce the nonlnear term f M) that depends on the nose n nput data s reduced. σ ( 5 OPTIMIZTION OF THE RCNN CLSSIFIER It takes a large amount of computaton to obtan the computatonal complexty of the RCNN classfer wth many nput data for each M. The tme nvolved n obtanng a computatonal complexty curve s proportonal to the number of complexty computaton. Fortunately, we do not have to obtan the locatons of all the ponts n the computatonal complexty curve f we have a pror knowledge on the general shape of the computatonal complexty curve n Fg. 4a. By reducng the number of complexty computatons, we can reduce the classfer desgn tme substantally, although we cannot reduce the computaton tme of K(M) much for each M. Frst, we note that the tme n obtanng the computatonal complexty curve for the noseless data s much shorter than that for the nosy data because the nput data s lmted only to class vectors n the noseless case. Therefore, we use the optmal number of anchor vectors for noseless data as the ntal guess M nt, whch s smaller than the optmal number of anchor vectors for nosy data. We denote an th estmate of M mn as M() whle M(0) = M nt. We explot the fact that the optmal pont s near the ntersecton pont of the lnear and nonlnear terms. If, whch s fσ (M) M, s postve, the optmal M s larger than the current estmate M(). Therefore, we update the current estmate wth (6). M( + 1) = M( ) + α ( K( ) 2 M( )). (6) If s negatve, the next estmate wll be decreased. Ths procedure contnues untl the dfference becomes too small to move by the step sze of 1.0, as shown n Fg. 5. The number of the updatng steps should be kept as small as possble because t requres much tme to fnd the expected search tme K(M()) for M() n each updatng step. We selected the update parameter α of 0.5, whch results n a smple reestmaton formula (7) to force the rato of M()/K() to 0.5. M( + 1) = 05. K( ). (7) lthough we assumed that the optmal M les near the ntersecton pont of two curves of M and fσ(m), the optmal pont s not located exactly at the ntersecton pont. Consequently, a correcton to the update rule (7) s requred. From the smulaton results n Secton 4, we know that the nonlnear component s a monotoncally nonncreasng functon and that t becomes flat as the nose n the data, σ, or the nput space dmenson, D, ncreases. Therefore, we assume that f ( M ) σ = N exp( a M ), where a should be selected accordng to the flatness of the nonlnear component f ( σ M ). Wth the smulatons, we confrmed that M mn les to the left of the M cross n most nontrval cases, whch corresponds to the condton of 0 < a M cross < 1. Therefore, the update rule needs to be modfed to (8) because the lnear term M s (50 δ) percent of the total search tme when t s mnmum. We used δ = 10 percent n the experment descrbed n the next secton. Note that δ should be selected carefully, based on the nose n the data. 50 δ M ( + 1) = 100 K ( ) (8) = 0.4 K ( ) for δ = 10percent 6 EXPERIMENTS We appled the optmzaton procedure n Secton 5 to desgn an Fg. 5. Sequence of the optmal M s estmate. Fg. 6. Numerals extracted from the box4 of hsf_0/f0024_4 n the NIST database. RCNN classfer for the handwrtten numerals n the NISTdatabase HWDB-1 [9]. We segmented the numerals n box3 through box5 wth connected component analyss [11]. ll the numerals were normalzed to as shown n Fg. 6. The classfer was traned untl percent classfcaton rate for 1,000 tranng vectors was acheved. The classfer has eght class vectors for each numerals, summng to 80 class vectors for 10 numerals. Ths conventonal NN classfer acheved 95.2 percent classfcaton rate for 1,000 test vectors. lthough the graph n Fg. 7a shows local mnma unlke those n Fg. 4, we obtaned a soluton of good qualty wth the proposed desgn algorthm. Ths algorthm was fnshed n fve steps, wth the sequence M(), as shown n Fg. 7b, fndng an RCNN classfer wth M mn = 20 and K mn = 48.7 whle M opt = 21, K opt = 46.8 for the optmal classfers obtaned, usng exhaustve search for all possble values of M. Wth the proposed fast desgn of the classfer, the desgn tme s decreased by 93 percent whle the search tme of the classfer s ncreased by about 2 percent compared to the optmal classfer. The search tme and desgn tme of the RCNN classfer compared to the conventonal NN classfer are lsted n Table 1. The RCNN classfer obtaned wth the proposed fast desgn has 38 percent reducton n the search tme compared wth the full search NN classfer. In contrast, although the classfer obtaned wth L s algorthm of three anchor vectors shows a very good result n mage VQ codng applcatons [5], t has only 20 percent reducton n the search tme for ths classfcaton problem, whch corresponds to the RCNN classfer wth three anchor vectors. Ths result supports the necessty of wde-range search for anchor vectors, especally n the classfcaton problem. Note that the number of anchor vectors requred n the classfcaton problem, especally wth hgh TBLE 1 SERCH TIME ND DESIGN TIME IN THE RCNN CLSSIFIER No. of dstance calculaton (relatve to full search %) No. of M values checked n classfer desgn Exhaustve desgn 46.8 (60%) Proposed desgn 48.7 (62%) 80 5 The number of anchor vectors (M) 21 20

5 566 IEEE TRNSCTIONS ON PTTERN NLYSIS ND MCHINE INTELLIGENCE, VOL. 20, NO. 5, MY 1998 (a) (b) Fg. 7. Computatonal complexty curves for the NIST database and desgn trace. (a) Computatonal complexty curve for N = 80. (b) Desgn trace of the classfer. nput dmenson and properly edted class vectors, s much larger than that n the mage VQ codng problem n order to obtan substantal reducton n classfcaton tme. 7 CONCLUSION For a gven NN classfer wth hgh-nput space dmenson and a large number of class vectors, we addressed the problem of the fast desgn of an RCNN classfer of near-mnmal computatonal complexty wth mnmal efforts. We explaned an algorthm of the RCNN classfer that explots the trangular nequalty and the concept of anchor vectors that are lmted to the class vectors. We employed a new dscrmnatve measure and a greedy algorthm n determnng the sequence of anchor vectors whch can be used to select an anchor vector set for a gven number of anchor vectors. Based on the shape of the computatonal complexty curve obtaned by experment, we ntroduced an algorthm for determnng a near-mnmal RCNN classfer effcently by recursvely locatng the ntersecton ponts of the lnear and nonlnear terms nstead of drectly fndng the mnmum value. We appled the proposed method to the classfcaton problem of handwrtten numerals n the NIST database and obtaned an RCNN classfer wth the search tme of 62 percent compared wth that of the gven fullsearch NN classfer. Ths s qute close to the 60 percent search tme of the optmal RCNN classfer obtaned by exhaustve search. The results showed the effectveness of the proposed method and ts potental for the fast desgn of the RCNN classfers. lthough we crcumvented the problem of computng the computatonal complexty for all M wth smplfcaton of K(M), further studes on ts analytc form are requred to tackle a more challengng NN classfer desgn problem. [3] [4] [5] [6] [7] [8] [9] F.P. Fsher and E.. Patrck, Preprocessng lgorthm for Nearest Neghbor Decson Rules, Proc. Nat l Electronc Conf., vol. 26, pp , Dec J.H. Fredman, F. Baskett, and L.J. Shustek, n lgorthm for Fndng Nearest Neghbors, IEEE Trans. Computers, vol. 24, pp. 1,000-1,006, Oct Gersho and R.M. Gray, Vector Quantzer and Sgnal Compresson. Boston: Kluwer cademc Publsher, W. L and E. Salar, Fast Vector Quantzaton Encodng Method for Image Compresson, IEEE Trans. CS for Vdeo Tech., vol. 5, no. 2, pp , pr G. Pogg, Fast lgorthm for Full-Search VQ Encodng, Electron. Lett., vol. 29, pp. 1,141-1,142, June C.-M. Huang, Q. B, G.S. Stles, and R.W. Harrs, Fast Full Search Equvalent Encodng lgorthm for Image Compresson Usng Vector Quantzaton, IEEE Trans. Image Processng, vol. 1, no. 3, pp , July NIST Specal Database 1(HWDB), Standard Reference Data, Natonal Insttute of Scence and Technology, 221/323, Gathersburg, Md. [10] S.-W. Ra and J.-K. Km, Fast Mean-Dstance-Orented Partal Codebook Search lgorthm for Image Vector Quantzaton, IEEE Trans. CS, vol. 40, no. 9, pp , Sept [11] W.K. Pratt, Dgtal Image Processng, 2nd ed. New York: John Wley, CKNOWLEDGMENTS The authors are grateful to the revewers for ther helpful comments about ths paper. Ths research was supported n part by the Insttute of Informaton Technology ssessment (IIT), Korea, under research grant IT2-I2. REFERENCES [1] R.O. Duda and P.E. Hart, Pattern Classfcaton and Scene nalyss. New York: John Wley, [2] B.V. Dasarathy, Nearest Neghbor (NN) Norms: NN Pattern Classfcaton Technques. Los lamtos, Calf.: IEEE CS Press, 1991.