A neural network with localized receptive fields for visual pattern classification

Transcription

1 Unversty of Wollongong Research Onlne Faculty of Informatcs - Papers (Archve) Faculty of Engneerng and Informaton Scences 2005 A neural network wth localzed receptve felds for vsual pattern classfcaton Son Lam Phung Unversty of Wollongong, phung@uow.edu.au Abdesselam Bouzerdoum Unversty of Wollongong, bouzer@uow.edu.au Publcaton Detals Phung, S. & Bouzerdoum, A. (2005). A neural network wth localzed receptve felds for vsual pattern classfcaton. In Suvsoft (Eds.), The Eght Internatonal Symposum on Sgnal Processng and Its Applcatons (pp ). USA: IEEE. Research Onlne s the open access nsttutonal repostory for the Unversty of Wollongong. For further nformaton contact the UOW Lbrary: research-pubs@uow.edu.au

2 A neural network wth localzed receptve felds for vsual pattern classfcaton Abstract We ntroduce a new neural network for 2D pattern classfcaton. The new neural network, termed as localzed receptve feld neural network (RFNet), conssts of a receptve feld layer for 2D feature extracton, followed by one or more 1D feedforward layers for feature classfcaton. All synaptc weghts and bases n the network are automatcally determned through supervsed tranng. In ths paper, we derve fve dfferent tranng methods for the RFNet, namely gradent descent, gradent descent wth momentum, reslent backpropagaton, Polak-Rbere conjugate gradent, and Levenberg-Marquadrt algorthm. We apply the RFNet to classfy face and nonface patterns, and study the performances of the tranng algorthms and the RFNet classfer n ths context. Dscplnes Physcal Scences and Mathematcs Publcaton Detals Phung, S. & Bouzerdoum, A. (2005). A neural network wth localzed receptve felds for vsual pattern classfcaton. In Suvsoft (Eds.), The Eght Internatonal Symposum on Sgnal Processng and Its Applcatons (pp ). USA: IEEE. Ths conference paper s avalable at Research Onlne:

3 A NEURAL NETWORK WITH LOCALIZED RECEPTIVE FIELDS FOR VISUAL PATTERN CLASSIFICATION Son Lam Phung and Abdesselam Bouzerdoum School of Electrcal, Computer and Telecommuncatons Engneerng Unversty of Wollongong Northfelds Av, Wollongong, NSW 2522, Australa E-mals: ABSTRACT We ntroduce a new neural network for 2D pattern classfcaton. The new neural network, termed as localzed receptve feld neural network (RFNet), conssts of a receptve feld layer for 2D feature extracton, followed by one or more 1D feedforward layers for feature classfcaton. All synaptc weghts and bases n the network are automatcally determned through supervsed tranng. In ths paper, we derve fve dfferent tranng methods for the RFNet, namely gradent descent, gradent descent wth momentum, reslent backpropagaton, Polak-Rbere conjugate gradent, and Levenberg-Marquadrt algorthm. We apply the RFNet to classfy face and nonface patterns, and study the performances of the tranng algorthms and the RFNet classfer n ths context. receptve feld 2D neuron 1D neuron H W 2D nput mage S Output Layer 1 Layer 2 Layer L 1. INTRODUCTION Artfcal neural networks have proven to be a powerful computatonal tool for many tasks: pattern classfcaton, data clusterng, functon approxmaton, data compresson, to name a few. The growng computaton powers have supported new and complex network archtectures for solvng dffcult cogntve tasks. In ths paper, we develop a new neural network archtecture, known as the receptve feld neural net (RFNet) that s specfcally desgned for the classfcaton of vsual patterns. The new neural network ntegrates a 2D feature extracton layer and feature classfcaton layers. Also n ths paper, we devse effcent tranng methods for the RFNet and apply the network to detect human faces n dgtal photos. Ths paper s organzed as follows. In secton 2, we descrbe the network model of the new neural network. In secton 3, we derve algorthms for supervsed tranng of the RFNet. In secton 4, we analyze the performances of RFNet tranng algorthms and RFNet classfer n the context of face and nonface classfcaton, and secton 5 s the concluson. 2. RFNET NETWORK MODEL The network model of the RFNet s llustrated n Fg. 1. An RFNet contans a 2D feature extracton layer followed by one or more 1D layers. The 2D feature extracton layer Fg. 1. Receptve feld neural network model. conssts of an array of neurons, each handles a square regon of the nput mage. Each 2D neuron s connected to pxels n an mage regon through synaptc weghts. The neuron s localzed to the mage regon and acts as a 2D feature extractor by applyng a nonlnear actvaton functon on the weghted sum of ts nput pxels. Each remanng layer s a feedforward layer that conssts of several neurons arranged n 1D. An 1D neuron apples an actvaton on the weghted sum of ts nputs, and the neuron s output s sent to neurons n the next 1D layer. The outputs of the last layer are taken as the network outputs. Suppose we need to process an mage X of sze H W pxels. The nput mage s dvded nto N 1 non-overlappng square regons, where N 1 H W S and S s the regon 2 wdth. Let x be the pxels n mage regon, and w 1 be the synaptc weghts to the regon. Let f l be the actvaton functon of layer l. The output of 2D neuron n layer 1 s computed as y 1 f 1 (w 1 x + b 1 ) (1) where b 1 s the bas of the 2D neuron. For the 1D feedforward layers, let w l and bl be the respectve synaptc weghts to and the bas of neuron n layer l, where l > 1. Let N l be the number of neurons n /05/$ IEEE 94

4 layer l. The output neuron n layer l s computed as : y l f l (w l y l 1 + b l ) : (2) The network outputs are the outputs y L of the last layer, where L s the number of layers. Compared to convolutonal neural network [1], the new RFNet s dfferent n that each mage regon s assgned to a dfferent receptve feld. Ths allows mage features wth strong spatal dependency to be extracted. Compared to the neural network used for face detecton by Rowley et al. [5], our RFNet has non-overlappng receptve felds that are fully connected to the next 1D feedforward layer. Therefore, the RFNet s connecton scheme s more systematc, whch allows tranng algorthms to be devsed and the network to be appled n dverse mage analyss tasks. 3. TRAINING ALGORITHMS FOR RFNET The RFNet can be desgned to approxmate a gven nputoutput mappng when a tranng set of nput samples and correspondng target outputs s avalable. Network tranng s formulated as an optmzaton problem n whch the objectve s to mnmze the mean-square-error (MSE) between the actual and desred outputs, through systematc modfcaton of network weghts and bases. There are numerous optmzaton algorthms that can be used for neural network tranng. In ths paper, we focus on fve algorthms that are commonly used. Let {x 1, x 2,..., x K } be the nput samples n the tranng set, and {d 1, d 2,..., d K } be the desred outputs. The error functon s defned as 1 E K N L e k 2 (3) where e k y L,k d k s the error for tranng sample k. The mean-square-error s treated as a functon E(w) of all network weghts and bases Gradent Descent Algorthm (GD) At each epoch, the network parameters are updated along the drecton of the steepest descent: w(t + 1) w(t) + w(t), (4) where w(t) α E(t), E s the error gradent, t s the epoch number, and α s a scalar learnng rate. We now derve the error gradent E for the RFNet. Let be the error senstvty of neuron n layer l for nput sample x k. It s defned as y l,k (5) The error senstvtes can be computed usng the error back-propagaton technque: For layer L and 1, 2,..., N L : δ L,k 2 e k (6) K N L For layer l, where 1 l < L, and 1, 2,..., N l : N l+1 δ l+1,k j w l+1 j, f l+1(w l+1 j y l,k + b l+1 j ) j1 Equaton (7) shows that the error senstvty of a neuron s the weghted sum of the error senstvtes that are propagated backwards from the subsequent layer. Once the error senstvtes are computed, the gradent can be obtaned through the chan rule of dfferentaton: (7) For 1D layer l, where 2 l L, 1, 2,..., N l and j 1, 2,..., N l 1 : w l,j b l f l (w l y l 1,k + b l ) y l 1,k j (8) f l (w l y l 1,k + b l ) (9) For the 2D layer (layer 1), 1, 2,..., N 1, and j 1, 2,..., S 2 : w 1,j b 1 δ 1,k f 1(w 1 x k + b 1 ) x k,j (10) δ 1,k f 1(w 1 x k + b 1 ) (11) 3.2. Gradent Descent wth Momentum Algorthm (GDMV) The GDMV algorthm specfes the weght update as w(t) λ w(t 1) (1 λ) α(t) E(t) (12) where λ s the momentum parameter, 0 < λ < 1. The role of the momentum term n (12) s to mantan the general trend of weght update. At each epoch, f the error E ncreases by some factor, the learnng rate α(t) s reduced. Vce versa, f E decreases by some factor, the learnng rate s ncreased. Ths varable learnng rate strategy leads to faster tranng n flat regons of the error surface Reslent Back-propagaton Algorthm (RPROP) Proposed by Redmller and Braun [4], the RPROP algorthm uses only the sgn of the gradent to determne the drecton of weght update; the gradent magntude s dscarded. The RPROP algorthm s desgned to address the slow speed of GD tranng when the gradent magntude becomes small. The weght update of the RPROP algorthm s gven by (t), f w (t) > 0 w (t) + (t), f w (t) < 0 (13) 0, otherwse 95

5 where (t) s the adaptve learnng rate for network parameter w, η nc (t 1), f w (t) w (t 1) > 0 (t) η dec (t 1), f w (t) w (t 1) < 0 (t 1), otherwse (14) In (14), η nc > 1 and 0 < η dec < 1 are two scalar terms. All adaptve learnng rates are ntalzed to the same value at tranng start Conjugate Gradent Algorthm (CG) Conjugate gradent algorthms update weghts along a search drecton s(t), whch s a combnaton of the prevous drecton and the negatve gradent: { E(1) for t 1 s(t) (15) E(t) + β(t) s(t 1) for t > 1 where β(t) s a scalar. In our work, the Polak-Rbere update for β(t) s used [2]: β(t) [ E(t) E(t 1)]T E(t) E(t 1) 2 (16) The search drecton s reset to the negatve gradent after every P epochs, where P s the number of weghts. The CG weght update s gven as w(t + 1) w(t) + α(t) s(t) (17) where α(t) s found through a lne search. We use the Charalambous lne search method [2], whch s based on cubc nterpolaton and bracketng of the mnmum Levenberg-Marquardt Algorthm (LM) The Levenberg-Marquardt algorthm s based on a secondorder approxmaton of the error functon. It has been appled to tran the multlayer perceptron by Hagan and Menhaj [3]. The weght update rule of the Levenberg- Marquardt algorthm can be wrtten as where w(t) [J T J + µi] 1 J T e (18) J s the Jacoban matrx that s made up from dervatves of the error e k ( 1, 2,..., N L and k 1, 2,..., K) w.r.t. each network parameter. e s a column vector consstng of all errors e k, 1, 2,..., N L and k 1, 2,..., K. µ s a regularzaton parameter. When µ s large, the weght update rule s smlar to the gradent descent method. When µ s small, the weght update s smlar to the Newton method. Computaton of the Jacoban matrx for the RFNet s smlar to computaton of the gradent E n (5)-(11). The only major modfcaton needed s the defnton of the error senstvtes n (5). 4. EXPERIMENTS AND ANALYSIS We analyze the RFNet and ts tranng algorthms n the context of face and nonface classfcaton. Face and nonface classfcaton s used n detectng the presence and the locaton of human faces n dgtal mages [1, 5]. Usually, wndows of the nput mage are scanned exhaustvely to determne f each wndow s a face or nonface pattern. Obvously, there are two man aspects n ths face detecton approach: () the classfer for dfferentatng face and nonface patterns, and () the post-processng strategy for combnng the results of wndow-scannng. Snce heurstcs used n post-processng can nfluence the outcome of face detecton, n ths paper we wll focus only on the facenonface classfcaton aspect of the RFNet. The neural network classfer wll be traned and evaluated on a large dataset of face and nonface patterns RFNet face-nonface classfer desgn The RFNet s desgned to process an nput wndow of sze pxels n gray scale. The nput wndow s dvded by 256 to brng pxel values to the range of [0,1). We desgn a RFNet that have three layers: a 2D layer wth receptve felds of sze 2 2 pxels, a hdden 1D layer wth 2 neurons an output 1D layer wth one neuron. We select the confguraton (16, 2) after analyzng several possbltes, ncludng (18, 3), (20, 2),(20, 4), (21, 3), (24, 4), (24, 6), (27, 3),(28, 4),(30, 5), and (32, 4). The total number of network weghts and bases s 453. The actvaton functon of the output layer s the hyperbolc tangent. The actvaton functons of other layers are the logstc sgmod. The RFNet s traned to produce outputs of 0.9 and 0.1 for face and nonface pattern, respectvely. Our experments used a dataset of 20,000 patterns, of whch 10,000 face patterns were manually cropped from web mages and 10,000 nonface patterns were randomly extracted from 2,000 photos that contan no human face. The fve-fold cross valdaton technque was used to compare tranng algorthm performances and estmate network generalzaton ablty. The entre dataset was dvded nto fve subsets. For each run, ten RFNets were traned on a combnaton of four subsets usng fve tranng algorthms, and then tested on the remanng ffth subset. Performance measures were averaged over the fve valdaton runs Comparson of RFNet tranng algorthms The tme evoluton of the tranng MSE, produced by the fve tranng algorthms, s shown n Fg. 2. To acheve a machne-ndependent comparson, we measure tme n terms of gdeu, whch s defned as the average tme taken to complete one GD epoch. The gdeu tme unt remans stable for a fxed-sze network and a fxed-sze tranng set. 96

6 Tranng MSE (log) GD 2.2 GDMV RPROP CG LM Tranng Tme (gdeu) Correct Detecton Rate (%) RFNet Template matchng 50 Nearest neghbor 3 nearest neghbor 5 nearest neghbor False Detecton Rate (%) Fg. 2. Tme evoluton of the tranng mean-square-error. Table 1. Memory usage of fve RFNet tranng algorthms Algorthm Memory Usage GD K P GDMV K P RPROP (K + 2) P CG (K + 3) P LM K P N L + P 2 Fgure 2 shows that all four algorthms GDMV, RPROP, CG and LM were faster than the standard GD algorthm. We found that the GDMV algorthm acheved a sgnfcant speed advantage over the GD algorthm only f the momentum parameter λ was wthn the range of (0.6, 0.9). Furthermore, mposng an upper lmt on the adaptve learnng rate α(t) made GDMV tranng more stable. The CG algorthm converged slghtly faster than the RPROP algorthm; both algorthms had faster speeds than the GDMV algorthm. The LM algorthm became faster than the CG algorthm only after 1200 gdeu. As shown n Table 1, among the fve algorthms, the LM algorthm requres the largest storage due to the computaton of the Jacoban and Hessan matrces. The RPROP and CG algorthms requre slghtly more storage than the GD algorthm, but ther convergence speeds on large tranng sets are comparable to that of the LM algorthm Analyss of RFNet face-nonface classfer The face-nonface classfcaton rates of the RFNet are shown n Fg. 3. The RFNets were traned usng the CG algorthm for a maxmum of 1, 500 epochs and a target MSE of Estmated over the fve valdaton runs, the RFNet acheved CDRs of 97.0% and 98.6% at FDRs of 5.0% and 10.0%, respectvely. It had a maxmum CR of 96.0%. For comparson purposes, we mplemented and tested two other face-nonface classfers usng the same data as for the RFNet: correlaton-based template matchng (TM) classfer, and k-nearest neghbor (k-nn) classfer. The RFNet clearly outperformed the TM classfer, whch had CDRs of 51.9% and 60.1% at FDRs of 5.0% and 10.0%, respectvely. The maxmum CR of the TM classfer was only 75.3%. Compared to the TM classfer, the RFNet conssts of several tranable 2D templates (e. receptve Fg. 3. Face-nonface classfcaton ROC curves of the RFNet, template matchng, and k-nn classfers. felds) that are compared to localzed mage regons. The RFNet classfer s also more accurate than the nonlnear k-nn classfer. The overall classfcaton rates of the 1- NN, 3-NN and 5-NN classfers were 85.5%, 84.4%, and 82.4%, respectvely. The k-nn classfer n our mplementaton requred about 390 tmes more storage than the RFNet classfer. 5. CONCLUSION We have ntroduced a new neural network n whch neurons wth receptve felds localzed to mage regons are traned as 2D feature extractors. The network also contans 1D feedforward layers for classfcaton of the extracted 2D features. We derve fve dfferent network tranng algorthms, whch have been shown capable of handlng large tranng sets. Results on the face-nonface classfcaton problem have shown that the new network s a promsng tool for mage classfcaton. 6. REFERENCES [1] C. Garca and M. Delaks, Convolutonal face fnder: A neural archtecture for fast and robust face detecton, IEEE Trans. PAMI, vol. 26, no. 11, pp , [2] M. T. Hagan, H. B. Demuth, and M. H. Beale, Neural Network Desgn. Boston, MA: PWS Publshng, [3] M. T. Hagan and M. B. Menhaj, Tranng feedforward networks wth the marquardt algorthm, IEEE Trans. Neural Networks, vol. 5, no. 6, pp , [4] M. Redmller and H. Braun, A drect adaptve method of faster backpropagaton learnng: The rprop algorthm, n IEEE Int. Conf. on Neural Networks, San Francsco, 1993, pp [5] H. A. Rowley, S. Baluja, and T. Kanade, Neural network-based face detecton, IEEE Trans. PAMI, vol. 20, no. 1, pp ,