Pattern Classification: An Improvement Using Combination of VQ and PCA Based Techniques

Transcription

1 Ameran Journal of Appled Senes (0): , 005 ISSN Sene Publatons Pattern Classfaton: An Improvement Usng Combnaton of and PCA Based Tehnques Alok Sharma, Kuldp K. Palwal and Godfrey C. Onwubolu Sgnal Proessng Lab., Grffth Unversty, Brsbane, Australa Department of Engneerng, Unversty of the South Paf, Suva, F Abstrat: Ths study frstly presents a survey on bas lassfers namely mnmum dstane lassfer (), vetor quantzaton (), prnpal omponent analyss (PCA), nearest neghbour (NN) and k-nearest neghbour (knn). Then vetor quantzed prnpal omponent analyss (PCA) whh s generally used for representaton purposes s onsdered for performng lassfaton task. Some lassfers aheve hgh lassfaton auray but ther data storage requrement and proessng tme are severely expensve. On the other hand some methods for whh storage and proessng tme are eonomal do not provde suffent level of lassfaton auray. In both the ases the performane s poor. By onsderng the lmtatons nvolved n the lassfers we have developed lnear ombned dstane () lassfer whh s the ombnaton of and PCA tehnques. The proposed tehnque s effetve and outperforms all the other tehnques n terms of gettng hgh lassfaton auray at very low data storage requrement and proessng tme. Ths would allow an obet to be aurately lassfed as qukly as possble usng very low data storage apaty. Key words: PCA, Classfaton auray,, total parameter requrement, proessng tme INTRODUCTION Pattern lassfaton/reognton s an area where we learn how to best famlarze the obets to the mahne and get atons or desons based on the observed ategores of the pattern. A pattern ould be human fae, sampled speeh, handwrtten or prnted dgts, any letter, gesture, spoken word, fnanal data, bometr data or any statstal data. Humans naturally lassfy/reognze patterns from the envronment n everyday lfe. A fve year old kd an adapt to dfferent type of obets or patterns and reat aordngly. Ths adaptaton s taken for granted untl we ome to teah a mahne to lassfy/reognze and provde atons or desons on the same patterns. The more the patterns avalable, the better the deson would be. Ths gves hope to desgn a lassfer system. For the last fve deades researh s gong on n ths feld to provde an optmum lassfer/reognzer. But the lassfer performane s stll far behnd the perepton of a human bran. However, pattern lassfaton/reognton plays a rual role n the areas lke bankng, multmeda ommunaton, data synthess, speeh or mage proessng, forens senes, omputer vson and remote sensng, data mnng, robots and artfal ntellgene. It emerged as an essental and ntegral part of daly lfe. The evolvng omputatonal demand n pattern lassfaton makes ths feld very hallengng and thus open for researh. For example n mage reognton, several thousands of multdmensonal patterns are requred for proessng whh makes the mplementaton of the lassfer system qute mpossble. There are two man ategores of pattern lassfaton () supervsed lassfaton: where the state of nature for eah pattern s known and () unsupervsed lassfaton: where the state of nature s unknown and learnng s based on the smlarty of patterns []. In ths study only supervsed pattern lassfaton proedures have been onsdered. A supervsed lassfaton ould be subdvded nto two man phases namely tranng phase and testng phase. In the tranng phase the lassfer s learned by known ategores (lasses) of patterns and n the lassfaton or the testng phase unknown patterns whh were out of the tranng dataset are assgned lass labels of tran patterns for whh the dstane from the test pattern to the prototype(s) s mnmum. The performane of a lassfer depends upon several fators. Some of the man fators are () number of tranng samples avalable to the lassfer, () generalzaton ablty.e. ts performane n lassfyng test patterns whh were not used durng the tranng stage, () lassfaton error - some measured value based on the norret deson of the lass labellng of any gven pattern, (v) omplexty - n some ases (due to lassfer desgn) the number of features or attrbutes (dmensons) are relatvely larger than the number of tranng samples usually referred as urse of dmensonalty, (v) speed - proessng speed of tranng and/or testng phase(s) and (v) storage - amount of parameters requred to store after the tranng phase, for lassfaton (testng) purposes []. For a gven lassfer model and a fxed number of tranng samples, the performane may depend on the generalzaton apablty (auray), speed and mplementaton ost (due to storage of nformaton). Correspondng Author: Alok Sharma, Sgnal Proessng Lab., Grffth Unversty, Brsbane, Australa 445

2 The number of parameters requred to perform lassfaton task (testng) after the tranng proedure, s referred as total parameters. For a gven lassfer we an assoate the total parameters to the mplementaton ost of the lassfaton system and the generalzaton apablty may depend upon the type of parameters (dstrbuton, values et.) used. The hgher the total parameters requred for lassfaton task the ostler the system would be. Another mportant fator n lassfer desgn s the speed or the proessng tme requred to do the task. It s possble n a lassfer that at two dfferent nstanes the total parameter requrement s same but the proessng tme dffers. We therefore want to redue the total parameters and proessng tme but at the same tme least sarfe the lassfaton auray. In other words, we searh for the optmal lassfaton auray or least lassfaton error, nvolvng as mnmum total parameters and proessng tme as possble. Ths would allow the system to lassfy/reognze an obet as qukly as possble at mnmum ost. Nearest Neghbour (NN) lassfer [] s the most smple lassfer found up tll now. In NN lassfer no speal proedure s requred to do tranng. All the avalable data (as maxmum as possble) s stored to perform lassfaton, where eah test pattern s ompared for smlarty wth all the avalable tranng data (pattern). The test pattern s assgned the lass label of that tranng pattern, whh s the losest to the test pattern. A maor drawbak of NN approah s ts large total parameter requrement to perform lassfaton task. For example, a dataset wth 0 lasses, havng 5000 vetors or patterns n eah lass wth 64 attrbutes or dmensons would requre total parameters as follows: total parameters = lass NoOfVe dmenson 6 = = 3. 0 If the dmenson s very hgh (e.g. n mage), then the total parameter requrement for NN approah wll be even more severe whh would restrt the pratal applaton of suh approah. It an also be seen that nrease n the total parameter does not always lead to better performane. When tran patterns and test patterns are losely mathed then auray obtaned by NN approah s good. But when the test patterns do not math wth tran patterns, NN approah provdes poor performane (n terms of auray). In the unmathed pattern ase the performane of the lassfer system does not mprove by nreasng the total parameters. The lassfaton auray of NN approah an be mproved by makng the deson of a test pattern for lass labellng based on k nearest patterns. Ths method s known as k-nearest Neghbour (knn) [] tehnque. The total parameter requrement for knn approah s same as that of NN approah exept for the omputatonal demand, whh s severe n the former approah. Am. J. Appl. S., (0): (0): , The mplementaton ost of the lassfaton system ould be redued by estmatng eah lass by a sngle prototype, usually a entrod. Ths would help n dereasng the total parameter requrement for the lassfaton task but ould be at the pre of lassfaton auray. Ths type of lassfer s known as mnmum dstane lassfer (). The goal of s to orretly label as many patterns as possble. It provdes mnmal total parameter requrement and omputatonal demand. The method fnds entrod of lasses and measures dstanes between these entrods and the test pattern. In ths method, the test pattern belongs to that lass whose entrod s the losest dstane to the test pattern. Takng the same above example of 0 lasses, the total parameter requrement for would be ust 640, whh s about /5000 as ompared to NN approah. Usually lassfaton auray s sarfed to get ths advantage of extremely low proessng tme and total parameter requrement. s used n many pattern lassfaton applatons [3-7] nludng dsease dagnosts [8], lassfaton of dgt mamograph mages [9] and optal meda nspeton [0]. The natural extenson of sngle prototype s mult prototype, where eah lass s estmated by several prototypes lke n vetor quantzaton () [,]. based lassfers are also referred as loal lassfers sne they partton eah lass nto several dsont regons or loal regons and estmate eah regon by a prototype (entrod) usually referred as odeword. The set of odewords s known as odebook of the system. The am of tehnque s to fnd the odebook that mnmzes the expeted dstorton between pattern x and the entrod of th dsont regon ( µ ).e. D = E[mn( x µ )] where E[ ]. denotes expetaton wth respet to x. So the tranng proedure s to fnd the odebook and store t for lassfaton task. Inreasng the number of odewords per lass would nrease the performane up to some extent but t would also augment the total parameter requrement and proessng tme. tehnque s appled n several areas of pattern ompresson and lassfaton [3], whh nlude mage lassfaton [4] speeh odng or speeh ompresson [5], speaker reognton [6], hgh range resoluton sgnature dentfaton [7] and mage odng [8]. Another way of performng lassfaton s by utlzng lnear subspae lassfers [9,0]. Here eah lass s represented by ts Karhunen-Loéve transform (KLT) [] or prnpal omponent analyss (PCA). The obetve of PCA s to fnd a global lnear transform of gven patterns n the feature spae and produe lassndependent or lass-dependent bass vetors. The frst bass vetor s n the dreton of maxmum varane of the gven data. The remanng bass vetors are mutually orthogonal and n order, maxmze the remanng varanes subet to the orthogonal

3 ondton. The prnpal axes are those orthonormal axes onto whh the remanng varanes under proeton are maxmum. These orthonormal axes are gven by the domnant egenvetors (.e. those wth the largest assoated egenvalues) of the ovarane matrx. Class-ndependent PCA fnds those h orthonormal axes (subspae dmenson) from d-dmensonal dataset ( h < d ), where h domnant egenvetors are from the t KLT of the data orrelaton matrx Σ= E[xx ] whh s n fat a ovarane matrx wth zero mean []. Classndependent PCA annot be used for lassfaton purposes sne all the lasses are sattered over the feature spae wth dfferent entrod values or mean and varanes for eah lass makng mpossble to preserve the ndvdual lass nformaton by a sngle KLT for the entre tran samples. Therefore domnant egenvetors are taken for eah lass separately (lass-dependent). For a -lass problem, ovarane matrx wll be gven by: t Σ = E [( x µ )( x µ ) ] for =,,... Where only those x, that belong to the th lass have been taken n the expetaton funton at a tme. It has been seen that the subspae lassfaton s further mproved by ts loal lnear extenson []. Here the performane depends upon the subspae dmenson and the number of loal regons. Kambhatla and Leen [] and Kambhatla [3] have shown loal lnear PCA or PCA for representaton purposes. The goal of PCA s to mnmze the mean squared reonstruton error E[ x x ˆ ] where ˆx s the reonstruted pattern of x. Kambhatla [3] showed PCA usng Euldean dstane (PCA-Eu) and PCA usng reonstruton dstane (PCA-re). PCA-re s a better tehnque than PCA-Eu for representaton purposes n terms of ahevng lesser reonstruton error, but ths ahevement omes wth the expense of hgher total parameter requrement and omputatonal demand. For example, takng the same 0 lass problem, where eah lass s subdvded nto 4 dsont regons (loal regons), ths would requre storage of dxd (64x64) egenvetor set for eah dsont regon together wth other parameters (entrod of dsont regon).e. total parameters = parameters due to egenvetors + parameters due to entrod total parameters (PCA-re) = (d d)*lass*level + (d )*lass*level total parameters (PCA-Eu) = (d h)*lass*level + (d )*lass*level Where the term level s the number of dsont regons or loal regons per lass and h<d. Ths yelds total parameters requrement for PCA-re.66x0 5 (for Am. J. Appl. S., (0): (0): , d=64), whereas 76 (for h = ) for PCA-Eu whh d+ s /( ) ompared to PCA-re. Although the h+ PCA-re model exhbts slght mprovement over PCA-Eu model, t severely nreases the total parameter requrement and omputatonal demand. Ths would nrease the mplementaton ost and proessng tme of the lassfaton system. Consderng the mplementaton ost and omputatonal demand we opted for an eonomal model (PCA-Eu) to tran the system. Hereafter PCA-Eu model wll be referred as PCA model. Some modfaton s requred n PCA model pror to use as a lassfer. The urrent PCA model frst parttons the data spae nto dsont regons and then performs loal PCA about eah luster (referred as a dsont regon of a lass) entre. Ths s deal for representaton purposes but for the lassfaton task a mnor hange n dstane measurement s requred whh should reflet the dstane of a test pattern from the entrod and domnant egenvetors of eah dsont regon onurrently. The PCA model as a lassfer does not exhbt very enouragng results but stll an be used to perform lassfaton task. Nonetheless t an be shown that PCA model as a lassfer behaves satsfatorly n terms of obtanng reasonably well perentage auray at low total parameter requrements and proessng tme. The performane of PCA as a lassfer ould be sgnfantly mproved by ombnng the lnear dstanes of and PCA. The normalzed reonstruton dstane measure x x ˆ and the normalzed dstane between the test pattern and the enter of dsont regon x µ, are ombned lnearly to form a new dstane measure for the lassfaton. Ths dstane measure would mnmze the ombnaton of the mean squared reonstruton error (MSE) E[ x x ˆ ] and the expeted dstorton E[ x µ ]. Eah dstane added together may have ts own loal regons n the feature spae where t performs the best. We have ntrodued ths lnear ombnaton of dstane () tehnque and shown n ths study that t s a better lassfer wth no extra total parameter requrement than PCA. Classfaton results obtaned by exhbt sgnfant mprovement over MCD,, PCA, NN and knn lassfers n terms of ahevng hgher perentage auray or lower lassfaton error and at the same tme mantanng the total parameters requrement and proessng tme as mnmum as possble. Consequently, ths would allow lassfaton or reognton of the obets as qukly as possble at mnmum ost. Conventonal lassfers: The style of notatons s adopted from Duda and Hart [4]. In all the dsussons ω denotes the state of nature or lass label of th lass n a -lass problem, χ denotes the set of n tran samples,

4 Am. J. Appl. S., (0): (0): , 005 Ω = { ω : =,, K, } be the fnte set of states of nature and let θ be the lass label of tran pattern or prototype suh that θ Ω.The set χ an be separated by lass nto subsets χ, χ,, χ, wth the samples n χ belongng to ω : χ = { x, x, K, xn} where x R d (d-dmensonal hyperplane) χ χ and χ χ χ = χ Let n denote the number of samples n the subset χ, therefore n = n. = Fgure llustrates the lass labellng of a test pattern and the relatonshp between the label of prototype ( θ ) and the label of lass ( ω ). The prototype ould be a tran pattern, a entrod, a KLT or a group of entrod and KLT dependng upon the type of lassfer s used. In Fg. two-lass problem s onsdered where eah lass onssts of 3 prototypes. Eah of the lass s assgned a unque label namely ωp and ωq suh that ( ωp, ωq) Ω. The lass labels of the prototypes are θ, θ, θ and θ l, θ m, θn suh that: k ω = θ = θ = θ p q ω = θ = θ = θ l m k n and total parameters = d n = d n () = It an be seen from equaton that total parameters depend upon the attrbute or dmenson d, number of lass and number of tran patterns. In many pratal applatons the values of d and n are very large whh severely affets the storage requrements and proessng tme, nreasng the ost and redung the speed of the lassfer system. knn lassfer: knn lassfer s a generalzed form of NN lassfer. In ths approah k nearest tran patterns to a test pattern x s olleted. The test pattern s assgned the lass label whh has the maorty of k olleted patterns. The tranng phase of the knn lassfer s smlar to NN lassfer where all the tranng patterns together wth ther lass label nformaton are stored for the later use. The total parameter requrement s also same as NN approah. The proessng speed of knn lassfer s slower than NN lassfer due to the searhng of k nearest patterns for eah of the test pattern. The lassfaton auray may mprove wth the nrease n the value k. Ths mprovement s usually observed when the test patterns and the tran patterns are losely mathed. However, n some ases when the test patterns and the tran patterns do not math the lassfaton auray s poor. In ths ase nreasng the value k may not mprove the lassfaton auray of the system. Fg. : Class labellng of a test pattern n a two-lass problem The lass label of prototype s assgned to a test pattern x whh s the losest to the prototype based on some dstane measurements or ondtonal probabltes. Therefore f L(x) denotes the lass label of a test pattern x then from the fgure L(x) = θ l = ωq. NN lassfer: The proedure for NN lassfer an be subdvded nto two man phases namely, tranng phase and testng or lassfaton phase. In the tranng phase all the avalable patterns χ wth ther orrespondng lass label nformaton are stored for lassfaton purpose. The total parameter requrement for NN approah s gven by: 448 lassfer: In lassfer eah lass χ s represented by sngle prototype, whh s usually the entrod of the lass n the feature spae. It requres mnmal total parameter requrement and least omputatonal demand. The total parameter requrement for s: total parameters = d Whh s / n as ompared to NN or knn lassfer. = Ths advantage of low total parameter requrement and fast omputaton may aheve by sarfng some lassfaton auray. lassfer: lassfer s the further extenson of lassfer. Here eah lass s represented by multple prototypes. parttons a lass nto several dsont regons n the feature spae usually known as Vorono regons []. The enter of Vorono regons (prototype) s referred as odeword of the lassfer and a set of odewords s known as odebook of the lassfer system. The am of s to produe a odebook that mnmzes the expeted dstorton E[ x µ ]. See Lnde et al. [] for detals. The total parameter requrement s d (Q ) where Q s the level of lassfer.e. number of dsont regons or odewords for eah of the lass..

5 Am. J. Appl. S., (0): (0): , 005 PCA lassfer: Class dependent PCA s onsdered for lassfaton where eah lass s represented by ts KLT. In a d-dmensonal feature spae let Σ and µ denote ovarane matrx and entrod of lass χ n a -lass problem respetvely, xˆ be the reonstruted pattern of x, then the goal of the tranng phase of PCA lassfer s to fnd egenvetors w suh that the followng rtera s satsfed: Σ w = λw () where λ denotes egenvalues orrespondng to w whh s obtaned by mnmzng MSE E[ x x ˆ ]. The total parameter requrement for PCA lassfer s: total paramaters = entrod _ paramters + egenvetor _ parameters total paramaters = d + ( d h) = d( h + ) Where h < d s the number of egenvetors used. PCA as a lassfer: In ths approah, frstly, the set of tran patterns are parttoned nto dsont regons by applyng tehnque for eah lass separately and then KLT s performed about eah of the dsont regon or loal regon enter []. The am of PCA s to mnmze MSE E[ x x ˆ ] n the loal regons. To llustrate tranng and lassfaton proedures let Q be the number of dsont regons or levels per lass. (Detals of the tranng proedure are gven n Kambhatla [3]. PCA an also be traned usng splttng tehnque [5] ). If PCA s used for representaton purposes then n the deodng step (here lassfaton) frstly the losest dsont regon to a test pattern x s omputed. One the losest regon s obtaned, next step s to use ts orrespondng egenvetor and entrod nformaton to ompute reonstruted pattern xˆ. For lassfaton PCA proedure would provde no better performane than tehnque sne the deson would le only on the losest dsont regon to the test pattern x and the omputaton of KLT for dsont regons may beome redundant. Therefore a proedure for deson makng of a test pattern should be adopted that uses both the entrod and dreton (egenvetor) nformaton n parallel. Classfaton: Step : Compute reonstruton dstane between a test pattern x and ts reonstruted pattern ˆx : = x xˆ t = ( I W W )( x µ ) for =,,...,( Q ) Step : Fnd the argument for whh the reonstruton dstane s mnmum: Q k = arg mn = Step 3: Assgn lass label ω r = θ k to the test pattern x, where θ k Ω. Thus, t an be seen that step omputes the error of reonstruton dstane by usng dreton and entrod nformaton n one sngle step for the lassfaton. Tranng Step : Take tran patterns χ χ of lass label ω at a tme for onsderaton, where =,,...,. Step : Apply tehnque and partton χ nto Q dsont regons; for all =,,...,. Step 3: For eah dsont regon ompute entrod µ and ovarane matrx Σ where =,,...,( Q). Step 4: Evaluate d h retangular matrx of egenvetors W = {w l :l =,,...,h} for eah dsont regon where h<d and w s from equaton ; arrange the obtaned egenvetors suh that ts orrespondng egenvalues are n desendng order. Let the lass label of egenvetor set W beθ Ω. Step 5: Store W and µ wth ther orrespondng lass nformaton for lassfaton. The total parameter requrement for PCA an be gven by: total paramters = parameters _ entrods + paramters _ egenvetors total paramters = Q d + Q ( d h) = Qd( h + ) Whh s Q tmes the total parameter requrement of PCA lassfer. 449 lassfer: The s a ombnaton of and PCA tehnques. Empral results show sgnfant mprovement of lassfer over prevously dsussed lassfers n terms of gettng hgher perentage auray wth total parameter requrement no more than PCA approah. In our approah the tranng phase of the lassfer s dental to PCA lassfer thus the total parameter requrement for approah s same as PCA approah. However the lassfaton proedure dffers. In the lassfaton phase the dstane used n lassfaton and the dstane used n PCA lassfaton are added together wth some weghtng to form a new dstane measure. Ths ombnaton or addton may redue expeted dstorton E[ x µ ] and MSE or root-mse E[ x x ˆ ], overall produng mproved results for the ombnaton. The mproved results aheved ould be due to eah of the onsttuent dstane performng the best n ther loal regons n the feature spae. The generalzaton apablty or lassfaton auray of a lassfer depends on the type of dstrbuton or values used for tranng and/or testng the lassfer. For e.g. f tranng patterns of eah lass

6 Am. J. Appl. S., (0): (0): , 005 s spherally dstrbuted, dense, well separated wth eah other and test patterns are losely mathed wth ther tran patterns then tehnques suh as,, NN and knn may perform the best; f outlers are present n the tranng patterns then tehnques suh as PCA or PCA may gve poor performane. However for Gaussan data wth mathng tran and test ondtons PCA may provde reasonably hgh lassfaton auray [] and PCA and may provde even better performane than PCA. In the presene of outlers and omplex dstrbutons (unmathed tran and test ondtons) may provde better performane than other tehnques. The onept of ombnaton of multple lassfers has been prevously appled by Xu et al. [6] for handwrtng reognton. They have llustrated the ombnaton usng some bas lassfers suh as Bayesan and knn and shown three ategores of ombnaton whh depend upon the levels of nformaton avalable from the lassfers. Jaobs et al. [7] suggested supervsed learnng proedure for systems omposed of many separate expert networks. Ho et al. [8] used multple lassfer system to reognze degraded mahne-prnted haraters and words from large lexons. Tresp and Tanguh [9] presented modular ways for ombnng estmators. Woods et al. [30] and Woods [3] presented a method for ombnng lassfers that uses estmates of eah ndvdual lassfer s loal auray n small regons of feature spae surroundng a test pattern. Zhou and Ima [3] showed a ombnaton of and mult layer pereptron (MLP) for Chnese syllables reognton. Almoglu and Alpaydn [33] used the ombnaton of two MLP neural networks for handwrtten dgt reognton. Kttler et al. [34,35] developed a ommon theoretal framework for ombnng lassfers whh uses dstnt pattern representatons. Breukelen van and Dun [36] showed the use of ombned lassfers for the ntalzaton of neural network. Alexandre et al. [37] ombned lassfers usng weghted average after Turner and Gosh [38]. Ueda [39] presented lnearly ombnng multple neural network lassfers based on statstal pattern reognton theory. Senor [40] used ombnaton of lassfers for fngerprnt reognton. Le et al. [4] demonstrated a ombnaton of multple lassfers for handwrtten Chnese harater reognton and Yao et al. [4] used a ombnaton based on fuzzy ntegral and Bayes method. Smlarly several other researh work on ombnatonal lassfers have been reported n the lterature. In our approah the tranng phase parameters µ (entrod) and W (egenvetor set) are stored wth the lass label θ Ω nformaton for the use n the lassfaton phase whh s same as the tranng phase of PCA approah. Let n a -lass problem eah 450 lass s separately parttoned nto Q dsont regons then the lassfaton phase of approah an be llustrated as follows: Classfaton Step : Compute the dstane between a test pattern x and the entrod µ of the dsont regon: = x µ for =,,...,(Q ) Step : Compute the reonstruton dstane between a test pattern x and ts reonstruted pattern ˆx : = x x ˆ = t (I W W )(x µ ) for =,,...,(Q ) Step 3: Normalze dstane and to elmnate the dfferene n ther ampltudes that would allow them to ontrbute equally n deson makng. ˆ Q /max( = ) and ˆ Q /max( = ) = = Step 4: Add dstane ˆ and ˆ : ˆ ˆ ˆ = α + ( α) for =,,...,(Q ), where α s a weghtng onstant n the range[0,]. Step 5: Fnd the argument for whh the ombned dstane s mnmum: Q k = argmnˆ = Step 6: Assgn lass label ω r = θ k to the test pattern x, where θ k Ω. The lassfaton phase of tehnque s smple, omputatonally nexpensve and attans hgh lassfaton auray or low lassfaton error. The dstane ˆ n the lassfaton phase depends on the weghtng onstant α and the two normalzed dstane ˆ and ˆ. The weghtng onstant α (n step 4) s a postve onstant n the range [0,]. Approprate value for α should be taken sne bad seleton may lead to poor lassfaton auray. The two normalzed dstane ˆ and ˆ are lassfaton dstane of and PCA tehnques respetvely. Choe of α: The optmum or lose to optmum performane by lassfer an be obtaned by seletng the approprate value of α emprally. We have used speeh data [43] and mage data [44,45] to selet the value of α. In ths study we have taken α as a numeral onstant, however, one an also take α as a probablst model whh would depend on a test pattern and the dstrbuton of tran patterns. Ths may nrease the omputaton and storage requrements. The dsusson on α as a probablst model s beyond the

7 Am. J. Appl. S., (0): (0): , Level Level Level 4 Level 8 Level 6 α = 0. α = 0. α = 0.3 α = 0.4 α = 0.5 α = 0.6 α = 0.7 α = 0.8 α = Dmenson Fg. : Classfaton auray for dfferent values of α on mage data sope of ths study. In Fg. and 3 lassfaton auray for tehnque s omputed for dmenson h and level Q, where h =,..., 4 and Q =,,4,8,6. The values of α are 0.,0.,...,0.9, where hoosng α values lose to 0. and 0.9 wll gve performane smlar to PCA approah and approah respetvely. Dvertng ether upwards ( α = 0.6,...,0.9 ) or downwards ( α = 0.4,...,0. ) from the enter value of α (0.5) wll make the dstane ˆ based for ˆ or ˆ respetvely. It an be observed from Fg. and 3 that at α = 0.5 lassfaton auray obtaned by tehnque (n Fg. and 3 denoted by bold lnes) s lose to optmum. Ths mples that when the dstane ˆ and ˆ ontrbute equally n the deson makng for a test pattern n the feature spae then the lassfaton auray s lose to optmum. Thus we have takenα = Level Level Level 4 Level 8 Level 6 α = 0. 7 α = 0. α = 0.3 α = α = 0.5 α = 0.6 α = α = 0.8 α = Dmenson Fg. 3: Classfaton auray for dfferent values of α on speeh data Level PCA Level Level 4 Level Dmenson Level 6 PCA 3 4 Fg. 4: Classfaton auray vs. dmensons and levels usng,, PCA, PCA and on mage dataset 45 Expermentaton: For all the experments two sets of mahne learnng orpuses have been utlzed namely TIMIT database [43] for speeh lassfaton and Sat- Image dataset [44,45] for mage lassfaton. From the TIMIT orpus a set of 0 dstnt monothongal vowels are extrated, then eah vowel s dvded nto three segments and eah segment s used n gettng melfrequeny epstral oeffents wth energy-deltaaeleraton (MFCC_E_D_A) feature vetors [46]. A total of 9357 MFCC_E_D_A vetors of dmenson 39 for tranng sesson and a separate set of 3 vetors for lassfaton are utlzed. The seond dataset s Sat- Image whh onssts of 6 dstnt lasses wth 36 dmensons. A sum of 4435 feature vetors s used to tran the lassfer and a dfferent set of 000 vetors s used for verfyng the performane of the lassfer. In the frst part of the expermentaton, lassfaton auray s measured for all the lassfers gven some fxed parameters. Here the auray s a funton of dmenson h and level Q, where Q =,,4,8,6and h =,,..4for all the levels, exept for Q = 8, where h =,,...,0. Level 8 ( Q = 8) s taken at random for dmenson h =,,...,0 to get a general understandng of how the dmenson affets the lassfaton auray f t s nreased ontnuously. Not all the tehnques depend upon both the dmenson h and level Q; depends upon levels, PCA depends upon dmensons,, NN and knn depend nether upon dmensons nor on levels, only PCA and depend upon dmensons as well as levels. Fg. 4 (mage dataset) and Fg. 5 (speeh dataset) llustrate lassfaton auray for,, PCA, PCA and tehnques and Table depts lassfaton auray for NN and knn tehnques. Usually the tehnque s a speal ase of when Q =, that s why t s represented n the olumn of Level n Fg. 4 and 5.

8 Am. J. Appl. S., (0): (0): , 005 Table : Classfaton auray for NN and knn tehnques on mage and speeh datasets Tehnque Classfaton auray Classfaton usng mage dataset auray usng speeh dataset NN knn Level Level Level 4 Level 8 Level PCA PCA 3 NN knn PCA PCA Dmenson Fg. 5: Classfaton auray vs. dmensons and levels usng,, PCA, PCA and on speeh dataset PCA 6 PCA log 0 (total paramters) NN knn (3,5,7,9,) Fg. 6.: Classfaton auray vs. log 0 (total parameters) on mage dataset It an be observed from Fg. 4 (mage dataset) that s gvng better lassfaton auray than PCA; s produng hgher lassfaton auray at Level and Level 4 than PCA, but PCA s showng mprovement over tehnque at level 8 and level 6. It s also lear that s performng better than,, PCA and PCA at all the levels and dmensons. Inreasng the dmenson at any gven level s mprovng the lassfaton auray of tehnque. At level 8 and dmenson 0 the lassfaton auray of s 89.% whh s very Proessng Tme Fg. 6.: Classfaton auray vs. proessng tme on mage dataset lose to NN and knn tehnques. It should be noted that NN and knn tehnques produe smlar lassfaton auray as tehnque but ther proessng tme and total parameter requrement are severely expensve. Furthermore, t an be observed from the experment on speeh data (Fg. 5) and Table that s gvng better lassfaton auray than NN tehnque; PCA s mprovng at dmenson over tehnque; PCA s produng better lassfaton auray over tehnque at levels and 4 for dmenson but deteroratng at level 8 and level 6. s exhbtng better performane than all the tehnques nludng NN and knn. The lassfaton auray s mprovng wth the nrease n dmenson at any gven level. The lassfaton auray by NN and knn s qute poor for speeh data. Ths may be due to the testng data not mathng wth ther tranng data. In the seond part of expermentaton, lassfaton auray s omputed as a funton of total parameters and proessng tme. Ths would gve 3D plot where x and y axes represent total parameters and proessng tme and z-axs represents lassfaton auray. For smplty, a 3D plot s splt nto two D plots, where one plot shows lassfaton auray versus total parameters and the other plot shows lassfaton auray versus proessng tme for the orrespondng values of total parameters. The level s taken as Q=,,4,8,6and dmenson h =,,...,0for mage dataset and h =,,...,for speeh dataset. Fgure 6. and 6. show lassfaton auray versus total parameters n logarthm sale and lassfaton auray versus proessng tme respetvely, usng all the tehnques on mage dataset. For tehnque, as presented n the Fg. 6. and 6., the frst value of lassfaton auray s 8.3% at total parameter (Fg. 6.) whh takes proessng tme of.94 unts (Fg. 6.). The next reported value of lassfaton auray n Fg. 6. and 6. s only those whh provde better lassfaton

9 auray than the present value,.e. those values are plotted next n the fgures whh are gvng mprovement n lassfaton auray ompared to the prevous value. Ths would help to desrbe that to aheve a ertan range of lassfaton auray what s the total parameter requrement and ts orrespondng proessng tme. Smlar strategy s opted for PCA and PCA tehnques. For tehnque there are only four levels and all of them are gven whh are denoted by,4,8 and 6 n the Fg. 6. and 6.. and NN have only one value and knn has got 5 values for k = 3,5,7,9,whh s depted n the same fgures. It an be observed from the Fg. 6. and 6. that has mnmal total parameter requrement and proessng tme but the lassfaton auray s qute poor around 76.6%. The other tehnques wth same total parameter requrement but wth dfferent proessng tmngs are PCA, and (at level ). Though the proessng tme s very low for PCA (around.53 to.99 tme unts), the performane s qute poor gvng lassfaton auray n the range of 69.4% to 73.3% whh s even lower than. Wth the same total parameter requrement gves muh better performane than PCA n terms of auray but the proessng tme nreases as the levels nrease towards 6. The lassfaton auray of PCA s qute poor at the begnnng. As the total parameter requrement nreases t gves reasonably well results but at the expense of hgh proessng tme. It s evdent that tehnque gves hgh lassfaton auray at low total parameter requrement and proessng tme, for e.g. t gves 85.4% auray at total parameters usng only 3.00 unts proessng tme whereas the maxmum auray obtaned by s 85.% at total parameters usng 3.4 unts proessng tme and PCA gves 84.9% at usng 3.8 unts proessng tme. The maxmum auray aheved by tehnque (when Q 6and h 0) s 90.0% at usng 48. unts proessng tme whh s very lose to NN tehnque (90.3%) and lose to the maxmum of knn (for k = 3 ) tehnque (90.5%). However the proessng tme for NN and knn tehnques are unts and from to 0.0 unts (for k = 3,5,7,9,) respetvely and the total parameter requrement for both the tehnques s , whh s qute expensve as ompared to and other tehnques. Fgure 7. and 7. show lassfaton auray vs. total parameters on logarthm sale and lassfaton auray vs. proessng tme respetvely for all the tehnques on speeh dataset. The plottng sheme s smlar to that appled for Fg. 6. and 6.. It s evdent from Fg. 7. and 7. that tehnque s performng better than all the other tehnques nludng NN and knn n terms of ahevng hgher lassfaton auray at low total parameter requrement and low proessng tme. The lassfaton auray of NN tehnque s even poorer Am. J. Appl. S., (0): (0): , PCA PCA knn log 0 (total paramters) Fg. 7.: Classfaton auray vs. log 0 (total parameters) on speeh dataset PCA 4 PCA Proessng Tme 6 NN 3 NN knn Fg. 7.: Classfaton auray vs. proessng tme on speeh dataset than, PCA and tehnques; ths means that nreasng total parameters does not always help n mprovng the lassfaton auray. The maxmum lassfaton auray for tehnque s 84.% at usng 8.74 unts proessng tme, whereas the nearest tehnque n terms of auray s knn whh s gvng 78.3% (for k = ) at usng unts proessng tme. It an be onluded from the experments on mage dataset and speeh dataset that tehnque outperforms, PCA,, PCA, NN and knn tehnques n terms of gettng reasonably aepted lassfaton auray and at the same tme mantanng mnmal total parameter requrement and proessng tme. Ths would enable the user to lassfy a gven obet aurately and qukly wth mnmal mplementaton ost. CONCLUSION A survey on bas lassfers namely,, PCA, NN and knn was gven. Ther lassfaton proedures were llustrated. Then we looked at PCA

10 tehnque whh s normally used for representaton purposes. We showed how to use PCA for lassfaton purposes. However, we found that PCA dd not gve very enouragng performane as a lassfer but ths gave us ntatve to develop ombned lassfers. Next we presented tehnque whh s the ombnaton of and PCA tehnques. By ombnng the lassfers we found that the performane mproved sgnfantly whh was not possble by usng ether or PCA ndvdually. The performane of tehnque s found to be better than all the other presented tehnques. Thus t an lassfy a gven obet more aurately at very low mplementaton ost and proessng tme, whh was demonstrated usng speeh and mage datasets. It was found that when the weghtng oeffent α was lose to 0.5 the tehnque gave lose to optmum performane,.e. when and PCA tehnques ontrbute equally n the deson makng of a test pattern then the performane s lose to optmum. REFERENCES. Jan, A.K., R.P.W. Dun and J. Mao, 000. Statstal pattern reognton: a revew. IEEE Trans. Pattern Anal. Mahne Intellgene, : Fukunaga, K., 990. Introduton to Statstal Pattern Reognton. Aadem Press In., Hartourt Brae Jovanovh, Publshers. 3. D Mao V. and F. Marano, 003. Automat lassfaton of neural spke atvty: an applaton of mnmum dstane lassfers. Cybernets and Systems, 34: Palk, P. and R.P.W. Dun, 003. Dssmlartybased lassfaton of spetra: Computatonal ssues. Real-Tme Imagng, 9: Sahn, F., 000. A radal bass funton approah to a olor mage lassfaton problem n a real tme ndustral applaton. PhD Thess, State Unversty, Vrgna. 6. Datta, P. and D. Kbler, 997. Symbol nearest mean lassfers. Pro. of the 4th Natl. Conf. on Artfal Intellgene, San Mateo, CA, pp: Grguolo, S., 994. Pxel-by-pxel lusterng for vegetaton montorng. Intl. Conf. on Alerte préoe et suv de l'envronment, Namey, Nger. 8. Lewensten, K. and M. Chonak, 004. Mnmum dstane lassfers n oronary artery dsease dagnosng. Modellng n Mehatrons, Kazmerz Dolny, Poland. 9. Lambrou, T., A.D. Lnney, R.D. Speller and A. Todd-Pokropek, 00. Statstal lassfaton of dgtal mamograms usng features from the spatal and wavelet domans. Medal Image Understandng and Anal., Portsmouth, UK. Am. J. Appl. S., (0): (0): , Toth, D., A. Condurahe and T. Aah, 00. A two-stage-lassfer for defet lassfaton n optal meda nspeton. 6th Intl. Conf. on Pattern Reognton (ICPR'0), 4: Lnde, Y., A. Buzo and R.M. Gray, 9. An algorthm for vetor quantzaton desgn. IEEE Trans. on Comm., COM-8, : Gray, R.M., 984. Vetor quantzaton. IEEE ASSP Magazne, pp: Wesel, R.D. and R.M. Gray, 994. Bayes rsk weghted and learnng. Pro. Data Compresson Conf. (DCC 94), UT, USA, pp: Potlapall, H., M.Y. Jasmha, H. Barad, A.B. Martnez, M.C. Lohrenz, J. Ryan and J. Pollard, 989. Classfaton tehnques for dgtal map ompresson. Pro. of the st Southeastern Symp. on System Theory, Tallahassee, FL, USA, pp: Makhoul, J., S. Rouos and H. Gsh, 985. Vetor quantzaton n speeh odng. Pro. of the IEEE, 73: Soong, F.K., A.E. Rosenberg and B. Juang, 987. A vetor quantzaton approah to speaker reognton. AT&T Tehnal Jnrl., 66: Dsung, T.P., 998. Applatons of unsupervsed lusterng algorthms to arraft dentfaton usng hgh range resoluton radar. Pro. IEEE Natonal Aerospae and Eletrons Conf., OH, USA, pp: Arvnd, R. and A. Gersho, 986. Low-rate mage odng wth fnte-state vetor quantzaton. In Pro. ICASSP pp: Oa, E., 983. Subspae Methods of Pattern Reognton. Researh Studes Press, New York. 0. Oa, E. and J. Parkknen, 984. On subspae lusterng. Seventh Intl. Conf. on Pattern Reognton, : Oa, E., 99. Prnpal omponents, mnor omponents and lnear neural networks. Neural Networks, 5: Kambhatla, N., Leen, T.K., 997. Dmenson reduton by loal PCA. Neural Computaton, 9: Kambhatla, N., 995. Loal models and Gaussan mxture models for statstal data proessng. PhD Thess, Oregon Graduate Inst. of S. and Tehnology. 4. Duda, R.O. and P.E. Hart, 973. Pattern Classfaton and Sene Analyss. John Wley and Sons, New York. 5. Sharma, A., K.K. Palwal and G.C. Onwubolu, 006. Splttng tehnque ntalzaton n loal PCA. J. Computer S., : (n prnt). 6. Xu, L., A. Krzyżak and C.Y. Suen, 99. Methods of ombnng multple lassfers and ther applatons to handwrtng reognton. IEEE Trans. on Systems Man. and Cybernets, :

11 Am. J. Appl. S., (0): (0): , Jaobs, R.A., M.I. Jordan, S.J. Nowlan and G.E. Hnton, 99. Adaptve mxtures of loal experts. Neural Computaton, 3: Ho, T.K., J.J. Hull and S.N. Srhar, 994. Deson ombnaton n multple lassfer systems. IEEE Trans. on Pattern Anal. and Mahne Intellgene, 6: Tresp, V. and M. Tanguh, 995. Combnng estmators usng non-onstant weghtng funtons. In G. Tesauro, D.S. Touretzky, T.K. Leen (Eds). Advanes n Neural Info. Proessng Systems 7, MIT press, Cambrdge. 30. Woods, K., K. Bowyer and W.P. Kegelmeyer, 996. Combnaton of multple lassfers usng loal auray estmates. IEEE Comp. So. Conf. Computer Vson and Pattern Reognton CVPR 96, pp: Woods, K., 997. Combnaton of multple lassfers usng loal auray estmates. IEEE Trans. Pattern Anal. Mahne Intellgene, 9: Zhou, L. and S. Ima, 996. Chnese all syllables reognton usng ombnaton of multple lassfers. ICASSP, 6: Almoglu, F. and E. Alpaydn, 997. Combnng multple representatons and lassfers for penbased handwrtten dgt reognton. Intl. Conf. Doument Analyss and Reognton, : Kttler, J., M. Hatef, R.P.W. Dun and J. Matas, 996. On ombnng lassfers. Intl. Conf. Pattern Reognton, : Kttler, J., M. Hatef, R.P.W. Dun and J. Matas, 998. On ombnng lassfers. IEEE Trans. Pattern Anal. Mahne Intellgene, 0: Breukelen van, M. and R.P.W. Dun, 998. Neural network ntalzaton by ombned lassfers. Intl. Conf. Pattern Reognton, : Alexandre, L.A., A.C. Camplho and M. Kamel, 000. Combnng ndependent and unbased lassfers usng weghted average. Intl. Conf. Pattern Reognton, : Turner, K. and J. Gosh, 999. Lnear and order statsts ombners for pattern lassfaton. In A. Sharkey, (Ed.). Combnng Artfal Neural Nets, Sprnger-Verlag, pp: Ueda, N., 000. Optmal lnear ombnaton of neural networks for mprovng lassfaton performane. IEEE Trans. Pattern Anal. Mahne Intellgene, : Senor, A., 00. A Combnaton Fngerprnt Classfer. IEEE Trans. Pattern Anal. and Mahne Intellgene, 3: Le, L., W. Xao-Long and L. Bng-Quan, 00. Combnng multple lassfers based on statstal method for handwrtten Chnese harater reognton. Intl. Conf. Mahne Learnng and Cybernets, : Yao, M., X. Pan, T. He and R. Zhang, 00. An mproved ombnaton method of multple lassfers based on fuzzy ntegrals. World Congress on Intellgent Control and Automaton, 3: Garofalo, S.G., L.F. Lor, F.M. Wllam, F.G. Jonathan, P.S. Davd and D.L. Nany, 986. The DARPA TIMIT aoust-phonet ontnuos speeh orpus drom. NIST. 44. Blake, C.L. and C.J. Merz, 988. UCI respostory of mahne learnng databases. Irvne, CA, Unversty of Calf., Dept. of Informaton and Comp. Sene. 45. Mhe, D., D.J. Spegelhalter and C.C. Taylor (Eds.), 994. Mahne Learnng, Neural and Statstal Classfaton. Ells Horwood. 46. Young, S., G. Evermann, T. Han, D. Kershaw, G. Moore, J. Odell, D. Ollason, D. Povey, V. Valthev and P. Woodland, 00. The HTK Book Verson 3., Cambrdge, England, Cambrdge Unversty. 455