CMAC: RECONSIDERING AN OLD NEURAL NETWORK. Gábor Horváth

Transcription

1 CMAC: RECONSIDERING AN OLD NEURAL NEWORK Gábor Horváth Budapest Unversty of echnology and Economcs, Department of Measurement and Informaton Systems Magyar tudósok krt.. Budapest, Hungary H-7. Abstract: Cerebellar Model Artculaton Controller (CMAC has some attractve features: fast learnng capablty and the possblty of effcent dgtal hardware mplementaton. Although CMAC was proposed many years ago, several open questons have been left even for today. Among them the most mportant ones are about ts modellng and generalzaton capabltes. he lmts of ts modellng capablty were addressed n the lterature and recently a detaled analyss of ts generalzaton property was gven. he results show that there are dfferences between the one-dmensonal and the multdmensonal versons of CMAC. he modellng capablty of a multdmensonal network s nferor to that of the one-dmensonal one. hs paper dscusses the reasons of ths dfference and suggests a new kernel-based nterpretaton of CMAC. It shows that a one-dmensonal bnary CMAC can be consdered as an SVM wth second order B-splne kernel functon. Applyng ths approach the paper shows that one-dmensonal and multdmensonal CMACs can be constructed wth smlar modellng capablty. Copyrght 3 IFAC Keywords: Neural Network, Support Vector Machnes, CMAC, Rdge regresson. INRODUCION he paper deals wth CMAC, a specal neural archtecture (Albus, 975. CMAC has some attractve features. he most mportant ones are ts extremely fast learnng capablty (hompson and Kwon, 995 and the specal archtecture that lets effectve dgtal hardware mplementaton possble (Mller et al., 99; Horváth and Deák, 993; Ker, et al., 995. CMAC archtecture was proposed n the mddle of the seventes and t has been mentoned as a real alternatve of ML (Mller et al., 99. he prce of the advantageous propertes s that ts modellng capablty s nferor to that of an ML (Commur et al., 995. hs especally true for multdmensonal case, as a multdmensonal CMAC can approxmate well only a functon belongng to the addtve functon set (Brown, et al., 993. A further defcency of CMAC s that ts generalzaton capablty s also nferor to that of an ML even for one-dmensonal cases. hs drawback was dscussed n (Szabó and Horváth,, where the real reason of ths property was dscovered and a modfed tranng algorthm was proposed for mprovng the generalzaton capablty. However, the other dsadvantage, the lmted modellng capablty n multdmensonal cases cannot be mproved usng the proposed modfcaton. hs paper wll deal wth ths queston. It wll show that the lmted modellng capablty comes from the archtecture of the multdmensonal CMAC. he archtecture can be modfed n such a way that the modfed verson wll have mproved modellng capablty (Lane at al., 99. However, ths modfcaton would ncrease the network complexty. hs ncreased complexty s unacceptable from the vewpont of realzaton f the nput dmenson s (much larger than two. CMAC can be nterpreted as a network where the output s formed as a sum of weghted bass functons. he orgnal, bnary CMAC apples rectangular (frst-order B-splne bass functons of fxed postons.

2 hs paper shows that an alternatve nterpretaton can also be gven, when the network s consdered as an SVM (support vector machne wth second-order B-splne kernel functons, where the postons of the kernel functons depend on the tranng data. hs nterpretaton can be appled for hgher-order CMACs (Chang Chng-san and Ln Chun-Sn, 996, (Ker et al., 995 wth hgher order bass functons (k th order B-splnes. In these cases CMACs correspond to SVMs wth properly chosen hgher order ((k- th order B-splne kernels.. A SHOR OVERVIEW OF HE CMAC CMAC s an assocatve memory type neural network, whch performs two subsequent mappngs. he frst one - whch s a non-lnear mappng - projects an nput space pont u nto an assocaton vector a. he second mappng calculates the output of the network as a scalar product of the assocaton vector a and the weght vector w: a w a + a w+ + w a + +3 w + 3 u a +4 w + u a 4 +5 w + 5 u 3 dscrete nput space Fg.. he mappngs of a CMAC y(u=a(u w ( C=4 u a a y a j a a j+ a j+ j+ 3 w j w j+ w j+ w j+ 3 a w assocaton vector weght vector he assocaton vectors are sparse bnary vectors, whch have only C actve elements, C bts of an assocaton vector are ones and the others are zeros. In ths case scalar products can be mplemented wthout multplcaton; the scalar product s nothng more than the sum of the weghts selected by the actve bts of the assocaton vector. y( u = ( w : a ( u = Σ y CMAC uses quantzed nputs, so the number of the possble dfferent nput data s fnte. here s a oneto-one mappng between the dscrete nput data and the assocaton vectors,.e. each possble nput pont has a unque assocaton vector representaton. Another nterpretaton can also be gven to the CMAC. In ths nterpretaton every bt n the assocaton vector corresponds to a bnary bass functon wth a fnte support of C quantzaton ntervals. hs means that a bt wll be actve f the nput value s wthn the support of the correspondng bass functon. hs support s often called as the receptve feld of the bass functon. An element of the assocaton vector can be consdered as the value of a bass functon for a gven nput, so the output of the bnary bass functon s one f an nput s n ts receptve feld and zero elsewhere: f u s n the receptce feld a ( u = of the th bass functon (3 otherwse he mappng from the nput space nto the assocaton vector should have the followng characterstcs: ( t should map two neghbourng nput ponts nto such assocaton vectors, where only a few elements -.e. few bts - are dfferent. ( as the dstance between two nput ponts grows, the number of the common actve bts n the correspondng assocaton vectors decreases. he nput ponts far enough from each other - further then the neghbourhood determned by the parameter C - should not have any common bts. hs mappng s responsble for the non-lnear property and the generalzaton of the whole system. he two mappngs are mplemented n a two-layer network archtecture. he frst mappng mplements a specal encodng of the quantzed nput data, ths layer s fxed; the tranable elements, the weght values whch can be updated usng the smple LMS rule, are n the second layer. he way of encodng, the postons of the bass functons n the frst layer, determne the generalzaton property of the network. In one-dmensonal cases every quantzaton nterval wll determne a bass functon, so the number of bass functons s approxmately equal to the number of possble dscrete nputs. However, f we follow ths rule n hgher dmensonal cases, the number of bass functons wll grow exponentally wth the dmenson, so the network may become too complex. As every selected bass functon wll be multpled by a weght value, the sze of the weght memory s equal to the total number of bass functons, to the length of the assocaton vector. In hgher dmensonal case the weght memory can be so huge that t cannot be mplemented. o avod ths hgh complexty the number of bass functons must be reduced. In a classcal hgher dmensonal CMAC ths reducton s acheved usng bass functons postoned only at the dagonals of the quantzed nput space as t s shown n Fgure. he shaded regons n Fgure are the receptve felds of dfferent bass functons. As t s shown the bass functons are grouped nto overlays. One overlay contans bass functons wth non-overlappng supports, but the unon of the supports covers the whole nput space. he dfferent overlays have the same structure; they consst of smlar bass functons n shfted postons. Every nput data wll select C bass functons, each of them on a dfferent overlay, so n an overlay one and only one bass functon wll be actve for every nput pont. he postons of the

3 overlays and the bass functons of one overlay can be represented by defnte ponts. onts of subdagonal onts of man dagonal Fg.. he bass functons of a two-dmensonal CMAC In the orgnal Albus scheme the overlay-representng ponts are n the man dagonal of the nput space, whle the bass functon postons are represented by the subdagonal ponts as t s shown n Fgure (black dots. In the orgnal Albus archtecture the number of overlays does not depend on the dmenson of the nput vectors; t s always C. hs means that n hgher dmensonal cases the number of bass functon wll not grow exponentally wth the dmenson. hs s an advantageous property from the pont of vew of mplementaton, however ths reduced number of bass functons s the real reason of the nferor modellng capablty of the multdmensonal CMAC, as reducng the number of bass functons the number of free parameters wll also be reduced (Brown and Harrs, 994. o avod ths unwanted effect we can get help from the approach of support vector machnes (SVMs (Vapnk, 995. It should be mentoned that n hgher-dmensonal cases further complexty reducton s requred. hs reducton s acheved by applyng a compressng new layer (Albus, 975, whch uses Hash codng. However, the effect of hashng can be neglected, when ts features are selected properly (Ellson, 99, so we wll not deal wth ths effect. 3. A BRIEF SUMMARY OF SVMs AND LS-SVMs Support Vector Machnes (SVMs were proposed recently by Vapnk (995 and are based on Statstcal Learnng heory (SL and Structural Rsk Mnmzaton (SRM prncple. SVMs can be constructed for lnear or non-lnear classfcaton tasks as well as for lnear or non-lnear regressons. Here only the regresson verson wll be summarzed. An SVM s constructed usng tranng data { x }, y = u quantzaton ntervals overlappng regons Regons of one overlay smlarly to the classcal neural networks. he goal of an SVM for regresson s to approxmate a (non-lnear functon (Smola et al., 998, where u the qualty of approxmaton s determned usng a loss or cost functon. he loss functon s representng the cost of the devaton from the target output d for each x nput. In most cases the ε nsenstve loss functon (L ε s used, but one can use other (e.g. non lnear loss functons too. he ε nsenstve loss functon s: for f ( x d < ε Lε ( y = (4 f ( x d ε otherwse Usng ths loss functon the approxmaton errors smaller than ε are gnored, whle the larger ones are punshed n a lnear way. Our goal s to gve an y = f ( x functon, whch represents the dependence of the output y on the nput x. In the SVM approach frst the nput vectors are projected nto a hgher dmensonal feature space, usng a set of non-lnear functons N M ϕ ( x : R R. he dmensonalty (M of the new feature space s not defned, t follows from the method (t can even be nfnte. he functon s estmated by projectng the nput data to the hgher dmensonal feature space as follows: where M w j j = y = ϕ = j ( x w ϕ( x w = [ w w,..., ] and ϕ = [ ϕ ( x ϕ ( x,..., ϕ ( x ] w M, (5, M It s assumed that ϕ (x, therefore w represents a bas term b. he goal of approxmaton s to fnd the parameter vector w n the representaton of Eq. (5. In ths optmzaton problem we have constrants gven n Eq. (6 (t comes from the ε nsenstve loss functon L ( f x, y ε ( and a further constrant of w c to keep w as short as possble (c s a constant. o deal wth tranng ponts outsde the ε boundary, the { } ξ = and { ξ } = slack varables are ntroduced: d w ϕ w ϕ( x ( x d ξ, ξ, ε + ξ, ε + ξ, =,,..., (6 he slack varables are ntroduced to descrbe the penalty for the tranng ponts lyng outsde the ε boundary. he measure of ths cost s determned by the loss functon. hs s solved by mnmzng the followng objectve functon: F( w, ξ, ξ = w w + ρ ( ξ + ξ (7 = he frst term stands for the mnmzaton of w, whle the ρ constant s the trade off parameter between ths and the mnmzaton of tranng data errors. hs constraned optmzaton can be defned as a Lagrangan functon. hs s often called as the prmal problem:

4 ( w,ξ,ξ, α, α, γ, γ = ρ ( ξ + ξ J α = α = = + w [ w ϕ( x d + ε + ξ ] [ w ϕ( x d + ε + ξ ] ( γ ξ + γ ξ = w (8 We have to mnmze (8 accordng to w and b and maxmze accordng to the Lagrange multplers. he prmal problem deals wth convex cost functon and lnear constrants, therefore from ths constraned optmzaton problem a dual problem can be constructed, whch can be solved more easly. he soluton can be obtaned usng quadratc programmng (Q. o do ths the Karush Kuhn ucker (KK condtons are used (Vapnk, 995. he dual problem s: Q ( α, α = d ( α α ε ( = ( α α ( α α K( x,x α + α = (9 = j= j j j Wth constrants: ( α α = α, α ρ, =,..., ( = From the dual problem t can be seen that the soluton does not depend drectly on the ϕ nonlnear functon set. Instead a kernel functon s used whch s formed as: K( x, x j = ϕ ( x ϕ( x j he result of the dual problem s the sets of the Lagrange multplers α and α. Fnally the response of the SVM can be determned as the weghted sum of the values of the kernel functon. ( K( x, x y = α α = ( he nonzero multplers ( ( α α, =,, mark ther correspondng nput data ponts as support vectors. It can be seen that the response depends only on the support vectors and not on all tranng data. hs property s an nterestng and mportant feature of SVM solutons. It shows that the soluton s obtaned as a sparse approxmaton. For constructng an SVM nstead of choosng the non-lnear functons, the kernel functon should be selected. o be a kernel a functon must satsfy the Mercer condton (Vapnk, 995. he most popular kernel functons are the Gaussan functons. In ths case the SVM corresponds to an RBF network where the centre vectors of the Gaussan bass functons are the support vectors. So the structure of a support vector machne and an RBF may be smlar although ther constructons are qute dfferent. he man drawback of SVM s ts hgh computatonal burden because of the requred quadratc programmng. Recently a least square (LS verson of SVM was proposed by Suykens (. LS-SVM can also be appled for both classfcaton and regresson. LS-SVM apples quadratc cost functon and equalty constrants nstead of the nequalty ones gven n Eq. (6. he optmsaton problem of LS-SVM s formulated as follows: F( w, e = w w + γ e, ( = where d = w ϕ( x + b + e. (3 From these equatons one can construct the Lagrangan, whch leads to the followng overall soluton: r b r = (4 + Ω C I α d [ ] [,...,] = [ ] Here d = d, d,..., d, α = α, α,.., α, r = and Ω K x, x., j ( j he response of the "network" can be obtaned as f ( x = α, = K( x x b + (5 It can be seen that the response for a gven nput can be obtaned smlarly to (, however, here nstead of quadratc programmng only a matrx nverson s requred to determne the Lagrange multplers. It can also be seen from ( that the real problem s to fnd such a weght vector that mnmzes the cost functon. An mportant dfference between Vapnk's SVM and LS-SVM s that the latter soluton s not sparse; all tranng ponts are used for gettng the soluton. o get sparse soluton, however, many dfferent approaches were proposed for example (Suykens et al.,, (Valyon and Horváth,. It can also be recognsed that LS-SVM and the method of rdge regresson (Saunders et al., 998 are almost equvalent, as the functon to be mnmzed n rdge regresson s: F( w, e = w w + γ e (6 = wth the constrant of d = w ϕ( x + e (7 hs means that the approxmaton error wll be: e = d w ϕ( x (8 Agan the Lagrangan can be constructed whch leads to the soluton: and α = Ω + I d (9 γ ( = ( x x f x = α K, ( he only dfference between LS-SVM and rdge regresson s that n rdge regresson bas s not used. Further t can be shown easly that the soluton of rdge regresson and a regularzed LS soluton are equvalent. he classcal LS soluton can be obtaned f (8 s used n (6. In ths case the optmal weght vector s obtaned from the mnmzaton of the regularzed LS problem.

5 w = Ω + I d ϕ( x γ ( = where Ω = K( x, x as before., j j [ ( ( ( ] he basc dfference between the two approaches s that n the LS soluton the weght vector w, whch s used n the M-dmensonal feature space, s obtaned drectly. Usng the Lagrange method the drect results are the Lagrange multplers. he weght vector w can be obtaned ndrectly from α. An mportant feature s that whle the dmenson of the feature space may be arbtrarly large, even nfnte, the dmenson of α s, whch s the number of the tranng samples. A more mportant feature s that usng the Lagrange approach the soluton ( s expressed as a lnear combnaton of dfferent poston kernel functons, where the centre parameters of the kernel functons are determned by the tranng ponts. Usng the kernel representaton to get the response of the network we do not need to determne the representaton n the feature space. he two solutons can be called prmal and dual solutons. rmal soluton can be preferred when the dmenson of the feature space s not too hgh and where there are no dffcultes wth the mplementaton of the ϕ = ϕ x, ϕ x,..., ϕm x non-lnear functons. On the other hand rdge regresson or LS- SVM approach should be preferred f the dmenson of the feature space s huge or f the sparse representaton has specal advantages. 4. CMAC AS A SUOR VECOR MACHINE he two dfferent approaches dscussed prevously can be appled n the case of the CMAC. he classcal bnary CMAC uses rectangular bass functons (frst-order B-splnes wth fxed postons, whch map the nput data nto the feature space. he assocaton vector corresponds to the feature space representaton. In one-dmensonal case the length of the assocaton vector s rather lmted, so CMAC can be mplemented effcently n the feature space. A one-dmensonal bnary CMAC can also be nterpreted as an LS-SVM (or more exactly as a rdge regresson soluton, because the classcal CMAC does not use bas term, where the kernel functon s obtaned from the fxed-poston frstorder B-splnes: the kernel functons wll be secondorder B-splnes where the centre parameters are the nput data ponts. If the support of the rectangular functons s C measured n quantums of the nput data, the support of a kernel functon s C. In onedmensonal case the kernel representaton of the CMAC has no specal advantages. However, n a multdmensonal case usng the prmal representaton we have to reduce the number of rectangular bass functons, the dmenson of the feature space. Wthout ths reducton the complexty of the multdmensonal CMAC would ncrease exponentally wth the nput dmenson. he reducton of the dmenson of the feature space n a classcal N-dmensonal CMAC s acheved by usng only C overlays of rectangular bass functons nstead of C N ones (and we apply Hash codng as t was dscussed n Secton. But ths reducton s the real reason of the lmted modellng capablty of the multdmensonal CMAC. If we used C N overlays we would get a multdmensonal CMAC wth the same modellng capablty as a one-dmensonal network. In the prmal space a CMAC wth C N overlays cannot be mplemented because of ts hgh complexty. However, n the dual space we use the kernel functon, and the complexty wll not ncrease even f the feature (prmal space s a very-large dmensonal one. he second-order B-splne kernel functon n twodmensonal case s shown n Fgure 3a. hs s the dscretzed verson of the contnuous second-order B- splne (Fgure 3b. accordng to the quantzed nput space of the CMAC a. b. Fg. 3. Second-order B-splne kernel functons for CMAC wth C=4. he new nterpretatons have some mportant consequences. Frst of all there wll be no sgnfcant dfference between the one-dmensonal and the multdmensonal cases. he modellng capablty of the multdmensonal verson wll be the same as that of the one-dmensonal CMAC wthout gettng so complex network archtecture that cannot be mplemented. In multdmensonal cases the SVM (rdge regresson representaton wll not be equvalent to the Albus CMAC, as t corresponds to a CMAC wth exponentally ncreasng number of overlays. Fgure 4. shows ths dfference n a smple example. he approxmaton error n the tranng ponts of the D snc functon s shown for the Albus CMAC n Fgure 4a. and for the kernel-based verson n Fgure 4b. It can be seen that n the latter case all tranng ponts can be represented exactly, whle the orgnal verson are not able to learn exactly all tranng data. a. Fg. 4. he tranng error of the D CMAC a. and the kernel network b. for the D snc functon. he kernel representaton and the rdge regresson verson can be constructed not only for bnary CMAC but for hgher-order networks too. In hgherorder CMACs nstead of the rectangular bass b

6 functons hgher-order B-splnes are used. Usng a k th -order B-splne n the orgnal CMAC (n the prmal space, the correspondng SVM versons wll use (k+ th order B-splnes as SVM kernels. he mproved modellng capablty has a prce. he kernel representaton needs hgher-order functons even n the case that corresponds to the bnary CMAC. herefore the network cannot be mplemented wthout usng multplers. hs drawback, however, at least partly can be compensated usng some effectve multpler archtecture (see e.g. Szabó et al., and an effcent constructon where all advantageous of ths multpler structure can be utlsed. 5. CONCLUSION In summary we can conclude that two nterpretatons of the CMAC networks can be used. he frst the orgnal one apples rectangular bass functons, where the bass functons mplement a mappng from nput space nto a "feature" space, and the soluton s obtaned as a lnear mappng from ths feature space nto the output. he SVM approach apples pecewse lnear B-splne bass functons as kernel functons and the soluton s obtaned as a mappng from the kernel space nto the output space. he frst verson s better (the network s rather smple, ts tranng s fast, etc n one-dmensonal case, but the second one can be preferred n multdmensonal cases as - although t may need more complex tranng algorthm - t has better modellng capablty. REFERENCES Albus, J.S. (975 "A New Approach to Manpulator Control: he Cerebellar Model Artculaton Controller (CMAC", ransacton of the ASME, Sep pp. -7. Brown, M. - Harrs, C.J. - arks, (993. "he Interpolaton Capablty of the Bnary CMAC", Neural Networks, Vol. 6, pp Brown, M. and Harrs, C.J. (994 "Neurofuzzy Adaptve Modelng and Control" rentce Hall, New York, 994. Chang Chng-san and Ln Chun-Sn (996: CMAC wth General Bass Functons. Neural Networks, Vol. 9, pp Commur S., Lews F.L. and Jagannathan S. (995: Dscrete-tme CMAC Neural Networks for Control Applcatons. roc. of the 34 th Conference on Decson & Control, New Orleans, LA, Vol.. pp Ellson, D. (99 "On the Convergence of the Multdmensonal Albus erceptron", he Internatonal Journal of Robotcs Research, Vol., pp Horváth, G. and Deák, F. (993 "Hardware Implementaton of Neural Networks Usng FGA Elements" roc. of he Internatonal Conference on Sgnal rocessng Applcaton and echnology, Santa Clara Vol. II, pp Ker, J.S. - Kuo, Y.H. - Lu, B.D. (995 "Hardware Realzaton of Hgher-order CMAC Model for Color Calbraton", roceedngs, of the IEEE Internatonal Conference on Neural Networks, erth, Vol. 4, pp Lane, S.H. - Handelman, D.A. and Gelfand, J.J (99 "heory and Development of Hgher- Order CMAC Neural Networks", IEEE Control Systems, Apr. 99. pp Mller,.W. III. Glanz, F.H. and Kraft, L.G. (99 "CMAC: An Assocatve Neural Network Alternatve to Backpropagaton" roceedngs of the IEEE, Vol. 78, pp Mller, W.. - Box, B.A. and Whtney E.C. (99 "Desgn and Implementaton of a Hgh Speed CMAC Neural Network Usng rogrammable CMOS Logc Cell Arrays", ANIS 3, pp Saunders, C.- Gammerman, A. and Vovk, V. (998, Rdge Regresson Learnng Algorthm n Dual Varables. Machne Learnng, roc of the Ffteenth Internatonal Conference on Machne Learnng, pp Smola, A.J. and Schölkopf, B. (998 A utoral on Support Vector Regresson, NeuroCOL echncal Report Seres NC R 998 3, Oct., 998. Suykens, J.A.K., Lukas, L. and Vandewalle J. ( Sparse approxmaton usng least squares support vector machnes roc. of the IEEE Internatonal Symposum on Crcuts and Systems ISCAS'. Vol. II, pp Suykens, J.A.K. ( Nonlnear Modelng and Support Vector Machnes, IEEE Instrumentaton and Measurement echnology Conference, Budapest, Vol. I, pp Szabó,. and Horváth, G. (999 CMAC and ts Extensons for Effcent System Modellng" Int. Journal of Appled Mathematcs and Computatons, Vol. 9, pp Szabó,. and Horváth, G. ( "CMAC Neural Network wth Improved Generalzaton Capablty for System Modellng" roc. of the IEEE Conference on Instrumentaton and Measurement, Anchorage, AK. Vol. II, pp Szabó,., Anton, L., Horváth, G. and Fehér, B. ( An effcent mplementaton for a matrx-vector multpler structure, n roc. of IEEE Internatonal Jont Conference on Neural Networks, IJCNN, Vol. II, pp hompson D.E. and Kwon S. (995: Neghbourhood Sequental and Random ranng echnques for CMAC. IEEE rans. on Neural Networks, Vol. 6, pp Valyon, J. and Horváth, G. ( "Reducng the Complexty and Network Sze of LS SVM Solutons" submtted to IEEE rans. on Neural Networks. Vapnk, V. (995 "he Nature of Statstcal Learnng heory", Sprnger, New York.

7

8