1996 Engineering Systems Design and Analysis Conference, Montpellier, France, July 1-4, 1996, Vol. 7, pp

Transcription

1 THE POWER AND LIMIT OF NEURAL NETWORKS T. Y. Lin Department f Mathematics and Cmputer Science San Jse State University San Jse, Califrnia tylin@cs.ssu.edu and Bereley Initiative in Sft Cmputing* Department f Electric Engineering and Cmputer Science University f Califrnia Bereley, Califrnia 9470 ABSTRACT The gal f this paper is t examine the internal f feedfrward neural wr cmputing. Fr this purpse the ntin f neural wrs and its architectures are frmally defined. The thery f apprximating functins by neural wrs are critically examined. A reasnable arguments are given t shw that n finite and fixed architecture can be a universal apprximatr; it is, hwever, nt a mathematical prf. This fact explains the tedius experimentatin f practitiners. Hence an infinite size architecture is prpsed. Mre interestingly, the training f neural wrs is fund t be merely a sphisticated methdlgy f interplatin. Mre precisely, training is a determinatin f the linear cmbinatin and selectin f prper activatin functins.. Neural Netwrs A typical neural wr (NN) can be represented as fllws: Input variables are x, x,..., x N. The input t the -th unit f the first hidden layer N = w i xi + b () where w are weights, i b are bias term, and =,,... N The utput f this unit is i = f ( ) () where, withut lss f generality, the activatin functin and input functin is assumed t be the same (Freeman and Sapura, 99). Output... Kernel ndes... M x x Input... The input t the -th unit f the (h +)-th hidden layer (h +) = w (h +) i i i h + b h (3) where w h are weights, i b h are bias terms, and =,,..., N (h + ) are the index f (h+)-th hidden layer. The utput f this unit is i (h + ) = (h +) f ( (h +) ) (4) The frmulas fr utput ndes are: (the cnstant last dentes the number f hidden layers) * This research is supprted by Electric Pwer Research Institute, Pal Alt, Califrnia

2 = w i last + b (5) = f ( ) (6) Cmpse (5) and (6) tgether, we have = f ( w last i + b ) (7) If we cmpse, (), (),...,(7), we get a ttal utput functin in terms f input variables, i.e., = G ( x, x,..., x N) (8). Gemetry f Mappings in Neural Netwrs We will interpret variables as pints in Euclidean spaces: E 0 = E 0 ( x, x,..., x N) is the Euclidean space that cnsists f pints represented by initial variables x, x,..., x N. E 0 is called initial input space. Similarly, the input variables at h-th layer h,..., h, h =,,... can als be regarded as pints in sme Euclidean spaces: I = I,,..., ), N I = I,,..., ), N.., I h= Ih h,..., h )..., O (,,..., + ). Equatin () defines a linear mapping frm the initial input space E0 t the first input space W 0 : E 0 ( x, x,..., x N) I,,..., ) N Similarly, Eq. (3) defines linear mappings: W h : Oh (i h, i h,..., i h ) (h +) Ih, (h +),..., (h +) ) N(h +) Fr cnvenience, in indexing, we may use O 0 fr E 0 ; we can regard the input variables as utput frm smewhere and get int the input layer. Equatins () and (4) represent very special type f mappings frm h-th space t itself (as Euclidean spaces I h = O h ; hwever, their crdinates are different). The mapping maps each crdinate (f the h-th space) int itself by the activatin functin f that crdinate: F h : Ih h,..., h ) Oh (i h, i h,..., i h ) Such mappings are called crdinate-wise mapping. When h=last, F last : Ilast last, last,..., last ) O (,,..., + ) We als need t cnsider cmpsitins f these mappings; each utput functin becmes a functin defined n initial variables ( x, x,..., x N): Fr each, we have the fllwing functins, I last = Ilast last, last,..., last ) = G 0 ( x, x,..., x N), We will call them the first input, secnd input, and h-th input space respectively. We als need t cnsider the spaces f utput variables (they are input t the next hidden layer): O = O(i, O = O (i,..., i,..., i ), N i,..., i ), N O h = Oh (i h, i h,..., i h ),..., r as mapping f spaces, we have G 0 : E 0 ( x, x,..., x N) O (,,..., + ) Mre generally, each utput functin is als a functin defined the input variables f each layers, h,..., h ). S fr each h =0,,,.. last, we have the mappings = G h h,..., h ), r as mapping f spaces, we have,

3 G h: Ih h,..., h ) O (,,..., + ) We have called E 0 ( x, x,..., x N) the initial input space, s we will call O (,,..., + ) the final utput space, r simply the final space. Definitin.. An architecture f neural wrs r simply architecture is a mathematical mdel specified by Eq. ()-(8), in which the ndes and their activatin functins are fixed, hwever, the weights w h i, b h are unspecified parameters. In gemetric terms, the spaces I h and the maps F h : Ih O h are fixed in an architecture, hwever, the linear maps W h are unspecified. Definitin.. A neural wrs is an architecture with specified weights, in ther wrds, all the weights are assigned by sme numerical values. In gemetric terms, the linear maps W h are unspecified. In ther wrds, an architecture is a parameterized family f neural wrs, in which the weights are the nly parameters. a neural wrs is an architecture with all parameters are specified. Definitin.3. The prcess f specifying a neural wr frm a given architecture using input data is called training. 3. Neural Netwrs f Small Architectures By an architecture f neural wrs, we mean the selectins f ndes, cnnectins (unspecified weights), and activatin functins. In ther wrds, the h-spaces and the crdinate-wise mappings are fixed; the nly unspecified nes are the linear mappings. By training f neural wrs, we mean an algrithm that determines the weights f each nde, r mre precisely, the linear mappings, O h I h, h = 0,,..., last. Nte h=0 and last+ are the initial input and final utput space respectively. 3.. Sme Linear Analysis In this sectin, we recall sme elementary facts in functinal analysis (Rudin, 973). First, we start with linear algebra. Let E = (, 0, 0), E = (0,, 0), and E 3 = (0, 0, ) be the basis f XYZ-space. Let Lspan(E, E ) be the linear space generated (r spanned) by tw vectrs E l and E ; it is the XY-plane. Let X=(,,). Is X ε Lspan(E, E )? Of curse n. In general, XYZ-space E 3 is 3-dimensinal space, while Lspan(E, E ) is -dimensinal. S E 3 Lspan(E, E ) Even if we allw apprximatin, then fr any given ε, the ε- neighbrhd N(Lspan(E, E )) still can nt cver the whle space Mrever, E 3 N(Lspan(E, E )). E 3 N(Lspan(E, E )) N(Lspan(E, E 3)) N(Lspan(E 3, E )). Let F(X) be ne f the usual functin space, such as, the space f analytic, differentiable C (X), cntinus C(X), r Lesbegue integrable L p (X) functins. It is an -dimensinal nrmed linear r Hilbert space with basis {S, S.,..., S...,}. Let FS = {S, S +.,..., S J+} be a finite subset f the basis. Prpsitin.. The clsure f the linear span in F(X), CLspan (FS ) = CLspan {S, S +.,..., S J+} is at mst -dimensinal. Given any psitive real number ε > 0, let N(CLspan(FS )) be an ε - neighbrhd f CLspan(FS ). Prpsitin.. H N(CLspan(FS)) Mre generally, let FS be any finite subset f the basis, whse cardinality is less than. Prpsitin.3. H N(CLspan(FS)), where run thrugh all subsets (f the basis) having cardinality less than. 3.. Neural Netwrs f Tw Ndes We will cnsider the neural wrs with tw ndes. By drpping the indices, Eq. ()-(8) are reduced t = w x + b = f(wx+b) S the set f all pssible neural wrs that can be trained frm this fixed architecture (the number f ndes and activatin functin is fixed) is ne-t-ne crrespnding t the set f all pssible pairs (w, b), where w and b are real numbers t be determined by training. Let us chse further that the activatin functin f t be the identity functin. T emphasize that weight w is a parameter nt a variable, we replace it by a. The equatins are transfrmed t = = a x + b The linear mapping W 0 becmes W 0 : x a x + b

4 We can view W 0 as a linear mapping frm initial input space ( real line) t final utput space (real line). Mrever, n matter hw smart the training algrithm that ne may have, the mapping determined by a trained neural wr is a linear mapping W 0. Essentially, what a training algrithm des is a determinatin f the values f tw parameters, a and b. This example indicates clearly that nce the architecture is fixed (i.e., activatin functin is fixed), real valued functins f single variable that can be apprximated by neural wrs f tw ndes are very limited. In this case, it is nly linear functins. This analysis, in a trivial way, eches the early negative example f Marvin Minsy and Seymur Papert. 3.. Neural Netwrs f Three Layers We will cnsider ne utput, tw inputs, ne hidden layer with N ndes(n is a fixed number, say 9) and n bias terms. Drpping the index f hidden layer and utput variables (since bth have ne bect nly), Eq. () -(8) are reduced t Let i = w x + w y = f ( ) N = w i = f ( ) f be the identity, then the equatins are further simplified t N = w f ( w x + w y) With such an architecture, culd we apply Par-Sandberg therem ( tentatively ignre the difference with radial-basis wrs) and claim that we can apprximate all functins f tw variables? The answer is bviusly n; n readers will believe that 9 Radial-Basisfunctins can express all functins f tw variables. Nevertheless, we will try t give a reasnably cnvincing arguments that such an architecture can nt generated all functins f tw variables. As we have cmmented, all that the neural wr training des is t determine the values f the weights w, w, w, b, b We divide the training in tw steps. First, we will assume that the weights w, w are determined. Let w varies freely, then we have the linear span CLspan( f, f,..., f9 ); these 9 functins are depended n w, w. Nw let us vary w, w freely and assume that we are s lucy that we have the basis f the functins space L p (X). S we have a family f CLspan( f, f,..., f9 ). By Prpsitin.3, all such N(CLspan( f, f,..., f9 )) can nt cver the L p (X); QED. Will the answer change, if we use An arbitrary, but a fixed N? Of curse nt! The family N(CLspan( f, f,..., f N )) still can nt cver the whle L p (X). Even thugh we give argument fr three layer architecture, it is bvius that similar arguments can be applied t general (but fixed) architecture. We hpe we give adequate arguments (but nt mathematical prf) t cnclude that Cnclusin N single neural wr with a fixed architecture can apprximate all functins in L p (X). Then what des Par-Sandberg type universal apprximatr therem say? It will be explained in next sectin. 4. Universal Apprximatrs Par and Sandberg use Radial Basis functins t frm a family f functins, dented by S : M q(x) = i = x wi K σ where () M is the number f ernel ndes, () x is an input vectr, an element in R r, (3) real numbers w i ε R are the weight frm the i-th ernel ndes t utput ndes, (4) M elements z i ε R r fr,,..., M, (5) a real value σ > 0, and (6) K is a radially symmetric ernel functin f a unit in the hidden layer; the neural wr cntains nly ne hidden layer. Therem Under prper cnditins, the family S is dense in L p (R r ) fr every p ε [0, ). This therem says that given ε > 0, fr each functin f in L p, there is a functin q defined by Eq. (9) such that f - q p < ε. We shuld stress that the index M in Eq. (9) may be different fr different functin f. In ther wrds, the family S is nt a finite set. S Par and Sandberg therem des nt guarantee that there is a single architecture f neural wr that can prduce all S K. In fact, we have argued in Sectin 3 that such an architecture des nt exist. S what is a universal apprximatr? It can be understd in tw ways. One is a sequence f architectures. Anther ne is t create an architecture f infinite size. 4.. A Sequence f Architectures Definitin 4.. A sequence f architectures is called a universal apprximatr, if it satisfies the fllwing prperties: zi (9)

5 () The size f each individual architecture is finite, but the sequence f their sizes is unbunded. () Fr any given functin f f a finite number f variables, and any ε > 0, there is a neural wr derived frm ne architecture, say A, f this sequence such that f - p < ε. where is a functin specified by Eq. (8), in which all weights have been assigned with sme numerical values. Fr practitiners, existence f universal apprximatrs guarantee nly that he can find a desirable neural wr by searching thrugh such a sequence f architectures. Briefly, try and errr is the nly methd. 4.. Architectures f Infinite Size Recall that a set is said t be a cuntable set if it has cuntably infinitely many elements. Let us cnsider ne utput, N inputs (a finite number), and ne hidden layer with cuntable ndes. Under this frmat, Eq. () - (8) have t be interpreted prperly fr infinite sum; we will assume that has been dne. Kernel ndes x Output x Input The hidden layers with cuntable ndes can nt be stred in a cmputer system; all cmputer systems have nly finite memry. S these ndes and their activatin functins have t be generated by effective prcedures. Essentially, we need an algrithm t prduce the Radial Basis functins. Par-Sandberg s riginal therem is nly a mathematical therem. They did nt address the cmputability. By chasing thrugh their prf, ne can mdify the prf int an algrithm (Lin 995). Definitin 4.. An architecture (f infinite size) that can apprximate all real valued functin by specifying its weights is called a universal apprximatr. Taing this view, we can have the fllwing generalized therem Prpsitin 4.3. If a family f functins frms a linear base f a functin space, such as L p, then there is a universal apprximatr prvided that such a family f linear base is Turing cmputable, that is, such a family can be generated by a halting Turing machine. Example f such linear base are the set f all mnmials, the set f all trignmetric functins, and sme classical special functins- we will reprts them in near future. We shuld als cautin the readers that mst f existing learning algrithms may nt wr fr such infinite size neural wrs. As we have remared that learning and training is t determine the weights. If the linear basis is nice enugh, we can determine in ther fashin. We will reprt ur findings in near future. 6. CONCLUSION -PRACTICES AND THEORIES Frm ur analysis, it is clear that there are ptential gaps between practices and theries. Many cmmercially available neural wr sftware systems are re-cnfigurable. Withut prper recnfiguring f the architectures, there is n universal apprximatr. Practitiners have cped with such gaps by try and errrs. They have t g thrugh many experiments t find the crrect activatin functins s that the target functin is in a suitable N(CLspan( f, f,..., f N )). We hpe ur prpsed infinite size neural wrs may imprve such a situatin. (6.) N single architecture can apprximate a class f reasnable functins, such as L p (6.) The ntin f learning in the thery f neural wrs is merely a methdlgy f determining prper linear cmbinatin f prper activatin functins. A learned r trained neural wr is a mathematical mdel that apprximates a target functin by a linear cmbinatin f activatin functins. (6.3) An input functin is ften represented as training data. A set f data defines a functin nly n a finite number f pints. S neural wrs is a methdlgy f interplating the given training data by a linear cmbinatin f activatin functins. If ne uses different activatin functin, ne will get different interplatin f input data. The classical pitfalls f interplatins may ccur fr neural wrs. We will reprt r finding in future wr. References Freeman, James and Sapura David, 99. Neural Netwrs- Algrithms, Applicatins, and Prgramming Techniques. Addisn- Wesley Publishing C. Lin, T. Y Universal Apprximatr - Turing Cmputability, Prceedings. Secnd Annual Jint Cnference n Infrmatin Sciences, Par, J. W. Sandberg, I. W., 99. Universal Apprximatin Using Radial-Basis-Functin Netwrs. Neural Cmputatin 3, Rudin W., 973. Functinal Analysis. McGraw-Hill, New Yr. Tsau Yung (T. Y.) Lin received his Ph.D frm Yale University, and nw is a Prfessr at San Jse State University. He has been chairs and members f prgram cmmittees in varius cnferences and wrshps, als served in editrial bards f several internatinal urnals. His interests include apprximatin thery (in

6 databases and nwledge-bases), data mining, data security, fuzzy sets, Petri s, and rugh sets.