Generalized perceptron learning rule and its implications for photorefractive neural networks

Transcription

1 Cheng et al. Vol. 11, No. 9/Setember 1994/J. Ot. Soc. Am. B 1619 Generalized ercetron learning rule and its imlications for hotorefractive neural networks Chau-Jern Cheng, Pochi Yeh,* and Ken Yuh Hsu Institute of Electro-Otical Engineering, National Chiao Tung niversity, 11 Ta Hsueh Road, Hsinchu, Taiwan 35, China Received Setember 23, 1993; revised manuscrit received Aril 5, 1994 We consider the roerties of a generalized ercetron learning network, taking into account the decay or the gain of the weight vector during the training stages. A mathematical roof is given that shows the conditional convergence of the learning algorithm. The analytical result indicates that the uer bound of the training stes is deendent on the gain (or decay) factor. A sufficient condition of exosure time for convergence of a hotorefractive ercetron network is derived. We also describe a modified learning algorithm that rovides a solution to the roblem of weight vector decay in an otical ercetron caused by hologram erasure. Both analytical and simulation results are resented and discussed. INTRODCTION Learning is one of the most intriguing roerties of the neural network. In the rocess of learning, both memory and information rocessing are involved. In a rimitive network such as the ercetron' the network can be trained to classify a number of inuts by suervised learning with examles. The caability of information rocessing in a neural network is mainly determined by its structure, i.e., the arrangement of neurons, and its interconnection. Once the structure of a neural network is given, the interconnection can be obtained by learning algorithms. The ercetron with a single layer structure is a basic neural network that can be trained to sort a set of inut atterns into category 1 (Cl) and category 2 (C2). A trained ercetron will erform the following attern classification when the network is interrogated with an inut vector X: W X > for X E C1, W X < for X E C2. In the learning rocess the interconnection weight W is modified according to a simle learning algorithm given by Wk+1 = Wk + ek'lxk, (1) where Xk is the inut attern vector, Wk is the interconnection weight vector, is a constant reresenting the learning rate, and Ek is an error signal denoted by ek = 1-1 E if Xk is correctly classified if Xk E C1 but is misclassified if Xk E C2 but is misclassified where the subscrit k is an integer (k = 1, 2, 3,... ) registering the number of tests (or interrogations). We note that the interconnection weight is changed whenever there is a misclassification of an inut vector. The rocess of weight vector udate continues until all the inut atterns are correctly classified. The convergence roof of the simle ercetron can be found in Refs. 2 and 3. A recent develoment in hotoinduced holograms in hotorefractive materials (e.g., LiNbO 3, Sr.Bal-,Nb 2 O6, and BaTiO 3 ) 4 and satial light modulators (SLM's) offers unique ossibilities for the otical imlementation of neural networks with a learning caability. The holograms can be recorded and erased in these media by an aroriate otical interferometric technique. This is ideal for imlementing the interconnection weight, which must be modified in the rocess of learning. Several otical architectures for imlementing the ercetronlike learning networks have been roosed and demonstrated.a 8 Learning is achieved by use of real-time holograhic techniques in hotorefractive crystals to record the modifiable interconnection weight, which is roortional to the amlitude of the hotoinduced hologram. By virtue of the dynamic resonse of the hotorefractive crystal, the hologram may decay during the learning rocess. The hologram decay leads to a decrease of the interconnection weight. We have shown that the holograhic decay may affect the convergence roerty of the ercetron. 9 Significant decay of the interconnection weight may lead to a divergence of the learning rocess. To overcome the hologram decay, the interconnection weight must be strengthened by otical techniques. Mathematically this is equivalent to adding a gain factor in the udate equation. Although some secial cases of otical ercetron with weight decay1 2 have been studied, a general ercetron theory that addresses the issue of gain or decay of the interconnection weight is not available. To illustrate the need, we oint out that the gain in the interconnection weight may also lead to a divergence of the learning rocess. In this aer we consider the convergence roerties of a generalized ercetron algorithm with an arbitrary gain factor e for the weight vector. The decay is reresented by < e < 1. An imortant secial case is the hotorefractive ercetron learning network, in which the gain factor = ex(-t/re) is less than unity /94/ $ Otical Society of America

2 16 J. Ot. Soc. Am. B/Vol. 11, No. 9/Setember 1994 GENERALIZED PERCEPTRON LEARNING ALGORITHM The weight udate equation of a generalized ercetron algorithm can be written as Wk+1 = Wk + Ek 77Xk, (3) where Xk is the inut vector, Wk is the weight vector, and Ek is an error signal at the kth interrogation, resectively. 7 is the learning rate, and, deending on its value relative to 1, is the gain or decay factor of the ercetron network. Following the ercetron learning algorithm and carrying out a similar analysis leading to the roof of convergence, we obtain a solution for the interconnection weight vector W,+, that must satisfy the following relation (see Aendix A): fl() < IW +112 < f2( ), (4) where is the total number of weight changes and NA-- 1-) 1( (5) f2(p) _ 2(1-22 with W being an arbitrary solution and min(w - Yk) /3 for all the attern vectors Yk in C, max(lyk1 2 ) a for all the attern vectors Yk in C, where C C1 C2. In Eqs. (7) and (8) Y is the inut attern redefined in Eq. (A3) of Aendix A, and we note that W+l is, in general, different from W. According to inequality (4) the squared magnitude of the weight vector W+l is confined in the region bounded by f,( ) and f2(p). We recall that is the total number of weight changes for training. The convergence of the learning rocess requires that an uer bound exist for the total number of udating stes. The existence of an uer bound for requires that a solution a exist that satisfies the following equation: A(P.) = MP.) - Cheng et al. (6) (7) (8) (9) C~' '4 C.' O. 1C )o~~~~~~~~~~~~~~~~~~~~~i 5s )O, ~~~~~~~~~~~~~~~~~~~~I it I2 1 - A/'/ I _--is..~~~~~~~~~~~~ ; I In other words, these two curves must intersect. The oint of intersection rovides an uer bound a for the total number of udating stes, which deends on the values of the gain factor 6 of the generalized ercetron algorithm. To understand the convergence roerty, we lot fi() and f2(p) as functions of for various values of 6 in Fig. 1. In the examle we used fi(l) =.5 and f2(1) = 5 with arbitrary units for heuristic exlanation. We note that the intersection occurs at a = 1 for the case t = 1. For cases in which the gain factors t =.9 and t = 1.1, the oint of intersection shifts toward > 1. In other words, either gain or decay of the weight vector will lead to a higher uer bound for the number of weight changes. sing Eqs. (5) and (6) and solving Eq. (9), we obtain the following exression for the uer bound Pa (O- l2) - m(l -)2 Pa= (1f - 2) + m(1-2 (1) where Pm is an uer bound of the total number of weight changes of the normal ercetron algorithm 2 defined by aiwi 2 Pm /32 sing Eqs. (5) and (6), we also find that (11) Fig. 1. er and lower bounds of the squared magnitude of the weight vector IW+11 2 versus udating stes of the generalized ercetron learning network for (a) 4 = 1., (b) e =.9, and (c) e = 1.1. Solid curves, fi(); dashed curves, f2(). The intersection a rovides an uer bound for. In the examle f,(l) =.5 and f2(1) = 5 (arbitrary units) were used. [2(1) f,1m=p (12) We note that Pm is greater than 1 and covers a wide range of numbers because of the large number of solution weights W. Equation (1) shows that the uer bound a is a function of the arameters t and Pm. We also note that the maximum number of udating stes Pa of the generalized ercetron algorithm is deendent

3 Cheng et al. Vol. 11, No. 9/Setember 1994/J. Ot. Soc. Am. B 1621 on the gain factor (or decay rate) and is indeendent of the learning rate -q. To illustrate this, Fig. 2 shows the comuter simulation results of the learning behaviors of the generalized learning algorithm described in Eq. (3) for various values of and 77. In the simulation we used 1 training atterns (Cl: {B, 3, 5, E, 6} and C2: {A, 2, C, 4, D}; each attern consists of 32 X 32 ixels) as shown in Fig. 2(g). In the learning rocess each iteration reresents a comlete cycle of interrogation of all 1 atterns. We note that, in our examle, either too much gain [see Fig. 2(e), t = 1.3] or too much decay [see Fig. 2(a), =.85] leads to a divergence in the learning rocess. Comaring Figs. 2(c) and 2(f), we also see that the convergence behaviors are identical for different values of the learning rate Y7. In Fig. 3 Pa is lotted as function of the gain factor f for various values of Pm. We note that for a given gain factor the uer bound Pa is an increasing function of Pm and the uer bound Pa of the generalized ercetron learning algorithm is always larger than that of the original one ( = 1). That is to say, the minimum Pa occurs at = 1, where lm of(1-2) - m(1 - )2 lim P = l(p Pm (13) This is the case of the normal ercetron (f = 1), which has the smallest uer bound. According to the roerties of the logarithm and analyzing Eq. (1), we find that a finite uer bound Pa for the total number of udating stes of the generalized ercetron learning algorithm exists only when t is bounded in a region given by where emin <~ St < max 6 max =P Pm 1 Pm + 1 (14) (15) (16) 1 X ZR W1_1 6 t el I CI 8t 6 4 I - (a) = 'I A,,.,/ f~~ -/, 16 4.) (d) * ( (e) (e) t ~~~~1.3 G 1.5 Cl IN _ (g) C2 I., I ' (c) I _ I \.~~~~~~~~' 16 c.s I- 4.) ) ( so (D. "I _1 I I. I.'. I. I Fig. 2. Learning behaviors of a generalized ercetron with 1 inut atterns (C1: {B, 3, 5, E, 6} and C2: {A, 2, C, 4, D}; attern consists of 32 X 32 ixels) and for various gain factors: (a) =.85; (b) f =.9; (c) = 1.; (d) -= 1.1; (e) f = 1.3, -=.5; (f) normal ercetron, = 1, 77 = 1; (g) training atterns. Each iteration reresents a comlete cycle of interrogation of all 1 atterns. Y11

4 1622 J. Ot. Soc. Am. B/Vol. 11, No. 9/Setember P., Fig. 3. er bound of udating stes of the generalized ercetron algorithm as a function of gain factor for Pm = 1,, 3 as indicated. In other words, the generalized ercetron learning algorithm may not converge if the gain factor 4 is out of the range secified by inequality (14). The range for the gain factor is deendent on the magnitude of Pmn. Table 1 lists the limits for various values of Pm. Figure 4 lots the limits as functions of Pm. The results in Fig. 4 indicate that, for a given set of inut atterns with a given classification, Pm covers a wide range of values, and the minimum of all these Pm's is useful in determining the range of the gain factor 4. It should be ointed out that the uer bound Pm deends on knowledge of solution weight W. Some values of Pm can be derived from Eq. (11), in which the weight vector W is obtained by the ercetron algorithm described in Eq. (1). By definition these values of Pm are all greater than min(m). In addition, the results of the simulation also include the number of weight changes for convergence. These values ofm as well as obtained by ercetron simulation rovide information about min(m). Secifically, < min(m) < Pm. In our case, Fig. 2(f), we obtained P = 7, Pm = alw,+i 2 /f3 2 = 368. This leads to 7 < min(m) Further simulation results are likely to decrease the range of ossible values of min( m). As mentioned earlier, information about min(m) is useful in determining the range of 4 for convergence. For Fig. 2(f) the range of 4 for convergence is at least.995 < 4: < 1.5, which we obtain by taking Pm = 368 as the worst case. The range of 4: for ercetron convergence is useful in the imlementation in which the interconnection weight is likely to suffer gain or loss because of ractical issues such as noise, electrical resistance, holograhic decay, and even system imerfection. The range of 4 for convergence would allow a finite range of tolerance in the system design. PHOTOREFRACTIVE PERCEPTRON NETWORK A secial case of interest is the hotorefractive ercetron, where the gain factor (or decay factor) is given by 4: = ex(-t/te), and the learning rate is given by -q = 1 - ex(-t/tr). as The weight udate equation is thus written W+l = ex(-t/te)w + [1 - ex(-t/r)]y, (17) where Y is the inut attern, W is the interconnection weight, and r and T T e are the rise and decay time of the hotorefractive crystal, resectively. t is the exosure time for each of the weight udates. Equation (17) indicates that the learning rate rq and decay rate 4 of the otical ercetron network are determined by the exosure time during the holograhic recording and the resonse times of the hotorefractive crystal. sing 4 = ex(-t/te) in Eq. (1), we obtain the following exression for the uer bound of the total number of weight changes of the hotorefractive ercetron: F Te 1 + Pm tanh(t/2re) 1 (18) Pa = e Ini I18 t L1 - Pm, tanh(t/2r,) According to Eq. (18), a finite uer bound requires that the exosure time of the otical ercetron network be written as < t < ln(pn + 1) (19) For Pm >> 1, inequality (19) can be written as tpm < 2Te. () Inequality () may rovide a guideline for the exosure time t rovided that min(m) is known. The hotorefractive ercetron will converge in a finite number of stes rovided that the exosure time is limited by inequality (). We note that inequality () is a sufficient condition for the convergence of a hotorefractive ercetron. Although there are infinite numbers of Pm, the minimum of all these Pm's rovides the guideline for the exosure time ,, Table 1. Limits of Gain Factor 4t Pm emin emax \ ~max nm ' O Fig. 4. Limits of gain factor e as functions of Pm. Solid curve, emax; dashed curve, emin. Pm Cheng et al.

5 Cheng et al. MODIFIED LEARNING ALGORITHM As a result of hologram decay during the training rocedure, the hotorefractive ercetron network may not converge or convergence may require more. This can be seen from the simulation results illustrated in Figs. 2(a)-2(f). Physically the hologram decay is consistent with the decay of the weight vector. When there is a large number of inut atterns, the weight decay may become too severe for the ercetron network to converge. To overcome the weight decay in a hotorefractive ercetron in the learning rocess, a method for weight restoration 3," 4 is required. The system can realize this by reading out the weight vector and then rerecording it otically in the same medium by using the technique of hase conjugation during each exosure for udate by revious records in holograhic associative memory.' 5 ' 1 6 Mathematically, the equation for the weight udate becomes W+1 = ex(-t/re)w + [1 - ex(-t/r)](y + AW), (21) where A is a constant. In other words, a revious weight vector is added to the inut vector to comensate for the weight decay. The equation can be rewritten as W+1 = {ex(-t/,r) + A[l - ex(-t/r)]}w + [1 - ex(-t/rr)]y. (22) According to the general results, inequality (14), the sufficient condition for the convergence of the ercetron becomes 6.mil < ex(-t/re) + A[l - ex(-t/tr)] < max. (23) The constant A can be roerly chosen to tailor the gain factor. For the case of a unity gain factor, i.e., ex(-t/te) + A[1 - ex(-t/rr)] = 1, we obtain A 1 - ex(-t/le) (24) 1-ex(-t/rr) For short exosure time we obtain A T rire. Since each constant A corresonds to a unique value of 4, the results of simulation are in exact agreement with those obtained in Fig. 2. CONCLSIONS In conclusion, we have considered the roerties of a generalized ercetron learning network, taking into account a gain factor in the udate of the interconnection weight. We have shown that the learning rate will not affect the convergence of the ercetron. It merely affects the scale of the inut vectors. The gain factor (or decay factor) may affect the convergence of the ercetron learning rocess. We have derived conditions for the convergence of the learning rocess. The range of the gain factor () for ercetron convergence is useful in hardware imlementation, where the interconnection weight may suffer gain or loss during the training stage because of ractical issues, which would allow a finite range of tolerance in the system design. We have also considered the se- Vol. 11, No. 9/Setember 1994/J. Ot. Soc. Am. B 1623 cial case of hotorefractive ercetrons in which the gain factor is an exonential term accounting for holograhic erasure during illumination. A feasible guideline for the exosure time of a hotorefractive ercetron network is resented. A modified hotorefractive ercetron with a comensation technique for overcoming the weight decay is roosed and analyzed. APPENDIX A. CONVERGENCE PROOF OF THE GENERALIZED PERCEPTRON We assume that the sets CI and C2 are linearly searable and that the union of these two subsets is the comlete training set C. Here, without loss of generality, we also assume that the ercetron network has a zero threshold value. That is, at least one solution weight vector W exists such that W X > for X E C1, W X < for X E C2. (Al) (A2) For the urose of roving the convergence, we define a set of new vectors such that for X E Cl for X E C2 Thus inequalities (Al) and (A2) can be written as W Y>O. The weight udate equation (3) now becomes W+1= (A3) (A4) 4:W + Y, (A5) where the attern Y is misclassified by the weight W and is now an integer registering the number of weight changes. We assume that there is no gain or decay of the weight vector during all the interrogations whose classifications are correct. In other words, 4 = 1, where ek =. If we start with the initial weight WI =, we obtain the weight vector after udating stes: W+1 = 4:- 1 7yl + ep-2, 7y y = 7 I 4P kyk k=1 (A6) Taking the dot roduct of the above equation with solution weight vector W, we obtain W Y = X _X + = IZ -Pk(W Yk). k=1 (A7) For a given solution weight vector W, let /3 be the minimum for all the inner roducts, i.e., min(w Yk) for all the attern vectors Yk, where 13 >. From Eqs. (A7) and (A8) we obtain (A8) (1 ) (A9)

6 1624 J. Ot. Soc. Am. B/Vol. 11, No. 9/Setember 1994 Cheng et al. sing the Cauchy-Schwarz inequality, lw,112 (W W+l) and inequalities (A9) and (A1), we obtain IWP ( _ We define a function of as sing Eqs. (A12) and (A18) and solving Eq. (A), we obtain the uer bound for : (A1) (1-2) - Pm(l - )2 P. = loge[ (1-2) + Pm(l - )2, (A21) (All) where Fm apwm 2 'g2 (A22) with Pm being the maximum number of finite udating stes of the normal ercetron algorithm. 2 From Eq. (AS) we have JWk±1 2 = 21IWk 2 + /W12 J (I - ) 2 1YkI 2 (A12) + 277(Wk - Yk). (A13) sing Wk - Yk < during training and 2f 7 >, we obtain JWk+11 2 _ 4: 2 IWkI 2 + fl 2 iykl2* (A14) By adding these inequalities for k = 1, 2,...,, we obtain k=1 Let the maximum magnitude of the attern vectors be written as This leads to max(iykj2) a. IW+l 2 ' 72 _ We define another function of as f2() a cg7( 2) Combining relations (All), (A12), (A17), obtain the following relation: fl() S IWP ' f2(p). and (A18), we Inequality (A19) indicates that the weight vector on comletion of the training W+1 will be localized in the region bounded by fi() and f2(). We recall that is the total number of udating stes leading to a trained ercetron. The total number of stes must satisfy the above equation. An uer bound for exists rovided that the following equation yields a finite solution.: fa(p.) = f2(po) - ACKNOWLEDGMENTS The research is suorted by the National Science Council, Taiwan, under contract NSC E Pochi Yeh acknowledges the suort of the K. T. Lee/K Y. Chin Foundation. *Permanent address, Deartment of Electrical and Comuter Engineering, niversity of California, Santa Barbara, California REFERENCES 1. F. Rosenblatt, Princile of Neurodynamics: Percetrons and the Theory of Brain Mechanisms (Sartan, Washington, D.C., 1962). 2. N. J. Nilsson, Learning Machines: Foundation of Trainable Pattern-Classifying Systems (McGraw-Hill, New York, 1965). 3. R.. Duda and P. E. Hart, Pattern Classification and Scene Analysis (Wiley, New York, 1973). 4. See, for examle, P. Yeh, Introduction to Photorefractive Nonlinear Otics (Wiley, New York, 1993). 5. D. Psaltis, D. Brady, and K. Wagner, "Adative otical networks using hotorefractive crystals," Al. Ot. 27, (1988). 6. E. G. Paek, J. Wullert, and J. S. Patel, "Holograhic imlementation of a learning machine based on a multicategory ercetron algorithm," Ot. Lett. 14, (1989). 7. J. Hong, S. Cambell, and P. Yeh, "Otical attern classifier with ercetron learning," Al. Ot. 29, (199). 8. K. Y. Hsu, S. H. Lin, C. J. Chen, T. C. Hsieh, and P. Yeh, "An otical neural network for attern recognition," Int. J. Ot. Comut. 2, (1991). 9. K. Y. Hsu, S. H. Lin, and P. Yeh, "Conditional convergence of hotorefractive ercetron learning," Ot. Lett. 18, (1993). 1. J. Hertz, A. Krogh, and R. Palmer, Introduction to the Theory of Neural Comutation (Addison-Wesley, Reading, Mass., 1991), Cha. 6, G. Parisi, "Asymmetric neural networks and the rocess of learning," J. Phys. A 19, L675-L68 (1986). 12. J. P. Nadal, G. Toulouse, J. P. Changeux, and S. Dehaene, 'Networks of formal neurons and memory alimsests," Eurohys. Lett. 1, (1986). 13. D. Brady, K. Hsu, and D. Psaltis, "Periodically refreshed multily exosed hotorefractive holograms," Ot. Lett. 15, (199). 14. Y. Qiao and D. Psaltis, "Samled dynamic holograhic memory," Ot. Lett. 17, (1992). 15. B. H. Soffer, G. J. Dunning, Y. Owechko, and E. Marom, "Associative holograhic memory with feedback using haseconjugation mirrors," Ot. Lett. 11, (1986). 16. Y. Qiao, D. Psaltis, C. Gu, J. Hong, P. Yeh, and R. R. Neurgaonkar, "Phase-locked sustainment of hotorefractive holograms using hase conjugation," J. Al. Phys. 7, (1991).