A Unified Approach on Fast Training of Feedforward and Recurrent Networks Using EM Algorithm

Transcription

1 2270 IEEE TRASACTIOS O SIGAL PROCESSIG, VOL. 46, O. 8, AUGUST 1998 [12] Q. T. Zhag, K. M. Wog, P. C. Yip, ad J. P. Reilly, Statistical aalysis of the performace of iformatio criteria i the detectio of the umber of sigals i array processig, IEEE Tras. Acoust., Speech, Sigal Processig, vol. ASSP-37, pp , A Uified Approach o Fast Traiig of Feedforward ad Recurret etworks Usig EM Algorithm Sheg Ma ad Chuayi Ji Abstract I this work, we provide a theoretical framework that uifies the otios of hidde represetatios ad movig targets through the expectatio ad maximizatio (EM) algorithm. Based o such a framework, two fast traiig algorithms ca be derived cosistetly for both feedforward etworks ad recurret etworks. I. ITRODUCTIO Although multilayer feedforward ad recurret eural etworks have foud may importat applicatios i sigal processig, traiig such etworks is usually slow due to the oliearity ad, possibly, the compact structure of these etworks. This hiders wide applicatios of these etworks i such importat sigal processig areas as speech ad commuicatio, where the etworks eed to be adapted rapidly to a ostatioary eviromet [4]. Oe of the iterestig approaches used to reduce the traiig time was based o hidde represetatios or movig targets [3], [6], [9]. Targets were obtaied adaptively for either hidde or recurret uits ad the used to decompose traiig a etire (multilayer or recurret) etwork ito traiig several sigle-layer feedforward etworks. The approach resulted i fast traiig algorithms for feedforward etworks with hard threshold uits [3] but was ot effective i traiig etworks with sigmoidal uits [6], [9]. I additio, the algorithms were heuristic, which was less desirable. O the other had, the expectatio-maximizatio (EM) algorithm [2] i statistics was developed based o so-called hidde (radom) variables. Although similar to movig targets, hidde (radom) variables were defied through a soud theoretical framework, ad the EM algorithms were developed based o a theoretical foudatio for maximum likehood estimatio (MLE). Whe applied to traiig treestructured etworks with the localized modules [5], the EM algorithm reduced the traiig time sigificatly. Motivated by the idea of movig targets ad the EM algorithms, we have developed i our previous work two fast traiig algorithms for feedforward [7] ad recurret etworks [8]. These algorithms have demostrated that EM algorithms ca also be applied effectively to layered ad recurret etworks to speed up traiig. However, sice the two algorithms have bee developed i separate cotexts, their iter-relatioships have ot bee carefully ivestigated. Mauscript received July 23, 1997; revised Jauary 5, This work was supported by the atioal Sciece Foudatio uder Grat ECS ad CAREER Grat IRI The associate editor coordiatig the review of this paper ad approvig it for publicatio was Prof. Yu-He Hu. The authors are with the Departmet of Electrical, Computer, ad Systems Egieerig, Resselaer Polytechic Istitute, Troy, Y USA. Publisher Item Idetifier S X(98) I this correspodece, we will focus o providig a uified framework, based o the EM algorithm, from which both algorithms ca be derived i a similar fashio. We will show that the two algorithms are based o a similar probabilistic formulatio o hidde targets. Such a formulatio will result i a EM algorithm for feedforward etworks that reduces the traiig a oliear etwork ito traiig idividual euros. By the Markov properties of recurret etworks ad the mea-field approximatio [11], a recurret etwork ca be ufolded at each EM step ito a equivalet feedforward etwork. The, the EM algorithm developed for feedforward etworks ca be directly applied to improve the traiig time. II. UIFIED OTATIOS Cosider the discrete time recurret etworks with a fully coected hidde layer as show i Fig. 1. The etwork has oe liear output, d iputs, ad L fully coected recurret (hidde) sigmoid uits. Let ~w (1) ad ~w (3) deote the weight vectors coectig the exteral iputs ad the other recurret uits to the th hidde uit, respectively, for 1 L: Let W (1) ad W (3) be the weight matrices cotaiig all ~w (1) ad ~w (3), respectively. Let ~w (2) be the weight vector coectig the recurret hidde uits to the output. f~x();t( +1)g =1 is a set of traiig samples, where ~x() is a (d +1)21 iput vector at the th time uit, 1 ad t( +1)is the correspodig desired output. Let h ( +1)be the output of the recurret hidde uit. The, h (+1) = g( ~w (1)T ~x()+ ~w (3)T ~ h()), where g(1) is a sigmoid fuctio. The output of the etwork y( +1) correspodig to a iput ~x() ca be expressed as y( +1) = ~w (2)T ~ h( +1): The recurret etwork is reduced to a feedforward etwork whe the recurret coectios do ot exist, i.e., W (3) =0: Therefore, we ca use the above otatios for both feedforward ad recurret etworks. 2 III. THE EM ALGORITHM The method we will use to develop our algorithm is based o the EM algorithm for MLE [2]. Let D I be a set of data observed directly, ad let 2 be a set of parameters characterizig the correspodig distributio of the radom variables. The maximum likelihood problem is to fid a optimal set of parameters 2 opt through maximizig the log likelihood of the data. Sice it ca be very difficult to maximize the origial log likelihood fuctio directly, the EM algorithm has bee itroduced to simplify this process. A set of hidde (radom) variables are itroduced. The data for the hidde variables are called missig data. Together with the missig data, the so-called icomplete data D I form a complete data set D c : The EM algorithm coverts the origial maximum likelihood procedure ito a two-step process: the E-step (expectatio) ad the M-step (maximizatio). I the E-step, the followig coditioal expectatio ca be computed (with respect to hidde variables) of the complete data log likelihood give icomplete data D I ad the previous parameters 2 (p) at the pth step Q(22 (p) )=Efl P (D c 2)D I ; 2 (p) g (1) where 2 are the parameters to be obtaied i the (ext) M-step. The otatio Q(22 (p) ) idicates that the coditioal expectatio : 1 The bias correspods to a iput x0() =1: 2 o delays exist for feedforward euros, i.e., the time +1reduces to X/98$ IEEE

2 IEEE TRASACTIOS O SIGAL PROCESSIG, VOL. 46, O. 8, AUGUST Fig. 1. Recurret etwork. is a fuctio of 2 ad that the expectatio is evaluated usig 2 (p) as parameters i the probability desity fuctio. P (1) represets a probability desity fuctio. E stads for the expectatio operatio. I the M-step, ew parameters 2 (p+1) are foud by maximizig the expected log likelihood Q(22 (p) ): The algorithm alterates betwee the E- ad M-steps ad termiates whe a certai covergece criterio is satisfied. Local covergece of the EM algorithm is guarateed [2]. There is a exteded survey o the EM algorithm i [1]. IV. PROBABILISTIC MODELS A. Choice of Hidde Variables For feedforward ad recurret etworks, we choose the hidde variables ~z to be either the desired hidde targets at the hidde layer for the feedforward etworks or the desired hidde states for recurret euros of the recurret etworks. The, ~z() ad ~z( +1)deote the correspodig missig data for the feedforward ad recurret etworks, respectively, correspodig to the iput ~x() (for 1 ). The complete data set therefore cotais the labeled traiig samples ad the desired hidde target, which is f~x();t();~z()g =1 for feedforward etworks, ad f~x();t( + 1);~z( +1)g =1 for recurret etworks. The icomplete data set is simply the traiig set f~x();t()g=1: For simplicity, we use the otatio f~xg; fyg ad f~zg to represet f~x()g =1; ft()g =1 or ft( +1)g=1; f~z()g =1 or f~z( +1)g=1, respectively, ad we use 2 to represet all the weights i a etwork. We assume that traiig samples (sequeces for recurret etworks) are draw idepedetly. So are the desired hidde targets. Usig the otatios, we have that the coditioal expectatio of the complete-data log likelihood defied i (1) ca the be expressed as Q(22 (p) ) = l P (f~xg2) + s P (f~zgftg; f~xg; 2 (p) ) l P (f~zg; ftgf~xg; 2) df~zg: B. Choice of Probabilistic Models To evaluate the expected log-likelihood Q(22 (p) ), we eed to obtai P (f~zgftg; f~xg; 2) ad P (f~zg; ftgf~xg; 2): I this work, Gaussia distributios are chose to facilitate the aalysis. 1) A Probabilistic Model for Two Layer Feedforward etworks: For two-layer feedforward etworks, we choose P (f~zgf~xg; 2) = B 1 exp(0 1E 1) (2) P (ftgf~zg; 2) = B 2 exp(02 2 E 2 ) (3) where E 1 = 6 =1 k~x() 0 ~ h()k 2, which measures the error betwee the desired ad the actual outputs of hidde uits. E 2 = 6 =1 (t() 0 ~z() T 1 ~w (2) ) 2, which characterizes the error betwee the desired ad the actual outputs of the etwork whe the desired hidde values are used as iputs to the secod layer. B 1 ad B 2 are ormalizatio costats. 1 ad 2 are weightig factors. The, we ca obtai [7] P (ftg; f~zgf~xg; 2) = A tz exp(0 1E 1 0 2E 2) (4) P (f~zgftg; f~xg; 2 (p) )=A =2 z =1 exp( 1 2 (~z() 0 ^~z()) T 6 01 (~z() 0 ^~z())) (5) where A tz = ( 2=)( 1=) L, ad A z =(1= (2) L det(6): 6 is a (L 2 L) covariace matrix whose iverse is 6 01 = 2( 1 I +

3 2272 IEEE TRASACTIOS O SIGAL PROCESSIG, VOL. 46, O. 8, AUGUST ~w (2) ( ~w (2) ) T ): The superscript p idicates the correspodig value at the pth step. I is a (L 2 L) idetity matrix. ^~z() is the expectatio of hidde targets ~z() give the th traiig pairs ad the previous parameter set 2 (p) : The th ( L) elemet of ^~z() ca be computed as [7] ^z () =h (p) 2w (2) () k ~w (2) k2 e() (6) where h (p) () represets the th elemet of ~ h(): e() is the traiig error of the th traiig sample satisfyig e() =t() 0 y (p) (): 2) A Probabilistic Model for Recurret etworks: For recurret etworks, we ote that the output of the recurret etwork ad the hidde states at the curret time step deped oly o the hidde states at the previous time step. This results i the Markov property that leads to the coditioal probability desity fuctios [8] P (f~zgftg; f~xg; 2) = P (~z( +1)~z();t( +1);~x(); 2) (7) =1 P (ftg; f~zgf~xg; 2) = P (t( +1);~z( +1)~z();~x(); 2): (8) =1 Similar Gaussio models to those used for feedforward etworks are chose as the coditioal distributios P (~z( +1)~z();~x(); 2) = B1 exp(01e1( + 1)) (9) P (t( +1)~z( +1); 2) = B2 exp(02e2( + 1)) (10) where B1;B2;1; ad 2 are the same as for feedforward etworks. E1( +1) = k~z( +1)0 ~ h 0 ( +1)k 2, where the th elemet of ~ h 0 (+1) for 1 L is h 0 (+1) = g(~x() T 1 ~w (1) +~z() T 1 ~w (3) ): Therefore, E1( +1)measures the error betwee the desired hidde states ~z( +1)ad ~ h 0 ( +1): Likewise, E2( +1)=(t( +1)0 ~z( +1) T 1 ~w (2) ) 2, which characterizes the error betwee the desired ad the actual output of the etwork whe the desired hidde states are used as iputs to the output layer. These Gaussia assumptios lead to P (t( +1);~z( +1)~z();~x(); 2) = A tz exp(01e1( +1)02E2( + 1)) (11) P (~z( +1)t( +1);~z();~x(); 2) = A z exp(0 1 2 (~z( +1) 0 ^~z( + 1)) T 01 (~z( +1)0 ^~z( + 1))) (12) where A tz ;A z ; 6; ad I are the same as for feedforward etworks. ^~z( +1)is the coditioal expectatio of ~z( +1)coditioed o t(+1);~z() ad ~x(): The th elemet of ^~z(+1) ca be expressed as ^z ( +1)=h 0 ( +1)+ 2w (2) 1 + 2k ~w (2) k2 e( +1) (13) with e( +1)=t( +1)0 ~ h 0 ( +1) T 1 ~w (2) : The, we ca obtai the two distributios P (f~zgftg; f~xg; 2) ad P (ftg; f~zgf~xg; 2) similar to those for feedforward etworks. V. APPLYIG THE EM ALGORITHM TO FEEDFORWARD AD RECURRET ETWORKS A. The E-Step for Feedforward etworks For feedforward etworks, the expected log likelihood ca be evaluated by usig the established probability models [7] as Q(22 (p) )=Efl P (ftg; f~zg; f~xg2)ftg; f~xg; 2 (p) g = E c 0 E p 0 E h 0 E o (14) wheree c is a costat, ad E p = 2 ( ~w (2) ) T ~w (2) :E h ad E o ca be expressed as E h = 1 E o = 2 (^z () 0 g(w T ~x())) 2 (15) (( ~w (2) ) T ^~z() 0 t()) 2 : (16) Therefore, the evaluatio of the expectatio of the log likelihood i the E-step is reduced to fidig the expected hidde targets through (6). B. The E-Step ad the Mea-Field Approximatio for Recurret etworks To evaluate Q(22 p ) for recurret etworks, we also eed to compute the expectatio of the hidde targets ~z(): Sice ~z() ow depeds o ~z( 0 1), due to the oliearity of sigmoidal fuctios, computig the expectatio directly is impossible. Therefore, the selfcosistet mea-field approximatio [11] is used to approximate the expectatio. 3 This approximatio replaces the coditio o ~z() by its expected value P (~z( +1)t( +1);~z();~x(); 2 p ) P (~z( +1)t( +1); ~z();~x(); ~ 2 p ) (17) where the th elemet ~z () of ~z() ~ ca be computed recursively through ~h ( +1)=g(~x() T 1 ~w (1) + ~z() ~ T 1 ~w (3) ) (18) ~z ( +1) h ~ 2w (2) ( +1) k ~w (2) k2 ~e( +1) (19) for every 1 L ad 1 : ~e( +1)=t( +1)0 ~ h( + 1) T 1 ~w (2) 1 ~ h( +1)is a L 2 1 vector whose th elemet is defied by (18). It is easy to observe that (19) is strikigly similar to (6). If ~w (3) s equal to zero so that the recurret etworks reduce to the feedforward etworks, ~z() ~ will equal to ^~z() for the feedforward etworks. The mea-field approximatio essetially ufolds a recurret etwork ito a two-layer feedforward etwork show i Fig. 2. The weights at the first layer of the ufolded etwork cosist of W (1) ad W (3) : The total iputs to the two-layer etwork cosist of the origial iput at time ad the estimated mea value of the previous hidde targets. The, the rest of the results will follow those for two layer feedforward etworks except that E h = 1 E o = 2 (~z () 0 g(~x() T 1 ~w (1) + ~ ~z() T 1 ~w (3) )) 2 (20) ( ~w (2) 1 ~ ~z() 0 t()) 2 : (21) 3 Please see [8] for more refereces.

4 IEEE TRASACTIOS O SIGAL PROCESSIG, VOL. 46, O. 8, AUGUST Fig. 2. Ufolded two-layer feedforward etwork. C. The M-Step ew weights are obtaied through maximizig the expected log likelihood Q(22 (p) ) i the M-step, which ivolves solvig two separated quadratic miimizatio problems ( ~w (1)(p+1) ;~w (3)(p+1) ) =arg mi ~w ;~w (~z ( +1) 0 g(~x() T 1 ~w (1) + ~ ~z() T 1 ~w (3) )) 2 (22) where ~w (1) ad ~w (3) stad for the ew weight vectors coectig the th uit to the iputs ad the other recurret hidde uits for 1 L: Solvig this maximizatio problem is equivalet to traiig idividual (orecurret) hidde uits i the ufolded etwork i parallel. I additio, we have ~w (2) = arg mi ~w ( ~w (2) ( +1)0 t( + 1)) 2 + E p (23) where ~w (2) are the ew weights coectig hidde uits with the output uit, ad E p = ( 2 k ~w (2) k 2 =2 ( 1 + 2k ~w (2) k 2 )) + ( 2 (k~w (2) k 2 k ~w (2) k 2 0 ( ~w (2) ) T ( ~w (2) ) ( ~w (2) ) T ~w (2) )=2 1 ( k ~w (2) k 2 )): This step is equivalet to traiig the output euro i the ufolded two layer feedforward etwork. Whe w (3) s are set to zero, we have the M-step for feedforward etworks. Therefore, the maor computatioal cost of the M-step for both the feedforward ad recurret etworks is i traiig idividual euros. This ca be doe rapidly through liear weighted regressios [7]. Fig. 3. Flowchart of the algorithm. D. Summary of the Algorithm I this work, we have established a uified framework to derive fast traiig algorithms for both feedforward ad recurret etworks. The key steps ca be summarized as follows. Probabilistic models are chose for hidde states for both two-layer feedforward etworks ad the recurret etworks. Based o these models, the idea of movig targets have bee formulated through the EM algorithm. For the feedforward etworks, the expectatio of hidde targets are computed at each E-step. I the M-step, they the serve as the targets at the hidde layer to trai the first layer weights. They are also used as the iputs to the output uit to trai the secod-layer weights. Therefore, traiig two-layer feedforward etworks is decomposed to traiig idividual euros. The mea-field approximatio effectively ufolds a recurret etwork ito a equivalet feedforward etwork at each EM step. Such a feedforward etwork uses the expected hidde states at the previous step as a part of the iputs, ad the expected hidde states at the curret step as the targets at the hidde layer. The, the maximizatio method developed for the feedforward etwork ca be applied to achieve fast traiig for each equivalet feedforward etwork. The maor steps of the uified EM algorithms are summarized i a flowchart give i Fig. 3. The traiig time has bee reduced 5 15 times by both algorithms o various bech-mark problems [7], [8]. Further ivestigatios are eeded o how widely our simple models for desired hidde states is applicable for real problems. ACKOWLEDGMET The authors would like to thak helpful commets from aoymous reviewers.

5 2274 IEEE TRASACTIOS O SIGAL PROCESSIG, VOL. 46, O. 8, AUGUST 1998 REFERECES [1] S. Amari, Iformatio geometry of EM algorithms for eural etworks, eural etworks, vol. 8, o. 9, pp , [2] A. P. Dempster,. M. Laird, ad D. B. Rubi, Maximum likelihood from icomplete data via EM algorithm, J. R. Statist. Soc., vol. B39, pp. 1 33, [3] T. Grossma, Learig, capacity ad implemetatio of eural etwork models, Ph.D. dissertatio, Weizma Ist. Sci., Rehovot, Israel, [4] S. Hayki ad L. Li, oliear adaptive predictio of ostatioary sigals, IEEE Tras. Sigal Processig, vol. 43, pp , Feb [5] M. Jorda ad R. A. Jacobs, Hierarchical mixture of experts ad the EM algorithm, eural Comput., vol. 6, pp , [6] A. Krough, G. I. Thorberaso, ad J. A. Hertz, A cost fuctio for iteral represetatios, i D. Touretzky, Ed., Advaces i eural Iformatio Processig Systems. Sa Mateo, CA: Morga Kaufma, 1990, pp [7] S. Ma, C. Ji, ad J. Farmer, A efficiet EM-based traiig algorithm for feedforward eural etworks, eural etworks, vol. 10, o. 2, pp , [8] S. Ma ad C. Ji, Fast traiig of recurret etworks based o the EM algorithm, submitted for publicatio. [9] R. Rohwer, The movig target traiig algorithm, Adv. eural Iform. Process. Syst., vol. 2, pp , [10] D. Saad ad E. Marom, Learig by choice of iteral represetatios: A eergy miimizatio approach, Complex Syst., vol. 4, pp , [11] J. Zhag, The mea-field theory i EM procedures for Markov radom fields, IEEE Tras. Sigal Processig, vol. 40, pp , Oct