SCALED CONJUGATE GRADIENT TYPE METHOD WITH IT`S CONVERGENCE FOR BACK PROPAGATION NEURAL NETWORK

Transcription

1 International Journal of Information echnoloy an Business Manaement 0-05 JIBM & ARF. All rihts reserve SCALED CONJUGAE GRADIEN YPE MEHOD WIH I`S CONVERGENCE FOR BACK PROPAGAION NEURAL NEWORK, Collee of computer sciences an Mathematics, Mosul University, Mosul, Iraq Mosul University, Iraq, Abstract Conjuate raient alorithms were wiely use in optimization, especially for lare scale optimization problems, because it was not require the storae for any matrices. he oal of the trainin is search an optimal set of a connection weihts in the manner that the error of net wor out put can be minimize. In this paper, we erive a propose formula of the conjuacy coefficient base on conjuacy conition, then we prove the sufficient escent an lobal converence properties for this formula. Comparative results for our propose alorithm an stanar bacpropaation (BP) are presente for two test problems an the results was encourae. Key wors : conjuate raient methos, fee forwar neural networ, Bac propaation(bp) networ, pure conjuacy conition..inroducion: he fee forwar neural networs with bac propaation (BP) trainin proceure have been use in various fiels of scientific researches an enineerin applications. he BP alorithm attempts to minimize the least square error of objective function, efine by the ifferences between the actual networ outputs an esire outputs [6]. he bac propaation trainin alorithm is a supervise learnin metho for multi layer fee forwar neural networ[6]. It is essentially a raient Havin trappe into local minima, bac propaation may lea to failure in finin a lobal optimal solution. Secon, the converence rate of bac propaation is still too slow even if learnin can be achieve. he bac propaation alorithm loos for the escen local optimization technique which involves bacwar error correction of the networ weihts. Despite the eneral success of Bac propaation in learnin the neural networs, Several neural networs, several major eficiencies are still neee to be solve. First, the bac propaation alorithm will ate trappe in local minima specially for non leaner separable problems[] such as the XOR problems [6]. minimum of the error function in weiht space usin the metho of raient escen. the combination of weiht which minimizes the error function is consiere to be a solution of the learnin problem. Since this metho

2 International Journal of Information echnoloy an Business Manaement 0-05 JIBM & ARF. All rihts reserve requires combination of the raient of the error function at each iteration step. he patch trainin of FFNN is consistent with theory of unconstraine optimization an can be viewe as the minimization of the error function E efine by..() Where is the sequare ifference error between the actual output value at the j- th out put layer neuron for pattern P an the taret output value.a traitional way to solve this problem is by an iterative raient base trainin alorithm which enerates a sequence of weihts {W} startin from an initial point use the recurrence [] () Where K is the current iteration usually calle epoch, is the learnin rate an is a escent search irection.i.e. since the appearance of bac propaation[6]..conjugae GRADIEN MEHOD: Conjuate Graient (CG) methos are willy use for unconstraine optimization especially when the imension is lare. We are concerne with the followin unconstraine minimization problem: Minimize f(x) (3) where λ >0 is a step size an is a search irection. Search irections are usually efine by: for for where enotes f(x ) an β is a scalar. (5) Where f:r n R is smooth an its raient (x)=f(x) is available. here are several ins of numerical methos for solvin eq.(3), which inclue the Steepest Descent (SD) metho, the Newton metho CG-metho an Quasi-Newton (QN) methos. he CG methos are our choice for solvin the larescale problems, because they o not nee any matrix storae. CG-methos, however, are iterative methos of the form: x + =x +λ (4) We can euce a formula for the scalar β : HS FR y y (6) (7) 3

3 International Journal of Information echnoloy an Business Manaement 0-05 JIBM & ARF. All rihts reserve PR y (8) AB 3 y y (4) DX (9) Perry v ) ( y y (5) DY LS AB y (0) y AB y () () y (3) his efinition of β ; in Eq. (6) ue to [8] ; β in Eq. (7) ue to [7]; β in Eq. (8) ue to[4] ; β in Eq. (9) ue to [5] ;β in Eq. (0) ue to [4]; β in Eq. () ue to [0] ;β 's in Eqs. (), (3), (4) ue to [] ; β in Eq. (5) ue to [3]. o establish the converence results of nonlinear CG-methos mentione above, it is usually require that the step λ efine in eq.(4) shoul satisfy the followin stron Wolfe conitions: f ( x ) f ( x ) ( x ) (6) (7) 3.PROPOSED CONJUGACY COEFFICIEN : Dai an Liao in 00[3] Propose a new formula that extene of Hestenes an Steifel metho as: DL t (8) Where, t is a positive scalar. In this paper we propose a new formula that moifie Al-Bayati an Al- 4

4 International Journal of Information echnoloy an Business Manaement 0-05 JIBM & ARF. All rihts reserve Assay() conjuate raient metho as:.(9).() ( ) Where s an t 0 are constant, for an exact line search is orthoonal to s, hence is reuce to AB metho. But if the line search is inexact then we can compute t by multiplyin equation (5) with y, we obtain the followin formula for computin t: now substitute the value of t in ()in equation (9) we et: ( ) ( ) (0) Now,if the irection is inexact(ils) then ( ) ( ) ( ) ( ) * ( ) +..() ( ) 5

5 International Journal of Information echnoloy an Business Manaement 0-05 JIBM & ARF. All rihts reserve an we use the last in equation () to prove the converence analysis of our alorithms. 4. CONVERGENCE ANALYSIS: In orer to establish the lobal converence analysis, we mae the followin assumptions for the objective function f. ASSUMPION () i. he level set { x f ( x) f ( x )} is boune, namely, there exists a constant B >0 such that x B for all x ii. In some neihborhoo N of, f is continuously ifferentiable, an its raient is lobally Lipschitz continuous, namely, there exists a constant L>0 such that ( x) ( y) L x y for all x,yn [8]. Proof: By inuction for = we have then 0, then we assume that 0 It follows from stron wolfe conition (6) an (7) that:. HEOREM () Suppose that is iven by (5) an ** which is efine in () then, the followin result is satisfies : c iviin both sie by inequality: an invert the 6

6 International Journal of Information echnoloy an Business Manaement 0-05 JIBM & ARF. All rihts reserve Now also it follows from stron wolfe conition (6) an 7) that y ( ) y ( ) ( ) ( ) an if we assume then we complete the proof 5. GLOBAL CONVERGENCE HEOREM: Uner Assumption ii, we ive a useful lemma which was essentially prove [7] LEMMA() : Suppose that x is a startin point for which Assumption () is satisfie. Consier any metho of the form (), where is a escent irection an satisfies Wolfe conitions (7) an (8) then we have : PROOF : Suppose that the conclusion oes not hol, that is to say their exist appositive constant such that for all. Since which is can be written as an since: HEOREM (3) : Suppose that x is a startin point for which Assumption () hols. Let { x, =,,...} be enerate by our metho. hen the alorithm either terminates at a stationary point or converes in the sense that lim inf 0 7

7 International Journal of Information echnoloy an Business Manaement 0-05 JIBM & ARF. All rihts reserve then such that b is a constant b = b an with this contraiction we complete the prove that is i i b 6. NUMERICAL EXPERIMENS: Now we present a numerical experiments whose objective function is compare with AB alorithms on the same set of unconstraine optimization test problem. For each test function (Anrei, 008). All alorithms implemente with the same line search an with the same parameters. he comparison is base on number of iteration (NOI), an number of function evaluation (NOF). Our alorithms has convere as soon as ². 0 6 able () shows the Comparison of alorithms with respect to NOI an NOF for n=000,n=0000,n=00000 respectively. able () Comparison between stanar AB metho an moifie AB metho with respect to (NOI an NOF) for n=000 est Functions stanar AB metho moifie AB metho NOI NOF Min NOI NOF Min Cubic E E-04 Sallow Function E E-4 8

8 International Journal of Information echnoloy an Business Manaement 0-05 JIBM & ARF. All rihts reserve Rosen E E-4 Beale E E-93 Noniaonal E E-6 Sum E E-08 Strait E E-07 Reciep E E-5 Woo Function E E-4 Panalty Function E E-03 otal able () Comparison between stanar AB metho an moifie AB metho with respect to (NOI an NOF) for n=0000 est Functions Stanar AB metho Moifie AB metho NOI NOF Min NOI NOF Min Cubic E E-3 Shallow Function E E-3 Rosen E E-3 Beale E E-6 Noniaonal E E-7 Sum 9 5.E E-09 Strait E E-6 Reciep E E-4 Woo Function E E-3 9

9 International Journal of Information echnoloy an Business Manaement 0-05 JIBM & ARF. All rihts reserve Panalty Function E E-0 otal able (3) Comparison between stanar AB metho an moifie AB metho with respect to (NOI an NOF) for n=00000 est Functions Stanar AB metho Moifie AB metho NOI NOF min NOI NOF Min Cubic E E-0 Sallow Function E E-5 Rosen E-3 * * * Beale E E-5 Noniaonal E E-00 Sum E E-08 Strait E E-5 Reciep E E-3 Woo Function E E-3 Panalty Function otal COCLUSION: he architecture of the FFNN is - 5- with simoi function in hien layer an linear function in output layer use to approximate y=sin(x)*cos(3x) in the interval [- pi,pi]. For the test problems, a table summarizin the performance of the 30

10 International Journal of Information echnoloy an Business Manaement 0-05 JIBM & ARF. All rihts reserve alorithms for simulations that reache solution is presente. Where the stanar parameters are the oal of error (GE), the minimum number of epochs (MIN/EP), the maximum number of epochs (MAX/EP), the averae value of epochs (AV/EP), the averae of total time (AV/M) bac-propaation (SBP) an the Propose Alorithm (P-CG-BPNN). he reporte an successful performance (SUCC/PERF). he succeee simulations out of (00) trials within the error function evaluations limit. able( 4): Results of Simulations by (Propose alorithm) when (GE=0.00) Alorithm GE AV/M MAX/EP MIN/EP AV/EP SUC/PERF SBP % Stanar AB % metho Moifie AB metho % 8. REFERENCES: AL - Bayati, A.Y. an AL-Assay, N.H. (986). Conjuate raient metho echnical Research, No (), school of computer stuies, Lees University. -Anrei, N., 008. An unconstraine optimization test Function Collection. Avance Moelin an Optimization. Romania, 0 (): Dai,Y.an Liao,L.(00),New conjuacy conition an relate non-liner CG metho.appl.math optim.,(43).sprin verla, New Yor 4-Dai, Y an Y, Yuan., 999. A Nonlinear conjuate raient metho with a stron lobal converence property. SIAM J. Optim, 0 () : 5-Dixon, L. G. W.,(975),"Conjuate Graient Alorithms Quaratic ermination Without Linear Searches", Journal of Inst. Of Mathematics An its Applications,5. 6-E.K. Blum, Approximation of Boolean function by simoial networs:part I:XOR an other twovariable function. Neural computation, 989.(4):p Fletcher, R. an C.M., Reeves.,964. Function minimization by conjuate raients. he Computer Journal, 7:

11 International Journal of Information echnoloy an Business Manaement 0-05 JIBM & ARF. All rihts reserve 8-Hestenes M. R. an E, Stiefel., 95. Methos of conjuate raientsfor solvin linear systems. Journal of Research of the National Bureau of Stanars, 49: Liu, D. an Story, C. (99), Efficient eneralize CG alorithms., part l : heory,. Journal of Optimization heory an Applications 0-Livieris, I.E. an Pintelas, P.,(009), Performance evaluation of escent CG methos for neural networ trainin, In Proceeins of 9th Hellenic European Research on Computer Mathematics an its Applications Conference (HERCMA'09). methos, In Inteer an Nonlinear Prorammin, J. Abaie(E), - Marco Gori an Alberto esi,on the problem of local minima in bac-propaation IEEE ransaction on pattern Analysis an Machine Intellience,99.4():p Perry, A. (978), A moifie conjuate raient alorithms, Operations Research, Vol. (6). 4-Pola E. an G, Ribiere., 969. Note sur la converence em ethoes e irections conjuu ees. Revue Francais Informatique et e Recherche Operationnelle, 3(6): Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (986). Learnin representations by bac-propaatin errors. Nature, 33, Rumelhart D.E., Hinton G.E., an Williams R.J..( 986), Learnin internal representations by error propaation. In D. Rumelhart an J. Mc- Clellan, eitors, Parallel Distribute Processin: Explorations in the Microstructure of Conition,paes 38 36, Cambrie, Massachusetts.. 7-Zoutenij, G. (970), Nonlinear prorammin, computational methos, In Inteer an Nonlinear Prorammin, J. Abaie(E), North- Holan. 8-Glibert, J.C. an Noceal,J., 99, Global Converence Properties of Conjuate raient methos for optimization, SIAM J. optimization, vol., no.,pp