Monte Carlo comparison of back-propagation, conjugate-gradient, and finite-difference training algorithms for multilayer perceptrons

Transcription

1 Rocheste Insttute of Technology RIT Schola Woks Theses Thess/Dssetaton Collectons 20 Monte Calo compason of back-popagaton, conugate-gadent, and fnte-dffeence tanng algothms fo multlaye peceptons Stephen Wehy Follow ths and addtonal woks at: Recommended Ctaton Wehy, Stephen, "Monte Calo compason of back-popagaton, conugate-gadent, and fnte-dffeence tanng algothms fo multlaye peceptons" (20). Thess. Rocheste Insttute of Technology. Accessed fom Ths Dssetaton s bought to you fo fee and open access by the Thess/Dssetaton Collectons at RIT Schola Woks. It has been accepted fo ncluson n Theses by an authozed admnstato of RIT Schola Woks. Fo moe nfomaton, please contact tscholawoks@t.edu.

2 Monte Calo Compason of Back-Popagaton, Conugate-Gadent, and Fnte-Dffeence Tanng Algothms fo Multlaye Peceptons By Stephen J. Wehy A Dssetaton fulfllment of the equements fo the Maste s Degee n Mathematcs Thess Advse: D. Aleando Engel Rocheste Insttute of Technology Depatment of Mathematcs and Statstcs Spng 20 Quate

3 MASTER S DISSERTATION OF STEPHEN JOSEPH WEHRY Apl, 7, 20 APPROVED: Dssetaton Commttee: Mao Pofesso D. Aleando Engel D. James Maengo D. Mchael Radn ROCHESTER INSTITUTE OF TECHNOLOGY 2

4 Table of Contents ABSTRACT...4 INTRODUCTION... 5 TEST PROBLEMS AND ENCODING SCHEMES... 9 PRECISION TEST - PATTERN RECOGNITION... 9 GENERALIZATION TEST - BIT COUNTING... 0 HEURISTICS USED... 0 EXAMPLE MULTILAYER PERCEPTRON XOR FUNCTION... 2 TRAINING ALGORITHMS TESTED... 6 THE BACK-PROPAGATION ALGORITHM...6 THE CONJUGATE-GRADIENT ALGORITHM... 2 THE FINITE-DIFFERENCE ALGORITHM...26 MONTE CARLO SIMULATION TEST METHODOLOGY SIMULATION RESULTS A Note About ρ Values Patten Recognton Bt Countng... 3 CONCLUSIONS AND FUTURE WORK REFERENCES APPENDICES APPENDIX A: PATTERN RECOGNITION CALCULATOR DIGITS APPENDIX B: BIT COUNTING TRAINING EXAMPLE SET... 4 APPENDIX C: DERIVATION RESULTING FROM HIGHER-ORDER Ρ VALUES APPENDIX D: RAW MONTE CARLO SIMULATION DATA D: Back-Pop Patten Recognton D2: C-G Patten Recognton D3: Fnte-Dffeence Patten Recognton D4: Back-Pop Bt Countng D5: C-G Bt Countng... 7 D6: Fnte-Dffeence Bt Countng

5 Abstact Monte Calo Smulaton s used to compae the pefomance of the Back- Popagaton, Conugate-Gadent, and Fnte-Dffeence algothms when tanng smple Multlaye Pecepton netwoks to solve patten ecognton and bt countng poblems. Twelve ndvdual smulatons wll be un fo each tanng algothm-test poblem combnaton, esultng n an oveall total of 72 smulatons. The andom elements n each Monte Calo smulaton ae to be the ndvdual synaptc weghts between layes, whch wll be unfomly dstbuted. Two othe factos, the sze of the hdden laye and the exponent of the eo functon, wll also be tested wthn the smulaton plan outlned above. 4

6 Intoducton Multlaye peceptons (heeafte denoted as MLPs) ae a well-studed type of neual netwok, datng back to the poneeng wok of Rosenblatt n Ths pape descbes a compason of thee tanng methods, back-popagaton, conugategadent, and fnte-dffeence algothms, used to tan smple MLPs to solve each of two dffeent poblems: patten ecognton and bt countng. Each of these tanng methods follow the supevsed leanng paadgm 2. n whch tanng examples ae pesented fom whch an eo sgnal s geneated. Ths eo sgnal s then used to tan the netwok to lean to solve the poblem at hand n dffeent (but elated) ways. All of the MLPs used n ths pape have the same geneal stuctue depcted n Fgue below. Fgue : 3-Laye Fully-Connected MLP Topology Each MLP n ths study conssts of thee layes, one nput laye, one hdden laye, and one output laye. The numbe of neuons n the nput and output layes ae detemned by the nput and output encodng schemes used to defne the poblem to be 5

7 solved; the numbe of hdden neuons was vaed as pat of ths study. All of the netwoks used n ths pape have the followng popetes n addton to those aleady mentoned: They ae fully-connected, meanng that each neuon n both the nput and hdden layes s connected to evey neuon n the subsequent laye 3.. They ae feed-fowad, meanng that dung computaton nfomaton flows n one decton fom the nput, though the hdden, and endng n the output laye. Ths means that thee s no use of onlne feedback 4.. An addtonal constant on the topology of the netwoks n ths study s that thee s no skppng of layes. All non-nput neuons utlze a nonlnea actvaton functon 5.. Input neuons ae fo nput only, hdden neuons ae fo computaton only, and output neuons ae fo computaton and output. In ths context computaton means the pocess of summaton of the nput sgnals to an ndvdual neuon then the applcaton of the actvaton functon. In the followng dscusson, as well as the est of ths pape, the notaton and temnology used wll closely follow that n Haykn (999). The computatonal stuctue of the th nput node s shown schematcally n Fgue 2. 6

8 Fgue 2: Computatonal Stuctue of Input Node The functon of an nput node s to eceve ts sngle nput sgnal and pass t on to each neuon n the hdden laye afte applyng the appopate synaptc weght 6. fo that connecton. The computatonal stuctue of the th hdden neuon s shown on the next page n Fgue 3. 7

9 Fgue 3: Computatonal Stuctue of Hdden Node Fgue 3 shows that a hdden node fst sums up ts nputs fom the pevous laye, adds ts bas, then apples the actvaton functon to ths value. The esultng value s then passed on to become pat of the nput of each neuon n the output laye, agan, afte the appopate synaptc weght has been appled 7.. The computatonal stuctue of the kth output neuon s shown below n Fgue 4. Fgue 4: Computatonal Stuctue of Output Neuon k 8

10 An output neuon acts ust lke a hdden neuon, except that no weght s appled to the output of the actvaton functon n these nodes 8.. Fnally, consde a few mscellaneous notes on the oveall popetes of the netwok. Fst, the functonal fom and paametes of φ(x) wee heustcally chosen fo two man easons. A sgmod functon, such as the hypebolc tangent functon used n ths study, has the necessay popetes of nonlneaty, monotoncty, antsymmetcty, and ange-boundedness. The paametes wee chosen to maxmze the cuvatue of ths functon at x = y = +, whch helps to pevent neuons fom beng dven to satuaton, a seous obstacle to the convegence of the tanng algothm. Ths leads to the second geneal popety of the netwoks used, whch s that both the nput and output spaces of each poblem ae encoded nto vectos x wth - < x < 9.. Test Poblems and Encodng Schemes Pecson Test - Patten Recognton The patten ecognton poblem used n ths study s fo the MLP to coectly dentfy ten smple fve-by-thee pxel calculato dgts detaled n Appendx A. To llustate the nput space encodng scheme, consde the ffteen pxel calculato dgt fo zeo, shown below n Fgue 5. Fgue 5: Patten Recognton Input Space Encodng Scheme As seen n Fgue 5, the ffteen pxel dgt s epesented by the coespondng ffteen component nput vecto. The dgt zeo, theefoe, s epesented by the nput vecto [,,,, 0,,, 0,,, 0,,,, ]. The nput vectos fo the emanng dgts ae obtaned n the same manne. The output space encodng scheme fo ths poblem s to smply epesent the numec value of each dgt by ts 4-bt bnay epesentaton. Thus the output coespondng to zeo would be [0, 0, 0, 0], that coespondng to nne would be 9

11 [, 0, 0, ], and so on. Fnally, the netwok wll be pesented wth one example of each of the ten dgts dung each epoch of tanng. Patten Recognton s a so-called pecson test because thee ae a small numbe of possble nputs, meanng that all of them can be pesented to the netwok dung each tanng epoch. Ths means that snce no genealzaton outsde of the tanng space wll be equed, the netwok s only task s to lean these examples to a hgh degee of accuacy. Genealzaton Test - Bt Countng The bt countng poblem used n ths study s to have the netwok count the numbe of s n an abtay bnay stng seven bts long. Natually the nput space fo ths poblem wll consst of vectos wth seven components. The output space of the netwok wll be a bnay epesentaton of the numec value of the answe. Snce thee ae a maxmum of seven s n a seven bt stng, the output space wll be a vecto wth thee components. In ths study thee ae 2 7 = 28 possble nput stngs, but only a subset of 68 them wll be used as tanng examples. Snce the netwok wll be tested aganst each of the 28 possble nputs, ths test pesents the netwok an addtonal challenge, that of genealzaton fom the known to the unknown. Heustcs Used Seveal heustcs fo mpovng the pefomance of MLPs have gown out of the cumulatve wok that has been done ove the yeas snce the ntoducton. Many have been ncopoated nto and used n ths study; they ae lsted below. Pe-pocessng of nput and output spaces Remove the mean of each to ensue that the components of the nput and output vectos ae not all postve o all negatve 0.. Eucldean Dstance Output calculaton both patten ecognton and bt countng ae poblems n whch vald outputs ae estcted to a dscete set of possble values. In ths study the aw output of the MLP, the vecto whose components ae the values of the output neuons, s compaed to each of the possble output vectos to detemne whch vald output s closest n a Eucldean- 0

12 dstance sense. That closest vald output vecto s what s used as the epoted output of the netwok. Synaptc weght ntalzaton - Calculate ntal values fo synaptc weghts accodng to a unfom dstbuton functon; use laye sze to detemne vaance of same.. In the Back-Pop Algothm, each synaptc weght has ts own ndvdually adustable leanng ate paamete 2..

13 Example Multlaye Pecepton XOR Functon As an llustatve example of usng an MLP s to consde the MLP dagammed below n Fgue 6, desgned to solve the Exclusve O, o XOR poblem. Fgue 6: XOR MLP Schematc The XOR functon, of couse, gves values accodng to the followng tuth table. Input 0 Input Output Value Table : XOR Tuth Table It eques two nput values, each ethe 0 o, and calculates a sngle output value, also ethe 0 o. Ths detemned the numbe of nput and output neuons n the desgn of the MLP n Fgue 7. Fo ths example a hdden laye of two neuons was also chosen. One possble example of an MLP taned to solve the XOR poblem, s gven n the followng table of paamete values. 2

14 Paamete Value Afte Successful Tanng W hdden,0, W hdden,0, W hdden,, W hdden,, W hdden,2, W hdden,2, W output,0, W output,, W output,2, Table 2: Example MLP Taned Paamete Values Followng the heustcs descbed n the pevous secton, the fst step n usng ths example MLP s to pe-pocess the nput values by subtactng out the mean of the nput space. The fou possble nput vectos n the XOR poblem ae (0,0), (,0), (0,), and (,). The mean value of the fst component of the possble nput space s (0++0+) / 4 = 0.5, and that of the second component s also (0+0++) / 4 = 0.5. Theefoe befoe we pesent the netwok wth an nput, we fst subtact the vecto (0.5, 0.5), esultng n the followng table of possble pe-pocessed nput values. Ognal Input Vecto Pe-Pocessed Input Vecto (Zeo-Mean) (0, 0) (-0.5, -0.5) (, 0) (0.5, -0.5) (0, ) (-0.5, 0.5) (, ) (0.5, 0.5) Table 3: Example MLP Zeo-Mean Input Values Fo ths example, the netwok wll be tested wth the nput (, ); see Appendx C fo example calculatons usng the othe thee possble XOR nputs. Fst, we use Table 3 to detemne the zeo-mean nput vecto that coesponds to (, ), whch s (0.5, 0.5). Ths nput vecto s assgned to the nput nodes n the example MLP, as seen n Fgue 7 on the next page. 3

15 Fgue 7: XOR MLP Calculaton: Input Laye Now the values of the hdden laye neuons must be calculated, usng the values of the nput laye along wth the synaptc weghts and bases that ae shown n Fgue 7. h = ϕ )( W ) + ( )( W + b h { } 0 ( 0 hdden,0,0 hdden,,0 ) hdden,0 h 0 = ϕ {(0.5)( ) + (0.5)( ) + ( ) } 2 = ϕ ) =.759 tanh( ( ) ) = ( 3 Smlaly, h { )( W ) + ( )( W + b } = ϕ ( 0 hdden,0, hdden,, ) hdden, h = ϕ {(0.5)( ) + (0.5)( ) } ( ) h = ϕ ) =.759 tanh ( ) 0., ( 3 = whch esults n the netwok state shown n Fgue 8 on the next page. 4

16 Fgue 8: XOR MLP Calculaton: Hdden Laye The next step s to calculate the value of the output neuon: o = ϕ h )( W ) + ( h )( W + { b } 0 ( 0 output,0,0 output,,0 ) output,0 o 0 = ϕ {( )( ) + ( )( ) } 2 = ϕ ) =.759 tanh( ( ) ) = o, 0 ( 3 whch esults n the fnal netwok state shown n Fgue 9. Fgue 9: XOR MLP Calculaton: Output Laye 5

17 The calculated output of the example MLP to the nput (, ) s , but the possble outputs of the actual XOR poblem ae estcted to ethe o 0. Ths s because ths MLP was taned usng zeo-mean output values (anothe of the heustcs lsted n the pevous secton), whch ae tabulated below n Table 4. Ognal Zeo-Mean Output Vecto Output Vecto (0) (-0.5) () (0.5) Table 4: Example MLP Zeo-Mean Output Values So, the possble zeo-mean output values fo the XOR poblem ae -0.5 o 0.5 to whch does the output value of ou example MLP coespond? Ths fnal step n the MLP calculaton nvolves computng the Eucldean dstance of ou output vecto to each of the possble zeo-mean output vectos, and choosng that whch s closest. We calculate ( ) (0.5) = [( ) (0.5)] 2 = [( ) ( 0.5) ] 2 = ( ) ( 0.5) =, so the fnal output of the example MLP to the XOR of (, ) s 0. Tanng Algothms Tested As mentoned n the Intoducton, both tanng methods used n ths study employ the supevsed leanng paadgm, n whch adustments ae made to the netwok s paametes due to eos n the output esponse to tanng examples 3.. Howeve, even small MLPs lke those n ths study have a lage numbe of such paametes n the fom of synaptc weghts between and bases of the neuons. Whch paametes should be adusted when, and by how much, dung tanng? Ths s what s known as the cedt assgnment poblem fo MLPs that tanng algothms ae desgned to solve 4.. The Back-Popagaton Algothm One of the fst computatonally effcent solutons to the cedt assgnment poblem, and stll one of the most poweful and popula neual netwok tanng algothms, s the back-popagaton (back-pop) algothm 5.. Back-popagaton s socalled because t adusts the fee paametes of the netwok by usng the famla Chan 6

18 Rule of Calculus to popagate the eo sgnal of each tanng example back, laye-bylaye, though the netwok. Hebban Leanng adustments ae made to the paametes of each laye n popoton to the contbuton of each ndvdual paamete to the oveall eo sgnal 6.. The back-pop fomulas ae deved as follows. Fst, assume an MLP wth N nput, M hdden, and P output neuons. Let d be the vecto (dmenson P) of desed esponse fo the tanng example pesented to the netwok as nput vecto (dmenson N). Gven the cuent state of the netwok (.e., the values of the weght and bas paametes fo each neuon), the netwok wll calculate an output esponse vecto o (dmenson P), the vecto of the output values of the output neuons. The output values wll dffe fom the desed esponse by an amount e = d o, k =,., P (.) k k k fo each output neuon k. Ths value s the eo sgnal fo the kth neuon n esponse to the cuent tanng example. An enegy-lke total eo n esponse to the cuent tanng example can be computed as E = P P ( dk ok ) = 2 ( e ) k k = k =. (2.) Ths s the eo sgnal that wll be popagated back though the netwok to effect tanng. Usng the Hebban Leanng analogy, we wsh to adust each paamete by an amount popotonal to ts ndvdual contbuton to ths total eo, as n E W = λ, (3.) W whee the value λ s the called the leanng ate paamete. The followng devaton 7. povdes a value fo E W. Consde a non-nput neuon. Let N be the numbe of neuons n the pevous laye (ndexed by the vaable ), and M be the numbe of neuons n the laye (ndexed by the vaable ) contanng neuon. Neuon wll eceve N+ total nput sgnals (see Fgues 3 and 4 above) fo ts calculaton, the outputs of the N neuons n the pevous laye plus the bas sgnal of neuon. Call these nput sgnals to neuon the values y, 7

19 = 0 to N, whee y 0 = s the sgnal whch, when multpled by the bas value b = W 0, gves the bas tem of the calculaton. Let W be the matx of synaptc weghts fom the pevous laye to the laye contanng neuon. The value v = N = 0 ( W )( y ) s called the nduced local feld of neuon. The output sgnal of neuon n esponse to the cuent tanng example s (4.) y = ϕ v ). (5.) ( Now, usng the Chan Rule of Calculus, E W = E e y v, (6.) e y v W whch gves an expesson fo the value we want n tems of values we can calculate. Fst, dffeentate equaton (2.) to get E = e e. (7.) Recall that fo non-nput neuon, e = d y, (8.) fom whch we get, va dffeentaton, e y =. (9.) To get the fouth facto of the ght sde of equaton (6.), we dffeentate equaton (5.) to get y v = ϕ ). (0.) ( v Fnally, by dffeentatng equaton (4.) we get v = W y. (.) Pluggng these fou patal devatves back nto equaton (6.) we get 8

20 E W = [ e ][ ϕ ( v )][ y ], (2.) whch gves the followng expesson fo equaton (3.): W = λ e ][ ϕ ( v )][ y ]. (3.) [ The value [ e ][ ϕ ( v )] s denoted by δ and s called the local gadent of neuon. Note that f neuon s an output node equaton (2.) s eadly evaluated, but due to the pesence of the desed esponse n e, futhe calculaton to defne ths value s necessay n the case whee neuon s a hdden neuon. The local gadent δ s defned as E δ = [ e ][ ϕ ( v )] = (4.) v E y = (5.) y v E = ϕ ( v ), (6.) y whee the Chan Rule s used n equaton (5.). Followng Haykn (999), note that n equaton (2.) the summaton s ove the output nodes k. Dffeentatng equaton (2.) yelds P E ek = e k. (7.) y k = y Agan the chan ule s used to get P E ek vk = e k. (8.) y k = vk y Snce neuon k s an output neuon, ecall fom equaton (8.) that ek = d k yk (8.) = d k ϕ v ). (9.) Theefoe we can dffeentate to get ( k 9

21 e v k k = ϕ v ) Usng equaton (4.) we can wte the nduced local feld of neuon k as v k = M = 0 ( k ([ W ][ y ]) k (20.), (2.) so we get that v k = y W k. (22.) Fnally, pluggng the two patals (20.) and (22.) back nto equaton (8.) yelds k = ([ e ][ ( v )][ W ]) P E = k ϕ k k (23.) y and k = ([δ ][ W ]) P E = k k. (24.) y Pluggng equaton (24.) nto equaton (5.) yelds what s known as the backpopagaton fomula, δ P = ϕ ( v ) ( δ W fo hdden neuon. Ths povdes the algothmc mechansm to calculate the local contbuton of hdden neuon to the oveall eo sgnal fo the cuent tanng example 8.. A pseudocode veson of the complete back-pop algothm used n ths study s shown n Table 5 on the next page. k = k k ) (25.) 20

22 Lne # 3-Laye MLP wth N nput, M hdden, and P output neuons Comments Intalze epoch counte = 0 2 Intalze (M+ by P) output laye weghts matx W output Unfom Dstbuton wth μ = 0, σ 2 = / M 3 Intalze (M+ by P) output laye leanng ate matx λ output Use an ntal value of Intalze (N+ by M) hdden laye weghts matx W hdden Unfom Dstbuton wth μ = 0, σ 2 = / N 5 Intalze (N+ by M) hdden laye leanng ate matx λ hdden Use an ntal value of Whle not done tanng do 7 Fo x = 0 to NUM_EXAMPLES 8 Set nput vecto = tanng example x 9 Set desed esponse vecto d 0 Compute hdden laye vecto h See Fgue 3 above. Compute output laye vecto o See Fgue 4 above. 2 Compute eo sgnal component E x P = 2 Ex 2 ( d o ) = 3 Calculate output laye local gadent vecto δ output δ output, k = ϕ ( voutput, k )( dk ok ); k =... P 4 Calculate hdden laye local gadents vecto δ hdden P δ hdden, = ϕ ( vhdden, ) ( δ output, kwoutput,, k ); =... M k = 5 Adust output laye weghts matx W output Woutput,. k = ( Woutput,. k ) pevous + ( λ, k )( δ output, k )( h ); =... M ; k =... P 6 Adust hdden laye weghts matx W hdden W = W ) + ( λ )( δ )( ); =... N; M hdden,. ( hdden,. pevous, hdden, =... 7 Adust ndvdual leanng ates matces λ hdden, λ output 8 Next x 9 Calculate Aveage Eo Value E avg NE E ( ) E, NE = NUM _ EXAMPLES avg = NE x = 20 If stoppng ctea s not met Then ncement epoch count Else done tanng s tue 2 Loop back to lne 4 22 Recod # epochs, total tanng tme, ntal aveage eo, and fnal aveage eo 23 STOP Table 5: The Back-Popagaton Algothm x The Conugate-Gadent Algothm The method of conugate gadents s an algothm fo exactly mnmzng the quadatc vecto equaton ½(x T Ax) bx + c = 0, when the n-by-n matx A s postvedefnte 9.. Ths algothm adusts the n-by- vecto x to mnmze the equaton along each of a constucted sequence of n A-conugate vectos, the esultng value of x beng the desed mnmzng value. If the matx A s known and postve-defnte the sequence of A-conugate vectos can be dectly calculated and the algothm fnshes n at most n steps. To begn ou dscusson of ths algothm, we must defne the concept of A- conugacy. A non-zeo set of N vectos {s 0, s, s 2,, s N- }s called A-conugate wth espect to the N-by-N postve defnte matx A f T s As = 0, (26.) 2

23 The concept of A-conugacy s a genealzaton of the moe commonly used dea of othogonalty of vectos. Lke a set of mutually othogonal vectos, a set of A-conugate vectos s lnealy ndependent. Ths can be poven by contadcton as follows. Wthout loss of genealty, assume we have the A-conugate set {s }, =0,,N-, and assume that s 0 = N = a s n othe wods, that the set of vectos s lnealy dependent. Left-multply Equaton (27.) by the matx A to get Agan left-multply by T s 0 to get As 0 = N = s (27.) a A. (28.) s T 0 As 0 = N = a s T 0 As, (29.) but because we assumed that the set {s }was A-conugate, the ght sde of Equaton (29.) s zeo, so we see that s T 0 As0 = 0. (30.) Ths s ou contadcton because by defnton s 0 s nonzeo and the matx A s postvedefnte, theefoe the set of A-conugate vectos must be lnealy ndependent 20.. The conugate-gadents algothm can be adapted fo use n a genec nonlnea mnmzaton poblem due to the theoetcal fact that any functon can be Taylo expanded to look lke a second-ode system n the neghbohood of a local mnmum. The tanng of an MLP can be consdeed as such a poblem, whee the aveage squaed eo ove a tanng epoch s used as the nonlnea functon to be mnmzed. In the neghbohood of a local mnmum, the eo suface s descbed by T T E ( w) w Aw b w c. (3.) Avg 2 + Clealy n the geneal neual netwok applcaton the matx A, the vecto b, and the scala c ae unknown, so we develop a method to calculate the sequence of A-conugate vectos wthout the need to explctly calculate any of these values

24 Gven the stuaton n Equaton (3.), the dea behnd conugate-gadents s to geneate a sequence of A-conugate seach vectos s 0, s, s 2,, s N- to adust the paamete vecto w usng the update equaton n n n s w w η + = +, n=0,,n-, (32.) whee the scala η s defned as the value that mnmzes the value of ) ( n n Avg s w E η + when both w n and s n ae held constant. Fst, defne the esdual vecto w n w Avg n n w w E Aw b = = = ) (, (33.) the gadent vecto of the eo suface evaluated at the cuent pont w on the eo suface. We begn the algothm by choosng an abtay ntal value of w 0, and settng ou ntal seach vecto 0 ) ( 0 0 w w Avg o w w E Aw b s = = = =. (34.) We ll use the conugate Gam-Schmdt 22. pocedue to constuct the emanng seach vectos such that = + = 0 k k k s s β. (35.) Take the nne poduct of Equaton (35.) wth the vecto As, 0 < <, to get ) ( ) ( ) ( 0 k k k s A s As s + = = β (36.) [ ] = + = 0 ) ( ) ( ) ( ) ( ) ( ) ( k k k s A s As As s β (37.) [ ] = + = 0 k T k k T T s A s As As s β. (38.) Recall that the set {s } s A-conugate, so As s T =, 0, (39.) so T s A s As β + = 0, (40.) 23

25 and solvng Equaton (40.) fo β gves T T As s As = β. (4.) Howeve, we want to be able to compute β wthout explctly usng o even knowng the matx A, so we must contnue ou calculaton. Fom the method of steepest descent we know that the esdual vectos we have defned ae elated by the ecuence elaton s A α = +. (42.) Takng the nne poduct ) ( ) ( ) ( ) ( ) ( ) ( s A = + α (43.) T T T s A α = + (44.) and eaangng Equaton (44.) yelds + = T T T As α, (45.) and theefoe + = = = othewse As T T T 0,,, α α. (46.) Pluggng Equaton (46.) nto Equaton (4.) gves + > + = = 0,, As s T T α β (47.) The paamete α - s the steepest descent paamete = T T As s s α, (48.) and = T T s As s α. (49.) Pluggng Equaton (49.) nto Equaton (47.) gves 24

26 = T T T T As s s As s β (50.) = T T s β, (5.) whee we dop the subscpt fom β as t s no longe necessay. We then take the nne poduct of the conugate Gam-Schmdt Equaton (35.) wth to get ) ( ) ( ) ( 0 k k k s s + = = β (52.) = + = 0 k T k k T T s s β, (53.) and when =, = + = 0 k T k k T T s s β. (54.) Howeve, we know that k s T k < =, 0, (55.) whch s the case wth each tem n the summaton n Equaton (54.), so T T s =. (56.) At long last, substtuton of Equaton (56.) nto Equaton (5.) gves what s known as the Fletche-Reeves equaton, = T T β. (57.) Ths s the equaton that, when used n Equaton (35.), allows the calculaton of the A- conugate seach vectos {s } usng only the values of the esdual vectos 23.. Pactcal esults have shown that a modfcaton of the Fletche-Reeves fomula s moe effectve n the geneal case. Ths s called the Polak-Rbee fomula 24., and s gven by ( ) = 0, max T T β. (58.) The eason s that the geneated seach vectos {s } ae only pefectly A-conugate when the functon E Avg (w) s a quadatc functon. Fo a hghly nonlnea eo functon o fo 25

27 values of w fa fom a local mnmum, ths tends not to be the case. Fo ths eason the geneated sequence of seach vectos to lose conugacy, whch means they ae no longe lnealy ndependent, and thus the seach can get stuck and fal to convege to a mnmum. The Polak-Rbee fomula avods ths ptfall by peodcally esettng the conugate-gadents seach at the cuent value of w by settng the adustment tem β to zeo, whch means the gadent s once agan used as the seach vecto (ecall that ths s how the conugate-gadents algothm stats n the fst place) 25.. A summay of the conugate-gadents algothm that was used n ths pape s shown below n Table 6. Lne # 3-Laye MLP wth N nput, M hdden, and P output neuons Comments Intalze epoch counte = 0 2 Intalze the weghts vecto w Unfom Dstbuton wth μ = 0, σ 2 = / M 3 Set Intal Leanng Rate η=0.0; Annealng Rate η ate=0.9 4 Set stoppng ctea α= Calculate ntal vectos g, s, and 0 All thee have dmenson [M*(N + P) + M + P] 6 Whle not done tanng do 7 Use Bent s Method to calculate value of η whch 8 Mnmzes E Avg(w+ηs) whee w and s ae held constant 9 Update weght vecto w + w + ηs 0 Use Back-Pop to calculate new gadent vecto g Set pev = Set = -g 2 Use Polack-Rbee fomula to calculate β T ( ) pev β = max 0, 3 Calculate new s = + βs 4 If stoppng ctea s not met Then ncement epoch count Stoppng cteon met when all tanng examples have been leaned wth accuacy. 5 Else done tanng s tue 6 Loop back to Lne 6 7 Recod # epochs, total tanng tme, ntal aveage eo, fnal aveage eo, and pecent coect 8 STOP T pev pev Pecent Coect s calculated afte tanng tme s computed. It s the numbe of coect answes n a test of all 28 possble nput values. Table 6: The Conugate-Gadent Algothm The Fnte-Dffeence Algothm The thd tanng algothm that wll be tested n ths study s the so-called fntedffeence algothm. Ths algothm s dentcal to back-pop except fo the computaton of gadents dung the back-popagaton phase. Instead of usng the local gadent calculatons n equatons (2.), (4.), and (25.), ths algothm wll dectly calculate the cental-dffeence appoxmaton to each gadent wth espect to each weght n the netwok. Thus, fo the fnte-dffeence appoxmaton algothm, E W = E( W + ε ) E( W ε ) 2ε, (59.) 26

28 whee E(W ) s the calculated eo value, aveaged ove a tanng epoch, at that patcula value of the ndvdual weght W. The stategc motvaton fo ths algothm s to utlze a multpocesso compute to calculate each cental-dffeence n paallel. Ths study, howeve, uses an mplementaton coded fo a sngle pocess on a sngle pocessng BUS, so the potental fo poductvty gans though paallel computaton ae mmedately fofet. Howeve, snce we ae nteested n testng only accuacy, genealzaton ablty, and numbe of tanng teatons, ths study wll stll be a vald test of the pefomance of the fnte-dffeence algothm elatve to the othe two. A summay of the fnte-dffeence algothm used n ths study s shown below n Table 7. Lne # 3-Laye MLP wth N nput, M hdden, and P output neuons Comments Intalze epoch counte = 0 2 Intalze (M+ by P) output laye weghts matx W output Unfom Dstbuton wth μ = 0, σ 2 = / M 3 Intalze (M+ by P) output laye leanng ate matx λ output Use an ntal value of Intalze (N+ by M) hdden laye weghts matx W hdden Unfom Dstbuton wth μ = 0, σ 2 = / N 5 Intalze (N+ by M) hdden laye leanng ate matx λ hdden Use an ntal value of Whle not done tanng do 7 Fo x = 0 to NUM_EXAMPLES 8 Set nput vecto = tanng example x 9 Set desed esponse vecto d 0 Compute hdden laye vecto h See Fgue 3 above. Compute output laye vecto o See Fgue 4 above. 2 Compute eo sgnal component E x P = 2 Ex 2 ( d o ) = 3 Calculate output laye cental-dffeence gadents E E( W + ε ) E( W ε ) = 2ε,, ; ε = W 5 Adust output laye weghts matx W output Woutput,. k = ( Woutput,. k ) pevous + ( λ, k )( δ output, k )( h ); =... M ; k =... P 6 Adust hdden laye weghts matx W hdden W = W ) + ( λ )( δ )( ); =... N; M hdden,. ( hdden,. pevous, hdden, =... 7 Adust ndvdual leanng ates matces λ hdden, λ output 8 Next x 9 Calculate Aveage Eo Value E avg NE E ( ) E, NE = NUM _ EXAMPLES avg = NE x = 20 If stoppng ctea s not met Then ncement epoch count Else done tanng s tue 2 Loop back to lne 4 22 Recod # epochs, total tanng tme, ntal aveage eo, and fnal aveage eo 23 STOP Table 7: The Fnte-Dffeence Algothm x 27

29 Monte Calo Smulaton Test Methodology The Monte Calo Smulaton desgned to test the effectveness of the thee tanng algothms s outlned n Table 8, below. Smulaton # (*) ρ Values (**) Hdden Laye Sze (***) Test Poblem Tanng Algothm -2 2, 4, 6 4, 0, 5, 20 Patten Recog. Back-Pop , 4, 6 4, 0, 5, 20 Patten Recog. C-G , 4, 6 4, 0, 5, 20 Patten Recog. Fnte-Dff , 4, 6 4, 6, 8, 0 Bt Countng Back-Pop , 4, 6 4, 6, 8, 0 Bt Countng C-G , 4, 6 4, 6, 8, 0 Bt Countng Fnte-Dff. (*) Each ndvdual smulaton n ths study s composed of 500 tals. A sngle tal n ths study s defned as one complete un though the tanng algothms outlned n Table, Table 2, o Table 3. (**) ρ s the exponent of the enegy-lke eo functon E to be mnmzed dung tanng. See Appendx C. (***) Both the hdden laye sze and ρ value wee vaed n ths study, but held constant dung an ndvdual smulaton. Fo example, a Hdden Laye Sze of 0 means that thee wee 0 neuons n the hdden laye dung that smulaton. Table 8: Monte Calo Smulaton Outlne As shown n Table 8, both the sze of the hdden laye and the exponent ρ of the enegylke eo functon E wee vaed to detemne the best value of each to use when compang the thee tanng algothms fo each poblem. See Appendx C fo the changes to the pecedng tanng algothm equatons that esult fom a ρ value geate than 2. The andomly-geneated, unfomly-dstbuted ntal values of the synaptc weghts wee the andom elements n each tal of a gven smulaton. The effectveness of the tanng algothms n each case was detemned by usng the data to answe two basc questons:. Dd the netwok lean the poblem suffcently well? 2. How much wok was equed to complete the tanng? Fo each tal n ths study, the followng was used as the stoppng ctea fo all of the thee tanng algothms tested. The supevsed-tanng tal was sad to convege f the netwok was able to lean of the tanng examples (0 fo patten 28

30 ecognton, 28 fo bt countng) n a gven epoch. If the netwok's pefomance n the tanng example set was less than, tanng contnues epoch-by-epoch untl ethe convegence o the had lmt of maxmum epochs was eached. Ths had lmt, detemned by empcal testng, was chosen to be 5,000 epochs fo patten ecognton and 5,000 epochs fo bt countng. Fnally, each of the smulatons descbed above wee pefomed on a Compaq Pesao F700 laptop compute wth a.90ghz AMD Athlon Dual-Coe Pocesso and 2.00GB of RAM. Smulaton Results A Note About ρ Values The pupose of ths study was to test the most compettve netwok confguatons fo each poblem-tanng algothm combnaton. In evey poblem-tanng algothm combnaton n ths study the best pefomance was acheved wth a ρ value of 2. In most cases the netwoks usng hghe values of ρ pefomed substantally wose, especally wth espect to convegence. It was theefoe decded to omt dscusson of ρ values geate than 2 fom the followng dscusson, although the elevant pefomance data fo these omtted tals can stll be found n Appendx D. Patten Recognton. Dd the netwok lean the poblem suffcently well? In the patten ecognton poblem, the netwok was pesented wth all 0 tanng examples, and the stoppng ctea was when all 0 tanng examples have been leaned. Theefoe, f a tanng tal dd not convege, the netwok was not able to lean the patten ecognton poblem. The convegence esults fo the patten ecognton tals (all ρ = 2) ae shown on the next page n Table 9. 29

31 Algothm Hdden Laye Sze Total Tals Convegent Tals Back-Popagaton Back-Popagaton Back-Popagaton Back-Popagaton Conugate-Gadent Conugate-Gadent Conugate-Gadent Conugate-Gadent Fnte-Dffeence Fnte-Dffeence Fnte-Dffeence Fnte-Dffeence Table 9: Patten Recognton Convegence Results As Table 9 shows, only 2 out of the 6,000 total patten ecognton tals faled to convege, both wth the conugate-gadent method. These two tals do not affect the oveall message of Table 9, howeve, whch s that the netwok was able to lean ths poblem equally well n all fou confguatons wth each of the thee tanng algothms. 2. How much wok was equed to complete the tanng? The wok equed to tan the netwok was measued by the numbe of epochs necessay to tan the netwok to convegence. The summay statstcs fo each of the twelve dstbutons geneated n ths study ae shown below n Table 0. Algothm Hdden Laye Sze Mn. Epochs Mean Epochs Medan Epochs Max. Epochs StDev Epochs Back-Popagaton Back-Popagaton Back-Popagaton Back-Popagaton Conugate-Gadent 4 (*) Conugate-Gadent Conugate-Gadent Conugate-Gadent 20 (*) Fnte-Dffeence Fnte-Dffeence Fnte-Dffeence Fnte-Dffeence (*) Only the 499 convegent tals wee used to calculate these summay statstcs. Table 0: Patten Recognton Epochs Summay Statstcs 30

32 Table 0 shows that the conugate-gadent algothm was by fa the most effcent n tems of numbe of epochs equed to lean the patten ecognton poblem. Among the conugate-gadent esults Table 0 shows that a hdden laye of 20 neuons was the best pefomng of the confguatons tested. Bt Countng. Dd the netwok lean the poblem suffcently well? In the bt countng poblem, the stoppng ctea fo each tanng algothm was the same as n patten ecognton stop tanng when the netwok acheves accuacy acoss the tanng set o the had lmt n epochs s eached. Table below s a summay of the convegence esults fo the bt countng tals (all ρ = 2) pefomed n ths study. Algothm Hdden Laye Sze Total Tals Convegent Tals Back-Popagaton Back-Popagaton Back-Popagaton Back-Popagaton Conugate-Gadent Conugate-Gadent Conugate-Gadent Conugate-Gadent Fnte-Dffeence Fnte-Dffeence Fnte-Dffeence Fnte-Dffeence Table : Bt Countng Convegence Results Table clealy shows that all of the thee tanng algothms showed bette convegence esults as the sze of the hdden laye was nceased fom 4 to 0 neuons. Fgue 0 below shows the same data n a gaphcal fomat hee one clealy sees the poo bt countng pefomance of the fnte-dffeence algothm elatve to the othe two. 3

33 conveged ddn't convege % Convegent Tals (500 Total pe Data Set) Back-Popagaton: 4 Back-Popagaton: 6 Back-Popagaton: 8 Back-Popagaton: 0 Conugate-Gadent: 4 Conugate-Gadent: 6 Conugate-Gadent: 8 Conugate-Gadent: 0 Data Set (Algothm: Hdden Laye Sze) Fnte-Dffeence: 4 Fnte-Dffeence: 6 Fnte-Dffeence: 8 Fnte-Dffeence: 0 Fgue 0: Pecent Convegent Tals by Algothm and Hdden Laye Sze Analyss of convegence behavo s a measue of how effectvely the netwok was able to lean the examples wthn tanng set. Fo bt countng, howeve, thee s anothe dmenson to the noton of how well the netwok was able to lean the poblem, and that s the genealzaton pefomance aganst the possble nputs that wee not pat of the tanng set. Ths pefomance was evaluated by testng the newly taned netwok aganst each of the 28 possble bt countng nputs and ecodng the esults on a scale fom 0 to coect. The summay statstcs fo esultng dstbutons of test scoes s shown below n Table 2. 32

34 Algothm Hdden Laye Sze Mn. % Coect Mean % Coect Medan % Coect Max. % Coect StDev % Coect Back-Popagaton Back-Popagaton Back-Popagaton Back-Popagaton Conugate-Gadent Conugate-Gadent Conugate-Gadent Conugate-Gadent Fnte-Dffeence Fnte-Dffeence Fnte-Dffeence Fnte-Dffeence Table 2: Bt Countng % Coect Summay Statstcs (All Data) Anothe useful way to compae the data that geneated Table 2 s to constuct and plot empcal cumulatve dstbuton functons fom the data, as seen n Fgues though 3, below Cummulatve % Tals (500 Total) Range of % Coect Fgue : Empcal CDF of Pecent Coect fo Back-Pop Bt Countng 33

35 Cummulatve % Tals (500 Total) Range of % Coect Fgue 2: Empcal CDF of Pecent Coect fo C-G Bt Countng Cummulatve % Tals (500 Total) Range of % Coect Fgue 3: Empcal CDF of Pecent Coect fo Fnte-Dff Bt Countng 34

36 Fom the pecedng fgues one can conclude that fo hdden laye szes of 6, 8, and 0 neuons the conugate-gadent algothm's statstcal pefomance was supeo to the othe two algothms. One cuous featue of these fgues, howeve, s that the back-pop algothm appeas to have outpefomed the othe two wth a hdden laye sze of How much wok was equed to complete the tanng? Agan the amount of wok equed to tan the netwok wll be evaluated by compang the dstbutons of the numbe of epochs necessay to tan the netwok. The summay statstcs of the dstbutons geneated dung the bt countng tals ae shown below n Table 3. Algothm Hdden Laye Sze Mn. Epochs Mean Epochs Medan Epochs Max. Epochs StDev Epochs Back-Popagaton 4 5,574,725,370 5,000 2,842 Back-Popagaton 6,4 5,346 4,630 5,000 3,062 Back-Popagaton 8,07 2,745 2,768, Back-Popagaton ,07,959 4, Conugate-Gadent ,837 3,227 5,000 6,24 Conugate-Gadent ,323,775 5,000 3,903 Conugate-Gadent 8 74, ,000,520 Conugate-Gadent , Fnte-Dffeence 4 2,48 4,649 5,000 5,000,82 Fnte-Dffeence ,6 4,949 5,000 5,322 Fnte-Dffeence ,822 3,676 5,000 4,792 Fnte-Dffeence ,263 2,332 5,000 2,879 Table 3: Bt Countng Epochs Summay Statstcs (All Data) As was done wth the pecent coect data, empcal cumulatve dstbuton functons wee ceated fom the dstbutons of epochs. These ae dsplayed below n Fgues 4 though 6. 35

37 Cummulatve % Tals (500 Total) Range of Epochs Fgue 4: Empcal CDF of Epochs fo Back-Pop Bt Countng Cummulatve % Tals (500 Total) Range of Epochs Fgue 5: Empcal CDF of Epochs fo C-G Bt Countng 36

38 Cummulatve % Tals (500 Total) Range of Epochs Fgue 6: Empcal CDF of Epochs fo Fnte-Dff Bt Countng As wth patten ecognton, so wth bt countng the conugate-gadent algothm equed fa fewe teatons fo each hdden laye sze than the othe two algothms by any statstcal measue. Theefoe, the esults of ths study suggest that the conugate-gadent algothm was supeo to the othe two algothms fo bt countng as well. Fgues 2 and 5 suggest that wthn the conugate-gadent data set, the netwok wth a hdden laye of 6 neuons gave the best oveall accuacy pefomance wth an acceptable tade-off n the dstbuton of epochs. Conclusons and Futue Wok All thee algothms wee able to teach the patten ecognton task wth neapefect accuacy egadless of the hdden laye sze chosen. The conugate-gadent algothm equed much fewe tanng epochs to do so by any statstcal measue. Theefoe the data suggests that the conugate-gadent algothm was the best of the thee algothms fo teachng the patten ecognton task. The bt countng poblem poved much moe dffcult fo all thee tanng algothms, but agan, the statstcal 37

39 esults of ths study demonstate the oveall supeoty of the conugate-gadent elatve to the othe two. The fact that a smple poblem lke bt countng should pove to be much hade fo the netwok to lean than patten ecognton begs some explanaton. One eason could have to do wth the concept of Hammng dstances. The Hammng dstance between two vectos s defned as the numbe of components n whch the two vectos dffe n value 26.. As a epesentatve example, Table 4 below shows the Hammng dstances between the vecto (0, 0, 0) and each of the othe vectos of 0 o of length 3. Vecto Vecto 2 Hammng Dstance (0, 0, 0) (0, 0, 0) 0 (0, 0, 0) (0, 0, ) (0, 0, 0) (0,, 0) (0, 0, 0) (0,, ) 2 (0, 0, 0) (, 0, 0) (0, 0, 0) (, 0, ) 2 (0, 0, 0) (,, 0) 2 (0, 0, 0) (,, ) 3 Table 4: Example Hammng Dstances The patten ecognton poblem n ths study has 0 unque nput vectos and 0 coespondng unque output vectos, a one-to-one nput-output functonal elatonshp. Ths can be ephased n tems of Hammng dstances as follows: n patten ecognton, when the Hammng dstance between two nput vectos s geate than zeo, the Hammng dstance between the coespondng output vectos wll also be geate than zeo. On the contay, the 7-dgt bt countng poblem has 28 unque vectos but only 8 unque output vectos, a many-to-one functonal elatonshp. Agan, n tems of Hammng dstances: n bt countng, when the Hammng dstance between two nput vectos s geate than zeo, the Hammng dstance between the coespondng output vectos wll be geate than o equal to zeo. The conectue that the esults of ths study suggest s that MLPs ae moe easly taned to pefom tasks whose functonal model s one-to-one than those whose model s one-to-many o many-to-one. Some suggestons fo futue wok ae lsted as follows: - Ty an ndvdually-adustable ρ value fo evey synaptc weght that would be an nceasng functon of the absolute value of the eo sgnal (desed actual). The 38

40 thought behnd the hghe ρ values s that lage eos would be taned out moe quckly by the coespondng hghe ode gadents that would esult. Howeve, ths s only tue when the absolute value of the eo sgnal s geate than o equal to one. When ths value s less than one, the lage ρ values would make the gadent much smalle, whch could cause small but appecable eo sgnals to pesst. - Ty a vaable/adaptve fnte-dffeence epslon to ty and mpove the pefomance of the fnte-dffeence algothm. Refeences.) Rosenblatt, F., 958. The Pecepton: A pobabalstc model fo nfomaton stoage and oganzaton n the ban, Psychologcal Revew, vol. 65, pp ) Haykn, S. Neual Netwoks: A Compehensve Foundaton. 2 nd Ed. New Jesey: Pentce-Hall; 999. pp ) Ibd., pp ) Ibd., pp ) Ibd., pp ) Ibd., pp ) Ibd., p ) Ibd., p ) Ibd., pp ) LeCun, Y., 993. Effcent Leanng and Second-ode Methods, A Tutoal at NIPS 93, Denve..) Haykn, 999. p ) Ibd., p ) Ibd., p ) Mnsky, M.L., 96. Steps towads atfcal ntellgence, Poceedngs of the Insttute of Rado Engnees, vol. 49, pp ) Rumelhat, D.E., G.E. Hnton, and R.J. Wllams, 986a. Leanng epesentatons of back-popagaton, n D.E. Rumelhat and J.L McCleland, eds., vol, Chapte 8, Cambdge, MA: MIT Pess. 6.) Hebb, D.O. The Oganzaton of Behavo: A Neuopsychologcal Theoy. New Yok: Wley; 949. p ) Haykn, 999. pp ) Ibd., pp ) Shewchuck, J.R An Intoducton to the Conugate Gadent Method Wthout the Agonzng Pan [Onlne]. Avalable fom: 2.cs.cmu.edu/s/spapes.html#cg. pp ) Haykn, 999. pp ) Ibd., pp ) Shewchuck, 994. pp ) Ibd., pp

41 24.) Polak, E., and G. Rbee, 969. Note su la convegence de methods de dectons conuguees, Revue Fancase Infomaton Recheche Opeatonnelle, vol. 6, pp ) Haykn, 999. pp ) Paul E. Black, Hammng dstance, n Dctonay of Algothms and Data Stuctues [Onlne], Paul E. Black, ed., U.S. Natonal Insttute of Standads and Technology. May 3, Avalable fom 40

42 Appendces Appendx A: Patten Recognton Calculato Dgts Fgue A: Ffteen Pxel Calculato Dgts 4

43 Appendx B: Bt Countng Tanng Example Set Bt Count Possble Tanng Examples (28) Numbe of Examples (68) Tanng Examples

44 Table B: Bt Countng Tanng Examples Appendx C: Devaton Resultng fom Hghe-Ode ρ Values The exponent of the eo functon E was vaed dung ths study, meanng that nstead of Equaton 2 we would have E = P P ( dk ok ) = 2 ( e ) k k = k = P P ρ ρ E = 2 ( d ) = 2 ( ), ρ = k ok ek k = k = (2.) 2,4,6. (2.C) The followng changes to the Back-Pop equatons ae then necessay. Recall fom Equaton 6 that but now E W = E e y v, (6.) e y v W P E ρ = 2 ( ek ) e (7.C.a) e k = E = ρ ( )( ρ )( e ) 2. (7.C) e The est of Equaton 6 s unchanged, so that now Equaton 2 becomes E W = ρ ρ [ ][ e ][ ϕ ( v )][ y ] 2 so that fo output neuons, Equaton 4 becomes ρ ρ δ = [ e ][ ϕ ( v [ ] )] 2, (2.C). (4.C) 43

45 Appendx D: Raw Monte Calo Smulaton Data D: Back-Pop Patten Recognton Data Set Total Tals Convegent Tals Mn Epochs Mean Epochs Medan Epochs Max Epochs StdDev Epochs , , Table D..: Summay Statstcs fo Numbe of Epochs by Data Set,000 mean mn max medan Max =,467 Max = 2, Numbe of Epochs Data Set (Hdden - Rho) Fgue D..: Plot of Summay Statstcs fo Epochs by Data Set (All Data) 44

46 Numbe of Epochs Data Set (Hdden - Rho) Fgue D..2: Plot of Standad Devaton of Epochs by Data Set (All Data) Cummulatve % Tals (500 Total) Range of Epochs Fgue D..3: Empcal CDF of Epochs fo Hdden Laye Sze 4 Netwoks by ρ Value 45

47 Cummulatve % Tals (500 Total) Range of Epochs Fgue D..4: Empcal CDF of Epochs fo Hdden Laye Sze 0 Netwoks by ρ Value Cummulatve % Tals (500 Total) Range of Epochs Fgue D..5: Empcal CDF of Epochs fo Hdden Laye Sze 5 Netwoks by ρ Value 46

48 Cummulatve % Tals (500 Total) Range of Epochs Fgue D..6: Empcal CDF of Epochs fo Hdden Laye Sze 20 Netwoks by ρ Value 47

49 D2: C-G Patten Recognton Data Set Total Tals Convegent Tals Mn Epochs Mean Epochs Medan Epochs Max Epochs StdDev Epochs , , , , , ,37 4 5, ,000, ,575 5,000 5,000 2, , , ,000 2, ,964 5,000 5,000 2,455 Table D.2.: Summay Statstcs fo Epochs by Data Set (All Data) Data Set Convegent Tals Mn Epochs Mean Epochs Medan Epochs Max Epochs StdDev Epochs , , Table D.2.: Summay Statstcs fo Epochs by Data Set (Convegent Tals Only) 48