Towards Multi-Layer Perceptron as an Evaluator Through Randomly Generated Training Patterns

Transcription

1 Proceedgs of the 5th WSEAS It. Cof. o Artfcal Itellgece, Kowledge Egeerg ad Data Bases, Madrd, Spa, February 5-7, 26 (pp ) Towards Mult-Layer Perceptro as a Evaluator Through Ramly Geerated Trag Patters JANIS ZUTERS Departmet of Computer Scece Uversty of Latva Raa bulv. 9, Rga, LV-5 LATVIA Abstract: - Mult-layer perceptro (MLP) s wdely used, because may problems ca be reduced to approxmato of fuctos. Patter evaluato, whch s dscussed ths artcle, belogs to ths rage of problems. MLP maages fucto approxmato problems qut well, however a mportat prerequste s a uformly dstrbuted set of trag patters. Ufortuately, such a set s ot always avalable. I ths artcle, the use of ramly geerated addtoal trag patters s examed to see whether ths mproves the trag result cases, whe ust postve patters are avalable. Key-Words: - Learg process, Mult-layer perceptro, Ramly geerated patters, Patter evaluato Itroducto The problem of patter evaluato ca be smply solved wth a mult-layer perceptro. We ust have to desg a etwork that approxmates the ukow evaluato fucto f( ) []: d = f (x), () where the vector x s the put ad the vector d s the output. We are gve the set of labeled examples (trag patters) Τ: Τ =, ) (2) { } d ( x = The fucto F( ) descrbg the put-output mappg actually realzed by the etwork should be close eough to f( ) a Eucldea sese over all puts: x F( x) f ( x) < ε, (3) where ε s a small postve umber. Such problems are perfect caddates for supervsed learg, t. al. MLPs. A problem arses f we have o set of patters adequately represetg f( ). [2] descrbes such a case wth solvg a tmetablg problem va a geetc algorthm (GA), where a eural etwork s teded to be a part of the ftess fucto wth GA order to support the evaluato (ratg) of solutos (). The dea s to tra the etwork o exstg (.e., vald) school, thus obtag the etwork, whch s able to evaluate. Ufortuately for ths approach we have o bad or ot very good examples of, so the avalable trag set s complete, ad the evaluato should be e as some kd of smlarty computato betwee the caddate tmetable ad vald. Fgure shows the schema of GA-based tmetablg, where the tmetable evaluato should be partally commtted to a eural etwork. Hard costrats 5. Modfcato of reproduced (mutato) 4. Choce of to be modfed ad reproducto. Geeratg the tal set of Set of urated Set of Termato codto satsfed Soft costrats 2. Ratg (ftess fucto) 3. Selectg Fg.. The schema for GA tmetablg [2] I ths artcle the author exames a method of orgazg a learg process (LP) through addg ramly geerated trag patters (TPs) to the trag set. The method s proposed order to compesate for the lack of a complete trag set.

2 Proceedgs of the 5th WSEAS It. Cof. o Artfcal Itellgece, Kowledge Egeerg ad Data Bases, Madrd, Spa, February 5-7, 26 (pp ) 2 Learg Process of a Neural Network The ablty of the etwork to lear from ts evromet ad to mprove ts performace through learg s the property of prmary sgfcace for eural etworks. [] The two aspects of the learg process are to be dstgushed:. Learg algorthm (LA). LAs dffer sgfcatly amog varous eural etwork models. 2. Orgazato of the LP. The latter ecompasses the way whch TPs are passed to the etwork ad correspodgly, to the LA. Usually the LP s arraged epochs. A epoch s oe sweep through all the patters the trag set, presetg them to the etwork,.e. the LA. Provded that the set of trag patters Τ s avalable (see (2)), the LP ca be geerally descrbed as follows (Fg. 2): % Τ - set of trag patters p := fetch the frst patter from Τ operate the learg algorthm o p p := fetch the ext patter from Τ whle p s avalable whle the stoppg crtero ot met Fg. 2. A learg process Fetchg of patters ca dffer terms of sequece some LAs requre ram order, others ot. 3 Orgazato of the Learg Process wth Icomplete Avalable Trag Set 3. The Problem Let s explore the followg problem: Compute the extet of smlarty betwee a gve alphaumerc symbol ad a fxed set of alphaumerc symbols Τ. I order to solve ths problem, we eed to desg a smlarty fucto for the fxed set Τ: Τ h : Α.. (4) { } where Τ Α s a set of alphaumerc symbols. Ths problem s the same as the oe dscussed Secto, ust replacg patters of symbols wth. To solve ths problem we could face the 2 followg sub-problems. Sub-problem (extremely mportat wth tmetable evaluato wth GA). Determe the crtero of smlarty betwee 2 patters (e.g., how close are symbols B ad 8 ). Sub-problem 2 (assumg that the problem s beg solved usg a eural etwork traed ust o postve patters). Overcome the lack of a complete trag set. The author proposes addg ramly geerated patters to the trag set to try to solve such problems. 3.2 Usuccessful Attempts The frst dea was to exame the Kohoe etwork. The competto prcple based o comparso betwee euros seemed very promsg ust to replace wer takes t all to everyoe takes as much as deserved accordg to gaed evaluato. Ufortuately varous attempts crashed the Kohoe etwork dd the clusterzato well, stll t was uable to fd out the evaluato of assgg a patter to some cluster. 3.3 The Proposed Method Trag MLPs wth Ramly Geerated Patters The secod dea, whch yelded results, was to use addtoal trag patters alog wth avalable oes. By ths approach we have to choose the ram rate τ {..}, whch deotes the proporto of the use of ramly geerated patters. The proposed supervsed learg method ca be descrbed as follows (see Fgure 3): % Τ - set of postve trag patters % τ - the ram rate {..} % s - sze of the trag set % determg the sze of a epoch (s2 s): s2 := s / ( - τ) for := to s2 rd := get ram value from terval.. f rd < τ the p := geerate ram patter d := else p := choose a patter from Τ ram d := operate the learg algorthm o [p, d] whle the stoppg crtero ot met Fg. 3. A supervsed learg process wth addtoal ramly geerated trag patters

3 Proceedgs of the 5th WSEAS It. Cof. o Artfcal Itellgece, Kowledge Egeerg ad Data Bases, Madrd, Spa, February 5-7, 26 (pp ) The proposed method was tested through the expermetato descrbed below. 4 Descrpto of the Expermetato The goal of the expermetato was to exame a method proposed by the author for orgazg the LP by addg ramly geerated TPs (solvg problems lke the problem, descrbed Secto 3.). The goal of each expermet s to obta a eural etwork that could be used for evaluato of patters,.e., oe, whch realzes the smlarty fucto (4). 4. Descrbg the Learg Process of the Expermetato The classc MLP wth the backpropagato learg algorthm was used the expermetato alog wth the author s proposed learg method (See Fg. 3). The core MLP operatg ad trag algorthm s brefly show as follows [3]. Propagato fucto: NET = = w o, (5) where NET s propagato value of the euro ; w the th weght of the euro (w bas of the euro ); o the th put sgal of the euro (o = ). Actvato fucto: o = ϕ( NET ), (6) where o output value of the euro ; ϕ( ) logstc actvato fucto. The geeral weght correcto rule: w = ηδ o, (7) where w correcto of the th weght of the euro ; η learg rate; δ local gradet of the euro (ot specfed ths paper); o the th put sgal of the euro. 4.2 Archtecture of Neural Networks Used Expermets I computer expermets eural etworks of the classc MLP archtecture were used: Sze of the put sgal 28. hdde layer wth 3, 5, or 7 euros. euro the output layer. Networks operate as descrbed above: see (5), (6), (7). Logstc fucto was used as a actvato m m m fucto wth the gaγ,,, where m 5 2 umber of euro puts. 4.3 Avalable Set of Trag ad Testg Patters The etworks were tested o black ad whte mages of a sze 32 4 pxels wth black had-wrtte dgts ad captal Lat letters depcted o a whte backgroud (Fg. 4). The umber of patter (symbol) types was =36 ( dgts + 26 letters). Let s deote the set of patter types as Γ. The umber of varats for each patter type was v=4. So, the total sze of the set of avalable patters Τ were v = 44. As we have 4 dfferet varats of each symbol (.e., four rows of patters), the for each expermet, oe of the rows served as the test set, the other three as the trag set. Fg. 4. A subset of the set of avalable patters Τ 4.4 Course of a sgle expermet A total of,25 expermets were coducted. Each expermet volves the desgato, trag, ad testg oe MLP.. Create a ew MLP accordg to the descrpto Secto Choose from Τ, at ram, three rows of avalable patters (Fg. 4) as a caddate set of trag patters Τ A Τ, so those of the remag row would be a test set Τ B Τ. 3. Choose at ram the ram rate τ amog values {,.2,.5,.7,.9}. τ= meas that the etwork s traed ust o selected patters wthout use of ramly geerated oes. 4. Choose at ram the amout of patter types c amog values {, 2, 4} the trag set. 5. Select patter types at ram for the trag set from 36 avalable dgts ad letters: Γ={q, q 2,..., q c } Γ. As three rows of four avalable fgures serve for trag, the chose patter types represet s = c 3 patters, so the trag set

4 Proceedgs of the 5th WSEAS It. Cof. o Artfcal Itellgece, Kowledge Egeerg ad Data Bases, Madrd, Spa, February 5-7, 26 (pp ) would cosst of s patters: Τ={p, p 2,..., p s } Τ A. 6. Tra the etwork usg the author s proposed method (Secto. 3.3) utl epochs are passed or the total error of output euros decreases uder ε=.. 7. Ram patters were geerated as black ad whte mages of a sze of 32 4 pxels, wth proporto of black at (also chose at ram). 8. Test the traed etwork o the set of test patters Τ B ad record the outputs, thus formg the expermetal results. The goal of a expermet to obta a etwork that s able to evaluate put patters Τ B (represets Γ ) wth respect to a fxed set Γ,.e. realzes a verso of the smlarty fucto h Γ. 5 Aalyss of the Proposed Trag Method Sub-problem 2 (Secto 3.) s stated to determe the crtero of smlarty betwee two patters. I ths case (aalyzg the trag method), such a crtero should be obtaed by exteral meas, ad the smlarty fucto that s based o t would the serve as a bechmark to measure the qualty of the soluto. 5. A Questoare Method to Obta the Smlarty Crtero As there exst o mafest formal crtera terms of smlarty betwee two patters, huma opo was chose as a crtero. A survey was arraged, ad eght respodets were asked to evaluate pars of symbols (a total of 63 pars each of 36 symbols wth each of the rest) wth the mark betwee ad (wth a mmum step of.), where meas very smlar, but absolutely dfferet (Fg. 5). All the respodets rated most of pars as. As the average result of all respodets, the smlarty measure for two patters g(, ) was obtaed: g : Γ Γ {..} (8) To reduce the computato, the fucto g(, ) was smplfed a maer so that the followg 2 equatos hold true: Γ : g(, ) = (9), Γ : g(, ) = g(, ) () Fg. 5. A excerpt of a qury form used the survey Now we ca defe the smlarty fucto wth respect to Γ: h ( ) = max g(, ) Γ, () Γ where Γ Γ the set of patter types, represetg the trag set, ad Γ the patter type. The troduced smlarty fucto represets the average evaluato of all respodets wth respect to Γ. 5.2 Represetg the Smlarty by the Smlarty Sequece Ufortuately, t was mpossble to exame expermetal results through comparg them drectly to the values yelded by the smlarty fucto. That was because of dsparate absolute output values amog expermets. There was a eed for a dfferet oto of the smlarty fucto. The dea s, stead of the smlarty fucto (), to represet the smlarty by the smlarty sequece wth respect to Γ: χ = χ, (2) = where Γ determes the type of the patter; χ determes the posto of the patter type the sequece, ordered accordg to the smlarty fucto: h( ) h( ) χ χ (3) 5.3 The Proposed Error Fucto to Evaluate The Trag Method Assume that the result of a expermet s also expressed as a ordered sequece of patter types: ψ = ψ, (4) = ordered accordg to the outputs of the etwork: F( ) F( ) ψ ψ (5) where F( ) the fucto realzed by the eural etwork traed o Γ.

5 Proceedgs of the 5th WSEAS It. Cof. o Artfcal Itellgece, Kowledge Egeerg ad Data Bases, Madrd, Spa, February 5-7, 26 (pp ) The we ca defe the sequetal error λ(, ) as dfferece betwee postos of two sequeces: ( χ ψ ) = λ ( χ, ψ ) = (6) Usg the smlarty sequece (2) stead of the smlarty fucto () s acceptable wth evaluato (ftess) fucto wth a GA, as the ftess fucto s volved determg, whch solutos are to be elmated or to be chose as parets, ad the absolute values of the evaluato are ot of great mportace. I the ext secto, the performed expermets are examed accordg to the sequetal error λ (6). 5.4 Aalyss of Expermetal Results ad Cocluso All the expermets were examed accordg to the sequetal error λ ad grouped by the umber of patter types c the trag set ad ram rate τ (see Secto 4.4). The summary of expermetal results s show Table. Table. Expermetal results as average of the sequetal error λ, grouped by the umber of patter types c ad the ram rate τ. sequetal error (λ) ram rate (τ) 2 umber of patter types Τ (c) To better evaluate the expermetal results, the smlarty sequeces, obtaed from separate respodets, ad the total smlarty sequece (2) also were examed. Ths was e by comparg them terms of sequetal error λ, ad the results were betwee 2.82 ad 5.99,.e., also rather far away from the deal value. Agast that backgroud, the acqured results for the proposed trag method look farly good. Although the mprovemet s small, we stll observe the followg beefts: There s a otceable effect of usg ust postve patters the trag set the mprovemet s small, but stable. The worst results were show by ram rate of value. Ths shows the effect of usg ramly geerated patters the learg process. The results ecourage the author to cotue research order to buld a eural etwork that would serve as evaluator of school, oe, whch would be cluded a geetc algorthm as a part of the ftess fucto. Refereces: [] S. Hayk, Neural etworks: a comprehesve foudato, 2d ed. Pretce-Hall, Ic, 999. [2] J. Zuters, A Adaptable Computatoal Model for Schedulg Trag Sessos, Aual Proceedgs of Vdzeme Uversty College ICTE Regoal Developmet, 25, pp [3] J. Zuters, A Exteso of Mult-Layer Perceptro Based o Layer-Topology, Proceedgs of the 5th Iteratoal Eformatka Coferece 5, 25, pp Accordg to the results show Table, the value of λ s early,.e., t s a great dstace away from the deal value of. It s ust a lttle better tha oe, whch ca be acqured by ramly geerated sequeces, whch yeld the λ value of approxmately 4.