Non-Linear Data for Neural Networks Training and Testing

Transcription

1 Proceedngs of the 4th WSEAS Int Conf on Informton Securty, Communctons nd Computers, Tenerfe, Spn, December 6-8, 005 (pp466-47) Non-Lner Dt for Neurl Networks Trnng nd Testng ABDEL LATIF ABU-DALHOUM MOHAMMED SADIQ AL-RAWI Computer Scence Dept, Kng Abdullh the Second School for Informton Technology Jordn Unversty Ammn 94, PO Box 3496, JORDAN Abstrct: - Hghly nonlner dt s re mportnt n the feld of rtfcl neurl networks It s not fesble to desgn neurl network nd try to clssfy some rel world dt drectly wth tht network N-bt prty s one of the oldest dt used to trn nd test neurl networks The smplest s the -bt prty lso known s the XOR clssfcton problem Some reserchers sy tht N-bt prty s though hghly nonlner t s smple tsk to lern by neurl networks, others were drfted to tlor specl purpose neurl networks to solve only the N-bt prty problem wthout explnng why there s such need Is t possble to udge the N-bt prty s smple dt due to the fct tht t cn be modeled by determnstc fnte ccepter? Moreover, should ptterns tht re n the form of context free whch requre pushdown utomton, or context-senstve nd recursvely enumerble tht requre Turng mchne be hrder to lern by neurl networks? The m of ths pper s to focus on nd propose some complex nonlner dt to be used n trnng nd testng of neurl networks The most mportnt n these prty dt s tht the developer cn tune the complexty of nonlnerty through vrous mounts of degrees; the user cn select vrous of ctegores, huge number of pttern smples, nd mny hybrd symbols Testng for vrous neurl networks nd ther generlzton nd blty to clssfy unseen ptterns cn be more effectve Expermentl results on the clssfcton of prme showed tht neurl networks cn lern the clssfcton of prme Key-Words: - Neurl Networks, nonlner seprble, hybrd N-prty, prme clssfcton, E problem, All symbol-prty, hybrd symbol-prty Introducton The N-bt prty s typcl clssfcton problem ddressed n neurl networks lterture N-bt prty s mppng defned on N dstnct bnry vectors tht ndctes whether the sum of the N components of bnry vector s odd or even (f the sum s odd the nput vector s clssfed s the frst ctegory, else nput vector belongs to the second ctegory) Ths smply constructed dt, lnerly nseprble, ws used by vrous reserchers n trnng nd testng neurl networks The prty problem s very dffcult tsk for neurl networks to lern wth generlzton [][][3][4] The m of N-bt prty s to test neurl network pproches tht would be ble to solve problems of unknown lnerty Neurl networks should do ths utomtclly wthout ny elegnt network topology desgn Wht s the mportnce of crefully desgnng network to solve the N-bt prty, especlly f such networks cnnot be trned wth bckpropgton (or ny other trnng method) One queston ntroduced n ths lterture whether N-bt prty clssfcton hs ny drect rel world pplcton? No nswer cn be found, nd f so, we cn perform the tsk wth logcl opertors, or smple counter The smplest prty problem s the - bt prty (lso known s the XOR problem) The clsscl Prllel Dstrbuted Processng textbook [] sttes tht mult lyer feedforwrd neurl network trned wth bckpropgton needs t lest N hdden lyer neurons to solve the N-bt prty Recent works show tht usng some shortcut connectons nd/or non-stndrd ctvton functon(s) the N-bt prty could be solved wth less thn N hdden neurons [] In fct, the problem cn be solved wth one output neuron f nondfferentble ctvton functon s used [3] More sophstcted studes [3][4] used the trple prty such tht three symbols re used {0,,} to generte strngs, nd f 0 s, s, nd s re even they re clssfed (ccepted) s the trget clss, otherwse they should be reected The trple prty benchmrk hd been used to test generlzton nd prunng of

2 Proceedngs of the 4th WSEAS Int Conf on Informton Securty, Communctons nd Computers, Tenerfe, Spn, December 6-8, 005 (pp466-47) recurrent neurl networks Mny works [5] emphsze tht N-bt prty s smple tsk for the neurl network to lern nd cn be performed v rndom serch method on recurrent neurl network Therefore, despte the complexty of N-bt prty, they [5] rgued tht ths s smple problem for neurl networks to lern nd more complcted (nonlner) dt re needed to trn nd test neurl networks However, [6][7][8][9][0][][][3] showed tht mny neurl networks expermenttons hve been performed usng N-bt prty dt The m of ths pper s to propose mny dt s wth complex nonlnerty Some dt my be derved from regulr, context-free, nd/or context senstve lnguges (knowng tht N-bt prty s form of regulr lnguge) All the proposed dt s re hghly nonlner nd they provde lrge dt benchmrk for trnng nd testng neurl networks Dt wth Complex Nonlnerly To nvestgte neurl networks blty to lern wth generlzton (ther blty to clssfy unseen smples), reserchers used N-bt prty nd other rtfcl dt In ths secton, we propose severl other dt s bsed on N-symbol-prty (N-prty) rther N-bt prty The N-prty s more complcted thn N-bt prty due to the use of mny symbols n the prty check Some of the proposed N-Prty problems re s descrbed n the followng sectons Hybrd N-Prty problem (HNP) Ths s dt tht mxes symbols tht re used for prty check {-, } nd the one tht s not used n prty check {0} the HNP Ths s defned s follows Defnton of HNP: Let = {,0, } be the of lphbets used to derve the strng s such tht s, s = N If the totl number of ctegores s four,, 4 tht denotes the frst to the fourth ctegory respectvely, nd - s nd s re counted for prty then the four ctegores re s gven below s : n ( s) s odd nd n -( s) s odd s : n ( s) s odd nd n -( s) s even, () s 4: n ( s) s even nd n -( s) s odd s : n ( s) s even nd n ( s) s even 3 - where n () s nd n - () s represent the number of s nd - s n s respectvely Note: = *, where s n empty strng nd * denotes the of strngs (pttern smples n ths pper) obtned by conctentng or more symbols from It s obvous tht the totl number of smples s n N Neurl networks trnng nd testng for the HNP hve been dscussed prevously n [4] All symbol prty (ASP) Ths s modfcton of the clsscl N-bt prty The AS P s defned s follows: Defnton of ASP: Let = { 0,,, n } be the of lphbets used to derve pttern smples such tht pttern smple s wth s = N If n =Σ, then the totl number of pttern smples s N n tht re clssfed nto c ctegores s follows s : D( s) = d s : D( s) = d " " s D( s) = d c : c, () n where d {0,,, } for =,, c such tht d d dc, D( s ) = DecmlOf(p), nd p = b ( s) b ( s) ( s) ( s) b n n b s bnry 0 number conctented from b ( s ) whch s gven by b t 0 the number of t n s s odd () s =, (3) the number of t n s s even for t = 0,,,, n It s obvous tht the bove ASP offers wde rnge of nonlnerty wth dfferent degrees of complextes My be t s better to express the ASP s functon of N nd n, therefore, we wrte ASP( N, n ) It s esy to show tht c = n for N = n Below, we demonstrte smple exmple of genertng some ASP dt The dt generted s hghly nonlner nd other hgher orders cn be generted too Exmple : Let Σ={0,,,3,4}, N = s =, nd n = 5 then the pttern smples {00,0,0,, 44} of t

3 Proceedngs of the 4th WSEAS Int Conf on Informton Securty, Communctons nd Computers, Tenerfe, Spn, December 6-8, 005 (pp466-47) ASP(,5) re clssfed nto eleven ctegores s shown below n Tble : Tble : The ASP(,5) dt , 0, 0, 03, 04,, 3, 4, 3, 4, 34,, 0, 33, Hybrd symbol N-Prty (HSNP) Ths s yet nother modfcton to the ASP n whch some of the symbols re only pdded to the pttern smples wthout beng counted for prty check, the sme s the {0} role n the HNP The defnton of HSNP s s follows Defnton of HSNP: Let = { 0,,, n } be the of lphbets used to derve pttern smples such tht pttern smple s Σ wth s = N The ncluded to be used n the prty check s sub from nd cn be selected s desred, let = {,,, r, s}, such tht k = The of symbols excluded from prty check s =, nd = n k If n =Σ, the totl number of pttern smples s n N tht re clssfed nto c ctegores s follows: s : D( s) = d s : D( s) = d, (4) " " s D( s) = d c : c where d s decml nteger obtned from d {0,,, k }, for =,, c such tht d d dc, D( s ) = DecmlOf(p), nd p = b ( s) b ( s) b ( s) b ( s) s bnry r s number conctented from b ( s ) whch s gven n (3) It s esy to show tht c = k for N = k Yet nother wde rnge of complex nonlnerty, the user should select how mny symbols to nclude n the prty check Agn, t s better to superscrpt the HSNP to be HSNP( N, n, k) To such cse, the t HNP s defned s HSNP(N,3, ), where = {,0,}, = {, }, ={0}, s = N Moreover, ASP(N,n) = HSNP( N,n,n) tht s ncludng ll the symbols results n the ASP Also, HSNP(N,, ) for = {0,}, = {}, = {0}, s = N s the N-bt prty dt Lst but not lest, HSNP(,,) s the so clled XOR dt For bpolr dt, we cn replce = {0,} wth = {,} where = { } wthout ny problem The below exmple demonstrtes the generton of HSNP(,5,) Exmple : Let Σ={0,,,3,4}, n = 5, = {, }, k =, nd N = s =, then the pttern smples {00,0,0,, 44} re clssfed nto four ctegores (snce N = k we wll hve k ctegores whch s four) ccordng to HSNP(,5,) s shown below n Tble : Tble : The HSNP (,5,) dt ,,, 33, 44, 03, 30, 04, 40, 34, 43 0, 0, 4, 4, 3, 3 0, 0, 3, 3, 4, 4, 4 Dchotomzed hybrd symbol N-Prty (DHSNP) Another modfcton cn be done to the HSNP( N, n, k) such tht the number of ctegores s two s follows: Defnton of DHSNP Let = { 0,,, n } be the of lphbets used to derve pttern smples such tht pttern smple s wth s = N The ncluded to be used n the prty check s sub from nd cn be selected s desred, let = {,,, r, s}, such tht k = The of symbols excluded from prty check s =,

4 Proceedngs of the 4th WSEAS Int Conf on Informton Securty, Communctons nd Computers, Tenerfe, Spn, December 6-8, 005 (pp466-47) therefore, = n k If n =Σ, then the totl number of pttern smples s nto two ctegores s follows N n tht re clssfed s : n ( p) s even, (5) s : n ( p ) s odd where p = b ( s) b () s b () s b () s b () s t r s s bnry number conctented from b t ( s ) whch s gven n (3) 5 The = (E) problem The =, s fmous dt used wth neurl networks We cn defne the = problem ccordng to the prevous nottons s below Defnton of E: Let =,,, } be the { 0 n of lphbets used to derve pttern smples such tht pttern smple s wth s = N The ncluded s = { } where Σ The totl N number of pttern smples s n cn be clssfed nto two ctegores s follows s : the number of n s s (6) s : otherwse The fmous (=) problem s the one such tht = nd ll other lphbets re to 0 whch s sub of the E problem gven bove 6 Prme number problem (PNP) Ths dt contns the bnry representton of decmls The tsk s tht neurl network should lern the clssfcton decml nto prme or not Defnton of PNP: Let = {0,} be the of lphbets used to derve the strng s such tht s, f the totl number of ctegores s two; denotes the frst ctegory nd denotes the second ctegory, then s DecmlOf( s) s prme (7) s otherwse A neurl network tht cn lern to clssfy the bove problem would be mrvels prme number genertor (provded tht t s effcent) whch cn be used n number theory nd n pplctons relted to publc key encrypton, or symmetrc encrypton Below s n exmple of lmted Exmple 3: Let Σ={0,}, s = 3, then the PNP v bnry representton s nputs to neurl networks Three nputs should be used for the neurl network, see the below PNP dt n Tble 3 Tble 3: The PNP for n= Exmple 4: Usng decml representton (0 to 7) s nputs to neurl networks, only one nput s needed See Tble 4 for llustrton Tble 4: The prme clssfcton problem usng decml representton The PNP wll be used n neurl network trnng nd testng n ths pper n order to show the complexty nd the lernng blty of ths dt 3 Expermentl Results The PNP dt presented n ths pper hs been used n trnng nd testng mny neurl networks wth dfferent networks topologes The tble below demonstrtes the performnce of usng the PNP dt n neurl networks trnng nd testng As for neurl networks, we used feedforwrd neurl network [] n trnng nd testng SCG 6-0- refers to the neurl network such tht the number of nputs s one, number of hdden neurons s 0, nd, the number of outputs s one (the nput strng s ether prme or not) The method used for optmzng the neurl network s the conugte grdent descent [][3] Results obtned usng the PNP for strngs of sze 6 re shown n Tble 5 As shown n Tble 5, decml re converted nto bnry ech of length 6 bt, then strngs representng decmls from 0 to 000 hve been used n trnng, nd strngs representng decmls from 3000 to nd strngs representng decmls from to hve been used n testng The m s to clssfy decml s beng prme number or not In Tble 6, we

5 Proceedngs of the 4th WSEAS Int Conf on Informton Securty, Communctons nd Computers, Tenerfe, Spn, December 6-8, 005 (pp466-47) mplemented the sme experments nd show tht usng decml s trnng dt for neurl networks, the network s unble to clssfy decml nto prme or not No successful lernng hppens nd ll networks we tred dd not converge Tble 5 Expermentl results of the clssfcton of decml nto prme or non-prme ccordng to ther bnry strng representton (the PNP problem) SCG 6-0- stnds for the feedforwrd neurl network wth 6 nputs, 80 hdden neurons, nd one output neuron tht s trned wth scled conugte grdent Neurl Network Used/ Topology Trnng Decml Testng Decml Correct Recognton on the testng Mn Error gol SCG / SCG / SCG / SCG / SCG / SCG / SCG / SCG (#0) /6 E-7 (#0) In ths experment, only 4 bts re used snce the decmls re from 0 to 5 Tble 6 Expermentl results of the clssfcton of decml nto prme or non-prme ccordng to decml vlue representton SCG -5- stnds for the feedforwrd neurl network wth one nput, fve hdden neurons, nd one output neuron tht s trned wth scled conugte grdent Neurl Network Used/ Topology Trnng Decml Testng Decml Correct Recognton on the testng Mn Error reched SCG / SCG /6 4E-7-0- SCG / (#) -80- SCG / SCG (#) / SCG (#) / (#) The mnmum error fter trnng s 05, no convergence hppened wthn more thn 0000 epochs (#) Sttstcl normlzton to zero men nd unt vrnce hs been performed so tht ll decmls re rel lyng n between - nd The prme number clssfcton needs more work to mprove the ccurcy of the clssfcton As cn be seen n Tble 5, of ccurcy clssfyng prme s more thn 95% for usng the trnng n testng, whle obtnng more thn 80% of ccurcy when usng dsont trnng nd testng s Ech of the bove experments s repeted up to 0 tmes nd the recognton ccurcy shown s the verge Thousnds of smples hve been used n trnng nd testng Mny other experments hve been performed usng only decmls from 0 to 5 In ths experment the neurl network dd lern both the decml representton dt nd the bnry (PNP) representton The flure for neurl networks to lern lrge tht re presented s drect or normlzed decmls suggests tht the neurl network (when lernng smll of decmls) s workng s n ssoctve memory knowng tht the number of weghts of the network n ths cse s greter thn the lerned decmls 4 Conclusons For ny clssfcton problem, the computer should perform heurstc serches on neurl networks to fnd the optmum (or best) weghts nd topology wth lttle humn nterventon s much s possble In ddton to the proposed new dt s, we conclude tht more elborted work should be done to trn nd test neurl networks, by tcklng hgh nonlner, hgh dmensonl dt, e, 0-bt prty (04 pttern smples), 0-bt prty (more thn one mllon pttern smple) These dt re to be clssfed ccordng to the proposed clssfcton problems stted n ths pper, e, HNP, ASP, HSNP, DHSNP, nd/or PNP Then, dvde the bulk of dt nto three prts; trnng, cross vldton, nd testng In dong ths t s possble to mesure the strtegc methods used to ntlze nd trn neurl networks As for the clssfcton of prme, t s very hrd problem to lern wth neurl networks, fndng better wys to tech neurl networks my serve other felds tht requre the generton of lrge prme effcently Neurl networks should lern the clssfcton tsk by lookng nto the bt ptterns rther thn usng some specl purpose lgorthm whether t s n effcent or exhustve serch for prme number method On usng decmls from to 000 (some re prme others re not) 83% of ccurcy hve been obtned for testng decmls wth the rnges to 65000, nd 74% of ccurcy hve been obtned for testng decmls

6 Proceedngs of the 4th WSEAS Int Conf on Informton Securty, Communctons nd Computers, Tenerfe, Spn, December 6-8, 005 (pp466-47) wth the rnges 3000 to 4000 It s obvous tht ll trnng nd testng s used re dsont It s mzng how neurl networks cn lern to clssfy prme from ther bnry representton, snce n well structured lgorthms the prme number should be non dvsble by ll decmls but tself nd one Rechng clssfcton rte of 74% or 83% s very promsng, yet needs more nvestgton It s worth mentonng tht neurl networks could not lern the prme clssfcton problem usng the drect decml representtons (s n Arbc numerl), therefore, t s our hope s we showed n the expermentl results tht the dt s complex nonlnerty but solvble wth neurl networks Trnng nd testng of ll the proposed dt s n ths pper s left s future work References: [] DE Rumelhrt, nd JLMcClellnd, Prllel Dstrbuted Processng, Vol, Cmbrdge, MA: MIT press, 986 [] RSetono, On The Soluton Of The Prty Problem By A Sngle Hdden Lyer Feedforwrd Neurl Network, Neurocomputng Vol6, No3, 997, pp 5-35 [3] M E Hohl, D Lu nd S H Smth, Solvng the N-bt prty problem usng neurl networks, Neurl Networks, Vol, No9, 999, pp 3-33 [4] CLGles, nd CWOmln, Prunng Recurrent Neurl Networks for Improved Generlzton Performnce, IEEE trnsctons on neurl networks, Vol5, No5, 994, pp [5] S Hochreter, nd J Schmdhuber, LSTN Cn Solve Hrd Long Tme Lg Problems, In Mozer, MC, Jordn, MI, Petsche, T eds, Advnces n Neurl Informton Processng Systems 9, NIPS'9, Cmbrdge MA: MIT Press, 997, pp [6] DLu, MEHohl, nd SHSmth, N-bt prty neurl networks:new solutons bsed on lner progrmmng, Neurocomputng, Vol48, 00, pp [7] E Lvretsky, On The Exct Soluton Of The Prty-N Problem Usng Ordered Neurl Networks, Neurl Networks, Vol 3, 000, pp [8] MZArslnov, DUAshglev, nd EEIsml, N-bt prty ordered neurl networks, Neurocomputng, Vol48, 00, pp [9] L Frnco, nd S A Cnns, Generlzton Propertes of Modulr Networks: Implementng the Prty Functon, IEEE trnsctons on neurl networks, Vol, No6, 00, pp [0] T Ntt, Solvng The XOR Problem And The Detecton Of Symmetry Usng A Sngle Complex-Vlued Neuron, Neurl Networks, Vol6, No,8, 003, pp 0-05 [] J Elmn, Fndng Structure n Tme, Cogntve Scence, Vol4, 990, pp 79- [] MF Mller, A Scled Conugte Grdent Algorthm for Fst Supervsed Lernng, Neurl Networks, Vol6, No4, 993, pp [3] MT Hgn, HB Demuth, nd MH Bele, Neurl Network Desgn, Boston, MA: PWS Publshng, 996 [4] M Al-Rw, A Neurl Network To Solve The Hybrd N-Prty: Lernng Wth Generlzton Issues, Neurocomputng, Vol68, 005, pp 73-80