Copuaonal Eln Copuaonalprocssng ln or nod fors a ghd su of d npus and passs h rsul hrough a non-lnar. Chapr 6 Mullar ural ors Bac Propagaon Algorh Th os popular hod for ranng ulplar ors n basd on gradn dscn n rror - gnrald dla rul, a naural nson of h LMS algorh. Transfr funcon f. Acvaon funcon d 3 Wha ar h ural ors? Mllons of non-lnar copuaonal lns oprang n paralll. Elns lnd b varabl ghs. Good prforanc va dns nrconncon. Slar o bologcal nural ors. Allo sulanous ploraon of svral hpohss Ap o achv huan -l prforanc n spch rcognon, ag rcognon Acvaon Funcons Lnar Bass Funcon Lnar bass funcon s a hprplan-p funcon. Ths s a frs-ordr lnar bass funcon. g, Radal Bass Funcon Radal bass funcon s a hpr sphr-p funcon. Ths nvolvs a scond-ordrnonlnar bass funcon. n g, d 4
Transfr Funcons Sp Funcon Rap Funcon Sgod Funcon Gaussan Funcon, f, f θ,,, f f c f > f f f f σ σ > θ < θ Prcpron Convrgnc Procdur Sp. Inal Wghs and Thrshold θ Sp. Prsn Inpu and Dsrd Oupu d. Sp 3. Calcula Acual Oupu. h Sp 4. Adap Wghs. d f? α hr, d Sp 5. Rpa b Gong o Sp. [ d ], ClassA ClassB,... d 5 7 Earl ural ors Sngl Lar Prcpron Eclln abl o rcogn spl parns. Parns hav o b lnarl sparabl. Wghs adusd usng Prcpron Convrgnc Procdur d f h ClassA ClassB Taono of ural ors as Classfrs Hopfld or Fd Wghs Bnar Parns Hang or ural or Classfr for Fd Parns Unsuprvsd Larnng ART- Suprvsd Larnng Sngl Lar Prcpron Connuous-Valud Parns Mul-Lar Prcpron Unsuprvsd Larnng Slf -Orgnng Faur Maps Opu Classfr Clusrng Algorh Gaussan Classfr - Classfr K-Mans Clusrng 6 8
A Brf Hsor of ural ors A Brf Hsor of ural ors 9 A Brf Hsor of ural ors A Brf Hsor of ural ors
A Brf Hsor of ural ors Bs-Knon ural ors Adapv Rsonanc Thor[978-86] Gall CarphnrU. of orhasrn, Sphn Grossburg U. of Boson. Parn rcognon, spcall hn parn s coplcad or unfallar o huans.radar or sonar Snsv o ranslaon, dsoron, chang n scals ocognron[978-84] Kunho FuusaHK Labs Handprnd characr rcognon Rqurs larg no. of procssng lns and conncons Bac Propagaon[974-85] Paul WrbosU. of Harvard, Davd ParrU. of Sanford, Davd RulharU. of Sanford Spch snhss fro ; Adapv conrol of roboc ars; Suprvsd ranng onl- corrc npu-oupu apls us b abundan. Bolan & Cauch Machns[985-86] ffr HnonU. of Torono, Trr Snosohns Hopns U., Harold Suaval Rs. Lab Parn rcognon for ags, sonar, radar. Bolan /c- long ranng. 3 5 Bs-Knon ural ors Prcpron[957] Fran RosnblaU. of Cornll. Tpd characr rcognon. Canno rcogn copl characrssuch Hangul; Snsv o dffrnc n scal, ranslaon, dsoron; Adaln/Madaln[96-6] Brnard WdroU. of Sanford. Adapv qualrs cho cancllrs n lphon lns. Assus a lnar rlaonshp b. npu and oupu; Slf-Organng Faur Map[98] Tuvo KohnnU. of Hlsn Maps on gorcal rgon such as a rcangl grd ono anohr such as an arcraf. Rqur nsv ranng. Bu or ffcv han an algorhc chnqus. Hopfld or[98] ohn HopfldCalforna Ins. of Tch. Rrval of copl daa or ags fro fragns. Dos no larng- ghs us b s n advanc. Tps of Wghd Conncons Thr ar four ps of ghd conncons : Fdforard, Fdbac, Laral, and T-Dlad Conncons Fdbac Conncons Laral Conncons Uppr Lar Fdforard Conncons Lor Lar 4 6
Tps of Wghd Conncons Th snapc conncons a b full or locallparall conncd. Fd Wghd Assocav Mor To ps of fd ghd assocav or orll b consdrd; On s of h fdforard p.g., Lnar and onlnar Assocav MorHang or Th ohr s h fdbac p.g., Hopfld or An assocav or or s a appng fro an npu spac o oupu spac Assocav or ors can b appld o hr auo-assocaon, or hroassocaon applcaons In h auo-assocaon applcaon, h dnsonal of h npu spac s qual o ha of h oupu spac. In h hro-assocaon applcaon, h dnson of npu and oupu spac ar n gnral dffrn. FullCross-bar Conncons LocallParall Conncons 7 9 Tranng Snapc Wghs Suprvsd Larnng Unsuprvsd Larnng Lnar Assocav Mor A lnar assocav orlam s a sngl-lar fdforard or. Th snapc gh ar W n LAM s drvd fro ang h corrlaon of h parn pars; Tranng Parns W Acual Oupu Tranng Parns W Acual Oupu hr, s h -h ln of h -h npu vcor s h -h ln of h -h dsrd vcor _ Th rspcv hrshold q s; θ Tachr Dsrd Oupu 8
Lnar Assocav Mor W W Tranng Parns ' ' Tranng Parns ' ' Lnar Assocav Mor W Thrshold θ I 4,3,4,3,-3,3,4,3,4' Ts Vcor ' f h W -θ? f h 4,-,4,-,,-,4,-,4',,,,,,,,' θ 3 onlnar Assocav Mor A nonlnar assocav or consss of hr pars;. Scor achng b asur of h nnr-produc or Hang dsanc.. onlnar procssng un for nnr slcon b hrsholdng or MAXET procssng. 3. Dsplang h slcd parn. B onlnar Crcus Wnnr Slcon A 4 Hang ors Th Hang or slcs a nnr fro h sord parns, {X,,...,M}, hch has h las Hang dsanc fro h npu vcor. Th lor sub calculas -HD o M aplar parns achng scors. Th uppr sub slcs ha nod h h au oupu. All nods us h hrshold-logc nonlnars.
Hang ors Oupus Vald afr Ma Convrgnc M- M u u T l u - u M } Ma Pcs Mau Fdbac Assocav Mor ors A fdbac or nds an raons bfor rrvng a fnal parn. Th os fdbac auo-assocaon or s h Hopfld odl, hch has h follong characrscs;. Snapc ghs ar prsord.. onlnar hrsholdng opraons ar usd n ach sag o produc bnar-valud sas. 3. Sa fdbacs ar usd so ha h sas can b ravl updad. 4. Iraons ll convrg o a soluon ha ns an nrg funcon pranng o h or. W } Calcula Machng Scors - Inpus Appld a T Zro 5 7 Hang or Algorhs Sp. Assgn Conncon Wghs and Offss In h lor sub:, θ,, M hr, s h conncon gh fro npu o nod ; θ s h hrshold n nod ; s -h ln of ranng parn. In h uppr sub:, Tl ε, l, M hr, T l s h conncon gh fro nod o nod l; Sp. Inal h Unnon Inpu Parn u f h u s h oupu of nod n h uppr sub a ; s ln of npu. Sp 3. Ira Unl Convrgnc Sp 4. Rpa b Gong o Sp. l ε <?,,, u fh u ε u, l M M, M Fdbac Assocav Mor ors Drvaon of Snapc Wghs Gvn M bnar-valud parns,.., or,,,m Th ghs of h Hopfld or ar drvd as, Th hrshold of h or ar gvn as θ Enrg Funcons and Convrgnc: M, Th nrg funcon or Lapunov funcon s dfnd as E θ Evr orgnal parn rprsns a local or global nu of h nrg funcon..., A or can ravl sarch for a local nu sa. 6 8
Fdbac Assocav Mor ors Th dffrnc of h nrg funcons bfor and afr a sa upda s E E E θ θ [ ][ θ ] [ ]{[ ]} u W hr, u θ Hopfld ors or s usd h bnar npu. or s usd as assocav or conn-addrssanl or or o solv opaon probls. o. of parns srod n ld snc no. of nods ncrass rapdl. Eplar parns ar unsabl f h shar oo an bs n coon h ohr plars 9 3 Fdbac Assocav Mor ors Hopfld ors In cas of a squnal upda, hr s onl on b updad a on On h -h b, E u u, Q A dscr vrson of h gradn s u θ Oupus Vald afr Convrgnc - E u u u u - u θ θ θ - θ - Inpus Appld a T Zro 3 3
Hopfld or Algorhs Sp. Assgn Conncon Snapc Wghs. M s h conncon gh fro nod o nod ;, hch can b or, s ln of h plar for class. Sp. Inal h Unnon Inpu Parn. s h ln of h npu parn and can b or ; s h oupu of nod a. Sp 3. Copu h Valu. Sp 4. Upda h Sa,, u?, f, u,, M,,,, u > u u < M Sp 5. Rpa b Gong o Sp 3 unl nod oupu ran unchangd h furhr raons. Copv Larnng ors Th copv larnngcl or's snapc ghs adaps accordng o unsuprvsd larnng ruls. CL odls can larn n h absnc of h achr's gudln..., Th ranng s basd clusvl on h ranng parns. Th class of CL ors ncluds, for apl, slf -organaon, adapv rsonanc, and nocognron Th ranng ruls ar h Hbban rul for h fdforard or and h nnr-a-allwta rul for h laral or. In ordr o pln copv larnng, h laral ors ar usuall nhbor. A nnr-a-all crcu s usd o slc a nnr, basd on a dsanc rc ovr h parn vcor spac. A un larns f and onl f ns h copon aong all h ohr uns. A basc copv larnng odl consss of fdforard and laral ors h fd oupu nods. Th npu and oupu nods ar assud o hav bnar valus or. Whn boh h -h npu and h -h oupu ar hgh, hn C ; ohrs C. Th srngh of h snapc gh conncng npu h oupu s dsgnd b. 33 35 Eapl for Hopfld ors Copv Larnng ors Consdr h o ranng parn vcors;,,,',,,, Th gh ar W s M Th hrshold θ s,,,- Ts parn vcor,,,,,, W, M Tranng Ruls Basd on orald Wghs In ordr o nsur a far copon nvronn, h su of all h ghs lnd o all h oupu nods should b norald. b h ghs conncd o an oupu nod, hn Σ. Thr ar caor and nhbor clls ha nrac srongl n h Malsburg's odl Th conncon srnghs ar a funcon of dsanc b. clls, hch hbs sgnfcan local naur. In Malsburg larnng rul, a nuron un shfs gh fro s nacv o acv npu coponns " If a un ns h copon, hn ach of s npu lns gv up so proporon g of s gh, and h gh s dsrbud quall aong h acv npu lns." Th larnng rul s; u Procssng:,,,'? f?,, u M,,, u > u u < g ], n [ f un ns on sulus f un loss on sulus hr, g s a sall posv consan, n s h no. of acv npu uns for sulus parn 34 36
Eapls of Copv Larnng Rul Adapv Rsonanc Thor A-p: conncons h C B-p: nnr's conncons C-p: losr's conncon Wnnng nod Adapv Rsonanc ThorART, ha as nroducd b Carpnr and Grossbrg, s a or sophscad clusrng chnqu for adapvl adusng h no. of clusrs. In addon o h forard or b. h npu nurons and oupu nurons, a bacard or s adopd for vglanc s. ART can b adapvl cra a n nuron for an ncong npu parn f s drnd b a vglanc s o b suffcnl dffrn fro h sng clusrs. Oupu gan STM F Y W X Rsonanc or Rs Mnal Larnng Rul: onl conncons of p A ar rand. Malsburg Larnng Rul: onl conncons of p A and p B ar rand. 3 La Larnng Rul: all conncons p A, B, and C ar rand _ gan T STM F W T ^X ρ X Machng Crron: Vglanc Parar 37 Inpu X 39 Vcor Quanr Adapv Rsonanc Thor A Vcor QuanrVQ s a ss for appng a s of vcors no a fn clusr for dgal sorag or councaon. VQs on of h vr frs unsuprvsd clusrng chnqus. A VQaps a s of dscr vcors no a rprsnaon suabl for councaon ovr dgal channl. Fro h codng prspcv, h goal of VQ s o oban h bs possbl fdl for h gvn coprsson ra. Oupu Y W X STM F Rs Th VQ procdur s suard blo:. Gvn a n parn, dnf h bs old clusr o ad h parn. Th crron s usuall h Eucldan dsanc.. Th cnrod of h slcd clusr s adusd o accooda s n br. 3. If non of h old clusrs can ad h n parn, a n clusr ll b crad for. 4. Rpa h procdur for h succssv parns. T W STM F Rsonanc T ^X ρ X Machng Crron: Vglanc Parar Inpu X 38 4
Adapv Rsonanc Thor Inpu vcor : Each npu X s an -dnsonal vcor,,...,, hr ach coponn s n [,]. Wgh Vcor: W,,..., and T,,..., ar h forard and bacard adapv gh vcor or LT racs of ach cagor. Th no. of cagors M s arbar. Inall... and... /. Parar: ART dnacs ar drnd b a choc parar α>; larnng parar β; and vglanc parar ρ Cagor choc: X^W Y I α W For ach npu I and cagor, h choc funcon Y s dfnd b Th cagor choc s ndd b, hr Y a{ Y,...,M}., hr, X, ^ AD If or han on Y s aal, h cagor h h salls nd s chosn. Adapv Rsonanc Thor Sp. Inalaon,, S ρ, ρ Sp. Appl Inpu X Sp 3. Copus Machng Scors, Sp 4. Slc Bs Machng Eaplar : Sp 5. Vglanc Ts M Y a{,... M} ρ, M If hn goo Sp 7; ls goo Sp 6 4 43 Adapv Rsonanc Thor Rsonanc or Rs: Rsonanc occurs f h ach funcon of h chosn cagor s h vglanc crron; ha s, f Msach rs occurs f ohrs. Thn h valu of h chos funcon T s rs - for h duraon of h npu prsnaon o prvn s prssn slcon durng sarch. A n nd s chosn. Th sarch procss connus unl rsonanc occurs Larnng: I ^ W I ρ Th gh W and T ar updad accordng o h quaon Adapv Rsonanc Thor Sp 6. Rs Bs Machng Eaplar Th oupu of h bs achng nod slcd n Sp 4 s polarl s o -, and no longr a par n h aaon of Sp 4. Thn goo Sp 3 Sp 7. Adap Bs Machng EplarRsonanc,.5 or W T n n β X ^ W β W X ^ T old old Fas larnng corrsponds o sng o β Inpu oralaon Opon: Prolfraon of cagors s avodd n analog ART f npus ar norald; ha s for so γ>, I > γ for all npus I. oralaon can b achvd b IX/ X old 4 44
Fd-forard Opraon Hddn lar MLBP: Mul-Lar Bac Propagaon d f,, d f f > Oupu lar nh n H f,, f f > 45 47 Fd-forard Opraon Fd-forard Opraon A spl 3-Lar or Inpu lar, Hddn lar, Oupu lar Eclusv OR Probl 46 48
Gnral Fd-forard Opraon n H g f n H d f f Bac Propagaon Algorh Th Bac PropagaonBP algorh s acuall a gnralaon of h las an squarlms algorh. Th BP algorh uss an rav gradn chnqu o n h an squar rror b. h dsrd oupu and h acual oupu of a ul-lar fdforard Prcpron. Th ranng procdur s nald b slcng sall rando ghs and nrnal hrsholds, prsnng all h ranng daa rpadl o or. Wghs ar adusd afr vr ral unl h sabl, and a h sa h cos funcon s rducd o an accpanc valu. 49 5 Gnral Fd-forard Opraon Sgod Funcons Sgod funcons of h for f ' f f f 5 5
Mul-lar Prcpron Bac Propagaon Algorh 53 55 Mul-lar Prcpron Bac Propagaon Algorh 54 56
Bac Propagaon Algorh Bac Propagaon Algorh 57 59 Bac Propagaon Algorh Mul-lar Prcpron C Oupu Lar W Hddn Lar W d Inpu Lar d 58 6
6 or Larnng Th Bac-propagaon larnng rul s basd on gradn dscn Crron funcon s hr, Th rav algorh rqurs ang a gh vcor a raon and upda as: C, hr η f θ 6 or Larnng Consdr h hddn-o-oupu ghs, Thn, h snsv for a oupu un s dfnd as: C δ θ δ Q 63 or Larnng Consdr h npu-o-hddn ghs, d C C C C C C θ δ δ Q 64 or Larnng Th larnng rul for h hddn-o-oupu gh s: Th larnng rul for h npu-o-hddn gh s: 4 4 4 3 44 4 c hr δ δ η η δ ηδ η η,
Error Bac Propagaon Th snsv for a hddn un s dfnd as δ c δ ω ω ω ω C δ C δ δ δ C d δ oupu hddn npu Bac Propagaon Tranng Algorh Sp 4. Calcula Error d for od δ Sp 5. Adap Wghs, ; Us a rcursv algorh sarng a h oupu nods and orng bac o h frs hddn lar. Adus ghs b η hr, s h gh fro nod o nod a ;, s h -h nod of npu and -h nod of hddn lar; η s a gan rlarnng ra;? δ s a rror r for nod ; <α<onu; Sp 6. Rpa b Gong o Sp ηδ α C δ α unl C < oal_ rror 65 67 Bac Propagaon Tranng Algorh Sp. Inal Wghs, and Offss q, q ; S all ghs and nod offss o sall rando valus. Sp. Prsn Inpu and Dsrd Oupus; Prsn a connuous valud npu vcor,,..., d and spcf h dsrd oupu,,..., C Larnng Ras Th opal larnng ra lads o rror nu n on larnng sp Th opal larnng ra s η op Sp 3. Calcula Acual Oupus; Us h sgod non-lnar funcon f Calcula lard oupus,,...,, and,,..., c. d f θ, f θ,,...,,,..., C, 66 68
Monu Enhancns o Gradn Dscn: Monu Larnng h onr η α 4 4 443 4 43 gradn dscn Monu 69 7 Enhancns o Gradn Dscn: Monu Mor Enhancns: Adapv Larnng Ras 7 7
Mor Enhancns: Adapv Larnng Ras Lng or Copl 73 75 Lng or Copl Lng or Copl 74 76
Trcs of h Trad Rpor Plo h dcson surfac funcon - usng BP or o of npu un :,, o of hddn un : o of oupu un :,, for class, for class, ohrs Tranng daa: Class : hp://vson..pusan.ac.r/parn Rcognon/class. Class : hp://vson..pusan.ac.r/parn Rcognon/class. ou nd o scal h ranng daa o [- ] b scal facor - - - 77 79 Trcs of h Trad 78