Chaper 4. Neural Neworks Based on Compeon Compeon s mporan for NN Compeon beween neurons has been observed n bologcal nerve sysems Compeon s mporan n solvng many problems To classfy an npu paern _1 no one of he m classes dea case: one class node has oupu 1, all oher 0 ; ofen more han one class _n nodes have non-zero oupu INPUT C_1 C_m CLASSIFICATION If hese class nodes compee wh each oher, maybe only one wll wn evenually wnner-akes-all. The wnner represens he compued classfcaon of he npu
Wnner-akes-all WTA: Among all compeng nodes, only one wll wn and all ohers wll lose We manly deal wh sngle wnner WTA, bu mulple wnners WTA are possble and useful n some applcaons Eases way o realze WTA: have an eernal, cenral arbraor a program o decde he wnner by comparng he curren oupus of he compeors break he e arbrarly Ths s bologcally unsound no such eernal arbraor ess n bologcal nerve sysem.
Ways o realze compeon n NN Laeral nhbon Mane, Mecan ha oupu of each node feeds y_ o ohers hrough nhbory connecons wh negave weghs Resource compeon < 0 oupu of _k s dsrbued o y_ y_ and y_ proporonal o w_k and w_k, as well as y_ and y_ w self decay k bologcally sound Learnng mehods n compeve neworks Compeve learnng Kohonen learnng self-organzng map, SOM Couner-propagaon ne Adapve resonance heory ART n Ch. 5 w < 0 _k y_ w < 0 w y_ w k
Fed-wegh Compeve Nes Mane Laeral nhbon beween compeors weghs : w 1 ε f oherwse acvaon funcon : f > 0 f 0 oherwse Noes: Compeon: erave process unl he ne sablzes a mos one node wh posve acvaon 0 < ε < 1 / m, oo small: akes oo long o converge oo bg: may suppress he enre nework no wnner where m s he # of compeors ε ε
Mecal Ha Archecure: For a gven node, close neghbors: cooperave muually ecaory, w > 0 farher away neghbors: compeve muually nhbory,w < 0 oo far away neghbors: rrelevan w 0 Need a defnon of dsance neghborhood: one dmensonal: orderng by nde 1,2, n wo dmensonal: lace
> < k k c k c w, dsance f 0, dsance f, dsance f weghs 2 1 ramp funcon : ma ma ma 0 0 0 acvaon funcon > < f f f f
eample : 0 0.0, 0.5, 0.8, 1.0, 0.8, 0.5, 0.0 1 0.0, 0.38, 1.06, 1.16, 1.06, 0.38, 0.9 2 0.0, 0.39, 1.14, 1.66, 1.14, 0.39, 0.0 Equlbrum: negave npu posve npu for all nodes wnner has he hghes acvaon; s cooperave neghbors also have posve acvaon; s compeve neghbors have negave acvaons.
Hammng Nework Hammng dsance of wo vecors, dmenson n, Number of bs n dsagreemen. In bpolar: and y y a d where : a s number of bs n agreemen n and y d s number of bs dfferen n and y d n a hammng dsance y 2a n a 0.5 y n larger y larger a shorer Hammng dsance of
Suppose a space of paerns s dvded no k classes, each class has an eampler represenave vecor e. An npu belongs o class, f and only f s closer o e han o any oher e,.e., e e Hammng ne s such a classfer: Weghs: le represen class w Y 0.5e, b 0. 5n The oal npu o y _ n b w 1 2 e Y n a
Upper layer: MAX ne akes he y_n as s nal value, hen eraes oward sable sae one oupu node wh hghes y_n wll be he wnner because s wegh vecor s closes o he npu vecor As assocave memory: each corresponds o a sored paern; Y paern connecon/compleon; sorage capacy oal # of nodes: k oal # of paerns sored: k capacy: k or k/k 1
Implc laeral nhbon by compeng lmed resources: he acvaon of he npu nodes y_1 y_ y_m W y _ n y _ n larger y _ n y β w y decay y akes larger share of w
Compeve Learnng Unsupervsed learnng Goal: Learn o form classes/clusers of eamplers/sample paerns accordng o smlares of hese eampers. Paerns n a cluser would have smlar feaures No pror knowledge as wha feaures are mporan for classfcaon, and how many classes are here. Archecure: Oupu nodes: Y_1,.Y_m, represenng he m classes They are compeors WTA realzed eher by an eernal procedure or by laeral nhbon as n Mane
Tranng: Tran he nework such ha he wegh vecor w. assocaed wh Y_ becomes he represenave vecor of he class of npu paerns Y_ s o represen. Two phase unsupervsed learnng compeng phase: apply an npu vecor randomly chosen from sample se. compue oupu for all y: y deermne he wnner wnner s no gven n ranng samples so hs s unsupervsed rewardng phase: he wnner s reworded by updang s weghs weghs assocaed wh all oher oupu nodes are no updaed repea he wo phases many mes and gradually reduce he learnng rae unl all weghs are sablzed. w
Wegh updae: w Mehod 1: Mehod 2 w w w w w -w_ a-w_ w_ a w_ w_ a-w_ w_ w_ a In each mehod, s moved closer o Normalzng he wegh vecor o un lengh afer s updaed w w w w
w s movng o he cener of a cluser of sample vecors afer repeaed wegh updaes Three eamplers: S1, S2 and S3 w_0 w_3 Inal wegh vecor w_0 Afer successvely raned S3 w_1 by S1, S2, and S3, S1 he wegh vecor changes o w_1, S2 w_2, and w_3 w_2
Eamples A smple eample of compeve learnng pp. 172-175 4 vecors of dmenson 4 n 2 classes 4 npu nodes, 2 oupu nodes S1 1, 1, 0, 0 S2 0, 0, 0, 1 S3 1, 0, 0, 0 S4 0, 0, 1, 1 Inalzaon: 0.6, wegh mar: Tranng wh S1 D1 S1 w 1.2 1 2.6 1 2.5 0 2 W.2.6.5.9.9 0 2.8.4.7.3 1.86 D 1 S1 w 2 0.98, class 2 wns w 2.8.4.7.3 1 1 0.6 0 0.8.4.7.3.92.76.28.12 hen W.2.6.5.9.92.76.28.12
W Smlarly, afer ranng wh S2 0, 0, 0, 1, n whch class 1 wns, wegh mar becomes A he end of he frs eraon each of he 4 vecors are used, wegh mar becomes Reduce 0.5 0.5 0.6 0.3 Repea ranng. Afer 10 eraons, wegh mar becomes 6.7e 17 1.000000 0.0 1.0 2.0e 16.4900000 0.0 0.5.5100000 2.3e 16 0.5 0.0 1.000000 1.0e 16 1.0 0.0 W W.08.24.20.96.032.096.680.980.92.76.28.12.970.300.110.048 S1 and S3 belong o class 2 S2 and S4 belong o class 1 w_1 and w_2 are he cenrods of he wo classes
Commens 1. Ideally, when learnng sops, each s close o he cenrod of a group/cluser of sample npu vecors. 2. To sablze w, he learnng rae may be reduced slowly oward zero durng learnng. 3. # of oupu nodes: oo few: several clusers may be combned no one class oo many: over classfcaon ART model laer allows dynamc add/remove oupu nodes w 4. Inal : w ranng samples known o be n dsnc classes, provded such nfo s avalable random bad choces may cause anomaly
Eample w_2 w_1 w 1 w 2 wll always wn no maer he sample s from whch class s suck and wll no parcpae n learnng unsuck: le oupu nodes have some conscence emporarly sho off nodes whch have had very hgh wnnng rae hard o deermne wha rae should be consdered as very hgh
Kohonen Self-Organzng Maps SOM Compeve learnng Kohonen 1982 s a specal case of SOM Kohonen 1989 In compeve learnng, he nework s raned o organze npu vecor space no subspaces/classes/clusers each oupu node corresponds o one class he oupu nodes are no ordered: random map cluser_1 w_2 w_3 w_1 cluser_2 cluser_3 The opologcal order of he hree clusers s 1, 2, 3 The order of her maps a oupu nodes are 2, 3, 1 The map does no preserve he opologcal order of he ranng vecors
Topographc map a mappng ha preserves neghborhood relaons beween npu vecors, opology preservng or feaure preservng. f 1 and 2 are wo neghborng npu vecors by some dsance mercs, her correspondng wnnng oupu nodes classes, and mus also be close o each oher n some fashon one dmensonal: lne or rng, node has neghbors or ±1mod n wo dmensonal:grd. recangular: node, has neghbors:, ± 1, ± 1,, or addonal ± 1, ± 1 heagonal: 6 neghbors ±1
Bologcal movaon Mappng wo dmensonal connuous npus from sensory organ eyes, ears, skn, ec o wo dmensonal dscree oupus n he nerve sysem. Renoopc map: from eye rena o he vsual core. Tonoopc map: from he ear o he audory core These maps preserve opographc orders of npu. Bologcal evdence shows ha he connecons n hese maps are no enrely pre-programmed or pre-wred a brh. Learnng mus occur afer he brh o creae he necessary connecons for approprae opographc mappng.
Two layer nework: Oupu layer: SOM Archecure Each node represens a class of npus Node acvaon : y w w Neghborhood relaon s defned over hese nodes Each node cooperaes wh all s neghbors whn dsance R and compees wh all oher oupu nodes. Cooperaon and compeon of hese nodes can be realzed by Mecan Ha model R 0: all nodes are compeors no cooperave random map R > 0: opology preservng map
SOM Learnng 1. Inalze W for all oupu nodes, and o a small value 2. For a randomly seleced npu sample/eampler deermne he wnnng oupu node J eher W J s mamum or w J w s mnmum 3. For all oupu node wh J R, updae he wegh w w w 4. Perodcally reduce and R slowly. 5. Repea 2-4 unl he nework sablzed. 2
Noes 1. Inal weghs: small random value from -e, e 2. Reducon of : Lnear: Geomerc: β may be 1 or greaer han 1 3. Reducon of R: R R 1 whle R > 0 should be much slower han reducon. R can be a consan hrough ou he learnng. 4. Effec of learnng For each npu, no only he wegh vecor of wnner J s pulled closer o, bu also he weghs of J s close neghbors whn he radus of R. 5. Evenually, w becomes close smlar o w ± 1. The classes hey represen are also smlar. 6. May need large nal R
Eamples A smple eample of compeve learnng pp. 172-175 4 vecors of dmenson 4 n 2 classes 4 npu nodes, 2 oupu nodes S1 1, 1, 0, 0 S2 0, 0, 0, 1 S3 1, 0, 0, 0 S4 0, 0, 1, 1 Inalzaon: 0.6, wegh mar: Tranng wh S1 D1 S1 w 1.2 1 2.6 1 2.5 0 2 W.2.6.5.9.9 0 2.8.4.7.3 1.86 D 1 S1 w 2 0.98, class 2 wns w 2.8.4.7.3 1 1 0.6 0 0.8.4.7.3.92.76.28.12 hen W.2.6.5.9.92.76.28.12
How o llusrae Kohonen map Inpu vecor: 2 dmensonal Oupu vecor: 1 dmensonal lne/rng or 2 dmensonal grd. Wegh vecor s also 2 dmenson Represen he opology of oupu nodes by pons on a 2 dmensonal plane. Plong each oupu node on he plane wh s wegh vecor as s coordnaes. Connecng neghborng oupu nodes by a lne oupu nodes: 1, 1 2, 1 1, 2 wegh vecors: 0.5, 0.5 0.7, 0.2 0.9, 0.9 C1, 2 C1, 1 C2, 1
Travelng Salesman Problem TSP by SOM Each cy s represened as a 2 dmensonal npu vecor s coordnaes, y, Oupu nodes C_ form a one dmensonal SOM, each node corresponds o a cy. Inally, C_1,..., C_n have random wegh vecors Durng learnng, a wnner C_ on an npu, y of cy I, no only moves s w_ oward, y, bu also ha of of s neghbors w_1, w_-1. As he resul, C_-1 and C_1 wll laer be more lkely o wn wh npu vecors smlar o, y,.e, hose ces closer o I A he end, f a node represens cy I, would end up o have s neghbors 1 or -1 o represen ces smlar o cy I,e., ces close o cy I. Ths can be vewed as a concurren greedy algorhm
Inal poson Two canddae soluons: ADFGHIJBC ADFGHIJCB
Addonal eamples
Couner propagaon nework CPN Basc dea of CPN Purpose: fas and coarse appromaon of vecor mappng y φ no o map any gven o s φ wh gven precson, npu vecors are dvded no clusers/classes. each cluser of has one oupu y, whch s hopefully he average of φ for all n ha class. Archecure: Smple case: FORWARD ONLY CPN, 1 z1 y1 v z w y k n z p y m from npu feaures o class from class o oupu feaures
Learnng n wo phases: ranng sample :y where y φ s he precse mappng Phase1: s raned by compeve learnng o become he represenave vecor of a cluser of npu vecors use sample only 1. For a chosen, feedforward o deermned he wnnng 2. 3. Reduce, hen repea seps 1 and 2 unl sop condon s me Phase 2: w s raned by dela rule o be an average oupu of φ where s an npu vecor ha causes z o wn use boh and y. 1. For a chosen, feedforward o deermned he wnnng 2. v new v old v old oponal 3. v w v new v old v old k new w old y w old k 4. Repea seps 1 3 unl sop condon s me k k z z
Noes A combnaon of boh unsupervsed learnng for n phase 1 and supervsed learnng for n phase 2. w Afer phase 1, clusers are formed among sample npu, each s a represenave of a cluser average. Afer phase 2, each cluser maps o an oupu vecor y, whch s he average of φ : cluser { } Vew phase 2 learnng as followng dela rule E w k w k yk w k where yk w k, because w k E 2 yk w k z 2 yk w k z when z wns w k w k I can be shown ha, when, v and w φ where s he mean of all ranng samples ha make wn v v
1 1 1 as rewreen be updaerule can ssmlar.wegh of proof Show only on v v w v k [ ] v v v v 11... 11 1 1 1 1 1 1 1 1 1 1 1 1 1 2 2 E E E E v E 1 1 1 ]...1 1 [1 1]...1 1 1 [ ] 1... 1 1 [ 1] [ 1 hen If are drawn randomly from he ranng se,
Afer ranng, he nework works lke a look-up of mah able. For any npu, fnd a regon where falls represened by he wnng z node; use he regon as he nde o look-up he able for he funcon value. CPN works n mul-dmensonal npu space More cluser nodes z, more accurae mappng.
If boh y Full CPN φ and s nverse funcon φ we can esablsh b-dreconal appromaon Two pars of weghs marces: V o z and U z o y for appro. map o y φ W y o z and T z o for appro. map y o φ When :y s appled on X and y on Y, hey can only deermne he wnner J or separaely for z, pp. 196 206 for more deals 1 y es * J z Jy