Internet Engineering. Jacek Mazurkiewicz, PhD Softcomputing. Part 3: Recurrent Artificial Neural Networks Self-Organising Artificial Neural Networks

Transcription

1 Internet Engneerng Jacek Mazurkewcz, PhD Softcomputng Part 3: Recurrent Artfcal Neural Networks Self-Organsng Artfcal Neural Networks

2 Recurrent Artfcal Neural Networks Feedback sgnals between neurons Dynamc relatons Sngle neuron change s transmtted to whole net Stable state s reached after the set of temporary states Stable state s avalable f strct assumptons are fxed to weghts Recurrent artfcal neural networks are equped by symmetrc nter-neurons connectons

3 Assocatve Memory computer memory as close as possble to human memory: assocatve memory to store patterns learnng procedure to nprnt the set of patterns retrevng phase output the stored pattern closest to the actual nput sgnal auto-assocatve: hetero-assocatve: Smoth Smth Smth Smth s face (Smth s feature)

4 Hopfeld Network (1) w Hammng dstance for a bnary nput: x 1 v 1 y 1 d H n 1 [ x (1 y ) + (1 x ) y ] x 2 v 2 y 2 Hammng dstance equals to zero f: y x x N-1 v N-1 y N-1 Hammng dstance s a number of not equal bts x N v N y N neuron

5 Retrevng Phase (1) each neuron performs the followng two steps: computes the coproduct: p N wpv k 1 ( k+ 1) ( ) updates the state: ( + 1) v p k where: w p v (k) p p 1 for ( k+ 1) p 0 v p( k) for ( k+ 1) p 0 1 for ( k+ 1) 0 weght related to feedback sgnal feedback sgnal bas p x 1 x 2 x N-1 x N v 1 v 2 v N-1 v N w y 1 y 2 y N-1 y N neuron

6 ntal condton: Retrevng Phase (2) v ( 0) x p p p process s repeated untl convergence, whch occurs when none of the elements changes state durng any teraton: x 1 v 1 w y 1 v ( k+ 1) v ( k) y p p p p x 2 v 2 y 2 converged state of Hopfeld net means that net has already reached one of attractors attractor - pont of a local mnmum of the energy functon (Lapunov functon): x N-1 v N-1 y N-1 1 N N E( x) w x x + x E x x T T ( ) W x + x N x N v N y N neuron

7 tranng patterns are presented one by one n a ftted tme ntervals Hebban Learnng durng each nterval nput data s communcated to neuron s neghbours N tmes x 1 v 1 w y 1 w 1 N 0 M ( m) ( m) x x dla m 1 dla x 2 v 2 y 2 convergence condton: w 0 w w p pp p p p x N-1 v N-1 y N-1 algorthm: easy, fast, low memory capacty: M max N x N v N y N neuron

8 correct weght values means: nput sgnal generates tself as output converged state avalable at once: one of possble solutons s: Pseudonverse Learnng W X X x 1 x 2 v 1 v 2 w y 1 y 2 ( T ) 1 W X X X X T x N-1 v N-1 y N-1 algorthm: sophstcated, hgh memory capacty: M max N x N v N y N neuron

9 Delta-Rule Learnng weghts are tuned step by step usng all learnng sgnals, presented n a sequence: + ( ) ( ) ( ) N W W x W x x , learnng rate algorthm s qute smlar to gradent methods used for Multlayer Perceptron learnng algorthm: sophstcated, hgh memory capacty: T x 1 x 2 x N-1 v 1 v 2 v N-1 w y 1 y 2 y N-1 M max N x N v N y N neuron

10 Retrevng Phase - Problems Input sgnals heavly corrupted by nose can follow to a false answer net output s far from learned/stored patterns Energy functon value for symmetrc states s dentcal (+1,+1,-1) (-1,-1,+1) both solutons offer the same acceptance factor Learnng algorthms can produce addtonal local mnma as lnear combnaton of learnng patterns Addtonal mnma are not fxed to any learnng pattern strongly mportant f the number of learnng patterns s sgnfcant

11 Example of Answers 10 dgts, 7x7 pxels Hebban learnng: 1 correct answer Pseudonverse & Delta-rule learnng: 7 correct answers 9 answers wth 1 wrong pxel 4 answers wth 2 wrong pxels

12 Hammng Network (1)

13 Hammng Network (2) Hammng net maxmum lkelhood classfer for bnary nputs corrupted by nose Lower Sub Net calculates N mnus the Hammng dstance to M exemplar patterns Upper Sub Net selects that node wth the maxmum output All nodes use threshold logc nonlneartes the outputs of these nonlneartes never saturate Thresholds and weghts n the Maxnet are fxed All thresholds are set to zero, weghts from each node to tself are 1 Weghts between nodes are nhbtory

14 Hammng Network (3) weghts and offsets of the Lower Sub Net: w x N 2 2 weghts n the Maxnet are fxed as: for 0 N 1 and 0 M 1 wlk 1 f f k k 1 1 for 0 l, k M and 1 M all thresholds n the Maxnet are kept zero

15 Hammng Network (4) outputs of the Lower Sub Net are obtaned as: N 1 w x 0 weghts n the Maxnet are fxed as: y ( ) f ( ) t Maxnet does the maxmsaton by evaluatng: for 0 N 1 and 0 M 1 0 for 0 M 1 t t t y + f y y ( ) ( ) ( ) t k k 1 for 0, k M 1 ths process s repeated untl convergence

16 Introducton learnng wthout a teacher data overload unsupervsed learnng: smlarty PCA algorthms classfcaton archetype fndng feature maps

17 Pavlov Experment FOOD (UCS) SALIVATION (UCR) FOOD + BELL (UCS + CS) SALIVATION (CR) BELL (CS) CS condtoned stmulus UCS uncondtoned stmulus SALIVATION (CR) CR condtoned reflex UCR uncondtoned reflex

18 Felds of Usng smlarty sngle-output net how close s nput sgnal to mean-learned-pattern PCA mult-output net, each output sngle prncpal component prncpal components responsble for smlarty actual output vector correlaton level classfcaton bnary mult-output wth 1 of n code class of closest data stored patterns fndng assocatve memory codng data compresson

19 Hebban Rule (1949) f neuron A s actvated n a cyclc way by neuron B neuron A s more and more senstve to actvaton from neuron B f(a) s any functon lnear for example X 1 W 1 X 2 w ( k + 1) w( k) + w( k ) W 2 u f(a ) y w x ( k) y ( k) X m W m

20 General Hebban Rule Problem: unlmted weght growth w F( x, y ) Soluton: set lmtatons (Lnsker) Oa s rule Lmtatons: Oa s rule: Hebban rule + normalsaton addtonal requrements w w [ w, w + ] y ( k) 0; m 0 w x ( k) y ( k)[ x ( k) y ( k) w ( k)] x y w x ( k) y 0; w ( k) ( k) 0

21 Prncpal Component Analyss - PCA Statstc loss compresson n telecommuncaton Karhuenen-Loeve approach Lnear converson nto output space wth reduced dmensons preserves the most mportant features of stochastc process x Frst component estmaton weghts vector usng Oa s rule: Other prncpal components by Sanger s rule: y Wx N K R x N N K K R W R y + ) ( ) ( ) ( k x W k x W k y N T N k W x k y 0 ) ( ) (

22 Neural Networks for PCA Oa s rule Sanger s rule n k w y x y w k l 1,..., 1,..., 1 n k w y x y w l 1,..., 1,..., 1

23 Sngle-layer Rubner & Tavan Network 1989 (1) One-way connectons Weghts: nput layer calculaton layer accordng to the Hebban rule w x y Internal connectons wthn calculaton layer accordng to the ant-hebb rule v y y

24 Rubner & Tavan Network 1989 (2) y 1 y 2 y 3 y 4 v 41 v v v 21 v 32 v 43 w 11 w 45 x 1 x 2 x 3 x 4 x 5

25 Pcture Compresson for PCA Large amount of nput data substtuted by lower amount combned n vector y and W Level of compresson number of PCA components man factor of the restored pcture qualty More prncpal components better qualty lower compresson level Pcture restored based on: 2 prncpal components compresson level: 28

26 Self-Organsng Artfcal Neural Networks Inter-neurons acton Goal: nput sgnals mapped nto output sgnals Smlar nput data are grouped Groups are separated Kohonen neural network leader! T. Kohonen from Fnland!

27 Concurrent Learnng WTA Wnner Takes All WTM Wnner Takes Most W X Y

28 WTA (1) Sngle layer of workng neurons The same nput sgnals x are loaded to all compettve neurons Startng weght values are random Each neuron calculates the product: The wnner s the neuron wth a maxmum output! u w x Neuron the wnner fnal output equals to 1 Other neurons set output values to 0

29 WTA (2) Frst presentaton of learnng vectors s the base to pont the wnner neuron Weghts are modfed by the Grossberg rule If the learnng vectors are smlar the same wnner neuron, the wnner s weghts are the mean values of nput sgnals W X

30 WTM (1) Wnner selecton lke n WTA Wnner s output s maxmum Wnner actvates the neghbourhood neurons Dstance from the wnner drves the level of actvaton Level of actvaton s a part of weght tunng algorthm All weghts are modfed durng learnng algorthm