Internet Engineering. Jacek Mazurkiewicz, PhD Softcomputing. Part 1: Introduction, Elementary ANNs

Transcription

1 Iere Egieerig Jacek azurkieicz, PhD Sofcompuig Par : Iroducio, Elemeary ANNs

2 Formal Iroducio coac hours, room No. 5 buildig C-3: oday: :00-4:00, Tuesday eve oly: 9:00 - :00, Wedesday eve oly: 3:00 5:00 slides:.zsk.ic.pr.roc.pl Professor Wikor Zi es: durig lecure - sofcompuig: - lecure + projec - projec mark 0% of fial mark, bous quesio!

3 Program Idea of iellige processig Fuzzy ses ad approimae reasoig Eper sysems - koledge base orgaizaio Eper sysems - reasoig rules creaio Eper sysems: ypical orgaizaio ad applicaios Arificial eural eorks: learig ad rerievig algorihms ulilayer percepro, RBF Kohoe eural eork, CNN Hopfield eural eork Hammig eural eork Arificial eural eorks: applicaios Geeic algorihms: descripio ad classificaio Geeic algorihms: basic mechaisms ad soluios

4 SUBJECT OBJECTIVES C. Koledge of arificial eural eorks i paer recogiio, digial sigals ad daa processig: opology of eorks, ifluece of parameers for eork behavior. C. Koledge of geeic algorihms used for daa pre- ad posprocessig. C3. Koledge of eper sysems reasoig rules ad koledge base creaio for differe asks. C4. Skills of special evirome usage for projec phase, modelig ad simulaio of sofcompuig sysems i case of differe scieific problems. SUBJECT EDUCATIONAL EFFECTS relaig o koledge: PEK_W0 kos he rules ad he idea of iellige processig. PEK_W0 defies he fuzzy ses ad udersads he idea of approimae reasoig. PEK_W03 defies he koledge base ad reasoig rules, kos he eper sysems cosrucio. PEK_W04 kos he archiecure of ypical arificial eural eorks srucures, learig ad rerievig algorihms, applicaios. PEK_W05 kos he descripio, classificaio, eamples of applicaios of geeic algorihms relaig o skills: PEK_U0 ca use he eviromes for projec phase, modelig ad simulaio of arificial eural eorks as ell as geeic algorihms i differe asks abou paer digial sigals recogiio. PEK_U0 ca use he eviromes for projec phase, modelig ad implemeaio of eper sysems o dedicaed fields of koledge. PEK_U03 ca use he eviromes for projec phase, modelig ad implemeaio of fuzzy ses ad fuzzy reasoig o dedicaed fields of koledge.

5 Lieraure B. Boucho euier, Fuzzy Logic ad Sof Compuig O. Casilo, A. Boarii, Sof Compuig Applicaios. Caudill, Ch. Buler, Udersadig Neural Neorks E. Damiai, Sof Compuig i Sofare Egieerig R. Hech-Nielse, Neurocompuig S. Y. Kug, Digial Neural Neorks D. K. Praihar, Sof Compuig S. N. Sivaadam, S. N. Deepa, Priciples of Sof Compuig A. K. Srivasava, Sof Compuig D. A. Waerma, A Guide o Eper Sysems D. Zhag, Parallel VLSI Neural Sysem Desig

6 Why Neural Neorks ad Compay? Sill i acive use No chace o solve some problems i oher ay Huma abiliy vs. classical programs Works as primiive huma s brai Arificial ielligece has poer! ANN + Fuzzy Logic + Eper Sysems + Rough Ses + A Algorihms = SofCompuig

7 The Sory 943 cculloch & Pis model of arificial euro 949 Hebb iformaio sored by biological eural es 958 Rosebla percepro model 960 Widro & Hoff firs eurocompuer - adalie 969 isky & Paper XOR problem sigle-layer percepro limiaios 986 cclelad & Rumelhar backpropagaio algorihm

8 Where Sofcompuig is i Use? Leers, sigs, characers, digis recogiio Recogiio of ship ypes daa from soar Elecric poer predicio Differe kids of simulaors ad compuer games Egie diagosic i plaes, vehicles Rock-ype ideificaio Bomb searchig devices

9 Neural Neorks Realisaio Se of coeced ideical euros Arificial euro based o a biological euro Hardare realisaio digial device Sofare realisaio simulaors Arificial eural eork idea, algorihm, mahemaical formulas Works i parallel No programmig learig process ecessary

10 Learig Nauczyciel Teacher Wih a Teacher Wekor Learig cech (dae vecor auki) Wyik Resul of klasyfikacji learig Parameers Klasyfikaor Weighs Wihou a Teacher Wekor Learig cech (dae vecor esoe) Parameers Klasyfikaor Weighs Wyik Resul of klasyfikacji learig

11 Sofcompuig vs. Classical Compuer Differe limiaios of sofcompuig mehods No sofcompuig: operaios based o symbols: ediors, algebraic equaios calculaios ih a high level of precisio Sofcompuig is very ice, bu o as uiversal as compuer

12 brai brai ser cerebellum Aaomy Foudaios () prologed cord Nervous Sysem -ays, symmerical se of srucures, divided io 4 pars: Spial Cord receivig ad rasmissio of daa spial cord Prologed Cord breahig, blood sysem, digesio Cerebellum moveme corol ervous sysem Brai (ca..3 kg) hemispheres feelig, hikig, moveme

13 Aaomy Foudaios ()

14 Aaomy Foudaios (3) Cerebral core hickess: mm, area: ca..5 m Cerebral core divided io 4 par lobes Each lobe is corrugaed Each hemisphere is resposible for half par of body: righ for lef par, lef for righ par Hemispheres are ideical i case of a srucure, bu heir fucios are differe

15 Aaomy Foudaios (4) Brai composed by fibres ih large umber of braches To ypes of cells i ervous issue: euros ad gley cells There are more gley cells: o daa rasfer amog euros caerig fucios Ca. 0 milliard euros i cerebral core Ca. 00 milliard euros i hole brai Neuro: dedries ipus, ao oupu, body of euro Neuro: housads of syapses coecios o oher euros

16 Aaomy Foudaios (5) Neuros i ork: chemical-elecrical sigal rasferrig cell geeraes elecrical sigals elecric pulse is chaged io a chemical sigal a he ed of ao chemical ifo passed by eurorasmiers 50 differe ypes of euros euros drive by a frequecy of hudreds of Hz euros are raher lo devices!

17 Aaomy Foudaios (6)

18 Biological ad Arificial Neural Nes Arificial eural eorks are a good soluio for: esig already ideified biological sysems paer recogiio aleraive cofiguraios o fid he basic feaures of hem Arificial eural eorks are primiive brohers of biological es Biological es have sophisicaed ieral feaures impora for heir ormal ork Biological es have sophisicaed ime depedeces igored i mos arificial eorks Biological coecios amog euros are differe ad complicaed os archiecures of arificial es are urealisic from he biology poi of vie os learig rules for arificial eorks are ureal i biology poi of vie os biological es e ca compare o already leared arificial es o realise fucio described i a very deailed ay

19 Liear ANN - ADALINE (ADAive Liear Neuro) 0 sigle euro s aser: y = j= j j + 0 y ~ ) = j= 0 T ( j j = ~ scalar descripio vecor descripio... + y umber of ipu euros K umber of oupu euros ~ = col ( 0,,..., ) = col(,,..., 0 = muli-oupu e: 0 )

20 Sigle-Layer uli-oupu Neork y y y K k-euro s aser: y K () = j= 0 kj j Ipu euro W kj Oupu euro y() T = colum y(x) = WX 0 0 K0 K K K W = 0 0 K 0 K K

21 Learig Procedure eperimeal daa: N - series N,...,, N K K K,...,, learig daa required asers, N K N fucio implemeed by e error fucio mea-square error: ( ) ( ) = = = N K k y k k W E ) ( = = = = N K k j k j jk W E 0 ) ( lookig for a miimum of E(W) fucio: 0 ) (, = kj j k W E

22 Pseudoiverse Algorihm = = = = N j j k j kj kj W E 0 ' ' 0 ) ( = = = = N j N j k j j kj j k 0 ' ', here: = N N X = N K N N K K T = K K K W fially: ( ) T X X W X T T T = T XW T = T X X) (X W T T T = pseudoiverse = τ T, X W τ T

23 y Gradie-Type Algorihms () ieraive approach: seps: radom eigh vecor e eigh vecor folloig: - E + repea process geeraig he sequece of eighs vecors: compoes of eigh vecors calculaed by error fucio: E ( ) = E ( ) = ( ) y ( + ) E( kj = kj ) kj E () = K k= j= 0 kj j k

24 sequeial approach: Gradie-Type Algorihms () E E ( + ) kj = kj = yk ( ) k * j = k * j kj kj error dela rule: k = yk ( ) algorihm: k Widro-Hoff rule: ( + ) kj = jk. se sar values by a radom ay for eample. calculae a e aser for available 3. calculae a error value k 4. calculae a e eigh vecor kj (+) accordig o he dela rule 5. repea seps. 4. uil E less ha required value k j

25 0 Percepro () he sory: i i y Rosebla (96) i classificaio ask Widro & Hoff (960) - ADALINE i aser: y( X ) = g g( T j j = ) j= 0 acivaio fucio: for a 0 g( a) = for a 0 bipolar for a 0 g( a) = 0 for a 0 uipolar 0 hreshold value a = j = 0 j j

26 error fucio: E jk percepro crierios: Percepro () does o eis, because g(a) is o differeiable compare he acual value of y i ad he required oupu value d i ad: if y i = d i he eighs values of W ij ad 0 are uchaged if y i = 0 ad he required value d i = updae he eighs as follo: W + ) = W ( ) +, b ( + ) = b ( ) +, ij ( ij j here: previous cycle, + acual cycle if y i = ad d i = 0 updae he eighs accordig o: W ij ( ij j + ) = W ( ), b ( + ) = b ( ) here: b i polariy, d i required euro s oupu sigal i i i i

27 summarisig:. look-up he ipu learig vecors Percepro (3). if classificaio is correc eighs are o chaged 3. if classificaio is rog: if = + add o he eigh values else subrac from he eigh values value of is o impora ca be se o, i oly scales i E = p k= ( k ) ( y i d ( k ) i ),

28 Percepro Problems () liear separabiliy: XOR problem isky & Paper (969): X I I I I Ou C C y()=0 X XOR Ou o-liear separable problem soluio: mulilayer e

29 Percepro Problems () mulilayer eork for XOR problem soluio: = s s = - = S