Learning & Neural Networks

Transcription

1 BCS-PAR Summer School on Paern Recogniion Eeer 003 Learning & Neural Neworks Regression & Learning Learning Paradigm &Nework Archiecure Percepron & Mulilaer Percepron RBF & SVM Learning & Generalisaion

2 BCS-PAR Summer School on Paern Recogniion Eeer 003 Regression & Learning Saisical Approach - Regression In simple saisical erms, learning refers o esimaion of he parameers of a probabili model or a paern classifier from a se of observed samples. p ξ,, if f, ξ > λ f, ξ 0, oherwise Baes eends he saisics o beond mere collecions of numbers. Baesian mehods view he parameers as random variables having some priori disribuions. Learning is an effec of revising our opinion abou he rue values of he parameers, afer observing addiional samples. p ξ p ξ p ξ χ p ξ p ξ dξ More general saisical mehod is so called regression, which ries o fi a funcion, linear or nonlinear in parameric or nonparameric forms, o he observaions. min f, ξ

3 BCS-PAR Summer School on Paern Recogniion Eeer 003 Regression & Learning Saisical Approach - Regression Eample: Manufacuring widges - linear regression model Form-Board Tes X i Widges/hr Y i Form-Board Tes Widges/hr Y' i a+bx i Y' i a+bx i X i Y i a3, b-.5 Y i -Y' i a4, b- Y i -Y' i Toal4.5 Toal55

4 BCS-PAR Summer School on Paern Recogniion Eeer 003 Regression & Learning Saisical Approach - Regression Eample: Manufacuring widges - Linear Regression Model - Leas Square Mehod min b a Yi Yi ' min X iyi X i Yi N N X i i b Y N N X X i i Y i a bx i The opimal values: b , a40.0 Y X

5 BCS-PAR Summer School on Paern Recogniion Eeer 003 Regression & Learning Arificial Inelligence Approach op-down approach AI ssems or ssems wih AI characers recreae/simulae human inelligence or informaion processing and response skills using cogniive, pschological or linguisic models. Represenaion, rules and semanics are he ke elemens. Learning, ofen referred o as Machine Learning, is o elici and refine he knowledge and covers a number of aciviies such as reasoning, undersanding, modelling and rule eracing. If... Then decision rees form he logic operaion. AI depends on a broad ineracion beween compuing and man disciplines and fields, including logic, pscholog, linguisics, philosoph, neuroscience, mechanical engineering, saisics, economics, and conrol heor, and man ohers. AI does no produce sand-alone ssems, bu insead adds knowledge and reasoning o eising applicaions, daabases, and environmens, o make hem more friendl, smarer, and more sensiive o user behaviour and changes in heir environmens.

6 BCS-PAR Summer School on Paern Recogniion Eeer 003 Regression & Learning Arificial Neural Neworks Approach boom-up approach NNs sared on he findings and discover in neurobilog and neuroscience. We graduall realise ha human brains bear lile resemblance o he von Neumann pe of compuers. The human brain has abou 0 billion inerconneed neurons, Each neuron is a cell ha uses biochemical reacions o receive, process and ransmi simulus. Each neuron is a simple processing uni, which receives hrough dendries and snapses simuli inpu from roughl 000 neighbouring neurons aons, cumulaes hese simuli in soma and fires hrough is aon o is oupu neurons if he aggregaed inpu is over a hreshold hillock. Neworks of hese cells form he basis of informaion processing.

7 BCS-PAR Summer School on Paern Recogniion Eeer 003 Human Visual Passwa There are abou 00 millions of phoosensiive cells in human reina, bu onl million opic nerves connecing beween reina and core.

8 BCS-PAR Summer School on Paern Recogniion Eeer 003 Regression & Learning Arificial Neural Neworks Approach boom-up approach Bilogical neurons

9 BCS-PAR Summer School on Paern Recogniion Eeer 003 Regression & Learning Arificial Neural Neworks Approach boom-up approach From regression poin of view, NNs are non- or semi-parameric approach o funcion approimaion. However, he funcion form is represened b a neural nework, or inerconneced unis, and parameers are he connecing weighs. The focus and aenion in NNs research have been in he archiecure of he nework and learning or raining mehod or algorihm ha adjuss he connecing weighs so ha he nework will perform a specific ask wih a required precision. In NN erms, Learning is a process b which he free parameers of neural neworks are adaped hrough a process of simulaion b he environmen in which he nework is embedded. The pe of learning is deermined b he manner in which he parameer changes ake place. Hakin 994

10 BCS-PAR Summer School on Paern Recogniion Eeer 003 Supervised Learning Learning Paradigms Error-correcion learning Reinforcemen learning Unsupervised Learning Hebbian learning Compeiive learning Self-organisaion min d, w w αe d,, ma Σreward { s, a} a π arg ma{ r s, a +γv s, a} w α αf, i, if neuron i wins wi 0, oherwise αi, if neuron i close o winner Global order can arise from local wi ineracions Turing 95 0, oherwise a

11 BCS-PAR Summer School on Paern Recogniion Eeer 003 Nework Archecures Weighs Weighs v Feed-forward Neworks Percepron and mulilaer percepron Radial basis funcion Suppor vecor machine Recurren Neworks Hopfiled newroks Bolzmann machine 3 Weighs : v v v 3 3 n- n : Inpu laer 3 v v 3 : v nh Hidden laer 3 Oupu laer n n v nh

12 BCS-PAR Summer School on Paern Recogniion Eeer 003 Percepron bias b w w S T w+b ϕ. ϕ T w+b n w n Inpu signals Snapic weighs Adder Acivaion funcion ϕv Oupu 0 v ϕv There are abou 00 millions of phoosensiive cells in human reina, bu onl million opic nerves connecing beween reina and core. 0 v

13 Percepron BCS-PAR Summer School on Paern Recogniion Eeer 003 Percepron Training - Leas mean square mehod: n k T k k w 0 w ] [ e E J e e e J w w e J ] [ d e J T w w w w w w α α α d e wihou hresholding or acivaion funcion

14 BCS-PAR Summer School on Paern Recogniion Eeer 003 Eample: Regression Inpu signal w bias b S Snapic weighs aw+b Adder Percepron Oupu w + w + α Form-Board Tes [ w b ] X i Widges/hr Y i b + b + α[ w b ] Observaion: Difficul o converge even for such a simple eample. I converges onl afer carefull Final vales : learning raes, e.g. α 0.00, α 0., someimes w ~.05 needs good iniialisaion, e.g. w 0-0., b0. b ~ 39.4 Oher limiaions: lineari, local minima,...

15 BCS-PAR Summer School on Paern Recogniion Eeer 003 Eample: classificaion Percepron b n w w w n S T w+b + w + w + α [ d ] b + b + α[ d ] Observaion: I can alwas achieve 00% classificaion rae on raining se, bu no alwas 00% on esing se. There are errors in abou in 3 runs for esing se. Or he average esing rae is afer man runs wih various groupings and iniial condiions 84.4%. Oher limiaions: lineari, local minima,... - sign T w+b 0

16 BCS-PAR Summer School on Paern Recogniion Eeer 003 Mulilaer Perceprons Biological neuronal neworks

17 BCS-PAR Summer School on Paern Recogniion Eeer 003 Mulilaer Perceprons Arifical neural neworks Weighs Weighs v v 3 n- : : v 3 3 n v nh Inpu laer Hidden laer Oupu laer

18 Sep : Sep : Sep 3: Sep 4: BCS-PAR Summer School on Paern Recogniion Eeer 003 Mulilaer Perceprons Training: Back-Propagaion algorihm Iniialisaion. Se all weighs and nodes biases o small random numbers. Presenaion of raining eamples. inpu [,, n ] T and desired oupu d[d, d,..d m ] T. Forward compuaion. Calculae he oupus of he hidden and oupu laer, k n h o o o h h h ϕ k w jkv j + bk v j ϕ j wij i + b j j Backward compuaion and updaing weighs. Compue he error erms, δ o k o o o n i ek ϕk ' ek k k k k dk k w o jk w o jk + αδ o k v j δ h j h j m k o k o jk ϕ ' δ w v v δ j j m k o k w o jk w h ij w h ij + αδ h j i Sep 5: Ieraion. Repea seps 3 and 4 and sop when he oal error reaches he required level.

19 BCS-PAR Summer School on Paern Recogniion Eeer 003 Mulilaer Perceprons Eample: Regression Form-Board Tes Widges/hr X i Y i Y hidden neurons, X linear oupu neuron ϕ v + e Limiaions: local minima, slow convergence, sensiive o iniial and oher parameers. av a 0 ϕv v

20 BCS-PAR Summer School on Paern Recogniion Eeer 003 Radial Basis Funcions The RBF model cam from wo fundamenal heories The Universal Approimaion Theor: an funcion can be approimaed wih arbirar precision b a weighed sum of a se of non-consan, bounded and monoone-increasing coninuous funcions, fˆ M i w i ϕ, ξ Cover s Separabili Theor: A comple paern-classificaion problem cas in a high-dimensional space nonlinearl is more likel o be linearl separable ha in a low high dimensional space.

21 BCS-PAR Summer School on Paern Recogniion Eeer 003 Radial Basis Funcions Weighs Weighs v v 3 n- n : Inpu laer : v 3 v nh Hidden laer ϕv 3 Oupu laer Properies: Hidden laer has gaussian nodes, Oupu laer is linear Eas o rain Fas o converge. 0 v

22 BCS-PAR Summer School on Paern Recogniion Eeer 003 Suppor Vecor Machines Linear classifier: sign T w+b. Which one is he bes, in he sense of generalisaion, i.e. performance on unseen daa?

23 BCS-PAR Summer School on Paern Recogniion Eeer 003 Suppor Vecor Machines Linear SVMs: Quadraic Programming Maimise he margin, i.e min w T w subjec o i it w+b- 0 N T Or min{ w w + C } ξ i i subjec o i it w+b -ξ I, & 0 ξ I, Using kernels can ransfer a nonlinear problem o a linear one.

24 BCS-PAR Summer School on Paern Recogniion Eeer Learning&Generalisaion Learning fiing & generalisaion: On one hand, we wan o ge as high classificaion or low error rae as possible for he raining daa se. On he oher hand, we would also like he rained nework o have good performance on unseen esing daa. 6 Y Which is beer? Y X X

25 Learning&Generalisaion BCS-PAR Summer School on Paern Recogniion Eeer 003 Bias & Variance dilemma: Φ d p d ] [ ] } { }][ { [ ] } { [ }] { [ ] } { } { [ ] [ d E E d E E d E E d }]} { {[ ] } { [ } ] {[ E E d E d E + bias variance A close fier/model ields small bias bu large variance, while a loose fier or a wild guess gives small variance bu large bias.

26 Furher Readings Bishop, C. Neural Neworks for Paern Recogniion, Oford, 995. Duda, R.O, D.G. Sork, P.E. Har, Paern Classificaion and Scene Analsis, John Wile & Sons, 00 Hakin, S. Neural Neworks: A Comprehensive Foundaion, Prenice Hall, 999. Kohonen, T., Self-Organising Maps, nd Ed. Springer, 997 Lippman, R.P., An Inroducion o Compuing wih Neural Neworks, IEEE ASSP Magzine, pp. 4-, April 987. Riple, B.D. Paern Recogniion and Neural Neworks, Cambridge, 996 Suon, R.S. and A.G. Baro, Reinforcemen Learning: An Inroducion, MIT Press, 998, A Bradford Book. www-anw.cs.umass.edu/~rich/book/he-book.hml. Vapnik, V.N., The Naure of Saisical Learning Theor, Springer, 995.