Introduction to Artificial Neural Networks EE /09/2015. Who am I. Associate Prof. Dr. Turgay IBRIKCI. What you learn from the course

Transcription

1 Introducton to Artfcal Neural Netorks EE-589 Who am I Assocate Prof Dr Turga IBRIKCI Room # 305 Tuesdas 3:30-6:00 2 (322) / 39 eembderslerordpresscom What ou learn from the course Course Outlne 3 Ho to approach a neural netork learnng classfcaton or clusterng Basc knoledge of the common lnear machne learnng algorthms Basc knoledge of Neural Netorks learnng algorthms A good understandng of neural netork algorthms 4 The course s dvded n to parts: theor and practce Theor covers basc topcs n neural netorks theor and applcaton to supervsed, unsupervsed and renforcement learnng 2 Practce deals th bascs of MATLAB and mplementaton of NN learnng algorthms We assume that ou kno MATLAB or ou ll learn ourself Course Gradng Project 5 Gradng the Class: Project 40% Proposal 5%( 3/03/205) Project paper 35% (CD -ll be n paper publcaton format) Presentaton (Week 4 ; 20 mns) Fnal Eam 20% (Week 5-We decde together) Homeorks 40% (At least 4 homeorks) Full attendng the class 0% (Bonus) 6 Choose a topc that s related to our research nterest and pertans to the course materal Dscuss th our supervsor The proposal should nclude the follong sessons (Due: 3/03) the project goal, the problems to be studed, overve of current methods, proposed methods, epected results, and references (about 4 pages: sngle space, fond sze = 2, references are not counted) References should be cted n the proposal A rtten fnal paper n the stle of a journal artcle n the follong slce s also requred Fnal paper s due b 2/2/205-soft&harcop Each student ll gve classroom presentaton-ppt about the fnal paper

2 Fnal Paper WARNING!!!!!!!!! 7 The paper should not eceed 8 pages (ecludng append) n the format provded at IEEE Manuscrpt Templates for Conference Proceedngshttp://eeeorg/conferencesevents/conferences/publshng/templ ateshtml : The paper should address the follong topcs: Introducton general descrpton of the project topc/research paper The problem statement h s an artfcal neural netork requred? What are the ssues? Objectve hat are ou proposng to do/sho? Approach descrbe the method ou have adopted Implementaton results nclude our graphs, tables, etc here Conclusons hat have ou learned? Ho can the algorthm be mproved? Append lstng of the source code (eclude these pages from the total page count) 8 IMPORTANT NOTE: We don t accept late materals at all We ll take our sgnature ever class durng the course that ou attend Academc Integrt Where to go for help All programmng s to be done alone! You can dscuss our programs th anone! Do not share code th anone else n the course (lookng at the code counts as sharng)! Comparson of homeorks ll be to catch cheaters! Mnmum penalt s 2-letter-grade-drop for the course for everone nvolved Feel free to send our code to turgabrkc@hotmalcom and ask me for help Please rte our name and coursenumber to the subject lne Brng our code to the class 9 0 Introducton to Artfcal Neural Netorks Fundamental Concepts of Intellgent Contents Fundamental Concepts of Intellgent Fundamental Concepts of ANNs Basc Models and Learnng Rules Neuron Models ANN structures Learnng Dstrbuted Representatons Conclusons 2 2

3 Intellgent machnes, or hat machnes can do What s Intellgence? Phlosophers have been trng for over 2000 ears to understand and resolve to Bg Questons of the Unverse: Ho does a human mnd ork? Can non-humans have mnds? These questons are stll unansered 3 Then e need to ask that hat Intellgence s 4 Intellgence: the capact to learn and solve problems (Websters dctonar) n partcular, the ablt to solve novel problems the ablt to act ratonall the ablt to act lke humans Artfcal Intellgence buld and understand ntellgent enttes or agents 2 man approaches: engneerng versus cogntve modelng Intellgent machnes Intellgent machnes 5 We can defne ntellgence as the ablt to learn and understand, to solve problems and to make decsons The goal of artfcal ntellgence (AI) as a scence s to make machnes do thngs that ould requre ntellgence f done b humans Therefore, the anser to the queston Can Machnes Thnk? as vtall mportant to the dscplne The anser s not a smple Yes or No 6 As humans, e all have the ablt to learn and understand, to solve problems and to make decsons; hoever, our abltes are not equal n dfferent areas Therefore, e should epect that f machnes can thnk, some of them mght be smarter than others n some as Turng Imtaton Game Turng Imtaton Game 7 One of the most sgnfcant papers on machne ntellgence, Computng Machner and Intellgence, as rtten b the Brtsh mathematcan Alan Turng over fft fve ears ago Hoever, t stll stands up ell under the test of tme, and the Turng s approach remans unversal Turng dd not provde defntons of machnes and thnkng, he just avoded semantc arguments b nventng a game, the Turng Imtaton Game 8 The mtaton game orgnall ncluded to phases In the frst phase, the nterrogator, a man and a oman are each placed n separate rooms The nterrogator s objectve s to ork out ho s the man and ho s the oman b questonng them 3

4 Turng Imtaton Game: Phase Turng Imtaton Game: Phase 2 In the second phase of the game, The man s replaced b a computer programmed to deceve the nterrogator as the man dd It ould even be programmed to make mstakes and provde fuzz ansers n the a a human ould If the computer can fool the nterrogator as often as the man dd, e ma sa ths computer has passed the ntellgent behavour test 9 20 Turng Imtaton Game: Phase 2 The Result of Turng Machnes 2 Turng beleved that b the end of the 20th centur t ould be possble to program a dgtal computer to pla the mtaton game Although modern computers stll cannot pass the Turng test, t provdes a bass for the verfcaton and valdaton of knoledge-based sstems A program ntellgence, n some narro area of epertse, s evaluated b comparng ts performance th the performance of a human epert To buld an ntellgent computer sstem, e have to capture, organze and use human epert knoledge n 22 some narro area of epertse What s Neural Netorkng?? Human Intellgence VS Artfcal Intellgence Artfcal neural netorks are composed of nterconnectng artfcal neurons (programmng constructs that mmc the propertes of bologcal neurons)

5 Human Intellgence VS Artfcal Intellgence Human Intellgence Artfcal Intellgence Human Intellgence VS Artfcal Intellgence Human Intellgence Artfcal Intellgence 25 Intuton, Common sense, Judgment, Creatvt, Belefs etc The ablt to demonstrate ther ntellgence b communcatng effectvel Plausble Reasonng and Crtcal thnkng Ablt to smulate human behavor and cogntve processes Capture and preserve human epertse Fast Response The ablt to comprehend large amounts of data quckl Humans are fallble The have lmted knoledge bases Informaton processng of seral nature proceed ver slol n the bran as compared to computers Humans are unable to retan large amounts of 26 data n memor No common sense Cannot readl deal th med knoledge Ma have hgh development costs Rase legal and ethcal concerns Human Intellgence VS Artfcal Intellgence Artfcal Intellgence VS Conventonal Computng Artfcal Intellgence Conventonal Computng 27 We acheve more than e kno We kno more than e understand We understand more than e can eplan (Claude Bernard, 9th C French scentfc phlosopher) 28 AI softare uses the technques of search and pattern matchng Programmers desgn AI softare to gve the computer onl the problem, not the steps necessar to solve t Conventonal computer softare follo a logcal seres of steps to reach a concluson Computer programmers orgnall desgned softare that accomplshed tasks b completng algorthms Artfcal ntellgence & Our socet Pscholog And Artfcal ntellgence Wh e need AI?? The functonalst approach of AI ves the mnd as a representatonal sstem and pscholog as the stud of the varous computatonal processes hereb mental representatons are constructed, organzed, and nterpreted (Margaret Boden's essas rtten beteen 982 and 988) To supplement natural ntellgence for eg e are buldng ntellgence n an object so that t can do hat e ant t to do, as for eample-- robots, thus reducng human labour and reducng human mstakes

6 Introducton to Artfcal Neural Netorks Fundamental Concepts of ANNs What s ANN? Wh ANN? ANN Artfcal Neural Netorks To smulate human bran behavor A ne generaton of nformaton processng sstem 3 32 Applcatons Pattern Matchng Pattern Recognton Assocate Memor (Content Addressable Memor) Functon Appromaton Learnng Optmzaton Vector Quantzaton Data Clusterng Tradtonal Computers are neffcent at these tasks although ther computaton Applcatons speed s faster Pattern Matchng Pattern Recognton Assocate Memor (Content Addressable Memor) Functon Appromaton Learnng Optmzaton Vector Quantzaton Data Clusterng The Confguraton of ANNs The Bologc Neuron An ANN conssts of a large number of nterconnected processng elements called neurons A human bran conssts of ~0 neurons of man dfferent tpes Ho ANN orks? Collectve behavor

7 The Bologc Neuron The Bologc Neuron Connecton pont tal Ectator or Inhbtor 37 chemcal 38 The Artfcal Neuron The Artfcal Neuron-Perceptron ( t ) a( f ) m m 40 f ( m ) m j m j j a( f ) 0 f 0 otherse The Artfcal Neuron 2 2 j postve ectator negatve nhbtor zero no connecton The Artfcal Neuron 2 2 Proposed b McCulloch and Ptts [943] M-P neurons 4 m m 42 m m 7

8 What can be done b M-P neurons? What ANNs ll be? A hard lmter A bnar threshold unt Hperspace separaton f ( ) 2 2 f ( ) 0 0 otherse ANN A neurall nspred mathematcal model Conssts a large number of hghl nterconnected PEs Its connectons (eghts) holds knoledge The response of PE(Process Element) depends onl on local nformaton Its collectve behavor demonstrates the computaton poer Wth learnng, recallng and, generalzaton capablt Three Basc Enttes of ANN Models Models of Neurons or PEs Models of snaptc nterconnectons and structures Tranng or learnng rules Introducton to Artfcal Neural Netorks Basc Models and Learnng Rules Neuron Models ANN structures Learnng Processng Elements Integraton Functons Etensons of M-P neurons What ntegraton functons e ma have? 47 What actvaton functons e ma have? 48 M-P neuron Quadratc Functon Sphercal Functon Polnomal Functon f net m j f 2 j j m f ( j j m j j j ) 2 j m m j k j k f jk j k j k 8

9 Actvaton Functons Actvaton Functons M-P neuron: (Step functon) Hard Lmter (Threshold functon) a( f ) 0 f 0 otherse a a( f ) sgn( f ) a f 0 f 0 49 f 50 f Actvaton Functons Actvaton Functons Ramp functon: Unpolar sgmod functon: a( f ) f 0 f 0 f f 0 a a( f ) e 5 f 5 f Actvaton Functons Eample: Actvaton Surfaces Bpolar sgmod functon: L 2 a( f ) f e 5 05 L 3 L 2 L L 2 L

10 Eample: Actvaton Surfaces Eample: Actvaton Surfaces L =0 00 L Regon Code 55 L 3 +4=0 L 2 =0 = 2 = 3 = 4 L L 2 L L L L L 2 L 3 Eample: Actvaton Surfaces Eample: Actvaton Surfaces L z=0 L 4 z L z=0 L 4 z 4 =25 L 3 L 3 z= L 2 L L 2 L 3 z= L 2 L L 2 L Eample: Actvaton Surfaces Unpolar sgmod functon: a( f ) e Eample: Actvaton Surfaces f M-P neuron: (Step functon) a( f ) 0 f 0 otherse L 4 z =2 =3 L 4 z L L 2 L 3 L L 2 L 3 =5 =

11 Introducton to Artfcal Neural Netorks ANN Structure (Connectons) Basc Models and Learnng Rules Neuron Models ANN structures Learnng 6 62 Sngle-Laer Feedforard Netorks Multlaer Feedforard Netorks 2 n 2 n m 2 2 2m n 22 n2 nm Output Laer Hdden Laer 63 2 m 64 Input Laer 2 m Pattern Recognton Multlaer Feedforard Netorks Sngle Node th Feedback to Itself Where the knoledge from? Learnng Input Classfcaton Analss Output Feedback Loop 65 66

12 Sngle-Laer Recurrent Netorks 2 n Multlaer Recurrent Netorks m Introducton to Artfcal Neural Netorks Basc Models and Learnng Rules Neuron Models ANN structures Learnng 70 Learnng Consder an ANN th n neurons and each th m adaptve eghts Weght matr: T T 2 W T n n n2 m 2m nm 7 Learnng Consder an ANN th n neurons and each th m adaptve eghts Weght matr: T T 2 W T n n n2 Ho? To Learn the eght matr m 2m nm 72 Learnng Rules Supervsed learnng Unsupervsed learnng Renforcement learnng 2

13 Supervsed Learnng () () (2) (2 ( k) T d d d Supervsed Learnng ) ( k (, ),(, ),,(, ) ), 73 Learnng th a teacher Learnng b eamples Tranng set T d d d () () (2) (2 ) ( k) (, ),(, ),,(, ( k ) ), 74 ANN W Error sgnal Generator d Unsupervsed Learnng Unsupervsed Learnng Self-organzng Clusterng Form proper clusters b dscoverng the smlartes and dssmlartes among objects ANN W Renforcement Learnng Renforcement Learnng 77 Learnng th a crtc Learnng b comments 78 ANN W Crtc sgnal Generator Renforcement Sgnal 3

14 We ant to learn the eghts & bas The General Weght Learnng Rule The General Weght Learnng Rule j j m-,m- Input: net m j Output: a( net ) j j j j m-,m- Input: net m j Output: a( net ) j j We ant to learn the eghts & bas The General Weght Learnng Rule The General Weght Learnng Rule j j m-,m- Input: net net m j j m j j j Let m = and m = j 82 m 2 Input: net jj 2 j j j m m-,m- net jj m = j m = Let m = and m = We ant to learn =(, 2,, m ) T The General Weght Learnng Rule The General Weght Learnng Rule j j m-,m- m = m = Input: net m j (t) =? j j 84 r Learnng Sgnal Generator d 4

15 The General Weght Learnng Rule The General Weght Learnng Rule 85 r Learnng fr( Sgnal,, d ) Generator d 86 ( t) r( t) r Learnng fr( Sgnal,, d ) Generator d ( t) r( t) The General Weght Learnng Rule We ant to learn =(, 2,, m ) T The General Weght Learnng Rule 87 ( t) r( t) r Learnng Rate Learnng fr( Sgnal,, d ) Generator d 88 ( ) r( t) t r f (,, d ) Dscrete-Tme Weght Modfcaton Rule: r f (,, d ) ( t ) ( t ) ( t ) ( t ) ( t ) ( t ) r Contnuous-Tme Weght Modfcaton Rule: d () t r() t dt Hebb s Learnng La Hebb s Learnng La Hebb [949] hpothess that hen an aonal nput from A to B causes neuron B to mmedatel emt a pulse (fre) and ths stuaton happens repeatedl or persstentl Then, the effcac of that aonal nput, n terms of ablt to help neuron B to fre n future, s someho ncreased T r f (,, d ) a( ) r ( t) r( t) j j 89 Hebb s learnng rule s a unsupervsed learnng rule

16 Introducton to Artfcal Neural Netorks Dstrbuted Representatons Dstrbuted Representatons Dstrbuted Representaton: An entt s represented b a pattern of actvt dstrbuted over man PEs Each Processng element s nvolved n representng man dfferent enttes Local Representaton: Each entt s represented b one PE 9 92 Act as a content addressable memor Eample Advantages Dog Cat Bread P 0 P P 2 P 3 P 4 P 5 P 6 P 7 P 8 P 9 P 0 P P 2 P 3 P 4 P Dog Cat Bread P 0 P P 2 P 3 P 4 P 5 P 6 P 7 P 8 P 9 P 0 P P 2 P 3 P 4 P P 0 P P 2 P 3 P 4 P 5 P 6 P 7 P 8 P 9 P 0 P P 2 P 3 P 4 P What s ths? Advantages Act as a content addressable memor Make nducton eas Advantages Act as a content addressable memor Make nducton eas Make the creaton of ne enttes or concept eas (thout allocaton of ne hardare) 95 P 0 P P 2 P 3 P 4 P 5 P 6 P 7 P 8 P 9 P 0 P P 2 P 3 P 4 P 5 Dog Cat Bread P 0 P P 2 P 3 P 4 P 5 P 6 P 7 P 8 P 9 P 0 P P 2 P 3 P 4 P 5 Fdo Dog has 4 legs? Ho man for Fdo? 96 P 0 P P 2 P 3 P 4 P 5 P 6 P 7 P 8 P 9 P 0 P P 2 P 3 P 4 P 5 Dog Cat Bread Doughnut Add doughnut b changng eghts 6

17 Advantages Act as a content addressable memor Make nducton eas Make the creaton of ne enttes or concept eas (thout allocaton of ne hardare) Fault Tolerance Dsadvantages Dog Cat Bread P 0 P P 2 P 3 P 4 P 5 P 6 P 7 P 8 P 9 P 0 P P 2 P 3 P 4 P Ho to understand? Ho to modf? Learnng procedures are requred Some PEs break don don t cause problem REFERENCES All materals ere collected from dfferent eb pages Thanks for the authors of these sldes 99 7