Connectionist Models. Artificial Neural Networks. When to Consider Neural Networks. Decision Surface of Perceptron. Perceptron

Transcription

1 Atfcal Neual Netos Theshol uts Gaet escet Multlaye etos Bacpopagato He laye epesetatos Example: Face ecogto Avace topcs Coectost Moels Cose humas Neuo stchg tme ~. seco Numbe of euos ~ Coectos pe euo ~ 4-5 Scee ecogto tme ~. seco feece step oes ot seem le eough must use lots of paallel computato! Popetes of atfcal eual ets (ANNs: May euo-le theshol stchg uts May eghte tecoectos amog uts Hghly paallel, stbute pocess Emphass o tug eghts automatcally Chapte 4 Atfcal Neual Netos Chapte 4 Atfcal Neual Netos Whe to Cose Neual Netos Iput s hgh-mesoal scete o eal-value (e.g., a seso put Output s scete o eal value Output s a vecto of values Possbly osy ata Fom of taget fucto s uo Huma eaablty of esult s umpotat Examples: Speech phoeme ecogto [Wabel] Image classfcato [Kaae, Balua, Roley] Facal pecto ALVINN ves 7 mph o hghays Shap Staght Shap Left Ahea Rght 4 He Uts 3x3 Seso Iput Reta Chapte 4 Atfcal Neual Netos 3 Chapte 4 Atfcal Neual Netos 4 X Pecepto Decso Suface of Pecepto X W W X X X X W W Σ x f x σ -othese X X f + x x > o( x,..., x -othese Sometmes e ll use smple vecto otato : f x > o(x -othese Chapte 4 Atfcal Neual Netos 5 Repesets some useful fuctos What eghts epeset g(x,x AND(x,x? But some fuctos ot epesetable e.g., ot lealy sepaable theefoe, e ll at etos of these... Chapte 4 Atfcal Neual Netos 6

2 hee Pecepto Tag Rule + η o x t c( x s taget value o s pecepto output η s small costat (e.g.,. calle leag ate Ca pove t ll covege If tag ata s lealy sepaable aη s suffcetly small [ ] Gaet Descet To uesta, cose smple lea ut, hee o + x x Iea :lea 's that mmze the squae eo E o D Whee D s the set of tag examples Chapte 4 Atfcal Neual Netos 7 Chapte 4 Atfcal Neual Netos 8 Gaet Descet Gaet Descet Gaet E[ ],,..., Tag ule : η E[ ].e., η Chapte 4 Atfcal Neual Netos 9 Chapte 4 Atfcal Neual Netos Gaet Descet o o o o o ( x, o x + Gaet Descet GRADIENT DESCENTag _ examples, η Each tag examples s a pa of the fom < x,t >, hee x s the vecto of put values a t s the t ag et output value. η s the leag ate (e.g.,. 5. Italze each to somesmall aom value Utl the temato coto s met, o - Italze each to zeo. - Fo each < x,t > tag _ examples, o Iput the stace x a compute output o * Fo each lea ut eght, o + η o x - Fo each lea ut eght, o Chapte 4 Atfcal Neual Netos Chapte 4 Atfcal Neual Netos

3 Summay Pecepto tag ule guaatee to succee f Tag examples ae lealy sepaable Suffcetly small leag ate η Lea ut tag ule uses gaet escet Guaatee to covege to hypothess th mmum squae eo Gve suffcetly small leag ate η Eve he tag ata cotas ose Eve he tag ata ot sepaable by H Icemetal (Stochastc Gaet Descet Batch moe Gaet Descet: Do utl satsfe:.compute the gaet ED[]. η E [ ] D Icemetal moe Gaet Descet: Do utl satsfe: - Fo each tag example D.Compute the gaet E[]. η E [ ] E D[ ] o D E [ ] o Icemetal Gaet Descet ca appoxmate Batch Gaet Descet abtaly closely f η mae small eough Chapte 4 Atfcal Neual Netos 3 Chapte 4 Atfcal Neual Netos 4 Multlaye Netos of Sgmo Uts Multlaye Decso Space h h 3 3 o h h x x Chapte 4 Atfcal Neual Netos 5 Chapte 4 Atfcal Neual Netos 6 X X X X W W W W Σ Sgmo Ut et x σ ( x s the sgmo fucto o σ ( et + e -x + e σ ( x Nce popety : σ ( x( σ ( x x We ca eve gaet escet ules to ta Oesgmo ut Multlaye etos of sgmo uts Bacpopagato et σ ( x + e -x The Sgmo Fucto output et put Sot of a oue step fucto Ule step fucto, ca tae evatve (maes leag possble Chapte 4 Atfcal Neual Netos 7 Chapte 4 Atfcal Neual Netos 8

4 Eo Gaet fo a Sgmo Ut - D o o o o et o et o o o But e o : So : o σ ( et o ( o et et et ( x x, D o o ( o x, Bacpopagato Algothm Italze all eghts to small aom umbes. Utl satsfe, o Fo each tag example, o. Iput the tag examplea compute the outputs. Fo each output ut δ o ( o o 3. Fo each he ut,, δ o ( o h + hee, η δ x h, h, outputs, h δ 4. Upate each eto eght, Chapte 4 Atfcal Neual Netos 9 Chapte 4 Atfcal Neual Netos Moe o Bacpopagato Gaet escet ove ete eto eght vecto Easly geealze to abtay ecte gaphs Wll f a local, ot ecessaly global eo mmum I pactce, ofte os ell (ca u multple tmes Ofte clue eght mometum α η δ x + α (, (,, Mmzes eo ove tag examples Wll t geealze ell to subsequet examples? Tag ca tae thousas of teatos -- slo! Usg eto afte tag s fast He Laye Repesetatos Outputs Iput Output Iputs Chapte 4 Atfcal Neual Netos Chapte 4 Atfcal Neual Netos He Laye Repesetatos Output Ut Eo ug Tag Outputs Iputs Iput Output Sum of squae eos fo each output ut Chapte 4 Atfcal Neual Netos 3 Chapte 4 Atfcal Neual Netos 4

5 He Ut Ecog Iput to He Weghts He ut ecog fo oe put Weghts fom puts to oe he ut Chapte 4 Atfcal Neual Netos 5 Chapte 4 Atfcal Neual Netos 6 Covegece of Bacpopagato Gaet escet to some local mmum Pehaps ot global mmum Mometum ca cause quce covegece Stochastc gaet escet also esults faste covegece Ca ta multple etos a get ffeet esults (usg ffeet tal eghts Natue of covegece Italze eghts ea zeo Theefoe, tal etos ea-lea Iceasgly o-lea fuctos as tag pogesses Chapte 4 Atfcal Neual Netos 7 Expessve Capabltes of ANNs Boolea fuctos: Evey Boolea fucto ca be epesete by eto th a sgle he laye But that mght eque a expoetal he umbe of puts he uts Cotuous fuctos: Evey boue cotuous fucto ca be appoxmate th abtaly small eo by a eto th oe he laye [Cybeo 989; Ho et al. 989] Ay fucto ca be appoxmate to abtay accuacy by a eto th to he layes [Cybeo 988] Chapte 4 Atfcal Neual Netos 8 Ovefttg ANNs Ovefttg ANNs Eo vesus eght upates (example Eo vesus eght upates (Example Eo Tag set Valato set 5 5 Eo Tag set Valato set Numbe of eght upates Numbe of eght upates Chapte 4 Atfcal Neual Netos 9 Chapte 4 Atfcal Neual Netos 3

6 Neual Nets fo Face Recogto left stt gt up Leae Neto Weghts left stt gt up Leae Weghts 3x3 puts 9% accuate leag hea pose, a ecogzg -of- faces 3x3 puts Typcal Iput Images Chapte 4 Atfcal Neual Netos 3 Typcal Iput Images Chapte 4 Atfcal Neual Netos 3 Alteatve Eo Fuctos Pealze lage eghts : E( o D outputs Ta o taget slopes as ell as values: E( ( t D outputs Te togethe eghts : e.g., phoeme ecogto o + γ, t + µ x o x Chapte 4 Atfcal Neual Netos 33 Recuet Netos x(t- x(t y(t+ Feefoa Neto y(t- x(t- y(t x(t y(t+ Recuet Neto x(t y(t+ Recuet Neto ufole tme Chapte 4 Atfcal Neual Netos 34 Neual Neto Summay physologcally (euos spe moel poeful (accuate, slo, opaque (ha to uesta esultg moel bas: pefeetal base o gaet escet fs local mmum effect by tal cotos, paametes eual uts lea lea theshol sgmo Chapte 4 Atfcal Neual Netos 35 Neual Neto Summay (cot gaet escet covegece lea uts lmtato: hypeplae ecso suface leag ule multlaye eto avatage: ca have o-lea ecso suface bacpopagato to lea bacpop leag ule leag ssues uts use Chapte 4 Atfcal Neual Netos 36

7 Neual Neto Summay (cot leag ssues (cot batch vesus cemetal (stochastc paametes tal eghts leag ate mometum cost (eo fucto sum of squae eos ca clue pealty tems ecuet etos smple bacpopagato though tme Chapte 4 Atfcal Neual Netos 37