Neural Networks The ADALINE

Transcription

1 Lat Lctu Summay Intouction to ua to Bioogica uon Atificia uon McCuoch an itt LU Ronbatt cton Aan Bnaino, Machin Laning, 9/ ua to h ADALI M A C H I L A R I G 9 /

2 cton Limitation cton aning u i not guaant to convg if ata i not inay aab. Wio-Hoff (96) Minimiz th o at th outut of th ina unit () ath than at th outut of th thho unit ( ). Aan Bnaino, a@i.it.ut.t Machin Laning, 9/ ADALI Aativ Lina mnt Saating hyan i quivant to th cton Aan Bnaino, a@i.it.ut.t Machin Laning, 9/

3 ADALI Aativ Lina mnt Laning u i iffnt fom th cton. Givn th taining t: {(, )},,..., minimiz cot function: ( ) ( ) ( ) Aan Bnaino, a@i.it.ut.t Machin Laning, 9/ ADALI - Simification Lt u coni that, fo vy attn: hu can it: Aan Bnaino, a@i.it.ut.t Machin Laning, 9/

4 ADALI Anaytic Soution Otimiz th cot function: Givn that: ( ) ( ) ( ) ( ) 3 [ ]...,...,, ( ) ( ) 4 ADALI Anaytic Soution Comut th gaint of cot function: ( ) ( ) Aan Bnaino, a@i.it.ut.t Machin Laning, 9/ ( )

5 ADALI Anaytic Soution Aan Bnaino, Machin Laning, 9/ ADALI Anaytic Soution Vy imotant! h atia ivativ of th o function ith ct a ight i ootiona to th um fo a attn of th inut on that ight mutii by th o. Aan Bnaino, a@i.it.ut.t Machin Laning, 9/

6 ADALI Anaytic Soution Givn that Aan Bnaino, Machin Laning, 9/ ( ) ADALI Anaytic Soution Aan Bnaino, a@i.it.ut.t Machin Laning, 9/

7 ADALI Anaytic Soution,...,, Aan Bnaino, Machin Laning, 9/ ADALI Anaytic Soution,...,, Aan Bnaino, Machin Laning, 9/ It i a ina ytm of + quation ith + unnon. Ho to ov it?

8 ADALI Mati otation [ ]... [ ],... Aan Bnaino, a@i.it.ut.t Machin Laning, 9/ ADALI Mati otation ( ) ( ) ( ) ( ) ( )( ) Aan Bnaino, a@i.it.ut.t Machin Laning, 9/ ( ) ( )( ) ( ) ( ) ( ) ( ) +

9 ADALI Mati otation ( ) ( ) ( ) ( ) + Aan Bnaino, a@i.it.ut.t Machin Laning, 9/ ( ) ( ) ( ) + ( ) ( ) ( ) + ADALI Mati otation Lt u intouc th avag oato < >: ( ) Aan Bnaino, a@i.it.ut.t Machin Laning, 9/ h cot function i ittn a: ( ) ( ) ( ) +

10 ADALI Mati otation Dfining: ( ) ( ) R σ th cot function i: ( ) ( ) ( ) R + σ Aan Bnaino, a@i.it.ut.t Machin Laning, 9/ ADALI Quaatic Cot ( ) R + σ R i a covaianc mati oitiv mi-finit h o function ufac i a aaboa. Aan Bnaino, a@i.it.ut.t Machin Laning, 9/

11 ADALI Gaint Vcto ( ) R + σ ( ) R Aan Bnaino, a@i.it.ut.t Machin Laning, 9/ ADALI Co Fom Soution * ( * ) R If R i oitiv finit, th minimum i uniqu. * R Aan Bnaino, a@i.it.ut.t Machin Laning, 9/

12 ADALI Co Fom Soution Co fom oution qui th invion of th covaianc mati, hich can b obmatic in high imnion. Aan Bnaino, a@i.it.ut.t Machin Laning, 9/ Gaint mtho a im an hav ov convgnc oti fo quaatic function. ( ) ( ) t t + η ADALI Gaint Ba Soution Rmmb ~ i bac: ( ) ( ) Aan Bnaino, a@i.it.ut.t Machin Laning, 9/

13 ADALI Gaint Ba Soution Aan Bnaino, Machin Laning, 9/ ( ) ( ) t t + η ( ) ( ) + t t η ADALI Batch Agoithm Initiaiz igth at abitay vau Dfin a aning at η. Rat: Fo ach attn in th taining t Aan Bnaino, a@i.it.ut.t Machin Laning, 9/ Ay to th aain inut Obv th outut an comut th o Fo ach ight, accumuat th ouct. Aft ocing a attn, uat ach ight by: ( ) ( ) + t t η

14 ADALI Batch Agoithm ADALI batch agoithm oti: Guaat to convg to ight t ith minimum qua o: givn ufficinty ma aning at η. vn hn taining ata contain noi. vn hn taining ata i not aab. Aan Bnaino, a@i.it.ut.t Machin Laning, 9/ ADALI Batch Agoithm ADALI batch agoithm qui th avaiabiity of a th taining ata fom th bginning. h ight a uat ony aft nting th ho taining ata. But human an continuouy! In om aication may ant to uat th ight immiaty aft ach taining attn i avaiab. Aan Bnaino, a@i.it.ut.t Machin Laning, 9/

15 ADALI Incmnta Agoithm Incmnta Agoithm aoimat th comt gaint by it timat fo ach attn. Comt (act) gaint ˆ Stochatic (aoimat) gaint Aan Bnaino, Machin Laning, 9/ ADALI Incmnta Agoithm Incmnta mo gaint cnt Batch mo gaint cnt: ( t + ) ( t ) ( t+ ) ( t ) η η Incmnta Gaint Dcnt can aoimat Batch gaint cnt abitaiy coy if η i ma ma nough.

16 ADALI Incmnta Agoithm Incmnta gaint cnt i ao non a tochatic gaint cnt. It i ao ca LMS agoithm o th Dta Ru. It i ba on an aoimation of th gaint o it nv go acty to th minimum of th cot function. Aft aching a vicinity of th minimum, it ociat aoun it. h amitu of th ociation can b uc by ucing η. Aan Bnaino, a@i.it.ut.t Machin Laning, 9/ ADALI - Comaion h ot ho th vau of on ight aong tim Batch Incmnta och attn och attn (th fu taining t) Aan Bnaino, a@i.it.ut.t Machin Laning, 9/

17 ADALI v cton Both ADALI Dta Ru an cton ight uat u a intanc of o Coction Laning. ADALI Dta Ru cton uat u ( t+ ) ( t ) ( t+ ) ( t ) ηη ( ) η ( y ) h ADALI ao abitay a vau in th outut vau ha th cton aum binay outut. h ADALI aay convg (givn ma nough η) to th minimum qua o, hi th cton ony convg hn ata i aab. Aan Bnaino, a@i.it.ut.t Machin Laning, 9/ ADALI Statitica Inttation h anaytica oution fo th ight a obtain avaging quantiti obtain fom th taining t. It i oib to ma a tatitica inttation of th oc: Inut: Obvaion of Ranom Vaiab X [, X,..., X,..., X ] Di Outut: Obvation of Ranom Vaiab D Outut: Obvation of Ranom Vaiab Y X Aan Bnaino, a@i.it.ut.t Machin Laning, 9/

18 ADALI Statitica Inttation h o function can b intt a an aoimation to th tatitica ctation [.]: [ ] ( ) ( y ) ( Y D) Soution * R Mati R an vcto can b intt a aoimation to th tatitica auto-covaianc an co-covaianc btn anom vaiab: R ( ) [ XX ] [ DX ] Aan Bnaino, a@i.it.ut.t Machin Laning, 9/ ADALI Statitica Inttation h LMS agoithm i ba on an intantanou timat of th gaint. hi timat can b mo by: gˆ( n) g( n) + ( n) h g (n) i a anom noi vcto. g LMS tochatic gaint cnt Aan Bnaino, a@i.it.ut.t Machin Laning, 9/

19 ADALI Statitica Inttation Un aonab conition, tochatic gaint mtho may convg to th act oution. Convgnc Conition [Mono an Ljung]: g (n) i zo man. attn qunc i anom. η(n) tn oy toa zo. n n η ( n) < η( n) Aan Bnaino, a@i.it.ut.t Machin Laning, 9/ ADALI Statitica Inttation yica aning at chu ( ) η n c n η ( n) η n + τ Aan Bnaino, a@i.it.ut.t Machin Laning, 9/