Neu Neto Intoducton Hung- ee
Reve: Supevsed enng Mode Hpothess Functon Set f, f : : (e) Tnng: Pc the est Functon f * Best Functon f * Testng: f Tnng Dt : functon nput : functon output, ˆ,, ˆ,
Neu Neto Wht does the functon hpothess set (mode) oo e? Wht s the est functon? Ho to pc the est functon? Ree t
Neu Neto Wht does the functon hpothess set (mode) oo e? Wht s the est functon? Ho to pc the est functon? Ree t
Neu Neto Fu onnected Feedfod Neto Input e e e Output vecto 3 vecto You cn s connect the neuons n ou on.
Neu Neto f Input e e e Output vecto 3 vecto Input e Hdden es Output e
Notton e nodes N e N nodes Output of neuon: e Neuon Output of one e: : vecto
Notton e nodes N e N nodes W e to e fom neuon (e ) to neuon (e ) N N
Notton e nodes N e N nodes : s fo neuon t e s fo neuons n e
Notton : nput of the ctvton functon fo neuon t e : nput of the ctvton functon the neuons n e e nodes N e N nodes N
Notton - Summ :output of neuon : eght :output of e W : eght mt : nput of ctvton functon : s : nput of ctvton functon fo e : s vecto
Retons eteen e Outputs e nodes N e N nodes
Retons eteen e Outputs nodes N e e nodes N W N
Retons eteen e Outputs nodes N e e nodes N
Retons eteen e Outputs W W e nodes N e N nodes
Functon of Neu Neto f W W W W vecto vecto, W, W, W W - W
Neu Neto Wht does the functon hpothess set (mode) oo e? Wht s the est functon? Ho to pc the est functon? Ree t
Fomt of Tnng Dt The nput/output of neu neto mode e vectos. Oect nd e shoud so e epesented s vectos. Empe: Hndtng Dgt Recognton Ech pe coesponds to n eement n the vecto : : 8 8 : fo n, : othese 8 8 = 784 dmensons 3 3 dmensons fo dgt ecognton
Wht s the Best Functon? Gven tnng dt: R f f ˆ, ˆ The est functon f * s the one ho mes f * s most cose to ˆ fo tnng empes f ˆ f ˆ R The est functon f * s the one mnmes. (f) evute the dness of functon f (f) s functon of functon (eo functon, cost functon, oectve functon )
Wht s the Best Functon? R f f f to ˆ ˆ The est functon f * s the one mnmes (f). Do ou e ths defnton of est? Queston Is the dstnce good mesue to evute the coseness? Refeence: Go, Pve, Ptc Doetsch, nd Hemnn Ne. "ossentop vs. squed eo tnng: theoetc nd epement compson." INTERSPEEH. 3. cose f cose to ˆ s testng dt
Wht s the Best Functon? f R ˆ ; Eo functon: R R ˆ, ˆ, ˆ, Gven tnng dt: ( functon of functon ) Ho to fnd the est pmete θ * tht mnmes (θ). f W W W ; f, W W,, W, Pc the est pmete set θ* (Hpothess Functon Set) Pc the est functon f*
Neu Neto Wht does the functon hpothess set (mode) oo e? Wht s the est functon? Ho to pc the est functon? Ree t
Posse Souton Sttement of poems: Thee s functon (θ) θ s set of pmetes θ = {θ, θ, θ 3, } Fnd θ * tht mnmes (θ) Bute foce? Enumete posse θ cuus? Fnd θ * such tht, * *,
Gdent descent Sttng Pmetes Hopefu, th suffcent tetons, e cn fn fnd θ* such tht (θ*) s mnmed.
Gdent descent one ve Fo smpfcton, fst consde tht θ hs on one ve Rndom stt t pont θ ompute (θ -ε) nd (θ +ε) If (θ +ε) < (θ -ε) θ = θ + ε
Gdent descent to ves Suppose tht θ hs to ves {θ, θ } (θ) Ho to fnd the smest vue on the ed cce?
To sees et h() e nfnte dffeente ound =. h h h! h h!
To sees h h! To sees fo h()=sn() ound =π/4 sn()=
sn()= To sees To sees fo h()=sn() ound =π/4 The ppomton s good ound π/4.
To sees One ve: Mutve:,,,, h h h h! h h h h When s cose to h h h When nd s cose to nd,,,, h h h h
Gdent descent to ves Red ce: (If the dus s sm),,, ' s, u ',,, v v s u,
Gdent descent to ves Red ce: (If the dus s sm) ' s u v Fnd θ nd θ to mnme (θ) d Smpe, ght?,
Gdent descent to ves Red ce: (If the dus s sm),,, v u v u s ' Fnd θ nd θ to mnme (θ) d, u,v v u To mnme (θ) v u
Gdent descent to ves, The esuts s ntutve, sn t t?,,,,,,
Gdent descent Hgh dmenson Spce of pmete set θ A The pont th mnmum (θ) on the s t θ = {θ, θ, θ 3, }
Gdent descent Sttng Pmetes ompute ompute ompute η s ced enng te η shoud e sm enough, ut shoud not e too sm.
Gdent descent - Poem Dffeent Inttons ed to dffeent oc mnmums Who s Afd of Non-onve oss Functons? http://vdeoectues.net/em7_ecun_/
Gdent descent - Poem Dffeent Inttons ed to dffeent oc mnmums To Empe
Neu Neto Wht does the functon hpothess set (mode) oo e? Wht s the est functon? Ho to pc the est functon? Ree t
Gdent descent fo Neu Neto R?, W W,, W, f R ˆ R R?
hn Rue se se f d d d d d d g f gs, t hs, t, s s s s
(chn ue) Gdent descent fo Neu Neto e (Output e) e - ˆ ˆ ŷ ŷ Empe: ˆ, ˆ ˆ
Gdent descent fo Neu Neto e (Output e) e - ˆ ˆ ŷ ŷ ˆ d d (constnt) (chn ue) Empe: ˆ,
Gdent descent fo Neu Neto e (Output e) e - ˆ ˆ ŷ ŷ (chn ue) ˆ d d Empe: ˆ,
Gdent descent fo Neu Neto ˆ (s nput s ) ˆ e (Output e) e - ˆ ˆ ŷ ŷ Empe: ˆ,
e - e - e (Output e) ˆ ˆ? - - - ŷ
(chn ue) Sum ove e e e - ˆ ˆ ŷ -
(chn ue) Sum ove e e - - - - e -
e -3 e - e - e (Output e) ˆ ˆ m - m - - ŷ m? m
m m m Sum ove e Sum ove e -
Summng ht e hve done Fo pmetes eteen e nd - Fo pmetes eteen e - nd - Fo pmetes eteen e -3 nd - m m Thee e effcent to compute the gdent cpopgton.
Refeence fo Neu neto hpte of Neu neto nd Deep enng http://neunetosnddeepenng.com/ch p.htm eun, Ynn A., et. "Effcent cpop." http://nn.ecun.com/ed/pus/pdf/ecun- 98.pdf Bengo, Yoshu. "Pctc ecommendtons fo gdent-sed tnng of deep chtectues. http://.o.umonte.c/~engo/ppes/ YB-tcs.pdf
Thn ou fo ou stenng!
Append
e--e - -- ---
e e d d e e e d d e e e d d (constnt)
(chn ue) Sum ove e e - - - - e -
(chn ue) Sum ove e e e - ŷ ŷ ŷ -
m m 3 m m
Gdent descent fo Neu Neto e (Output e) e - ˆ ˆ ŷ ŷ (s nput s ) Empe: ˆ,
Wht s the Best Functon? f f f R ; ˆ (Hpothess Functon Set) The est functon f * s the one mnmes. Dffeent θ Dffeent f Dffeent Oectve functon s functon of θ (θ) The est functon θ * s the one mnmes (θ). Ho to fnd θ *?
Notton N W W
Notton N W W