Neural Network Introduction. Hung-yi Lee

Transcription

1 Neu Neto Intoducton Hung- ee

2 Reve: Supevsed enng Mode Hpothess Functon Set f, f : : (e) Tnng: Pc the est Functon f * Best Functon f * Testng: f Tnng Dt : functon nput : functon output, ˆ,, ˆ,

3 Neu Neto Wht does the functon hpothess set (mode) oo e? Wht s the est functon? Ho to pc the est functon? Ree t

4 Neu Neto Wht does the functon hpothess set (mode) oo e? Wht s the est functon? Ho to pc the est functon? Ree t

5 Neu Neto Fu onnected Feedfod Neto Input e e e Output vecto 3 vecto You cn s connect the neuons n ou on.

6 Neu Neto f Input e e e Output vecto 3 vecto Input e Hdden es Output e

7 Notton e nodes N e N nodes Output of neuon: e Neuon Output of one e: : vecto

8 Notton e nodes N e N nodes W e to e fom neuon (e ) to neuon (e ) N N

9 Notton e nodes N e N nodes : s fo neuon t e s fo neuons n e

10 Notton : nput of the ctvton functon fo neuon t e : nput of the ctvton functon the neuons n e e nodes N e N nodes N

11 Notton - Summ :output of neuon : eght :output of e W : eght mt : nput of ctvton functon : s : nput of ctvton functon fo e : s vecto

12 Retons eteen e Outputs e nodes N e N nodes

13 Retons eteen e Outputs nodes N e e nodes N W N

14 Retons eteen e Outputs nodes N e e nodes N

15 Retons eteen e Outputs W W e nodes N e N nodes

16 Functon of Neu Neto f W W W W vecto vecto, W, W, W W - W

17 Neu Neto Wht does the functon hpothess set (mode) oo e? Wht s the est functon? Ho to pc the est functon? Ree t

18 Fomt of Tnng Dt The nput/output of neu neto mode e vectos. Oect nd e shoud so e epesented s vectos. Empe: Hndtng Dgt Recognton Ech pe coesponds to n eement n the vecto : : 8 8 : fo n, : othese 8 8 = 784 dmensons 3 3 dmensons fo dgt ecognton

19 Wht s the Best Functon? Gven tnng dt: R f f ˆ, ˆ The est functon f * s the one ho mes f * s most cose to ˆ fo tnng empes f ˆ f ˆ R The est functon f * s the one mnmes. (f) evute the dness of functon f (f) s functon of functon (eo functon, cost functon, oectve functon )

20 Wht s the Best Functon? R f f f to ˆ ˆ The est functon f * s the one mnmes (f). Do ou e ths defnton of est? Queston Is the dstnce good mesue to evute the coseness? Refeence: Go, Pve, Ptc Doetsch, nd Hemnn Ne. "ossentop vs. squed eo tnng: theoetc nd epement compson." INTERSPEEH. 3. cose f cose to ˆ s testng dt

21 Wht s the Best Functon? f R ˆ ; Eo functon: R R ˆ, ˆ, ˆ, Gven tnng dt: ( functon of functon ) Ho to fnd the est pmete θ * tht mnmes (θ). f W W W ; f, W W,, W, Pc the est pmete set θ* (Hpothess Functon Set) Pc the est functon f*

22 Neu Neto Wht does the functon hpothess set (mode) oo e? Wht s the est functon? Ho to pc the est functon? Ree t

23 Posse Souton Sttement of poems: Thee s functon (θ) θ s set of pmetes θ = {θ, θ, θ 3, } Fnd θ * tht mnmes (θ) Bute foce? Enumete posse θ cuus? Fnd θ * such tht, * *,

24 Gdent descent Sttng Pmetes Hopefu, th suffcent tetons, e cn fn fnd θ* such tht (θ*) s mnmed.

25 Gdent descent one ve Fo smpfcton, fst consde tht θ hs on one ve Rndom stt t pont θ ompute (θ -ε) nd (θ +ε) If (θ +ε) < (θ -ε) θ = θ + ε

26 Gdent descent to ves Suppose tht θ hs to ves {θ, θ } (θ) Ho to fnd the smest vue on the ed cce?

27 To sees et h() e nfnte dffeente ound =. h h h! h h!

28 To sees h h! To sees fo h()=sn() ound =π/4 sn()=

29 sn()= To sees To sees fo h()=sn() ound =π/4 The ppomton s good ound π/4.

30 To sees One ve: Mutve:,,,, h h h h! h h h h When s cose to h h h When nd s cose to nd,,,, h h h h

31 Gdent descent to ves Red ce: (If the dus s sm),,, ' s, u ',,, v v s u,

32 Gdent descent to ves Red ce: (If the dus s sm) ' s u v Fnd θ nd θ to mnme (θ) d Smpe, ght?,

33 Gdent descent to ves Red ce: (If the dus s sm),,, v u v u s ' Fnd θ nd θ to mnme (θ) d, u,v v u To mnme (θ) v u

34 Gdent descent to ves, The esuts s ntutve, sn t t?,,,,,,

35 Gdent descent Hgh dmenson Spce of pmete set θ A The pont th mnmum (θ) on the s t θ = {θ, θ, θ 3, }

36 Gdent descent Sttng Pmetes ompute ompute ompute η s ced enng te η shoud e sm enough, ut shoud not e too sm.

37 Gdent descent - Poem Dffeent Inttons ed to dffeent oc mnmums Who s Afd of Non-onve oss Functons?

38 Gdent descent - Poem Dffeent Inttons ed to dffeent oc mnmums To Empe

39 Neu Neto Wht does the functon hpothess set (mode) oo e? Wht s the est functon? Ho to pc the est functon? Ree t

40 Gdent descent fo Neu Neto R?, W W,, W, f R ˆ R R?

41 hn Rue se se f d d d d d d g f gs, t hs, t, s s s s

42 (chn ue) Gdent descent fo Neu Neto e (Output e) e - ˆ ˆ ŷ ŷ Empe: ˆ, ˆ ˆ

43 Gdent descent fo Neu Neto e (Output e) e - ˆ ˆ ŷ ŷ ˆ d d (constnt) (chn ue) Empe: ˆ,

44 Gdent descent fo Neu Neto e (Output e) e - ˆ ˆ ŷ ŷ (chn ue) ˆ d d Empe: ˆ,

45 Gdent descent fo Neu Neto ˆ (s nput s ) ˆ e (Output e) e - ˆ ˆ ŷ ŷ Empe: ˆ,

46 e - e - e (Output e) ˆ ˆ? ŷ

47 (chn ue) Sum ove e e e - ˆ ˆ ŷ -

48 (chn ue) Sum ove e e e -

49 e -3 e - e - e (Output e) ˆ ˆ m - m - - ŷ m? m

50 m m m Sum ove e Sum ove e -

51 Summng ht e hve done Fo pmetes eteen e nd - Fo pmetes eteen e - nd - Fo pmetes eteen e -3 nd - m m Thee e effcent to compute the gdent cpopgton.

52 Refeence fo Neu neto hpte of Neu neto nd Deep enng p.htm eun, Ynn A., et. "Effcent cpop." 98.pdf Bengo, Yoshu. "Pctc ecommendtons fo gdent-sed tnng of deep chtectues. YB-tcs.pdf

53 Thn ou fo ou stenng!

54 Append

55 e--e

56 e e d d e e e d d e e e d d (constnt)

57 (chn ue) Sum ove e e e -

58 (chn ue) Sum ove e e e - ŷ ŷ ŷ -

59 m m 3 m m

60

61 Gdent descent fo Neu Neto e (Output e) e - ˆ ˆ ŷ ŷ (s nput s ) Empe: ˆ,

62 Wht s the Best Functon? f f f R ; ˆ (Hpothess Functon Set) The est functon f * s the one mnmes. Dffeent θ Dffeent f Dffeent Oectve functon s functon of θ (θ) The est functon θ * s the one mnmes (θ). Ho to fnd θ *?

63 Notton N W W

64 Notton N W W