The rise of neural networks. Deep networks. Why many layers? Why many layers? Why many layers? 24/03/2017

Transcription

1 Th rs of ural ors I h md-s, hr has b a rsurgc of ural ors, mal du o rasos: hgh compuaoal por bcam avalabl a lo cos va gral-purpos graphcs procssg us (GPGPUs). maor plars l Googl, crosof, ad Facboo, dd o aalz hr hug amou of daa. Ths rsurgc ld o a ural modl, o as Dp Larg, for rag ors of ma lars calld Dp Nors. Dp ors Dp ors rfr o ors h mor ha 3 lars. Th ohrs ar calld shallo ors. Boh ar fd-forard ors rad h a suprvsd larg paradgm. pu lar hdd lar 1 hdd lar hdd lar 3 oupu lar Wh ma lars? Wh should us ma lars f h uvrsal appromao horm provd ha a ural or h hr lars ca approma a fuco? Wh ma lars? or prcsl, h uvrsal appromao horm sas ha, gv a fuco f() of h pu vcor adarror >, hr ss a umbr N of hdd us ad a s of ghs ha ca approma h fuco h h gv rror. Hovr, dos o sa ho o fd such ghs! I ohr ords, v hough a 3-lar or ca horcall approma a fuco, fdg h rgh ghs could a vr log m for compl fucos. O, bu h a or h ma lars could solv h sam problm mor asl? Wh ma lars? Cosdr a 3-lar or for hadr dg rcogo from h NIST mag s. Wh do us 1 oupu uros ad o 4 ( bar cod)? pu uros 8 1 oupu uros 1

2 Wh ma lars? Tpcall, ha h hdd uros lar s o rcogz h prsc of lmar faurs h pu mag. Wha hs uro has o lar s o gra hs faurs b crasg hos ghs. Wh ma lars? If hav 4 oupus, ach uro has o lar h b of h bar cod, ad hr s o as a o rla hs formao o smpl faurs of h pu mag. = 1 1 8, 9 1 4, 5, 6, 7 1, 3, 6, 7 1 1, 3, 5, 7, 9 Wh ma lars? Hovr, s much asr o lar a bs rprsao b addg a ra lar ha mas h covrso. Each lar lars a mor sophscad daa rprsao: smpl faurs dgs bar cod Wh ma lars? L s cosdr a mor compl problm as fac rcogo. Assumg h 3-lar or s abl o lar classfg facs, s qu hard o udrsad from h ghs ha h or rall lard. I s a fac srous bhavor I ohr ords, h fal bhavor s qu msrous: ghs ar dscovrd auomacall ad do o udrsad ho h or s dog s ob. W cao ma prdcos! Would ou rus a auoomous car ha as drvg dcsos basd o a ural or ha obod udrsad? A br approach A br a o approach fac rcogo s o dcompos h problm o sub-problms: s hr a h op lf? s hr a h op rgh? s hr a os h mddl? s hr a mouh h boom mddl? If h asr o svral of hs qusos s s, h coclud ha h mag s ll o b a fac.

3 Faur dcors Sub-faur dcors Hc h archcur bcoms: E dcor h op lf E dcor h op rgh Nos dcor h mddl ouh dcor h boom For ampl, h E dcor ca b dcomposd o moduls for dcg bro, lashs, rs, ad so o: Ebro dcor Elashs dcor Irs dcor E shap dcor Clarl, sub-problms ca also b dcomposd o dc smplr faurs, ad so o. Faur composo Faur composo Hc h or archcur bcoms: lmar shaps smpl faurs macro faurs Th rsul s a or of ma lars (dp ural or), h arl lars dcg vr smpl faurs o h mag, ad lar lars buldg mor ad mor compl absracos: fac N-class classfr Ths da ca b rad, ul faurs ar dcomposd o ma vr smpl faurs locad small rgos of pls. Faur composo Th rsul s a or of ma lars (dp ural or), h arl lars dcg vr smpl faurs o h mag, ad lar lars buldg mor ad mor compl absracos: N-class classfr To ma problms Uforual, crasg h umbr of hdd lars lads o o o problms: 1. Vashg grad: as add mor ad mor hdd lars, Bacpropagao bcoms lss ad lss ffcv o h lor lars, sc h grad bcoms smallr ad smallr (hc vashg). 1. Ovrfg: I ors h a larg umbr of uros (hc, ma dgrs of frdom) h or ds o f h rag daa oo closl, prformg rall ll o h rag s, bu vr poorl o ohr ampls. Ths problms rmad usolvd ul 6, h svral mhods hav b dvlopd o lar dp ural ors. 3

4 Vashg grad Vashg grad Ths problm has b formall dfd for h frs m 1991 b Spp Hochrr hs masr hss. If ra a or for dgs classfcao h a crasg umbr of hdd lars, s ha h accurac dos o mprov as pcd. To udrsad ha s gog o, l s moor h grad of h uros ach hdd lar, sc gvs a dcao of ho qucl ach uro s larg. Each uro updas s ghs accordg o: 8 6 Accurac (%). δ 1 δ. δ l If δ l s h grad vcor of lar l, s orm δ l gvs a rough masur of h spd larg of lar l: # hdd lars lar l δ δ l l δ.3.17 Vashg grad oorg δ l for a or of 4hddlars,ahsarof larg g: Ths s h suao a h bgg of larg hdd lar Ipu uros: 88=784 Oupu uros: 1 Hdd uros: 3 pr lar Larg ra: η =.1 Rgularzao: λ = 5. -bach sz: m = 1 a_pochs: 3 Vashg grad oorg δ l durg larg g: Spd of larg δ l Ths s cofrms ha h grad dcrass poall as mov bacard hrough h hdd lars. l = 4 l = 3 l = l = 1 No ha h spd of larg dcrass poall procdgs from h oupu o h pu Numbr of pochs Wh dos grad vash? Wh dos grad vash? To udrsad h ssu, cosdr h smpls dp or: E b l L E '( al ) '( a ) l1 L.5 σ (a) To udrsad h h grad vashs, l s drv E/ b, ha s, h grad of h bas of h frs hdd uro: E E '( a) 3 '( a3) 4 '( a4) 5 '( a5) b Ecp for E/ 5, s a produc of rms of h form σ (a ) If alz ghs usg a Gaussa(,1), h < 1 σ (z ) < Hc, h produc of ma such rms dcrass poall h h umbr of rms! a 4

5 Wh dos grad vash? L E E '( al ) '( a ) bl l1 L You could argu ha f ghs gro durg rag, h could o logr b ru ha σ (a ) <1/4. Idd, f h rms g larg ough, grar ha 1, h do hav a vashg grad, bu a plodg grad. Wh dos grad vash? No ha h grad s mor ll o vash ha plodg usg sgmod uros. I fac, o plod d o hav σ (a ) 1, bu hs s o so as o happ, bcaus σ (a) also dpds o : σ (a)=σ ( + b) Hc: larg larg a small σ (a) Th ol a o ma σ (a ) 1 s f h pu falls h a small rag of valus. Somms ha happs, bu mor of dos o. Hc, h grad s mor ll o vash. Improvg Bacpropagao Svral chqus hav b proposd o mprov Bacpropagao o ma suabl for dp ors. Th clud: A br choc of h Loss fuco; A br choc of h acvao fuco; Rgularzao mhods o addrss ovrfg ad mprov gralzao. Loss fuco W hav s ha al gh valus srogl affc h larg spd: E =.6, b =.9 E =., b =. 3 poch 3 poch Th problm s du o h fac ha s proporoal o E/, ad, sc h E/ s proporoal o (a). 1 E ( ) 5

6 A br loss fuco A br loss fuco W sad ha hs problm ca b solvd b alzg h ghs h small radom valus. Aohr approach s o rplac h quadrac cos fuco for compug h rror E =½() h a dffr loss fuco C, o as h cross-rop fuco: C l (1 )l (1 ) No ha C sasfs o proprs of good cos fucos: 1. C >, sc boh log argums ar h rag (,1);. C h. Hovr, hs s ru ol f ca b hr or 1, as mos classfcao problms. Ul h quadrac rror fuco, h drvav of h crossrop loss fuco dos o dpd o (a): C l (1 )l (1 ) C C a a C ( ) No ha.r.. E / ol h rm (a) s mssg: C 1 1 (1 ) '( a) (1 ) a a Sc = C/, h largr h rror, h fasr h uro ll lar! E ( ) (a) If ru h prm h h cross-rop loss fuco, g (h =.5): E A br loss fuco =.6, b =.9 3 poch E =., b =. 3 poch No ha for dffr loss fucos h valus of cao b compard. Cross-rop loss fuco Th cross-rop fuco usd bfor s rlav o a sgl oupu uro, bu ca asl b dfd for h oupu lar ad for h r rag s (global loss): For a sgl oupu uro o ampl For h oupu lar L o ampl For h oupu lar L o h r rag s: C L 1 l C C 1 C C 1 (1 )l (1 ) Th cross-rop fuco s arl alas h bs choc, provdd h oupu uros ar sgmod uros. Whr dos C drv from? Wouldlogrdof (a) h rror fuco: Soouldlohava loss fuco C such ha: C '( a) C Whch mas: (1 ) E ( ) (a) C C a ( ) a C (a) = ( ) B addg a h omaor g: (1 ) (1 ) 1 (1 ) (1 ) 1 So go: Whr dos C drv from? C C 1 1 Igrag.r.. hav: Tha s: C d (1 ) C ( 1 )l (1 ) d 1 l d 6

7 Is sgmod h bs? O problm of h sgmod fuco s ha uro oupus ar alas posv ad accordg o Bacpropagao, gh varaos ar compud as =. Sc h lms of ar all posv, all ghs of a uro hr cras or dcras oghr dpdg o. Th ah fuco A fuco of usd o rplac h sgmod s h ah: ah( ) 1 Tha s a problm, sc som of h ghs ma d o cras hl ohrs d o dcras. Tha ca ol happ f h lms of ca hav dffr sgs Ths rasog suggss rplacg h sgmod b a fuco ha allos boh posv ad gav acvaos. -1 Th ah fuco No ha: 1 ah( / ) ah( ) () 1 ( ) Tha s, ah s us a rscald vrso of h sgmod fuco. Proof: ah( 1 1 (1 1 ) 1 () 1 1 ) I s orh obsrvg ha: Th ah fuco Sc ah rags from -1 o 1, pus ad oupu d o b ormalzd dffrl ha h sgmod ors. Bacpropagao ad sochasc grad dsc ca also b appld o a or of ah uros. A or of ah uros ca lar a fuco. Esv prms shod ha h rspc o sgmod uros ah provds ol small mprovms. Thr s o formal proof ha sas hch of h o fucos allos larg fasr or gralzg br for a applcao. Ar hr br fucos? I boh sgmod ad ah, uros sop larg h h saura, sc f (a) rducs h grad. Isad of chagg h loss fuco from quadrac o cross-rop, aohr approach could b o choos a acvao fuco f (a) such ha s drvav dos o dcras h h acvao. Th rcfd lar fuco f ( ) ma(, ) f () A commol usd fuco h hs faur s h rcfd lar fuco. 7

8 Th rcfd lar fuco Sofma uros No ha: Icrasg ghs ad h pu acvao, a rcfd lar u ll vr saura, so hr s o larg slodo. O h ohr had, h h acvao s gav, h grad vashs, so h uro sops larg. Bacpropagao ad sochasc grad dsc ca also b appld o a or of rcfd lar uros. A or of rcfd lar uros ca lar a fuco. Eprms shod ha rcfd lar us ca achv cosdrabl bfs, ovr sgmod ad ah uros. Thr s o proof sag h such us ar prfrabl. A sofma uro s ormall usd for h oupu lar. Th acvao s h ghd sum of s pus, bu h oupu s compud b h so-calld sofma fuco, dfd as: 1 a f ( a ) hr s h umbr a uros h oupu lar. No ha h oupus ar posvs ad alas sum up o 1. Hc, h oupu from a sofma lar ca b s as a probabl dsrbuo, hr ach oupu s h smad probabl ha h corrc oupu s. Ths propr s qu cov for classfcao! C 1 Sofma uros Sofma uros ar usd combao h h so calld log-llhood loss fuco, dfd as: acual arg oupu oupu l I hs ampl, hav: C l.3 Ths dfo s soud, sc for a par of class, ( =1): 1 C s lo C s hgh 1 =.7 =.1 3 =.8 4 =.7 5 = = =1 3 = 4 = 5 = 1 1 Sofma uros L for h prvous cross-rop loss fuco, h drvav C /a dos o dpd o (a): C C a 1 C l To drv hs rsuls, compu: a C a C 1 Sc dpds o all ohr oupus, o compu /a hav o dsgush o cass: = ad. Sofma uros Sofma uros Cas = : Cas : a a a a a a a ( ) ( ) a a a a a a a 1 a a a a a (1 ) a a C a C a C a C a 1 1 ) ( f = f ( ) (1 ) (1 ) 1 = 1 8

9 Sofma uros I summar, a sofma oupu lar h log-llhood cos s qu smlar o a sgmod oupu lar h cross-rop cos. 1 a a C 1 l C a I ma suaos boh approachs or ll. Th ol ral dffrc b h o cass s ha a sofma lar provds smas o classfcao probabls. Ho o addrss ovrfg Ovrfg s h suao hch a ural or sars larg oo much dals of h TS, loosg h abl o gralz o ampls. Ths occurs h h or has a ma hdd uros compard h h TS sz. accurac A TS A VS Ths s a sg ha h or s ovrfg h TS, larg h os o daa! pochs Ho o addrss ovrfg Ovrfg s a srous problm h rag larg ors h ma hdd uros (l dp ors). Svral chqus hav b proposd o addrss ovrfg ad mprov gralzao. Th clud: Earl soppg; L rgularzao; L1 rgularzao; Dropou; Avragg or vog chqus; Arfcal paso of h rag s. A mhod cosdrd o avod ovrfg s arl soppg. No ha som o rval udgm s rqurd o drm h o sop, loog a h rd of boh TS ad VS: Ho o addrss ovrfg accurac STOP A TS A VS pochs Ho o addrss ovrfg O of h bs approachs for rducg ovrfg s o cras h sz of h TS. Whoughragdaas dffcul o ovrf, v for a vr larg or. Uforual, rag daa ca b psv or dffcul o acqur, so hs s o alas a praccal opo. Aohr approach s o rduc h umbr of hdd uros (hc h umbr of dgrs of frdoms). Hovr, larg ors hav h poal o b mor porful ha small ors, so hs s a opo do l o adop. Rgularzao mhods Forual, hr ar ohr mhods o rduc ovrfg, v h hav a fd or ad fd rag daa. Ths ar o as rgularzao mhods. Rgularzao mhods d o prv ghs o rach hgh valus durg larg. Bu, h pg ghs a lo valus hlps avodg ovrfg? 9

10 Wh rgularzao hlps L rgularzao A or h small ghs s lss ssv o small pu chags, so lars basd o pars s mor of TS: B coras, a or h larg ghs s mor ssv o small varaos ad ds o lar os pcular h TS: For hs raso, gh rgularzao hlps ors o gralz br from ha h lar. Th mos commo o s o as L rgularzao or gh dca. Th da s o add a ra rgularzao rm R o h cos fuco: C = C + R hr C s h orgal cos fuco ad R s gv b: R hr λ >s h rgularzao paramr ad s h sz of h rag s. No ha R dos o clud bass. L rgularzao Iuvl, h ffc of R s o p ghs small. Larg ghs ll ol b allod f h cosdrabl mprov C. Bascall, rgularzao balacs b fdg small ghs ad mmzg h orgal cos fuco. Th rlav mporac of hs lms dpds o λ: mz C 1 λ Small ghs To udrsad h R rducs ovrfg, l s compu h ghs varao b drvg h rgularzd cos fuco. C C C C C C L rgularzao C C 1 Ths s acl h sam as h grad dsc rul, cp ha ach gh s rscald b a facor 1 gh dca L rgularzao C ( ) 1 ( 1) No ha, bg < gh_dca < 1, a a frs glac sms ha should poall dcras oards zro. Bu ha s o ru, sc ma cras for h ohr rm. Bass ar o subc o rgularzao sc of dos o chag h rsuls vr much. Usg L rgularzao, h TS rror dcrass as hou R, bu h accurac o h VS cous o cras. Icrasg h TS sz ad h umbr of hdd us ma furhr mprov h accurac. accurac L rgularzao A TS A VS h R A VS hou R pochs 1

11 L1 rgularzao L1 rgularzao Aohr form of rgularzao s h L1 rgularzao, hch h rgularzao rm s dfd as: R C C sg( C ( ) ( 1) sg( 1) ) Wh s larg, Wh s small, L1 rgularzao shrs h gh much lss ha L rgularzao. L1 rgularzao shrs h gh much mor ha L rgularzao. Th rsul s ha L1 rgularzao ds o cocra h ghs a rlavl small umbr of hgh-mporac cocos, hl h ohr ghs ar drv oard zro. Dropou Dropou s a radcall dffr chqu for rgularzao, proposd b Ho al. 1. Ul L1 ad L rgularzao, dropou dos o modfs h cos fuco, bu h or slf b radoml dlg half of h hdd uros a vr larg rao. Dropou Dropou s a radcall dffr chqu for rgularzao, proposd b Ho al. 1. Ul L1 ad L rgularzao, dropou dos o modfs h cos fuco, bu h or slf b radoml dlg half of h hdd uros a vr larg rao. m-bach m m ms m-bach m m ms Dropou I h oprao phas, all hdd uros ar acv, hc c h uros usd h larg phas. To compsa for ha, ghs ougog from h hdd uros ar halvd: = Dropou Wh dos dropou hlp rducg ovrfg? Usg dropou s l rag svral ors h h sam TS ad h avragg h rsuls. Th avragg schm s qu ffcv (alhough psv) o rduc ovrfg, bcaus dffr ors ovrf dffr as, hus avragg s l flrg h os bhavor. Dropou has b vr succssful mprovg h prformac of ural ors, rachg vr hgh accurac h usd oghr h rgularzao mhods. 11