18.7 Artificial Neural Networks

Transcription

1 Artfcl Neurl Networks Neuroscence hs hypotheszed tht mentl ctvty conssts prmrly of electrochemcl ctvty n networks of brn cells clled neurons Ths led McCulloch nd Ptts to devse ther mthemtcl model of the neuron lredy n 1943 Roughly spekng, t fres when lner combnton of ts nputs exceeds some (hrd or soft) threshold Hence, t mplements lner clssfer A neurl network s ust collecton of unts connected together The propertes of the network re determned by ts topology nd the propertes of the neurons Nmes for the reserch feld: connectonsm, prllel dstrbuted processng, neurl computton, nd computtonl neuroscence =1 w 0, Actvton Functon Output Lnks w, g Input Lnks Input Functon Output g n 0 w, 1

2 312 Neurl network structures Neurl networks re composed of nodes or unts connected by drected lnks A lnk from unt to unt serves to propgte the ctvton from to Ech lnk lso hs weght w, ssocted wth t, whch determnes the strength nd sgn of the connecton Ech unt hs dummy nput 0 = 1 wth n ssocted weght w 0, (lner regresson model) Ech unt frst computes weghted sum of ts nputs: n n 0 w, 313 Then the unt pples n ctvton functon g to ths sum to derve the output g( n ) g 0 The ctvton functon g s typclly ether A hrd threshold, n whch cse the unt s clled perceptron, or A logstc functon, n whch cse the term sgmod unt s sometmes used Both of these nonlner ctvton functon ensure the mportnt property tht the entre network of unts cn represent nonlner functon n w, 2

3 314 There re two fundmentlly dstnct wys to connect ndvdul neurons together to form network A feed-forwrd network hs connectons only n one drecton.e., t forms drected cyclc grph Every node receves nput from upstrem nodes nd delvers output to downstrem nodes; there re no loops A feed-forwrd network represents functon of ts current nput It hs no nternl stte other thn the weghts themselves A recurrent network feeds ts output bck nto ts own nputs Ths mens tht the ctvton levels of the network form dynmcl system tht my rech stble stte or exhbt osclltons or even chotc behvor Moreover, the response of the network to gven nput depends on ts ntl stte, whch my depend on prevous nputs Hence, recurrent networks cn support short-term memory Interestng models of the brn, but dffcult to understnd 315 Feed-forwrd networks re usully rrnged n lyers, such tht ech unt receves nput only from unts n the mmedtely precedng lyer Multlyer networks hve one or more lyers of hdden unts tht re not connected to the outputs of the network A network wth ll the nputs connected drectly to the outputs s clled sngle-lyer neurl network, or perceptron network Let us thnk of feed-forwrd neurl network s functon h w (x) prmetrzed by the weghts We cn express the output s functon of the nputs nd the weghts As long s we cn clculte the dervtves of such expressons wth respect to the weghts, we cn use the grdent-descent loss-mnmzton method to trn the network 3

4 316 Becuse the functon represented by network cn be hghly nonlner composed, s t s, of nested nonlner soft threshold functons we cn vew neurl networks s tool for dong nonlner regresson Wth sngle, suffcently lrge hdden lyer, t s possble to represent ny contnuous functon of the nputs wth rbtrry ccurcy Wth two lyers, even dscontnuous functons cn be represented Unfortuntely, for ny prtculr network structure, t s hrd to chrcterze exctly whch functons cn be represented nd whch ones cnnot We cn bck-propgte the error from the output lyer to the hdden lyers The bck-propgton process emerges drectly from dervton of the overll error grdent 317 Lner clssfers Let f(x) be lner functon of ts rgument vector x = (x 1,,x m ) T f ( x) w x b w x b w x b Input x s clssfed postve, f f(x) 0 nd otherwse x s clssfed negtve 1, sgn( f ( x)) 1, The hyperplne determned by the equton w x+b = 0 dvdes the nput spce n two hlfspces If for n exmple (x, y) nd hypothess h t holds tht yh(x) > 0, then the exmple hs been correctly clssfed = yh(x) s the mrgn of the exmple w.r.t. h m 1 f ( x) 0 otherwse 4

5 Support Vector Mchnes Vldmr Vpnk & collegues, 1990 s The SVM frmework s currently the most populr pproch for off-the-shelf supervsed lernng Three dvntges of SVMs: 1.SVMs construct mxmum mrgn seprtor decson boundry wth the lrgest possble dstnce to exmple ponts. Ths helps them to generlze well 2.SVMs crete lner seprtng hyperplne, but they hve the blty to embed the dt nto hgher-dmensonl spce usng the so-clled kernel trck. Often dt tht re not lnerly seprble n the orgnl nput spce re esly seprble n the hgher-dmensonl spce. The hgh-dmensonl lner seprtor s ctully nonlner n the orgnl spce. Ths mens the hypothess spce s gretly expnded over methods tht use strctly lner representtons SVMs re nonprmetrc method they retn trnng exmples nd potentlly need to store them ll. In prctce they often end up retnng only smll frcton of the number of exmples sometmes s few s smll constnt tmes the number of dmensons. SVMs combne the dvntges of nonprmetrc nd prmetrc models: they hve flexblty to represent complex functons, but they re resstnt to overfttng Consder lnerly seprble bnry clssfcton problem. There s n nfnte number of decson boundres consstent wth ll the dt From the pont of vew of 0/1 loss ech would be eqully good Logstc regresson would fnd some seprtng lne, the exct locton of whch depends on ll the exmple ponts The key nsght of SVMs s tht some exmples re more mportnt thn other, nd tht pyng ttenton to them cn led to better generlzton 5

6 320 Insted of mnmzng expected emprcl loss on the trnng dt, SVMs ttempt to mnmze expected generlzton loss Under the probblstc ssumpton tht the s-yet-unseen ponts re drwn from the sme dstrbuton s the prevously seen exmples, there re rguments suggestng tht we mnmze generlzton loss by choosng the seprtor tht s frthest wy from the exmples we hve seen so fr Ths s clled the mxmum mrgn seprtor The mrgn s twce the dstnce from the seprtor to the nerest exmple pont Trdtonlly SVMs use the conventon tht clss lbels re +1 nd 1 The seprtor s defned s the set of ponts {x w x + b = 0} Grdent descent n the spce of w nd b could be used to fnd the prmeters tht mxmze the mrgn whle correctly clssfyng ll the exmples 321 There s nother pproch to solvng the problem There s n lterntve representton clled the dul representton, n whch the optml soluton s found by solvng 1 rgmx k y yk x xk 2, k subect to constrnts 0 nd y = 0 Ths s qudrtc progrmmng optmzton problem, for whch there re good softwre pckges Once we hve found the vector we cn get bck to w wth the equton w = x, or we cn sty n the dul representton The frst mportnt property of the bove dul optmzton problem s tht the expresson s convex: t hs sngle globl mxmum tht cn be found effcently 6

7 322 Second, the dt enter the expresson n the form of dot products of prs of ponts Ths s lso true of the equton for the seprtor tself; once the optml hve been clculted, t s h ( x) sgn y ( x x ) b The thrd mportnt property s tht the weghts ssocted wth ech dt pont re zero except for support vectors the ponts closest to the seprtor Becuse there re usully mny fewer support vectors thn exmples, SVMs gn some of the dvntges of prmetrc models 323 However, we would not usully expect to fnd lner seprtor n the nput spce x, but we cn fnd lner seprtors n hghdmensonl feture spce F(x) smply by replcng x x k n the dul representton optmzton problem wth F(x ) F(x k ) Kernel functon K llows us to fnd lner seprtors n the hgherdmensonl feture spce F(x) smply by replcng x x k wth kernel functon K(x, x k ) Thus we cn lern n the hgher-dmensonl spce, but we compute only kernel functons rther thn the full lst of fetures for ech dt pont As n exmple, F(x ) F(x k ) = (x x k ) 2 when we use three fetures: f 1 = x 12 f 2 = x 2 2 f 3 = 2 x 1 x 2 Kernel trck: Pluggng the kernels nto the optmzton problem optml lner seprtors cn be found effcently n feture spces wth bllons of (or, n some cses, nfntely mny) dmensons 7