Feedforard neural netork IFT 725 - Réseau neuronau
Septemer Astrat6, 22 ugolaroelle@userrookeasepte verste de Serrooke Sep Mat for my sldes Feedforard neural netork ARTIFICIAL NEURON aroelle@userrookea Septemer 6, 22 Astrat Astrat Mat for my sl Tops: onneton egts, as,>atvaton funton Mat for my neural netork sldes Feedforard Mat for my sldes Feedforard neural netork Neuron Septemer 6, 22 nput atvaton: Mat for my sldes Feedforar Astrat Mat for my sldes Feedfor a > Mat for my sldes g(a g( > a Mat for m a Mat for my sldes Feedforard neural netork > a a g(a g(a g( g( > a g(a d d a a Astrat Neuron (output atvaton g(a g( g(a g( d dg( d g(a d g(a g( g(a( dforard neural netork d d d d d d d d > are te onneton egts { {{ { { s te neuron as g( { salled te atvaton funton g( g( g( { 2 g( { g(a
g(a g( Astrat Mat for my sldes Feedforard neural netork ARTIFICIAL NEURON a > Tops: onneton egts, as, atvaton funton g(a g( Mat for my sldes Feedfor Mat for my a a g(a g(a g( { range determned y g( y { - - - 2 a { g( as only anges te poston of te rff (from asal Vnent s sldes 3
ARTIFICIAL NEURON Tops: lnear atvaton funton erforms no nput squasng Not very nterestng { g(a a 4
ARTIFICIAL NEURON Tops: sgmod atvaton funton Squases te neuron s nput eteen and Alays postve Bounded Strtly nreasng g(a sgm(a ep( a 5
ARTIFICIAL NEURON Tops: yperol tangent ( tan atvaton funton Squases te neuron s nput eteen - and Can e postve or negatve Bounded Strtly nreasng g(a tan(a ep(a ep( a ep(aep( a ep(2a ep(2a 6
ARTIFICIAL NEURON Tops: retfed lnear atvaton funton Bounded elo y (alays postve Not upper ounded Strtly nreasng Tends to gve neurons t sparse atvtes g(a reln(a ma(,a 7
ARTIFICIAL NEURON Tops: apaty, deson oundary of neuron Could do nary lassfaton: t sgmod, an nterpret neuron as estmatng also knon as logst regresson lassfer f greater tan 5, predt lass oterse, predt lass ues p(y deson oundary s lnear 2 R (smlar dea an apply t tan 2 R 2 (from asal Vnent s sldes 8
ARTIFICIAL NEURON Tops: apaty of sngle neuron Can solve lnearly separale prolems OR (, 2 AND (, 2 AND (, 2 2 2 2 9
ARTIFICIAL NEURON Tops: apaty of sngle neuron Can t solve non lnearly separale prolems XOR (, 2 XOR (, 2 2? AND (, 2 AND (, 2 unless te nput s transformed n a etter representaton
g(a reln(a ma(, a ugolar tan(a g(a sgm ep( a ep(2a ep(a ep( a g(a ep(2a ep(a g(a a ep(aep( a ep( (a g(a a tan(a ep(aep( g(a ep(aep( a ep(2a g(a sgm(a a ep(2a g(a g(a ma(, a g(a ma(, a g( (, a g(a sgm(a ep( a g(a a g(a ma(, a ma(, S g(a sgm(a ep( a g(a tan a n(a ma(, a g(a reln(a ma(, a,j j g(a reln(a ep(a ep( a ma(, a g(a tan(a g(a tan(a ep(aep( a g(a reln(a ma(, a Tops: sngle dden layer neural netork g(a reln(a ma(, a ep(a ep( p(y g(a (out (2 g(a tan(a g(a ma(, a f g ( g(a ma layer nput atvaton: jhdden g( ep(aep( p(y g(a ma(, g( g( g(a reln(a a a a a j,jma(, j g( g(a ma(,ag(a reln Mat for my sldes Feedf g(a reln(a g( j j,j a,j j,j j > Mat for m j,j f o( j Hdden >,j layer atvaton: a j ma(,,jreln(a g(a g(a g( g(a g( a a g(a g(a g(a g( a a j,j j g(a g( Output g(a( layer atvaton: a a a a a j,j a j j,j,j a a > f o > (out > d o g ( g(a( d j >,j g(a f o (out > NEURAL NETORK (out o g ( o g ( output atvaton funton g(a a a
NEURAL NETORK Tops: sngle dden layer neural netork Réseau de neurones z 2 - - z k - y y 2 - - - 2 as - y -4 7-5 5 sorte k y kj 2 j aée j - - - 2 entrée 2 (from asal Vnent s sldes 2
NEURAL NETORK Tops: sngle dden layer neural netork 2 s oues R R 2 R2 R 2 (from asal Vnent s sldes 3
NEURAL NETORK Tops: sngle dden layer neural netork z 2 y 2 z y 3 y y y 2 y 3 y 4 y 4 2 (from asal Vnent s sldes 4
NEURAL NETORK Tops: unversal appromaton Unversal appromaton teorem (Hornk, 99: a sngle dden layer neural netork t a lnear output unt an appromate any ontnuous funton artrarly ell, gven enoug dden unts Te result apples for sgmod, tan and many oter dden layer atvaton funtons Ts s a good result, ut t doesn t mean tere s a learnng algortm tat an fnd te neessary parameter values! 5
NEURAL NETORK Tops: softma atvaton funton For mult-lass lassfaton: e need multple outputs ( output per lass e ould lke to estmate te ondtonal proalty p(y e use te softma atvaton funton at te output: o(a softma(a strtly postve ep(a ep(a ep(a C ep(a > sums to one redted lass s te one t gest estmated proalty 6
F H p(y p(y De parte p(y p(y > > { ep(a ep(a ep(a > Unve ep(a softma(a o(a p(y ep(a ep(a > o(a softma(a ep(a ep(a ep(a ep(a ep(a ep(a Tops: multlayer neural netork o(a softma(a o(a softma(a ep(a ep(a ep(a ugolaro ep(a g(a a f > NEURAL NETORK C C ep(a f p(y o(a softma(a p(y Could ave layers: L (3 dden f p(y ep(a f C ep(ac (3 ep(a C Se g(a sgm(a > (3 (3 ep( a > p(y ep(a ep(a (k ( C (3 (3 ( > a layer nput ep(a (3 (3 ep(a ep(a k> o(a softma(a ep(a C for ep(a ep(a atvaton o(a f C softma(a ep(a o(a softma(a > ep(a ep(a ep(a ep(a ep(a ep(a ep( (k ( f C a g(a (k ( o(a softma(a ( (k g(a tan(a a ( ep(a ( ep(a (3 (3 a ( ep(aep( p(y p(y (L (3 (3 (L f o(a f f > p(y f g(a g(a g(a ep(a ep(a (k ( o(a softma(a (k ( ep(a ep(a a (3 ( ( ep(a ep(a g(a ma(, a C dden a o(a softma(a Mat for my sldes Feedfor layer atvaton (k from L: (L (3 ep(a (3 (L to ep(a (3 (3> for ep(a Mat my (L ep(a (L C (L (L o(a f o(a o(a softma(a f f g(af ep(a o(a ep(a ( (k a a ( g(a reln(a ma(, g(a (k ( (L (L (3 f a (k ( ( o(a f ( a a a C f (k ( (3 (L (L a ( g(a g(a o(a f g( (3 g( (3 g(a g(a output atvaton g(a layer (kl: (L g(a (L (L (L o(a f o(a f (k ( a ( (k ( a (L ( (L (L (L o(a f o(a f,j d g(a g(a (L j g(a (L (L (L d 7
NEURAL NETORK Tops: parallel t te vsual orte edges mout nose eyes fae 8
BIOLOGICAL NEURONS Tops: synapse, aon, dendrte e estmate around and te numer of neurons n te uman ran: tey reeve nformaton from oter neurons troug ter dendrtes te proess te nformaton n ter ell ody (soma tey send nformaton troug a ale alled an aon te pont of onneton eteen te aon ranes and oter neurons dendrtes are alled synapses 9
BIOLOGICAL NEURONS Tops: synapse, aon, dendrte Sgnal transmsson Computaton Sgnal reepton Synapses Aon Cell ody Dendrtes Oter neurons (from Hyvärnen, Hurr and Hoyer s ook 2
BIOLOGICAL NEURONS Tops: aton potental, frng rate An aton potental s an eletral mpulse tat travels troug te aon: ts s o neurons ommunate t generates a spke n te eletr potental (voltage of te aon an aton potental s generated at neuron only f t reeves enoug (over some tresold of te rgt pattern of spkes from oter neurons Neurons an generate several su spkes every seonds: te frequeny of te spkes, alled frng rate, s at araterzes te atvty of a neuron - neurons are alays frng a lttle t, (spontaneous frng rate, ut tey ll fre more, gven te rgt stmulus 2
BIOLOGICAL NEURONS Tops: aton potental, frng rate Frng rates of dfferent nput neurons omne to nfluene te frng rate of oter neurons: dependng on te dendrte and aon, a neuron an eter ork to nrease (ete or desrease (nt te frng rate of anoter neuron Ts s at artfal neurons appromate: te atvaton orresponds to a sort of frng rate te egts eteen neurons model eter neurons ete or nt ea oter te atvaton funton and as model te tresolded eavor of aton potentals 22
BIOLOGICAL NEURONS Huel & esel eperment ttp://youtueom/at?v8vdff3egfg&featurerelated 23
CONCLUSION e ave seen te most ommon: atvaton funtons netork topologes (layer-se e ould easly ave desgned more omplated atvaton funtons and topologes: ould get more nspraton from neurosene Hoever, tose dsussed ere tend to ork fne 24