CSCE 496/896 Lecture 2: Basic Artificial Neural Networks. Stephen Scott. Introduction. Supervised Learning. Basic Units.

Transcription

1 (Adaped from Vinod Variyam, Ehem Alpaydin, Tom Michell, Ian Goodfellow, and Aurélien Géron) learning is mos fundamenal, classic form machine learning par comes from he par labels for examples (insances) Many ways o do supervised learning; we ll focus on arificial neural neworks, which are he basis for deep learning ssco@cseunledu / / ANNs Properies Consider humans: Properies arificial neural nes (ANNs): Toal number neurons 0 0 Neuron swiching ime 0 3 second (vs 0 0 ) Connecions per neuron Many neuron-like swiching unis Many weighed inerconnecions among unis Highly parallel, disribued process Scene recogniion ime 0 second 00 inference seps doesn seem like enough ) massive parallel compuaion Emphasis on uning weighs auomaically Srong differences beween ANNs for ML and ANNs for biological modeling 3 / 4 / When o Consider ANNs Hisory ANNs Inpu is high-dimensional discree- or real-valued (eg, raw sensor inpu) Oupu is discree- or real-valued Oupu is a vecor values Possibly noisy daa Form arge funcion is unknown Human readabiliy resul is unimporan Long raining imes accepable The Beginning: Linear unis and he Percepron algorihm (940s) Spoiler Aler: sagnaed because inabiliy o handle daa no linearly separable Aware usefulness muli-layer neworks, bu could no rain The Comeback: Training muli-layer neworks wih agaion (980s) Many applicaions, bu in 990s replaced by large-margin approaches such as suppor vecor machines and boosing 5 / 6 /

2 Hisory ANNs (con d) Ouline The Resurgence: Deep archiecures (000s) Beer hardware and sware suppor allow for deep (> 5 8 layers) neworks Sill use agaion, bu Larger daases, algorihmic improvemens (new loss and acivaion funcions), and deeper neworks improve performance considerably Very impressive applicaions, eg, capioning images The Ineviable: (TBD) Oops learning Basic ANN unis Linear uni Linear hreshold unis Percepron raining rule separable problems and mulilayer neworks agaion acivaion funcions Puing everyhing ogeher 7 / Thank a gamer oday 8 / from Examples from Examples (con d) Le C be he arge funcion (or arge concep) o be learned Think C as a funcion ha akes as inpu an example (or insance) and oupus a label Goal: Given raining se X = {(x, y )} N = where y = C(x ), oupu hypohesis h H ha approximaes C in is classificaions new insances Each insance x represened as a vecor aribues or feaures Eg, le each x =(, x ) be a vecor describing aribues a car; = price and x = engine power In his example, label is binary (iive/aive, yes/no, /0, +/ ) indicaing wheher insance x is a family car : Engine power x x : Price 9 / 0 / Thinking abou C Thinking abou C (con d) Can hink arge concep C as a funcion In example, C is an axis-parallel box, equivalen o upper and lower bounds on each aribue Migh decide o se H (se candidae hypoheses) o he same family ha C comes from No required o do so Can also hink arge concep C as a se iive insances In example, C he coninuous se all iive poins in he plane Use whichever is convenien a he ime : Engine power x e e C p p : Price / /

3 Hypoheses and Error Hypoheses and Error (con d) 3 / A learning algorihm uses raining se X and finds a hypohesis h H ha approximaes C In example, H can be se all axis-parallel boxes If C guaraneed o come from H, hen we know ha a perfec hypohesis exiss In his case, we choose h from he version space = subse H consisen wih X Wha learning algorihm can you hink o learn C? Can hink wo ypes error (or loss) h Empirical error is fracion X ha h ges wrong Generalizaion error is probabiliy ha a new, randomly seleced, insance is misclassified by h Depends on he probabiliy disribuion over insances Can furher classify error as false iive and false aive 4 / Linear Uni (Regression) Linear Threshold Uni (Binary Classificaion) Linear Uni Linear Threshold Uni Percepron Training Rule x x n w w w n x 0 = w 0 n wi x i i=0 ŷ = f (x; w, b) =x > w + b = w + + w n x n + b Linear Uni Linear Threshold Uni Percepron Training Rule x x n w w w n x 0 = w 0 n wi x i i=0 + if f (x; w, b) > 0 y = o(x; w, b) = oherwise n if w > 0 o = { i x i=0 i - oherwise (someimes use 0 insead ) Puing 5 / Things Each weigh vecor w is differen h If se w 0 = b, can simplify above Forms he basis for many oher acivaion funcions Puing 6 / Things Linear Threshold Uni Decision Surface Linear Threshold Uni Non-Numeric Inpus Linear Uni Linear Threshold Uni Percepron Training Rule + + x + - (a) - - Represens some useful funcions Wha parameers (w, b) represen g(, x ; w, b) =AND(, x )? - x + - (b) + Linear Uni Linear Threshold Uni Percepron Training Rule Wha if aribues are no numeric? Encode hem numerically Eg, if an aribue Color has values Red, Green, and Blue, can encode as one-ho vecors [, 0, 0], [0,, 0], [0, 0, ] Generally beer han using a single ineger, eg, Red is, Green is, and Blue is 3, since here is no implici ordering he values he aribue Puing 7 / Things Bu some funcions no represenable Ie, hose no linearly separable Therefore, we ll wan neworks unis Puing 8 / Things

4 Percepron Training Rule ( Algorihm) Where Does he Training Rule Come From? Linear Regression Linear Uni Linear Threshold Uni Percepron Training Rule where x j Ie, if (y w 0 j w j + (y ŷ ) x j is jh aribue raining insance y is label raining insance ŷ is Percepron oupu on raining insance > 0 is small consan (eg, 0) called learning rae ŷ) > 0 hen increase w j wr x j, else decrease Can prove rule will converge if raining daa is linearly separable and sufficienly small Recall iniial linear uni (no hreshold) If only one feaure, hen his is a regression problem Find a sraigh line ha bes fis he raining daa For simpliciy, le i pass hrough he origin Slope specified by parameer w x y Puing 9 / Things 0 / Where Does he Training Rule Come From? Linear Regression Where Does he Training Rule Come From? Convex Quadraic Opimizaion Goal is o find a parameer w o minimize square loss: J(w )=55w 734w mx J(w )= ŷ X y m = w x y = = =(w 8) +(w 465) +(3w 79) +(4w 0) +(5w ) = 55w 734w / / Minimum is a w 485, wih loss 053 Wha s special abou ha poin? Where Does he Training Rule Come From? Where Does he Training Rule Come From? Example Recall ha a funcion has a (local) minimum or maximum where he derivaive is 0 d dw J(w )=0w 734 In our example, iniialize w, hen repeaedly updae w 0 = w (0 w 734) Seing his = 0 and solving for w yields w 485 Moivaes he use gradien descen o solve in high-dimensional spaces wih nonconvex funcions: w 0 = w rj(w) is learning rae o moderae updaes is a vecor parial i i n i= Could also updae one a = w (x ) x y 3 / 4 /

5 Where Does he Training Rule Come From? Handling The XOR Problem E[w] 5 J(w) Using linear hreshold unis x B: (0,) D: (,) 5 / n In general, define loss funcion J, compue gradien J wr J s parameers, hen apply gradien descen w XOR General 6 / A: (0,0) g (x) < 0 > 0 < 0 > 0 C: (,0) g (x) Represen wih inersecion wo linear separaors x g (x) = + x / g (x) = + x 3/ = x R : g (x) > 0 AND g (x) < 0 = x R : g (x), g (x) < 0 OR g (x), g (x) > 0 Handling The XOR Problem (con d) Handling The XOR Problem (con d) XOR General Le z i = ( 0 if g i (x) < 0 oherwise Class (, x ) g (x) z g (x) z B: (0, ) / / 0 C: (, 0) / / 0 A: (0, 0) / 0 3/ 0 D: (, ) 3/ / Now feed z, z ino g(z) = z z / z D: (,) < 0 > 0 g(z) XOR General In oher words, we remapped all vecors x o z such ha he classes are linearly separable in he new vecor space Hidden Layer w 30= / z w 3= w 50= / Σw x i 3i i w 53= Inpu Layer w 3= Σw w = i 5izi x 4 w 4= Σw x i i 4i w 54= z w 40= 3/ Oupu Layer This is a wo-layer percepron or wo-layer feedforward neural nework 7 / A: (0,0) B, C: (,0) z 8 / Can use many nonlinear acivaion funcions in hidden layer Handling General Training Muliple Layers XOR General By adding up o hidden layers linear hreshold unis, can represen any union inersecion halfspaces Firs hidden layer defines halfspaces, second hidden layer akes inersecion (AND), oupu layer akes union (OR) Mulilayer In a muli-layer nework, have o une parameers in all layers In order o rain, need o know he gradien he loss funcion wr each parameer The agaion algorihm firs feeds forward he nework s inpus o is oupus, hen propagaes back error via repeaed applicaion chain rule for derivaives Can be decomed in a simple, modular way 9 / Alg 30 /

6 Given a complicaed funcion f ( ), wan o know is parial derivaives wr is parameers Will represen f in a modular fashion via a compuaion graph Eg, le f (w, x) =w 0 x 0 + w Eg, w 0 = 30, w = 0, x 0 = 0, = 40 Mulilayer Mulilayer Alg Alg 3 / 3 / So wha? Can now decome gradien calculaion ino basic operaions = If g(y, z) =y + @z = Via chain @a =(0)(0) =0 Mulilayer Mulilayer Alg Alg 33 / 34 / The Basics If h(y, z) = z Via chain @x 0 = 0w 0 = 30 How does his help us wih muli-layer ANNs? Firs, le s replace he hreshold funcion wih a coninuous approximaion w x 0 = Mulilayer Alg 35 / So for x = [0, 40] >, rf (w) =[0, 40] > Mulilayer Alg 36 / x x n w w n w 0 (ne) is he logisic funcion (ne) = (a ype sigmoid funcion) Squashes ne ino [0, ] range n ne = w i x i=0 i = f(x; w,b) + e ne o = (ne) = + ē ne

7 The Compuaion Graph The Le f (w, x) =/ ( +exp( (w 0 x 0 + w = 0( /h )= 0073 Mulilayer Mulilayer Alg Alg 37 / 38 / The = 0073() = = 0073 exp(d) = 0966 Mulilayer Mulilayer Alg Alg 39 / 40 / The = 0966( ) =0966 and so on: Noe = (c)( (c)), = Mulilayer Mulilayer Alg 4 / So for x = [0, 40] >, rf (w) =[0966, 07866] > Alg 4 /

8 Weigh Updae Mulilayer Mulilayer If ŷ = (w x ) is predicion on raining insance x wih label y, le loss be J(w) = (ŷ y = ŷ y @ = ŷ y So updae rule is In general, = ŷ y ŷ ŷ x w 0 = w ŷ ŷ ŷ y x Mulilayer Tha updae formula works for oupu unis when we know he arge labels y (here, a vecor o encode muli-class labels) Bu for a hidden uni, we don know is arge oupu! Alg 43 / w 0 = w ŷ ŷ ŷ y x Alg 44 / w ji = weigh from node i o node j Oupu Hidden Mulilayer Le loss on insance (x, y ) be J(w) = P n i= (ŷ i y i ) Weighs w 5 and w 6 ie o oupu unis s and weigh updaes done as before Eg, w 0 53 = 53 = w 53 ŷ ( ŷ )(ŷ y ) 3 Mulilayer Mulivariae chain rule says we sum pahs from J o w 4 @c = ([ŷ ( ŷ )(ŷ y )] [w 54 ][ 4(a)( 4(a))] Alg 45 / Alg 46 / + [ŷ ( ŷ )(ŷ y )] [w 64 ][ 4(a)( 4(a))]) x Hidden Hidden Mulilayer Alg 47 / Analyical soluion is messy, bu we don need he formula; only need o compue gradien The modular form a compuaion graph means ha once we @d we can plug hose values in and compue gradiens for earlier layers Doesn maer if layer is oupu, or farher back; can run indefiniely backward agaion error from oupus o inpus Define error erm hidden node h as X h ŷ h ( ŷ h ) w k,h k, kdown(h) where ŷ k is oupu node k and down(h) is se nodes immediaely downsream h Noe ha his formula is specific o sigmoid unis Mulilayer Alg 48 / We are propagaing back error erms from oupu layer oward inpu layers, scaling wih he weighs Scaling wih he weighs characerizes how much he error erm each hidden uni is responsible for Process: Submi inpus x Feed forward signal o oupus 3 Compue nework loss 4 Propagae error back o compue loss gradien wr each weigh 5 Updae weighs

9 agaion Algorihm Sigmoid Acivaion and Square Loss agaion Algorihm Noes Mulilayer Alg 49 / Iniialize weighs Unil erminaion condiion saisfied do For each raining example (x, y ) do Inpu x o he nework and compue he oupus ŷ For each oupu uni k 3 For each hidden uni h k ŷ k ( ŷ k)(y k ŷ k) h ŷ h ( ŷ h) 4 Updae each nework weigh w j,i X kdown(h) w j,i w j,i + w j,i w k,h where w j,i = j x j,i and x j,i is signal sen from node i o node j k Mulilayer Alg 50 / Formula for assumes sigmoid acivaion funcion Sraighforward o change o new acivaion funcion via compuaion graph Iniializaion used o be via random numbers near zero, eg, from N (0, ) More refined mehods available (laer) Algorihm as presened updaes weighs afer each insance Can also accumulae w j,i across muliple raining insances in he same mini-bach and do a single updae per mini-bach ) Sochasic gradien descen (SGD) Exreme case: Enire raining se is a single bach (bach gradien descen) Oupu Hidden Oupu Hidden 5 / Given hidden layer oupus h Linear uni: ŷ = w > h + b Minimizing square loss wih his oupu uni maximizes log likelihood when labels from normal disribuion Ie, find a se parameers ha is mos likely o generae he labels he raining daa Works well wih GD raining Sigmoid: ŷ = (w > h + b) Approximaes non-differeniable hreshold funcion More common in older, shallower neworks Can be used o predic probabiliies Smax uni: Sar wih z = W > h + b Predic probabiliy label i o be P smax(z) i =exp(z i )/ j exp(z j) Coninuous, differeniable approximaion o argmax Oupu Hidden 5 / Recified linear uni (ReLU): max{0, W > x + b} Good defaul choice In general, GD works well when funcions nearly linear Variaions: leaky ReLU and exponenial ReLU replace z < 0 side wih 00z and (exp(z) ), respecively Logisic sigmoid (done already) and anh Nice approximaion o hreshold, bu don rain well in deep neworks since hey saurae Puing Everyhing Hidden Layers Puing Everyhing Universal Approximaion Theorem How many layers o use? Deep neworks build poenially useful represenaions daa via comiion simple funcions Performance improvemen no simply from more complex nework (number parameers) Increasing number layers sill increases chances overfiing, so need significan amoun raining daa wih deep nework; raining ime increases as well Accuracy vs Deph Accuracy vs Complexiy Any boolean funcion can be represened wih wo layers Any bounded, coninuous funcion can be represened wih arbirarily small error wih wo layers Any funcion can be represened wih arbirarily small error wih hree layers Only an EXISTENCE PROOF Could need exponenially many nodes in a layer May no be able o find he righ weighs Highlighs risk overfiing and need for regularizaion 53 / 54 /

10 Puing Everyhing Iniializaion Puing Everyhing Opimizers Previously, iniialized weighs o random numbers near 0 (from N (0, )) Sigmoid nearly linear here, so GD expeced o work beer Bu in deep neworks, his increases variance per layer, resuling in vanishing gradiens and poor opimizaion Gloro iniializaion conrols variance per layer: If layer has n in inpus and n ou oupus, iniialize via uniform over [ r, r] or N (0, ) q q 6 r = a and = a nin+nou nin+nou Acivaion a Logisic anh p 4 ReLU Variaions on gradien descen opimizaion: Momenum opimizaion AdaGrad RMSProp Adam 55 / 56 / Puing Everyhing Momenum Opimizaion Puing Everyhing AdaGrad 57 / Use a momenum erm o keep updaes moving in same direcion as previous rials Replace original GD updae w 0 = w where m = w 0 = w m, m + rj(w) rj(w) wih Using sigmoid acivaion and square loss, replace w ji = j x ji wih w ji = j x ji + w ji Can help move hrough small local minima o beer ones & move along fla surfaces 58 / Sandard GD can oo quickly descend seepes slope, hen slowly crawl hrough a valley AdaGrad adaps learning rae by scaling i down in seepes dimensions: w 0 = w rj(w) p s +, where s = s + rj(w) rj(w), and are elemen-wise muliplicaion and division and = 0 0 prevens division by 0 s accumulaes squares gradien, and learning rae for each dimension scaled down Puing Everyhing RMSProp Puing Everyhing Adam AdaGrad ends o sop oo early for neural neworks due o over-aggressive downscaling RMSProp exponenially decays old gradiens o address his Adam (adapive momen esimaion) combines Momenum opimizaion and RMSProp m = m +( )rj(w) s = s +( )rj(w) rj(w) 3 m = m/( ) where w 0 = w rj(w) p s +, s = s +( )rj(w) rj(w) 4 s = s/( ) 5 w 0 = w m p s + Ieraion couner used in 3 and 4 o preven m and s from vanishing Can se = 09, = 0999, = / /