Neural Networks. Class 22: MLSP, Fall 2016 Instructor: Bhiksha Raj

Transcription

1 Neural Networs Class 22: MLSP, Fall 2016 Instructor: Bhsha Raj

2 IMPORTANT ADMINSTRIVIA Fnal wee. Project presentatons on 6th 18797/

3 Neural Networs are tang over! Neural networs have become one of the major thrust areas recently n varous pattern recognton, predcton, and analyss problems In many problems they have establshed the state of the art Often exceedng prevous benchmars by large margns 18797/

4 Recent success wth neural networs Some recent successes wth neural networs A bt of hyperbole, but stll /

5 Recent success wth neural networs Some recent successes wth neural networs A bt of hyperbole, but stll /

6 Recent success wth neural networs Some recent successes wth neural networs A bt of hyperbole, but stll /

7 Recent success wth neural networs Some recent successes wth neural networs A bt of hyperbole, but 18797/11755 stll.. 7

8 Recent success wth neural networs Captons generated entrely by a neural networ 18797/

9 Successes wth neural networs And a varety of other problems: Image recognton Sgnal enhancement Even predctng stoc marets! 18797/

10 So what are neural networs?? Voce sgnal N.Net Transcrpton Image N.Net Text capton Game State N.Net Next move What are these boxes? 18797/

11 So what are neural networs?? It began wth ths.. Humans are very good at the tass we just saw Can we model the human bran/ human ntellgence? An old queston datng 18797/11755 bac to Plato and Arstotle.. 11

12 Observaton: The Bran Md 1800s: The bran s a mass of nterconnected neurons 18797/

13 Bran: Interconnected Neurons Many neurons connect n to each neuron Each neuron connects out to many neurons 18797/

14 The bran s a connectonst machne The human bran s a connectonst machne Ban, A. (1873). Mnd and body. The theores of ther relaton. London: Henry Kng. Ferrer, D. (1876). The Functons of the Bran. London: Smth, Elder and Co Neurons connect to other neurons. The processng/capacty of the bran s a functon of these connectons Connectonst machnes emulate ths structure 18797/

15 Connectonst Machnes Neural networs are connectonst machnes As opposed to Von Neumann Machnes Von Neumann Machne Neural Networ PROCESSOR PROGRAM DATA NETWORK Processng unt Memory The machne has many processng unts The program s the connectons between these unts Connectons may also defne memory 18797/

16 Modellng the bran What are the unts? A neuron: Soma Dendrtes Axon Sgnals come n through the dendrtes nto the Soma A sgnal goes out va the axon to other neurons Only one axon per neuron Factod that may only nterest me: Neurons do not undergo cell dvson 18797/

17 McCullough and Ptts The Doctor and the Hobo.. Warren McCulloch: Neurophyscan Walter Ptts: Homeless wannabe logcan who arrved at hs door 18797/

18 The McCulloch and Ptts model A sngle neuron A mathematcal model of a neuron McCulloch, W.S. & Ptts, W.H. (1943). A Logcal Calculus of the Ideas Immanent n Nervous Actvty, Bulletn of Mathematcal Bophyscs, 5: , 1943 Threshold Logc Note: McCullough and Ptts orgnal model was actually slghtly dfferent ths model s actually due to Rosenblatt 18

19 The soluton to everythng Fran Rosenblatt Psychologst, Logcan Inventor of the soluton to everythng, aa the Perceptron (1958) A mathematcal model of the neuron that could solve everythng!!! 19

20 Smplfed mathematcal model Number of nputs combne lnearly Threshold logc: Fre f combned nput exceeds threshold Y = 1 f w x + b > 0 0 else 18797/

21 Smplfed mathematcal model A mathematcal model Orgnally assumed could represent any Boolean crcut Rosenblatt, 1958 : the embryo of an electronc computer that [the Navy] expects wll be able to wal, tal, see, wrte, reproduce tself and be conscous of ts exstence 18797/

22 Perceptron X X Y X -1 0 X Y X X Y Y Boolean Gates But 18797/

23 Perceptron No soluton for XOR! Not unversal! X??? X Y Y Mnsy and Papert, /

24 A sngle neuron s not enough Indvdual elements are wea computatonal elements Marvn Mnsy and Seymour Papert, 1969, Perceptrons: An Introducton to Computatonal Geometry Networed elements are requred 18797/

25 Mult-layer Perceptron! X X Y X Y XOR Y -1-1 Hdden Layer X Y The frst layer s a hdden layer 18797/

26 Mult-Layer Perceptron ( A&X&Z A&Y )&( X & Y X&Z ) X Y Z A Even more complex Boolean functons can be composed usng layered networs of perceptrons Two hdden layers n above model 1 21 In fact can buld any Boolean functon over any number of nputs 18797/

27 Mult-layer perceptrons are unversal Boolean functons A mult-layer perceptron s a unversal Boolean functon In fact, an MLP wth only one hdden layer s a unversal Boolean functon! 18797/

28 Constructng Boolean functons wth only one hdden layer x 1 x 2 y 1 y 2 x 1 x 2 y 1 y 2 + x 1 x 2 y 1 y 2 + x 1 x 2 y 1 y terms 2 n unts 2n nputs Any Boolean formula can be expressed by an MLP wth one hdden layer Any Boolean formula can be expressed n Conjunctve Normal Form The one hdden layer can be exponentally wde But the same formula can be obtaned wth a much smaller networ f we have multple hdden layers 18797/

29 Neural Networs: Mult-layer Perceptrons Voce sgnal N.Net Transcrpton Image N.Net Text capton In realty the nput to these systems s not Boolean Inputs are contnuous valued Sgnals may be contnuous valued Image features are contnuous valued Pxels are mult-valued (e.g ) 29

30 MLP on contnuous nputs The nputs are contnuous valued Threshold logc: Fre f combned nput exceeds threshold Y = 1 f w x + b > 0 0 else 18797/

31 A Perceptron on Reals w 1,w 2 x 2 y = 1 f w x T 0 else x 1 A perceptron operates on real-valued vectors Ths s just a lnear classfer x 2 x /

32 Booleans over the reals x 2 Can now be composed nto networs to compute arbtrary classfcaton boundares x 1 The networ must fre f the nput s n the coloured area 18797/

33 Booleans over the reals x 2 x 1 x 2 x 1 The networ must fre f the nput s n the coloured area 18797/

34 Booleans over the reals x 2 x 1 x 2 x 1 The networ must fre f the nput s n the coloured area 18797/

35 Booleans over the reals x 2 x 1 x 2 x 1 The networ must fre f the nput s n the coloured area 18797/

36 Booleans over the reals x 2 x 1 x 2 x 1 The networ must fre f the nput s n the coloured area 18797/

37 Booleans over the reals x 2 x 1 x 2 x 1 The networ must fre f the nput s n the coloured area 18797/

38 Booleans over the reals 3 N x x =1 AND y N? y 1 y 2 y 3 y 4 y x 2 x 1 The networ must fre f the nput s n the coloured area 18797/

39 Booleans over the reals OR AND AND x 2 OR two polygons A thrd layer s requred x 1 x 1 x /

40 How Complex Can t Get An arbtrarly complex decson boundary Bascally any Boolean functon over the basc lnear boundares Even wth a sngle hdden layer! 18797/

41 Composng a polygon N y =1 AND N? y 1 y 2 y 3 y 4 y 5 x 2 x The polygon net Increasng the number of sdes shrns the area outsde the polygon that have sum close to N 18797/

42 Composng a crcle No nonlnearty appled! + The crcle net Very large number of neurons Crcle can be of arbtrary dameter, at any locaton Acheved wthout usng a thresholdng functon!! 42

43 Addng crcles No nonlnearty appled! + The sum of two crcles sub nets s exactly a net wth output 1 f the nput falls wthn ether crcle 18797/

44 Composng an arbtrary fgure + Just ft n an arbtrary number of crcles More accurate approxmaton wth greater number of smaller crcles A lesson here that we wll refer to agan shortly /

45 Story so far.. Mult-layer perceptrons are Boolean networs They represent Boolean functons over lnear boundares They can approxmate any boundary Usng a suffcently large number of lnear unts Complex Boolean functons are better modeled wth more layers Complex boundares are more compactly represented usng more layers 18797/

46 Lets loo at the weghts x 1 y = 1 f w x T 0 else x 2 x 3 x N y = 1 f x T w T 0 else What do the weghts tell us? The neuron fres f the nner product between the weghts and the nputs exceeds a threshold 18797/

47 The weght as a template x 1 x 2 X T W > T cos θ > T X w x 3 x N θ < cos 1 T X The perceptron fres f the nput s wthn a specfed angle of the weght Represents a convex regon on the surface of the sphere! The networ s a Boolean functon over these regons. The overall decson regon can be arbtrarly nonconvex Neuron fres f the nput vector s close enough to the weght vector. If the nput pattern matches the weght pattern closely enough 18797/

48 The weght as a template W X X y = 1 f w x T 0 else Correlaton = 0.57 Correlaton = 0.82 If the correlaton between the weght pattern and the nputs exceeds a threshold, fre The perceptron s a correlaton flter! 18797/

49 The MLP as a Boolean functon over feature detectors DIGIT OR NOT? The nput layer comprses feature detectors Detect f certan patterns have occurred n the nput The networ s a Boolean functon over the feature detectors I.e. t s mportant for the frst layer to capture relevant patterns 49

50 The MLP as a cascade of feature detectors DIGIT OR NOT? The networ s a cascade of feature detectors Hgher level neurons compose complex templates from features represented by lower-level neurons Rs n ths perspectve: Upper level neurons may be performng OR Loong for a choce of compound patterns 50

51 Story so far MLPs are Boolean machnes They represent Boolean functons over lnear boundares They can represent arbtrary boundares Perceptrons are correlaton flters They detect patterns n the nput MLPs are Boolean formulae over patterns detected by perceptrons Hgher-level perceptrons may also be vewed as feature detectors Extra: MLP n classfcaton The networ wll fre f the combnaton of the detected basc features matches an acceptable pattern for a desred class of sgnal E.g. Approprate combnatons of (Nose, Eyes, Eyebrows, Chee, Chn) Face 18797/

52 MLP as a contnuous-valued regresson h 2 h 1 h n x 1 1 T 1 T 1 T 2 T f(x) T 1 T 2 x h 1 + h 2 x h n MLPs can actually compose arbtrary functons to arbtrary precson Not just classfcaton/boolean functons 1D example Left: A net wth a par of unts can create a pulse of any wdth at any locaton Rght: A networ of N such pars approxmates the functon wth N scaled pulses 52

53 MLP as a contnuous-valued regresson + h 1 h 2 h 1 h 2 h n h n MLPs can actually compose arbtrary functons Even wth only one layer To arbtrary precson The MLP s a unversal approxmator! 18797/

54 Mult-layer perceptrons are unversal functon approxmators X, Y N.Net Z, Colour A mult-layer perceptron s a unversal functon approxmator Horn, Stnchcombe and Whte 1989, several others 18797/

55 What s nsde these boxes? Voce sgnal N.Net Transcrpton Image N.Net Text capton Game State N.Net Next move Each of these tass s performed by a net le ths one Functons that tae the gven nput and produce the requred output 18797/

56 Story so far MLPs are Boolean machnes They represent arbtrary Boolean functons over arbtrary lnear boundares MLPs perform classfcaton MLPs can compute arbtrary real-valued functons of arbtrary real-valued nputs To arbtrary precson They are unversal approxmators 18797/

57 Buldng a networ for a tas 18797/

58 These tass are functons Voce sgnal N.Net Transcrpton Image N.Net Text capton Game State N.Net Next move Each of these boxes s actually a functon E.g f: Image Capton 18797/

59 These tass are functons Voce sgnal Transcrpton Image Text capton Game State Next move Each box s actually a functon E.g f: Image Capton It can be approxmated by a neural networ 59

60 The networ as a functon Input Output Inputs are numerc vectors Numerc representaton of nput, e.g. audo, mage, game state, etc. Outputs are numerc scalars or vectors Numerc encodng of output from whch actual output can be derved E.g. a score, whch can be compared to a threshold to decde f the nput s a face or not Output may be mult-dmensonal, f tas requres t 18797/

61 The networ s a functon Gven an nput, t computes the functon layer wse to predct an output 18797/

62 A note on actvatons x 1 x 2 x 3 x N sgmod tanh Prevous explanatons assumed networ unts used a threshold functon for actvaton In realty, we use a number of other dfferentable functons Mostly, but not always, squashng functons 62

63 The entre networ Inputs (D-dmensonal): X 1,, X D Weghts from th node n l th layer to j th node n l+1 th l layer: W,j Complete set of weghts {W l,j ; l = 0.. L, = 1 N l, j = 1 N l+1 } Outputs (M-dmensonal): O 1,, O M 18797/

64 Mang a predcton (Aa forward pass) Illustraton wth a networ wth X N-1 hdden layers f N O 18797/

65 Forward Computaton X I 1 f 1 O 1 I N-1 O N-1 IN O N f 1 f 1 f N f N f 1 I W 1 j 1 j X f N 1 For the jth neuron of the 1 st hdden layer: Input to the actvaton functons n the frst layer 18797/

66 Forward Computaton X I 1 f 1 O 1 I N-1 O N-1 IN O N f 1 f 1 f N f N f 1 I W 1 j 1 j X f N 1 For the jth neuron of the 1 st hdden layer: Input to the actvaton functons n the frst layer O 1 j f ( 1 1 I j ) Output of the jth neuron of the 1 st hdden layer: 18797/

67 Forward Computaton I -1 O -1 I O I N-1 O N-1 IN O N f N f N f f N 1 For the jth neuron of the th hdden layer: Input to the actvaton functons n the th layer I j W j O /

68 Forward Computaton I -1 O -1 I O I N-1 O N-1 IN O N f N f N f f N 1 For the jth neuron of the th hdden layer: Input to the actvaton functons n the th layer I j W j O 1 O j Output of the jth neuron of the th hdden layer: f ( I j 18797/ )

69 Forward Computaton I -1 O -1 I O I N-1 O N-1 IN O N f N f N I W 1 j 1 j X f f N 1 ITERATE FOR = 1:N for j = 1:layer-wdth Output O N s the output of the networ O I j j W f ( I j j ) O /

70 The problem of tranng the networ The networ must be composed correctly to produce the desred outputs Ths s the problem of tranng the networ 18797/

71 The Problem of Tranng a Networ f(x) X Y = f(x) X Y = f(x) A neural networ s effectvely just a functon What maes t an NNet s the structure of the functon Taes an nput Generally mult-dmensonal Produces an output May be un- or mult-dmensonal Challenge: How to mae t produce the desred output for any gven nput 18797/

72 Buldng by desgn Y X 2 X 2 X 1 X 1 Soluton 1: Just hand desgn t What wll the networ for the above functon be? We can hand draw ths one /

73 Tranng from examples Most functons are too complex to hand-desgn Partcularly n hgh-dmensonal spaces Instead, we wll try to learn t from tranng examples Input-output pars In realty the tranng data wll not be even remotely as dense as shown n the fgure It wll be several orders of magntude sparser 73

74 General approach to tranng Blue lnes: error when functon s below desred output Blac lnes: error when functon s above desred output Defne an error between the actual networ output for any parameter value and the desred output Error can be defned n a number of ways Many Dvergence functons have been defned Typcally defned as the sum of the error over ndvdual tranng nstances 18797/

75 Recall: A note on actvatons x 1 x 2 x 3 x N sgmod tanh Composng/learnng networs usng threshold actvatons s a combnatoral problem Exponentally hard to fnd the rght soluton The smoother dfferentable actvaton functons enable learnng through optmzaton 75

76 ERROR Mnmzng Error Problem: Fnd the parameters at whch ths functon acheves a mnmum Subject to any constrants we may pose Typcally, ths cannot be found usng a closed-form formula 18797/

77 The Approach of Gradent Descent Iteratve soluton: Start at some pont Fnd drecton n whch to shft ths pont to decrease error Ths can be found from the slope of the functon A postve slope movng left decreases error A negatve slope movng rght decreases error Shft pont n ths drecton The sze of the shft depends 18797/11755 on the slope 77

78 The Approach of Gradent Descent Mult-dmensonal functon: Slope replaced by vector gradent From current pont shft n drecton opposte to gradent Sze of shft depends on magntude of gradent y y x f x y x f f ), ( ), ( 18797/

79 The Approach of Gradent Descent Steps are smaller when the slope s smaller, because the optmum value s generally at a locaton of near-zero slope The gradent descent algorthm W W η W E X; W Untl error E(X;W) converges η s the step sze 18797/

80 The Approach of Gradent Descent Steps are smaller when the slope s smaller, because the optmum value s generally at a locaton of near-zero slope Needs computaton of gradent of error w.r.t. networ parameters The gradent descent algorthm W W η W E X; W Untl error E(X;W) converges η s the step sze 18797/

81 Gradents: Algebrac Formulaton W N f N O X W 1 f 1 f 1 W f f N 1 Weghts matrx Actvaton functon The networ s a nested functon o = f N (W N f N 1 W N 1 f N 2 W N 2 f (W f 1 ( f 1 W 1 X ) )) E = Dv y, o = y o /

82 A note on algebra Y f N O Dv(O,Y) E W N W X W 1 f 1 f 1 I f f N 1 I W f 1 W E E f T I 1 Usng f to represent the output of the actvatons rather than O for easer nterpretaton Usng subscrpts rather than superscrpts to represent layer 18797/

83 A note on algebra E E I f I f X f 1 W 1 f f N 1 W N f N O Y E Dv(O,Y) I ', ',2 ', L f f f f I E f f f E f f I f ',1 ',1 ', ', ',1 ' L f f f E E f I f ' 18797/

84 Gradents: Algebrac Formulaton Y f N O Dv(O,Y) E W N X W 1 f 1 Generc rule ' T W E f E f f 1 T f 1 s the vector of outputs of the -1 th layer f E s the gradent of the error w.r.t. the vector of outputs of the th layer ' f f 1 W f s the vector of dervatves of the actvatons of the th layer w.r.t ther nputs W f N 1 E 1 ' f1 f 1 E X taen at current nput W f /

85 Gradents: Algebrac Formulaton Y f N O Dv(O,Y) E W N X W 1 f 1 f 1 W f f N 1 ' T W E f E f f 1 What s ths? 18797/

86 A note on algebra Y f N O Dv(O,Y) E W N W X W 1 f 1 f 1 I f f N 1 I W f 1 f 1 I W T f 1 f f 1 I I f W T ' f 0 0 0,1 f 0 ',2 0 0 ', L 18797/ f 0 0 0

87 Gradents: Algebrac Formulaton Y f N O Dv(O,Y) E W N X W 1 f 1 f 1 W f W +1 f +1 ' T W E f E f f 1 f E ( T ' ) f f E W 1 f 1 1 f 1 f 1 E T ' f E W f E f 18797/

88 Gradents: Algebrac Formulaton ' T W E f E f f 1 T ' f E W f 1 1 f1 E f N W N O Y Dv(O,Y) E X W 1 f 1 f N E O f 1 W Output layer f Dv( O, Y ) f N 1 Dv( O, Y O1 Dv( O, Y O N ) ) 18797/

89 Gradents: Algebrac Formulaton Y f N O Dv(O,Y) E W N X W 1 f 1 f 1 W f f N 1 ' T ' T ' T ' T W E f W W WN N ODv O Y 1 f 1 2 f 2... f (, ) f /

90 Gradents: Bac Propagaton Y f N O Dv(O,Y) E W N X W 1 f 1 f 1 W f f N 1 Forward pass: Intalze f X 0 For = 1 to N: f f W f ) ( 1 Output O f N 18797/

91 Gradents: Bac Propagaton Y f N O Dv(O,Y) E W N X W 1 f 1 f 1 W f f N 1 Error dvergence: For = N downto 1: f N E Dv O ( O, Y ) ' T W E f E f f 1 (actual recurson) 1 T ' f E W f f E 91

92 BP: Local Formulaton Y f N O Dv(O,Y) E W N X W 1 f 1 f 1 W f f N 1 The networ agan 18797/

93 Gradents: Local Computaton I -1 O -1 I O I N-1 O N-1 IN O N f N Dv(O,Y) E f N f f N 1 Redrawn Separately label nput and output of each node 18797/

94 Forward Computaton X I 1 f 1 O 1 I N-1 O N-1 IN O N f 1 f 1 f N f N f 1 I W 1 j 1 j X f N /

95 Forward Computaton I -1 O -1 I O I N-1 O N-1 IN O N f N f N I W 1 j 1 j X f f N 1 I j W j O /

96 Forward Computaton I -1 O -1 I O I N-1 O N-1 IN O N f N f N I W 1 j 1 j X f f N 1 I j W j O 1 O j f ( I j 18797/ )

97 Forward Computaton I -1 O -1 I O I N-1 O N-1 IN O N f N f N I W 1 j 1 j X f f N 1 ITERATE FOR = 1:N for j = 1:layer-wdth O I j j W f ( I j j ) O /

98 Gradents: Bacward Computaton I -1 O -1 I O I N-1 O N-1 IN O N f N Dv(O,Y) E f N f f N /

99 Gradents: Bacward Computaton I -1 O -1 I O I N-1 O N-1 IN O N f N Dv(O,Y) E f N E Dv( O, Y ) O O N N f f N /

100 Gradents: Bacward Computaton I -1 O -1 I O I N-1 O N-1 IN O N f N Dv(O,Y) E f N E Dv( O, Y ) O O N N f f N 1 E I N do di N N E O N f ' N ( I N ) E O N 18797/

101 Gradents: Bacward Computaton I -1 O -1 I O I N-1 O N-1 IN O N f N Dv(O,Y) E f N f f N 1 E Dv( O, Y ) O O E I N N f ' N ( I N N E ) O N E N O 1 j I O N j N 1 E I N j j W E I N N N 1 N Note: j I O W other N j j j 18797/ stuff

102 Gradents: Bacward Computaton I -1 O -1 I O I N-1 O N-1 IN O N f N Dv(O,Y) E f N E I f f ' ( I ) E O f N 1 E Dv( O, Y ) O O E I N N E N O 1 f ' N ( I N N E ) O N N E Wj I j N j 18797/

103 Gradents: Bacward Computaton I -1 O -1 I O I N-1 O N-1 IN O N f N Dv(O,Y) E f N f f N 1 E Dv( O, Y ) O O E I N f ' ( I N E ) O E O 1 j I O j 1 E I j j W E I j j 18797/

104 Gradents: Bacward Computaton I -1 O -1 W j I O I N-1 O N-1 IN O N f N Dv(O,Y) E f N E W j I W j j f E I j O 1 f N 1 E I j E Dv( O, Y ) O O E I N E O 1 f ' ( I N E ) O E Wj I j j 18797/

105 Gradents: Bacward Computaton I -1 O -1 I O I N-1 O N-1 IN O N f N Dv(O,Y) E f N Intalze: Gradent w.r.t networ output E Dv( O, Y ) O O N N For = N..1 For = 1:layer-wdth E I E O E 1 f ' ( I E Wj I j E ) O E /11755 Wj 105 I j O

106 Bacpropagaton: Multplcatve Networs Some types of networs have multplcatve combnaton I -1 O -1 Z W Forward: 1 1 j O O z Bacward: j j z E O z E O z O E j z E O O E /

107 Overall Approach For each data nstance Forward pass: Pass nstance forward through the net. Store all ntermedate outputs of all computaton Bacward pass: Sweep bacward through the net, teratvely compute all dervatves w.r.t weghts Actual Error s the sum of the error over all tranng nstances E = Dv y(x), o(x) = E x x Actual gradent s the sum or average of the dervatves computed for each tranng nstance x W E = x W E(x) W W η W E 18797/

108 Issues and Challenges What does t learn? Speed Wll not address ths Does t do what we want t to do? Next class: Varatons of nnets MLP, Convoluton, recurrence Nnets for varous tass Image recognton, speech recognton, sgnal enhancement, modellng language /