EIE6207: Deep Learning and Deep Neural Networks

Size: px

Start display at page:

Download "EIE6207: Deep Learning and Deep Neural Networks"

Dorcas Pitts
5 years ago
Views:

1 EIE6207: Deep Learning and Deep Neural Networs Man-Wai MAK Dept. of Electronic and Information Engineering, The Hong Kong Polytechnic University mwma References: mwma/papers/bp.pdf S.Y. Kung, M.W. Ma and S.H. Lin, Biometric Authentication: A Machine Learning Approach, Prentice Hall, 2005, Chapter 5. M.W. Ma, Deep Learning: Concepts, Tools, and Applications, IEEE Hong Kong Section, Robotics and Automation/Control Systems Joint Chapter, The Hong Kong Polytechnic University, 16 March ( mwma/papers/ieee-racs-seminar18.pdf) November 9, 2018 Man-Wai MAK (EIE) Deep Learning and DNN November 9, / 50

2 Overview 1 Motivations 2 Deep Learning and DNN Neural Networs NN as Nonlinear Classifier Deep Neural Networs 3 Gradient Descent Simple Gradient Descent Stochastic Gradient Descent 4 Bacpropagation Loss Function: Mean Square Error Loss Function: Cross Entropy 5 Convolutional Neural Networs Man-Wai MAK (EIE) Deep Learning and DNN November 9, / 50

3 Motivations Deep learning is currently the most exciting area in AI, perhaps because of the success of AlaphaGO and Master. Deep learning finds its way in many disciplines, e.g., Power Electronics: Neural Networ Applications in Power Electronics and Motor Drives: An Introduction and Perspective, Autonomous Vehicle: End-to-End Learning for Self-Driving Cars, Nvidia. Multimedia: DNNs win almost all competitions in computer vision, face recognition, speech recognition, natural language processing, etc. Big-Data Analytics: Recommendation systems, customer relationship management. Why does deep learning wor so well? Why you need to learn deep learning? Man-Wai MAK (EIE) Deep Learning and DNN November 9, / 50

4 Definitions AI, Machine Learning, and Deep Deep learning is a branch of machine learning algorithms for training Learning deep neural networs. Deep Learning Machine Learning Artificial Intelligence 5 Conventionally, we have shallow networs such as multi-layer perceptrons (discussed later), SVMs, and GMMs. Deep neural networs have narrow structure but many layers of non-linear processing elements. Man-Wai MAK (EIE) Deep Learning and DNN November 9, / 50

5 What Maes DNN Successful? More complex models can be build, provided that we have more data In the big data era, we have a large amount of data With GPUs and the cloud, we also have a large amount of computational resources Open source efforts (e.g., Theano, TensorFlow, Kaldi, PyTorch, CNTK) also facilitate the development of complex models. Man-Wai MAK (EIE) Deep Learning and DNN November 9, / 50

Neural Networs Neural Networs Artificial Ar&ficial neural neural networs networs (ANNs) (ANNs) are aare family a family of statistical of sta&s&cal learning models learning inspired models by

6 Neural Networs Neural Networs Artificial Ar&ficial neural neural networs networs (ANNs) (ANNs) are aare family a family of statistical of sta&s&cal learning models learning inspired models by biological inspired neural by biological networs neural (the brain). networs (the They brain). are used to approximate some unnown functions that depend on athey largeare number used of to inputs approximate some unnown func&ons that Thedepend simpleston type a large of neural number networs, of inputs called perceptrons, has one neuron The only: simplest type of neural networs has one neuron only! # x = # # "# x 1 x 2 x 3 $ & & & %& h w,b (x) =f(w T x + b) =f X3 w ix i + b i=1 f(z) = 1 1+e z 15 Man-Wai MAK (EIE) Deep Learning and DNN November 9, / 50

Geometric Interpretation of Perceptrons Geometric Interpretation For 2-D inputs, we may use coordinate geometry to understand how a Slopes-intercept form of a plane: perceptron wors.

7 Geometric Interpretation of Perceptrons Geometric Interpretation For 2-D inputs, we may use coordinate geometry to understand how a Slopes-intercept form of a plane: perceptron wors. x1 -slope <latexit sha1_base64="s+kna/148abgczadp+omdb7mffu=">aaacinicbzdlsgmxfiyzxmu9bp0eywfrswzrdcnuhtsok9qfugje3b0gqyjblphfomvosv4fb37ssv4monmw1nyvt/opdxn3m4yy8zrtxvc9naxlldw09s5hd3nre2xx39qtaxorqcpfcqpgmniw0ophhtn6pcgsmnma7t+m+7uhqst4b0zrrqluddhuaqsvbgnuqfr0tgw4gtuyicoqwijzy1tlngouonjcrhqvuzit4e8ff8fpigvtlwp1ptiwjbq0n4urhu9fppugzrhdjrtxppgiprrlzyshhq3uomxxrbvhxascovrddaift3i0fc66hadlig09pzvbh5x68rm85lk2fhfbsaumhtsyhxccd2wzrynhqwuikgbfcpkusmtxhmchara3fn49gearfgu8v/lvzxo6scddseroay+uaalcavkoaiieaiv4bw8oc/ou/phfe5hl5x05wdmypn+bbvsobc=</latexit> z x2 -slope z-intercept <latexit sha1_base64="daqjcoinapwqchdypgvwjirye=">aaacinicbzdnsgmxemezftb6terrs7auflhsfevqtglxwr2a9qyjgnahiabjclk69jn8cv8ba969yaebe8+iwm7b9s6mothf2ayyr9hngnev/o0vlk6tp6zio7ubw9s+vu7ve1bwhfsk5vhwmnouspbxddkf1sfemkc13l8z12spvgmw3szghlog7ioowgy6xapc/xonahwobp1aerutfszhr36bacpccas4ogrcnffwjgexwu8hb9iob+5psy1jlghocedan3wvmq0ekcmip6nsm9y0qqspurrhmusc6lyy+dii5q3shh2pbiygtts/ewswg8ftp0cmz6er43f/2qn2hquwwlo9qewxdwioyrf2cbkuomh1pardf7kyq9pbax1swzlvimstyuf96cragwc75x8o/oc6xr1j4moarh4b44akuwc0ogwog4am8gffw5w7786h8zltxxlsmqmwe873l71zobg=</latexit> z = m1 x1 + m2 x2 + b <latexit sha1_base64="nuhx8yxnaxrh6r4ktzgydip0x4=">aaacehicbzdlssnafizpvnz6i7pw4wawcijqilorii6cvnbxqanytkdtennazewvos/gkbnxvttz6bm59eqdtfrb1hwmf/zmhm/mhcwdko863tbs8srq2xtgobm5t7+zae/snfaes0dqjesxbavaus4wndocthjjsqg4bqadm3g/+uclynf0r4cj9qtursxbgt+fbh05xwxfro6gwjv2koyiw7zjtdizci+dmuijcnd/+6xrgoaackxum3xsbsxyaz4xru7kskjpgmci+2duzyuovlw+m0ilxuiimpalio4n7dypdqqmhcmywlqv5nt879eo9xhpzexke1c0ujhypgm0tgn1mare86ebtcqzb0wyum2mq2cyuqo6ijxz2pybealblrln2781l1oo+naedwdkfgwgvu4rzquacci3ibv3iznq1368p6ni4uwfnoaczi+voficuacg==</latexit> z = m2 x 2 + b b <latexit sha1_base64="a56ltu50zucgz3wpc0w/bqtrc=">aaacrhicbzbnswmxeiazftb1q+rrs7aublhsfevhaixwusf2wrtsirpqsfsyrzabv0r/n/avitt17e69i2i5q1ygbz96zyzixx5xp43mpzszs3pzcym7jxv5zxvvpb2w2tewuoxuiuvsxggnkwutrhhlol2nfcccnvhnyafvkvkmxldmh5ma4guitzlbbrhfmz4qaiqh/2bo5bezytls1ht2ihlbdspfuay5go3eiavud3mhyn9ddkmc94jw8c8c/4grrafruw/9lusjiighncdyt34tncjlgoh2ydvrnebbl3rlsuicaqddpzpisxapqo7utmmdbyrpzdsjltuc2wnbtlx+ndvjp7xaywmexsliotqymyodrnodqshyehayombxvarhf7fshuuykewn9nrqcxdc1pvi/lfglxlj90r++ughepzzwpbyafsah8cgio4btvqbwtcgqfwbj6de+fvexpej6mztrazbabc+fge24srpg==</latexit> x1 mx m x2 y <latexit sha1_base64="wyecycvp5xcgkcrmymcrj00i4=">aaacj3icbzdlsgmxfiyz9vbrrerstbaubhgykyjuhkiblxxsbdoyznjmg5rlgsdehb+bk+glvduxnd6poyxha29ccb//c+mgdxg876czmrq2vpgdo3tb2zu5ffp6hpmshmqlgyqroh0orrqaqggyasskih4zuw/7nuf5/ieptke7nmcztrqcrhqy6ug7xyfr3gw4hnu8idqwsptbxtndioryz5rjmixytififzzxmwrcbn8gbtclspd/axutgrboddp3ytnotium7uzlwgsi9xhxdk0kbanup1o/wcrat0ycsvtwhgrp07sku9zchtpm09oltbh4x62zmoiynvirj4yipd0ujqwacccmwq5vbbs2ticwovatepeqqthyk+euhhyus6b4ixysq63+p7r350xytcze7lgcbyde+cdc1agt6acqgcdj/acxsgb8+y8ox/o57q148xmdsfcon+/ov6iq==</latexit> x1 <latexit sha1_base64="wyecycvp5xcgkcrmymcrj00i4=">aaacj3icbzdlsgmxfiyz9vbrrerstbaubhgykyjuhkiblxxsbdoyznjmg5rlgsdehb+bk+glvduxnd6poyxha29ccb//c+mgdxg876czmrq2vpgdo3tb2zu5ffp6hpmshmqlgyqroh0orrqaqggyasskih4zuw/7nuf5/ieptke7nmcztrqcrhqy6ug7xyfr3gw4hnu8idqwsptbxtndioryz5rjmixytififzzxmwrcbn8gbtclspd/axutgrboddp3ytnotium7uzlwgsi9xhxdk0kbanup1o/wcrat0ycsvtwhgrp07sku9zchtpm09oltbh4x62zmoiynvirj4yipd0ujqwacccmwq5vbbs2ticwovatepeqqthyk+euhhyus6b4ixysq63+p7r350xytcze7lgcbyde+cdc1agt6acqgcdj/acxsgb8+y8ox/o57q148xmdsfcon+/ov6iq==</latexit> <latexit sha1_base64="bfyl/9dqng08euahcwimyuzww=">aaacj3icbzdlsgmxfiyz9vbrrerstbaubhgykyjuhkiblxxsbdoyznjmg5rlgsdehb+bk+glvduxnd6poyxha29ccb//c+mgdxg876czmrq2vpgdo3tb2zu5ffp6hpmshmqlgyqroh0orrqaqggyasskih4zuw/7nuf5/ieptke7nmcztrqcrhqy6ug7xyfr3gw4hnu8idqwsptbxtndioryz5rjmixyvgybc/1jggxwz9bacyieur/wh2je06ewqxp3fs92lrtpazfzo5sjzrecpdrlzqtcssjbqetf41g0sodgelluxg4uf9opihrpesh7eti9pribsz+v2smjrpsp1teisectw9fcyngwrfjsemvwyynlscsqh0rxd2edbwyrrir/lrcn+ogxlucu5vuf6d+ef8vxmniw4asfgbpgaptblaiaksdgcbyav/dmpdvvzofzow3nolozqzaxzvcvovkig==</latexit> z = m1 x 1 + b z = f (x1, x2 ) <latexit sha1_base64="1gnhunkaqefrce3fsf877lbfmbs=">aaacvhicbzblswmxfixtqy9atvzdugmwgqkwmsloplb047kcfubbhitn2gaygzkmtb364/wtgls3bnxvwvqh1eqfc1/ouzcbdo4408z1x1joemv1bt2zd3c2s7t5hf3glrgita6vyqfaachbsumgg01akbky0yz+uj74zueqnjphnrlgtcvqfcgcrpcxp9vf0cv4xtwypsecr9sqwwjz4swogllqxkmuyzoofscwqpcdonvybezeqdh161yrgfl7gld1rwl3hzkib51fz8a6cnssxoaahhwrc9nzldbcndclcf6msarog8ohvathgiqxu3myywhwr9gagle3qwkn6cynbquuhwhzsinpxy95e/m9rxya47cysgjdqzi7fmqcggnicieu5qyprsaigl2r5d0ule2nx/xcfinlwhemsr/ivgues5je/2vfc9msetaqfgebwbd1yakrgbnvahbdybn/aopllpqu8n7azorp3ufgcf/con9wxwl6z7</latexit> z b <latexit sha1_base64="oaxcprt5/yyfpgu3tbpzzgslio=">aaacm3icbzdlsgmxfiyz9vbrrerstbaublhmfeercu3livyc7rlsnk0du0mq5krtpfxzfwfdzqvtwvt76d6qw0rqcofpznlvw45ewb1/1wemvrg5tbye3uzu7e/h68kiizaqilrpjpaphpclnas0bzithyoigtmt4t7dpf59oozgtyaquibanuc1myegsv56zvsscb8d/ztnph5y3llefu1jdynmplzsxir2lhraaf3uxn864oxcacbw8owtapep+etxosrijghcdz1zw1nm0bkmmlt/aayhid3vo3wkabnxneprhecxapqxbutmdjyqfydijlqecgw7btjdvvybip/v6pfpxzdfosroqgzhwphhbojj4bbflougd6wgihi9q2qdjfcxfhbf65gmupzu7xlc1ahs95bs57umwub+f2jmejoavnwanxoauqqmuaqhp4bw8gxfnxfl0xs7xrdxhzgeowui43z+lyqz6</latexit> 1 x2 <latexit sha1_base64="bfyl/9dqng08euahcwimyuzww=">aaacj3icbzdlsgmxfiyz9vbrrerstbaubhgykyjuhkiblxxsbdoyznjmg5rlgsdehb+bk+glvduxnd6poyxha29ccb//c+mgdxg876czmrq2vpgdo3tb2zu5ffp6hpmshmqlgyqroh0orrqaqggyasskih4zuw/7nuf5/ieptke7nmcztrqcrhqy6ug7xyfr3gw4hnu8idqwsptbxtndioryz5rjmixyvgybc/1jggxwz9bacyieur/wh2je06ewqxp3fs92lrtpazfzo5sjzrecpdrlzqtcssjbqetf41g0sodgelluxg4uf9opihrpesh7eti9pribsz+v2smjrpsp1teisectw9fcyngwrfjsemvwyynlscsqh0rxd2edbwyrrir/lrcn+ogxlucu5vuf6d+ef8vxmniw4asfgbpgaptblaiaksdgcbyav/dmpdvvzofzow3nolozqzaxzvcvovkig==</latexit> 6 Man-Wai MAK (EIE) Deep Learning and DNN November 9, / 50

Geometric Interpretation of Perceptrons Apply non-linear activation function on z: (z) z (z) = 1 1+e z z z0 σ b x2 x1 <latexit

8 Geometric Interpretation of Perceptrons Apply non-linear activation function on z: (z) z (z) = 1 1+e z z z0 σ b x2 x1 <latexit sha1_base64="bfyl/9dqng08euahcwimyuzww=">aaacj3icbzdlsgmxfiyz9vbrrerstbaubhgykyjuhkiblxxsbdoyznjmg5rlgsdehb+bk+glvduxnd6poyxha29ccb//c+mgdxg876czmrq2vpgdo3tb2zu5ffp6hpmshmqlgyqroh0orrqaqggyasskih4zuw/7nuf5/ieptke7nmcztrqcrhqy6ug7xyfr3gw4hnu8idqwsptbxtndioryz5rjmixyvgybc/1jggxwz9bacyieur/wh2je06ewqxp3fs92lrtpazfzo5sjzrecpdrlzqtcssjbqetf41g0sodgelluxg4uf9opihrpesh7eti9pribsz+v2smjrpsp1teisectw9fcyngwrfjsemvwyynlscsqh0rxd2edbwyrrir/lrcn+ogxlucu5vuf6d+ef8vxmniw4asfgbpgaptblaiaksdgcbyav/dmpdvvzofzow3nolozqzaxzvcvovkig==</latexit> <latexit sha1_base64="wyecycvp5xcgkcrmymcrj00i4=">aaacj3icbzdlsgmxfiyz9vbrrerstbaubhgykyjuhkiblxxsbdoyznjmg5rlgsdehb+bk+glvduxnd6poyxha29ccb//c+mgdxg876czmrq2vpgdo3tb2zu5ffp6hpmshmqlgyqroh0orrqaqggyasskih4zuw/7nuf5/ieptke7nmcztrqcrhqy6ug7xyfr3gw4hnu8idqwsptbxtndioryz5rjmixytififzzxmwrcbn8gbtclspd/axutgrboddp3ytnotium7uzlwgsi9xhxdk0kbanup1o/wcrat0ycsvtwhgrp07sku9zchtpm09oltbh4x62zmoiynvirj4yipd0ujqwacccmwq5vbbs2ticwovatepeqqthyk+euhhyus6b4ixysq63+p7r350xytcze7lgcbyde+cdc1agt6acqgcdj/acxsgb8+y8ox/o57q148xmdsfcon+/ov6iq==</latexit> x1 <latexit sha1_base64="wyecycvp5xcgkcrmymcrj00i4=">aaacj3icbzdlsgmxfiyz9vbrrerstbaubhgykyjuhkiblxxsbdoyznjmg5rlgsdehb+bk+glvduxnd6poyxha29ccb//c+mgdxg876czmrq2vpgdo3tb2zu5ffp6hpmshmqlgyqroh0orrqaqggyasskih4zuw/7nuf5/ieptke7nmcztrqcrhqy6ug7xyfr3gw4hnu8idqwsptbxtndioryz5rjmixytififzzxmwrcbn8gbtclspd/axutgrboddp3ytnotium7uzlwgsi9xhxdk0kbanup1o/wcrat0ycsvtwhgrp07sku9zchtpm09oltbh4x62zmoiynvirj4yipd0ujqwacccmwq5vbbs2ticwovatepeqqthyk+euhhyus6b4ixysq63+p7r350xytcze7lgcbyde+cdc1agt6acqgcdj/acxsgb8+y8ox/o57q148xmdsfcon+/ov6iq==</latexit> x2 σ <latexit sha1_base64="bfyl/9dqng08euahcwimyuzww=">aaacj3icbzdlsgmxfiyz9vbrrerstbaubhgykyjuhkiblxxsbdoyznjmg5rlgsdehb+bk+glvduxnd6poyxha29ccb//c+mgdxg876czmrq2vpgdo3tb2zu5ffp6hpmshmqlgyqroh0orrqaqggyasskih4zuw/7nuf5/ieptke7nmcztrqcrhqy6ug7xyfr3gw4hnu8idqwsptbxtndioryz5rjmixyvgybc/1jggxwz9bacyieur/wh2je06ewqxp3fs92lrtpazfzo5sjzrecpdrlzqtcssjbqetf41g0sodgelluxg4uf9opihrpesh7eti9pribsz+v2smjrpsp1teisectw9fcyngwrfjsemvwyynlscsqh0rxd2edbwyrrir/lrcn+ogxlucu5vuf6d+ef8vxmniw4asfgbpgaptblaiaksdgcbyav/dmpdvvzofzow3nolozqzaxzvcvovkig==</latexit> z0 (z) = max{z, 0} Man-Wai MAK (EIE) Deep Learning and DNN 7 November 9, / 50

Neural Networs Neural Networs with 1 Hidden Layer A neural networ is

W (1) 2< 3 3 A neural networ is Bias unit W (1) Hidden layer W (2) 2< 1 3

a (2) 1 a (2) 2 a (2) 3 Both input and output layers can have multiple

9 Neural Networs Neural Networs with 1 Hidden Layer A neural networ is constructed by hooing many neurons together. W (1) 2< 3 3 A neural networ is constructed by hooing many neurons together. Bias unit W (1) Hidden layer W (2) 2< 1 3 Output layer Both input and output layers can have mul<ple nodes. a (2) 1 a (2) 2 a (2) 3 Both input and output layers can have multiple nodes. o(x) Man-Wai MAK (EIE) Deep Learning and DNN November 9, / 50

10 Neural Networs with 1 Hidden Layer (z) z y z10 x o = F (x, y) z20 y 1 1 x Hidden Layer Man-Wai MAK (EIE) Deep Learning and DNN 8 November 9, / 50

11 Neural Networs with 1 Hidden Layer A Neural Networ with Multiple Neurons z0 (z) = max{z, 0} y z10 x o = F (x, y) z20 y 1 1 x Hidden Layer Man-Wai MAK (EIE) Deep Learning and DNN 9 November 9, / 50

12 Comparing Activation Functions You may observe the effect of different activation functions through Tensorflow Playground ( Man-Wai MAK (EIE) Deep Learning and DNN November 9, / 50

13 Comparing Activation Functions Man-Wai MAK (EIE) Deep Learning and DNN November 9, / 50

14 Neural Networs Forward propagation: Forward propaga,on: Neural Networs General equations General for equa,on forward propagation: forward propaga,on: a (l) o (l) o (0) x = (w(l) )T o (l 1) + b (l) = f(a(l) ), l = 1, 2 18 Man-Wai MAK (EIE) Deep Learning and DNN November 9, / 50

NN as Nonlinear Classifier NN as Nonlinear Classifier Networs can have multiple

Feature Extrac4on 2 3 x 1 x = 4 x 2 5 x 3 x 1 x 2 x 3 P (M x) P (F x) Hidden

n } N we need n=1 where training x n 2< samples: 3 and t n 2< 2 {x n, t n } N n=1,

15 NN as Nonlinear Classifier NN as Nonlinear Classifier Networs can have multiple layers and multiple outputs: Networs can have mul4ple layers and mul4ple outputs: Feature Extrac4on 2 3 x 1 x = 4 x 2 5 x 3 x 1 x 2 x 3 P (M x) P (F x) Hidden layers To train this networ, we need training samples To train this{x networ, n, t n } N we need n=1 where training x n 2< samples: 3 and t n 2< 2 {x n, t n } N n=1, where x n R 3 and t n R 2 19 Man-Wai MAK (EIE) Deep Learning and DNN November 9, / 50

16 NN as Nonlinear Classifier To ensure that the output nodes produce posterior probabilities given the input vectors, we typically apply the softmax function on the output nodes: ) P (Class = x) y = o (L) = exp(a(l) exp(a(l) ) where a (L) is the activation of the -th output node (at layer L). Man-Wai MAK (EIE) Deep Learning and DNN November 9, / 50

17 Deep Neural Networs Both SVM and GMM can be considered as networs with one hidden layer Deep neural networs (DNNs) are networs with many hidden layers (22 layers in GoogLeNet) Since the 80s, researchers new that DNNs with many hidden layers are very powerful as long as they can be trained. From the 80s to 2005, NNs were trained by the bacpropagation (BP) algorithm. Unfortunately, BP fails to train any networs with more than one hidden layer. People found that shallow networs such as SVM and GMM perform much better than NN with multiple layers. Man-Wai MAK (EIE) Deep Learning and DNN November 9, / 50

18 Deep Neural Networs This situation has changed in 2006 when G. Hinton discovered a new approach to training DNNs. The method uses unsupervised learning to initialize the DNN and use BP to fine-tune the networ weights. During the unsupervised learning stage, the networ is built one layer at a time (from bottom to top) by stacing a number of Restricted Boltzmann Machines (RBM) The RBM is trained by an algorithm called contrastive divergence. For details of RBM and contrastive divergence, see mwma/papers/rbm.pdf Man-Wai MAK (EIE) Deep Learning and DNN November 9, / 50

19 DNN for Classification Internal Representation in DNNs P(neutral x) = 0.2 P(anxiety x) = 0.7 P(happy x) = 0.09 P(angry x) = 0.01 x Identifying light and dar Identifying edge and simple shapes Identifying complex shapes and obects Identifying obects that can be used for recognizing facial expression 10 Man-Wai MAK (EIE) Deep Learning and DNN November 9, / 50

Internal Representation in DNNs Feature Representa,on in DNNs Data Data presented presented to the tonetwor the networ are raw arepixel rawvalues. pixel values.

20 Internal Representation in DNNs Feature Representa,on in DNNs Data Data presented presented to the tonetwor the networ are raw arepixel rawvalues. pixel values. But, internally But, internally the networ generates the networ much higher generates level much features. higher For level example, features. there For might example, be a node there in the networ mighthat be responds a node into the networ presence that of diagonal responds lines to in the an presence image or, of at an even diagonal higher level, linesto inthe an presence image or, of at faces an even higher level, to the presence of faces. 36 Man-Wai MAK (EIE) Deep Learning and DNN November 9, / 50

21 DNN for Dimension Reduction DNN Encoding Face (Autoencoder) Reduce high-dim facial images to 30-dim vectors by autoencoder: Man-Wai MAK (EIE) Deep Learning and DNN November 9, / 50 38

DNN for Dimension Reduction DNN for Dimension Reduc0on Reduce 784 dimension in handwri5en digit images to 2 Reducedimension: 784 dimension 784 1000 500 250 2 250 500 1000 784 in handwritten digit

22 DNN for Dimension Reduction DNN for Dimension Reduc0on Reduce 784 dimension in handwri5en digit images to 2 Reducedimension: 784 dimension in handwritten digit images to 2 dimension: : Input Hidden Output PCA DNN 39 Man-Wai MAK (EIE) Deep Learning and DNN November 9, / 50

23 Simple Gradient Descent Assume that we have a set of weights in the form of a weight vector w and an error function E(w). Our obective is to minimize E(w) with respect to w based on the gradient of E: w (t+1) = w t η E(w (t) ) where t is the iteration index (epoch) and η is a small positive learning rate. Techniques that use the whole data set at once are called batch methods. At each step the weight vector is moved in the direction of the greatest rate of decrease of the error function. Man-Wai MAK (EIE) Deep Learning and DNN November 9, / 50

24 Simple Gradient Descent Gradient Descent for 1D Man-Wai MAK (EIE) Deep Learning and DNN November 9, / 50

25 Stochastic Gradient Descent There is an on-line version of gradient descent that has proved useful in practice: N E(w) = E n (w) The update formula becomes: n=1 w (t+1) = w t η E n (w (t) ) This update is repeated by cycling through the data either in sequence or by selecting points at random with replacement. There are intermediate scenarios in which the updates are based on batches of data points. This is called mini-batches in the deep learning community. Man-Wai MAK (EIE) Deep Learning and DNN November 9, / 50

Bacpropagation for Training Multilayer Networs Bac- propaga*on Algorithm The networ weights W (1) and W (2) in multiple-layer neural networ below The can networ be trained weights by the W in

26 Bacpropagation for Training Multilayer Networs Bac- propaga*on Algorithm The networ weights W (1) and W (2) in multiple-layer neural networ below The can networ be trained weights by the W in Bac-propagation a mul5ple- layer neural (BP) networ algorithm. can be trained by the Bac- propaga5on (BP) algorithm. W (2) W (1) X 18 Man-Wai MAK (EIE) Deep Learning and DNN November 9, / 50

27 Loss Function: Mean Square Error Output Layer: y = o (2) = h(a (2) ) h(z) = e z a (2) = w (2) o(1) Hidden Layer: o (1) = h(a (1) ) a (1) = i w (1) i o(0) i = i w (1) i x i where x i is the i-th elements of the input vector x. Man-Wai MAK (EIE) Deep Learning and DNN November 9, / 50

28 Loss Function: Mean Square Error We aim to minimize the sum of square error between the target output and the actual output: E tot = 1 (y n, t n, ) 2 2 n = 1 ( ) o (2) 2 2 n, t n, = 1 2 n n E n To minimize E tot with respect to W (2), we compute the gradient E tot w (2) = n E n w (2) Man-Wai MAK (EIE) Deep Learning and DNN November 9, / 50

29 Error Gradient in SGD In stochastic gradient descent (SGD), we compute the error for each training vector: E = 1 (y t ) 2 = 1 ( ) o (2) t Error gradient for a training vector: where E w (2) δ (2) = E = a (2) = E a (2) a (2) w (2) = E o (2) o (2) a (2) ( o (2) t ) h (a (2) ) = = δ (2) o(1) (1) ( ) o (2) (2) h(a t a (2) ) Man-Wai MAK (EIE) Deep Learning and DNN November 9, / 50

30 Error Gradient in SGD Error gradient w.r.t. hidden-layer weights where δ (1) = E a (1) = E w (1) i = = E a (1) a (1) w (1) i E a (2) δ (2) w(2) h (a (1) ) a (2) a (1) = δ (1) x i (2) = δ (2) a (2) o (1) o (1) a (1) = h (a (1) ) δ (2) w(2) The above derivative is based on the fact that E depends on a (2) and that each of the a (2) depends on a (1). Man-Wai MAK (EIE) Deep Learning and DNN November 9, / 50

31 Error Gradient in SGD Weight update formula for output-layer weights: w (2) w (2) w (2) w(2) w(2) w(2) η E w (2) ηδ(2) o(1) η(y t )h (a (2) )o(1) (3) where h (a (2) ) = y (1 y ) for non-linear sigmoid output. Man-Wai MAK (EIE) Deep Learning and DNN November 9, / 50

32 Error Gradient in SGD Weight update formula for hidden-layer weights: w (1) i w (1) i η E w (1) i w (1) i w (1) i ηδ (1) [ w (1) i w (1) i η x (1) i h (a (1) ) δ (2) w(2) ] x i (4) where h (a (1) ) = o (1 o (1) ). Man-Wai MAK (EIE) Deep Learning and DNN November 9, / 50

33 Loss Function: Cross Entropy Consider a BP networ with a single output node with output denoted as y. Using Eq of Bishop s boo, E tot = n {t n log y n + (1 t n ) log(1 y n )} where n is the sample index and t n as the target output for the n-th sample. The instantaneous error is E n = t n log y n (1 t n ) log(1 y n ) To simplify notation, we drop the subscript n: E = t log y (1 t) log(1 y) Man-Wai MAK (EIE) Deep Learning and DNN November 9, / 50

34 Loss Function: Cross Entropy Instantaneous error gradident: where δ (2) 1 = E = = a (2) 1 E w (2) 1 = E [ t y + 1 t 1 y = E a (2) 1 a (2) 1 w (2) 1 o (2) 1 o (2) 1 a (2) 1 = E y = δ (2) 1 o(1) y a (2) 1 ] h (a (2) 1 ) h(z) = 1 t(1 y) + (1 t)y [ ] h(a (2) 1 y(1 y) ) 1 h(a (2) 1 ) = y t y = h(a (2) E w (2) 1 = (y t)o (1) 1 ) 1 + e z Man-Wai MAK (EIE) Deep Learning and DNN November 9, / 50

35 Loss Function: Cross Entropy Similarly, where E w (1) i = E a (1) a (1) w (1) i = δ (1) x i Update Formulae: δ (1) = E a (1) = E a (2) 1 a (2) 1 a (1) = (y t)w (2) 1 o(1) (1 o (1) ) w (2) 1 w (2) 1 w (1) i w (1) i η η(y t)o(1) = δ (2) a (2) 1 1 o (1) [ (y t)w (2) 1 o(1) (1 o (1) ) o (1) a (1) ] x i Man-Wai MAK (EIE) Deep Learning and DNN November 9, / 50

36 Cross Entropy for Multi-class [Optional Notes] The cross-entropy loss function can also be used in networs with multiple outputs. Assume that the softmax function is used for the output nodes: ) y = o (2) = exp(a(2) exp(a(2) Because y depends on a (2) where = 1,..., K, we need to compute the derivative of y w.r.t. a (2), i.e., where y a (2) = I exp(a (2) = y I y y I = ) ) exp(a(2) ) exp(a (2) ( ) 2 exp(a(2) ) { 1 = ; 0 otherwise. ) exp(a(2) ) Man-Wai MAK (EIE) Deep Learning and DNN November 9, / 50 (5)

37 Cross Entropy for Multi-class [Optional Notes] The instantaneously cross-entropy is E = K t log y. (6) =1 Eq. 6 suggests that E is a function of y = 1,..., K, and according to Eq. 5, y depends on a (2) r r = 1,..., K. Therefore, we need to use the chain rule to compute E : w (2) E w (2) = K r=1 E a (2) r a (2) r w (2) Man-Wai MAK (EIE) Deep Learning and DNN November 9, / 50

38 Cross Entropy for Multi-class [Optional Notes] E a (2) r = K =1 E y y a (2) r = E y r y r a (2) + E y r r y a (2) r = t r y r (1 y r ) t y (I r 1)y r (using Eq. 6) y r y r = t r (1 y r ) t (0 1)y r r = t r (1 y r ) + y r r K = t r + y r t + t r y r r t K = t r + y r t = y r t r (One-of-K encoding) =1 Man-Wai MAK (EIE) Deep Learning and DNN November 9, / 50

39 Cross Entropy for Multi-class [Optional Notes] Therefore, we have E w (2) = r E a (2) r a (2) r w (2) = (y t )o (1) a(2) r w (2) = 0 r. Similarly, E a (1) i = = K =1 =1 E a (2) a (2) o (1) i o (1) i a (1) i K (y t )w (2) i (1 o (1) o(1) i ) Man-Wai MAK (EIE) Deep Learning and DNN November 9, / 50

40 Cross Entropy for Multi-class [Optional Notes] Instantaneous error gradient: E w (1) i = s = s = E a (1) s a (1) s w (1) i (y t )w (2) s o(1) s (1 o (1) s (y t )w (2) o(1) (1 o (1) )x i ) a(1) s w (1) i Update Formula w (2) w(2) η(y t )o (1) K w (1) i w (1) i η =1 (y t )w (2) o(1) (1 o (1) )x i Man-Wai MAK (EIE) Deep Learning and DNN November 9, / 50

cublas, cudnn, curand CPU GPU They are all

41 Deep Learning Software Stac [Optional Notes] Deep Learning Software Stac Keras pytorch CNTK (Microsoft) TensorFlow (Google) Eigen::Tensor NumPy Theano (U. of Montreal) Gpu array Torch (Faceboo) Caffe/C affe2 (Bereley AI Researcy) BLAS cublas, cudnn, curand CPU GPU They are all free! 11 Man-Wai MAK (EIE) Deep Learning and DNN November 9, / 50

42 Convolutional Neural Networs A convolutional neural networ (CNN) is a type of machine learning program thats well-suited for image classification tass, lie spotting the early signs of cervical cancer and recognition of obects from images. A CNN consists of an input and an output layer, as well as multiple hidden layers. The hidden layers of a CNN typically consist of convolutional layers, pooling layers, fully connected layers and normalization layers. Man-Wai MAK (EIE) Deep Learning and DNN November 9, / 50

43 Why CNN? If we use a DNN for handwritten digit recognition, we need to stac (flatten) the rows or columns of an image to form vector Man-Wai MAK (EIE) Deep Learning and DNN November 9, / 50

44 Why CNN? This method wors fine but we have problem when the digit is off-centered The flattened vector is now completely different from the training data. A good solution is to use convolution as human do not loo at an image at the pixel level. Instead, we loo for the spatial structure of an image. This means that it does not matter where the 8 is as long as it is an 8. Man-Wai MAK (EIE) Deep Learning and DNN November 9, / 50

45 CNN for Digit Recognition Man-Wai MAK (EIE) Deep Learning and DNN November 9, / 50

How CNN Detect Patterns in Images https://www.quora.

46 How CNN Detect Patterns in Images Man-Wai MAK (EIE) Deep Learning and DNN November 9, / 50

47 Recent Development in Deep Learning Xavier Initialization (Glorot and Bengio, 2010): A simple but effective weight initialization method With Xavier initialization, per-layer generative pre-training is not necessary Rectified Linear Unit, ReLU (Glorot and Bengio, 2011): Rectifier units are closer to biological neurons They can avoid the gradient vanishing problem. DNNs with ReLU as non-linear units do not require pre-training on supervised tass with large labeled datasets. Generative Adversarial Networs, GAN (Goodfellow, 2014): Use opposite obective functions during training. The generator attempts to synthesize images that can fool the discriminator into believing that the synthetic images are real. Man-Wai MAK (EIE) Deep Learning and DNN November 9, / 50

48 Recent Development in Deep Learning Batch Normalization (Ioffe and Szegedy, 2015): During BP training, apply z-norm to the output of each layer, using the statistics within a mini-batch. Can achieve the same accuracy with 1/14 of the training steps Residual Networs ((He et al. 2015)): Aim to solve a common problem in DNNs: When depth increases, accuracy gets saturated and then degrades. Networs are trained to predict the residual H(x)x instead of H(x), where x is the input and H(x) is the desired mapping of the application. Can train networs up to 1000 layers. Variational Autoencoders (Kingma and Welling, 2014): Combine DNNs with probabilistic models (bottlenec-layer nodes are stochastic) Wor very well in generating complicated data. Man-Wai MAK (EIE) Deep Learning and DNN November 9, / 50

49 References G. Hinton and R.R. Salahutdinov, Reducing the dimensionality of data with neural networs, Science, vol. 313, no. 5786, pp , Y. Bengio, Learning deep architectures for AI, Foundations and Trends in Machine Learning, vol. 2, no. 1, pp. 1127, Y. LeCun, Y. Bengio and G.E. Hinton, Deep Learning, Nature, vol. 521, pp , May P. Vincent, H. Larochelle, I. Laoie, Y.H. Bengio, and P.A. Manzagol, Staced denoising autoencoders: Learning useful representations in a deep networ with a local denoising criterion, The Journal of Machine Learning Research, vol. 11, pp , D. Erhan, Y. Bengio, A. Courville, P.A. Manzagol, P. Vincent, and S. Bengio, Why does unsupervised pre-training help deep learning?, Journal of Machine Learning Research, 11(Feb), G. E. Hinton, S. Osindero, and Y. W. Teh. A fast learning algorithm for deep belief nets. Neural Computation, 18: , Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., and Bengio, Y. (2014). Generative adversarial nets. In Advances in neural information processing systems (pp ). Man-Wai MAK (EIE) Deep Learning and DNN November 9, / 50

50 References S. Ioffe and C. Szegedy, Batch normalization: Accelerating deep networ training by reducing internal covariate shift, in Proceedings of the 32nd International Conference on Machine Learning (ICML), 2015, pp X. Glorot and Y. Bengio, Understanding the difficulty of training deep feedforward neural networs., in Aistats, 2010, vol. 9, pp K. M. He, X. Y. Zhang, S. Q. Ren, and J. Sun, Deep residual learning for image recognition, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp D. P. Kingma, and M. Welling. Auto-encoding variational bayes. arxiv preprint arxiv: (2014). Man-Wai MAK (EIE) Deep Learning and DNN November 9, / 50

Learning Deep Architectures for AI. Part II - Vijay Chakilam

Learning Deep Architectures for AI - Yoshua Bengio Part II - Vijay Chakilam Limitations of Perceptron x1 W, b 0,1 1,1 y x2 weight plane output =1 output =0 There is no value for W and b such that the model