Agenda. What is Machine Learning? Learning Type of Learning: Supervised, Unsupervised and semi supervised Classification

Transcription

1 Agenda Artificial Intelligence and its applicatins Lecture 6 Supervised Learning Prfessr Daniel Yeung danyeung@ieee.rg Dr. Patrick Chan patrickchan@ieee.rg Suth China University f Technlgy, China Learning Type f Learning: Supervised, Unsupervised and semi supervised Classificatin Definitin s Bayes, Decisin Tree K-NN, NN SVM, MCS What is Machine Learning Changes in the system that are adaptive in the sense that they enable the system t d the same task (r tasks drawn frm a ppulatin f similar tasks) mre effectively the net time Tm Michael Mitchell 997 Chair f the Machine Learning Department at Carnegie Melln University Herbert Aleander Simn (983) Turing Award 975 Nbel Prize in Ecnmics 978 Imprving sme task T based n eperience E with respect t perfrmance measure P Machine Learning An Eample Salmn / Sea Bass A fish packing plant wants t autmate the prcess f srting incming fishes (Salmn / Sea Bass) n a belt accrding t species Salmn Sea Bass 3 4

2 Prcedure Sensing Prcedures: Measuring Device, e.g. camera Object islatin, nise reductin, Feature etractin, selectin Classificatin, Regressin, Crss-validatin, Bstrap, Object Sensing Image Preprcessing Refined Image Feature Etractin Input Features Mdel Learning Output Evaluatin Mdel Digitize the bject t the frmat which can be handled by machines 5 6 Preprcessing Feature Etractin Refine the data What can cause prblems during sensing E.g. lighting cnditins, psitin f fish n the cnveyr belt, camera nise, etc. What kind f infrmatin can distinguish ne specie f fish frm the ther E.g. length, width, weight, number and shape f fins, tail shape, etc. Eperts (Fisherman) may help A salmn is usually shrter than a sea bass Length is chsen (as a feature) as a decisin criterin 7 8

3 Feature Etractin Feature Etractin 5 is selected as the threshld Althugh sea bass is lnger than salmn in general, there are many eceptins The eperts may be wrng! Histgrams f the length feature fr tw types f fish in Training Samples l* Hw abut ther features E.g. lightness 5.5 is selected as the threshld Using lightness as a feature is much better than using length Histgrams fr the lightness feature fr the types f fish in Training Samples Salmn Sea Bass Salmn Sea Bass 9 0 Feature Etractin Mdel Learning Hw abut using tw features Lightness: Width: Sea Bass Many techniques (classifiers) are available Discuss in detail later Is it (a linear classifier) t simple Large errr n training set Salmn Artificial Intelligence and its Applicatins: Lecture 6

4 Mdel Learning Mdel Learning A mre cmple classifier: It classifies training samples perfectly Will it be t fit t training samples (verfitting) Lk mre reasnable Nt t cmple Gd in classifying the training and unseen samples 3 4 Mdel Learning Evaluatin T simple Perfrmance n the training samples is nt gd T cmple Cannt be generalized t the unseen samples Tradeff between accuracy f training samples and cmpleity Ultimate Objective Perfrmance n unseen samples (Generalizatin Ability) Cannt be calculated but just be estimated Evaluate whether a trained mdel generalizes the knwledge frm training samples t future unseen samples Measurement Using training samples / test samples Repeat eperiments 5 6

5 Evaluatin Evaluatin Errr Classificatin Errr : Class Mean Square Errr : Real number Cst Csts f misclassifying n different classes may be different Fr eample: Case : Cmpany s view Salmn is mre epensive than sea bass. Selling Salmn with the price f sea bass will be a lss If a fish is a salmn, it is classified as sea bass HIGH cst If a fish is a sea bass, it is classified as salmn LOW cst 7 8 Evaluatin Evaluatin Fr eample: Case : Custmer s view Custmers wh buy salmn will be very upset if they get sea bass; Custmers wh buy sea bass will nt be upset if they get the mre epensive salmn If a fish is a salmn, it is classified as sea bass LOW cst If a fish is a sea bass, it is classified as salmn HIGH cst Hw wuld these cst cnsideratins affect ur decisin Salmn Sea Bass Case Case Salmn Sea Bass Mre seabass Mistaken as salmn Salmn Sea Bass Mre salmn Mistaken as seabass 0 9 0

6 Pattern Classificatin Prblem Artificial dataset Pattern Classificatin Prblem Will classifier recgnize unseen samples Artificial Dataset Red Pints Unseen Samples Artificial Dataset O O O O O : Training Sample in Class X: Training Sample in Class : Training Sample in Class X: Training Sample in Class Pattern Classificatin Prblem Trains a classifier t apprimate the unknwn inpututput system using the training dataset. Fr neural netwrks, usually dne by minimizing the MSE between netwrk utputs and desired utputs in the training dataset --- the Training Errr R emp the average errr f the finite training dataset R emp = l l ( b) ( b) ( f( X ) F( X )) b= () where l, fand F dente number f training samples, classifier utput and desired utput respectively Pattern Classificatin Prblem Instead f minimizing the training errr R emp nly, the ultimate gal f classifier training is t crrectly predict the class / categry f future unseen samples. R true the epected errr fr all samples ( f( X) F( X) ) p( X) dx R true = R gen the generalizatin errr fr unseen samples (3) R gen =R true -R emp R gen nt cmputable, nly estimated by Empirical Methds Analytical Mdels () V. Vapnik, Statistical Learning Thery, Wiley, 988 R.O. Duda, P. E. Hart and D.G. Strk, Pattern Classificatin, Wiley, 00 3 V. Vapnik, Statistical Learning Thery, Wiley, 998 R.O. Duda, P. E. Hart and D.G. Strk, Pattern Classificatin, Wiley, 00 4

7 Generalizatin Errr Mdel R gen nt cmputable, nly estimated Empirical Mdel K-fld Crss-Validatin (CV) Leave-One-Out Crss-Validatin (LOOCV) Analytical Mdel Infrmatin Criteria VC-Dimensin based Analytical Mdel The artificial dataset Artificial Dataset : Training Sample in Class X: Training Sample in Class Analytical Mdel trained by minimizing R emp nly Easily ver-fitting, high R gen Artificial Dataset Analytical Mdel trained by minimizing R emp nly Easily ver-fitting, high R gen Artificial Dataset Unseen testing sample F() : Training Sample in Class X: Training Sample in Class : Training Sample in Class X: Training Sample in Class

8 Analytical Mdel trained by minimizing R emp nly Easily ver-fitting, high R gen Artificial Dataset Analytical Mdel trained by minimizing cmpleity nly Easily under-fitting and ver minimizing the cmpleity Artificial Dataset : Training Sample in Class X: Training Sample in Class : Training Sample in Class X: Training Sample in Class Analytical Mdel trained by minimizing h nly Easily under-fitting and ver minimizing the cmpleity (h) Artificial Dataset Analytical Mdel trained by minimizing h nly Easily under-fitting and ver minimizing the cmpleity (h) Artificial Dataset : Training Sample in Class X: Training Sample in Class : Training Sample in Class X: Training Sample in Class

9 Analytical Mdel trained by minimizing bth R emp and h Analytical Mdel trained by minimizing bth R emp and h Artificial Dataset Artificial Dataset f() : Training Sample in Class X: Training Sample in Class : Training Sample in Class X: Training Sample in Class Type f Learning Pattern Classificatin Prblem Central questin in PR: Find the best f t apprimate F Supervised Learning A teacher tells yu whether the answer is crrect Reinfrcement learning N teacher but having feedback Tell yu a reward f an actin but nt which actin is crrect Unsupervised Learning N teacher 35 36

10 K-Nearest Neighbr (K-NN) K-Nearest Neighbr (K-NN) A new pattern is classified by a majrity vte f its neighbrs, with the pattern being assigned t the class mst cmmn amngst its k nearest neighbrs Eample f -nn Algrithm: Given an unseen sample Green Star Calculate the distance between unseen sample and each training sample Select the K nearest training samples Majrity vte frm these K samples K = Triangle K = 3 Circle K = 5 Triangle Unseen Sample K-Nearest Neighbr (K-NN) K-Nearest Neighbr (K-NN) Nisy Data 39 Circle A simple linearly separable dataset Obviusly, unseen sample in questin shuld be identified as Triangle Hwever, if there is a nise sample net t this unseen sample, then a -NN classifier will classify it wrngly as a Circle Larger K can slve this prblem Triangle 39 Is larger K always better Obviusly, this unseen sample will be identified as a Circle by a 5-NN classifier But a 9-NN classifier will classify it as a Triangle Hwever, 5-NN classifier will classify it crrectly 40 Circle Triangle 40

11 K-Nearest Neighbr (K-NN) Advantages: Very simple All cmputatins deferred until classificatin N training is needed Disadvantages: Difficult t determine K Affected by nisy training data Classificatin is time cnsuming Need t calculate the distance between the unseen sample and each training sample Decisin Tree (DT) One f the mst widely used and practical methds fr inductive inference E.g. C4.5 Apprimates discrete-valued functins (including disjunctins) 4 4 Decisin Tree (DT) Decisin Tree (DT) D we g t play tennis tday Decisin Regin: > a If Outlk is Sunny AND Humidity is Nrmal Yes N Yes If Outlk is Overcast Yes > b If Outlk is Rain AND Wind is Weak Yes b N Yes Other situatin N a 43 44

12 Decisin Tree (DT) Internal ndes can be multivariate Mre than ne features are used Shape f Decisin Regin is irregular a + b + c > 0 N Yes Decisin Tree (DT) Hw t determine which features shuld be used in a nde Infrmatin Gain Current entrpy Gain( X, A) = H( X) H( X A) Entrpy after using feature A Reductin in entrpy (reduce uncertainty) due t srting n a feature A Entrpy is a measure f uncertainty (Smaller value, less uncertainty,, mre infrmatin gain) where H( X) = n i= p( i)lg p( i ) Xis a randm variable with nutcmes { i : i=,,n} p() is the prbability mass functin f utcme Decisin Tree (DT) Decisin Tree (DT) LOOP: Select the best feature (A) accrding t Infrmatin Gain Fr each value f A, create new descendant f nde Srt training samples t leaf ndes STOP when training samples perfectly classified 47 N > # Yes STOP > N STOP Yes STOP Rule etractin frm trees A decisin tree can be used fr feature etractin (e.g. seeing which features are useful) Interpretability Easy t understand Cmpact and fast classificatin methd 48

13 Neural Netwrk (NN) A very simplified mdel f the brain Human Brain Neural Netwrk (NN) Cmpsed f neurns which cperate tgether input utput input utput Artificial NN Basically a functin apprimatr 49 Transfrms inputs int utputs t the best f its ability Neural Netwrk (NN) 50 I I I d input w w w d f synapse neurn Ө utput Artificial Neural Netwrk (ANN) Hw des a neurn wrk Cmpsed f neurns which cperate tgether The utput f a neurn is a functin f the weighted sum f the inputs plus a bias (ptinal) unit which always emits a value f r - as a threshld. bias I w I w f Ө = f(w I + w I + + w d I d + bias) I d w d activatin functin 5 5

14 Artificial Neural Netwrk (ANN) Activatin Functin Artificial Neural Netwrk (ANN) XOR Eample The functin ( f ) is the activatin functin Eamples: y = bias ( 0.7y 0.4 ) y= sgn y Linear Functin 53 Output is the same as input Differentiable f () = Sign Functin Decisin Making Nt differentiable f () > 0 - < 0 Sigmid Functin Smth, cntinuus, and mntnically increasing (differentiable) f () = /( + e - ) 54 y = +.5 Artificial Neural Netwrk (ANN) XOR Eample Artificial Neural Netwrk (ANN) XOR Eample y = + + y = = = y = sgn(++0.5) = sgn(.5) = y = sgn(+-.5) = sgn(0.5) = y = sgn( ) = sgn(-.3) = - ( 0.7y 0.4 ) y= sgn y = - = - y = sgn( ) = sgn(-.5) = - y = sgn(---.5 ) = sgn(-3.5) = - y = sgn( ) = sgn(-.3) = - The first hidden unit: OR gate = The secnd hidden unit: AND gate y y = +.5 The final utput unit: AND NOT gate: ( 0.7y 0.4 ) y= sgn y y = y AND NOT y = ( OR ) AND NOT ( AND ) = XOR 55 56

15 Artificial Neural Netwrk (ANN) Architecture Artificial Neural Netwrk (ANN) Architecture: Eample w w w 3 Σ g w w w 3 Σ g Σ g Σ g Input Layer Hidden Layers Output Layer A simple three-layer neural netwrk Input Layer: input units Hidden Layer: 3 hidden units Output Layer: utput units Input Layer Hidden Layers Output Layer = length f a fish (salmn/seabass) = lightness f a fish (salmn/seabass) w ij = weight fr assigning imprtance f each input t a neurn Hidden neurns = Cmbinatin f length and lightness discriminant g = final utput Artificial Neural Netwrk (ANN) Learning Artificial Neural Netwrk (ANN) Learning: BP Algrithm net 59 j = d m= w w 3 w w jm Input Layer Hidden Layers Output Layer m g ( ) y j= f net j k f n net y f w' net w f g f f g f f net' = w' ki d k i= m= = w n i= im w' m ki y i ( ' ) g k= f net k Weights can be determined by training Reduce the errr between the desired utputs and the NN utputs f training samples Back-prpagatin algrithm is the mst widely used methd fr determining the weight 60 Natural etensin f LMS algrithm Prs: Simple and general methd Cns: Slw and trapped at lcal minima

16 Artificial Neural Netwrk (ANN) Learning: BP Algrithm Calculatin f the derivative flws backwards thrugh the netwrk Hence, it is called backprpagatin Supprt Vectr Machine (SVM) Which ne is the best linear separatr These derivatives pint in the directin f the maimum increase f the errr functin (find ut where ma errr being made and g back t try t decrease this errr) A small step (learning rate) in the ppsite directin will result in the maimum decrease f the (lcal) errr functin: 6 E w' = w+ α w where αis the learning rate & E the errr functin 6 Supprt Vectr Machine (SVM) Supprt Vectr Machine (SVM) A clever sheep dg wh is herding its sheep It runs between the sheep and tries t separate the black sheep and white sheep Clever Sheep dg The sheep dg keeps running The sheep start t grw wl The dg feels that the gap between black sheep and white sheep is narrwer 63 White Sheep Black Sheep 64

17 Supprt Vectr Machine (SVM) Supprt Vectr Machine (SVM) The wls becme bigger and bigger Finally, nly ne path is left.. Sheep Vectrs Largest Margin Serendipitusly, the sheep invented bth large margin classificatin and sheep vectrs Frm: Learning with Kernels, Schölkpf & Smla Supprt Vectr Machine (SVM) Supprt Vectr Machine (SVM) Which ne is the best linear separatr Similar t the sheep, the wl is grwing This shuld be the best classifier 67 68

18 Supprt Vectr Machine (SVM) On the ther hand, we can seek a classifier with the largest margin We will have similar result Supprt Vectr Machine (SVM) Maimum Margin ONLY depends n few samples, called Supprt Vectrs SVM: Linearly Separable SVM: Nn-Linearly Separable Prblem can be frmulated as a Quadratic Optimizatin Prblem and slve fr w and b minimize w, b subject t where w y w i i =...n T ( +b) i and y={, } y() = w+b y() = y() = - ξ ξ ξ 3 > > < Errr Errr Crrect ξ 4 ξ 5 ξ ξ 4 ξ ξ 5 3 ξ > < Errr Crrect Slack Variable (ξ) is added as a punishment t allw a sample in / far away frm the margin Prblem becmes: Minimize w, b subject t y i w + C N i= ξ T ( w i b) i i + ξ i =... N ξ i 0 7 Margin width = w 7 ξs f ther samples are 0 where C: tradeff parameter between errr and margin

19 SVM: Kernel Kernel is a functin which maps input int high dimensinal feature space SVM: Kernel Cnstruct linear SVM in feature space Similar t the linearly separable case but change all inner prducts t kernel functins Φ minimize w, b subject t w T yi( w φ( i) +b) i =... N Feature Space Multiple System (MCS) Multiple System (MCS) Practically, n idea which kind and which parameters f classifiers is the best fr a prblem. A number f different kinds f classifiers with different parameters are trained A best classifier is selected t use accrding t sme criteria Selecting the best classifier is nt necessarily the ideal chice Different classifiers may cntain different valuable infrmatin Ptentially valuable infrmatin may be lst by discarding results f less-successful classifiers Selecting a wrng classifier will definitely lead t errneus classificatin result 75 76

20 Multiple System (MCS) Multiple System (MCS) A single, trained classifier may nt be adequate t handle tday s increasingly cmple prblems Cmbining classifiers may be a better chice X X Single f result f f f L Fusin result Base classifier / Individual classifier MCS Fusin Methd May depend n X Sample X is fed int each individual classifier Each individual classifier makes it s wn decisin Final decisin is made by cmbining all individual decisins X f f f L Fusin Final Result Multiple System (MCS) Multiple System (MCS) Three factrs affecting the perfrmance (accuracy) f a MCS: Accuracy f base classifiers Hw gd are the base classifiers Diversity amng individual classifiers Hw different are the decisins reached by the classifiers Fusin Methd Hw t cmbine classifiers A methd t arrive at a grup (MCS) decisin Tw categries accrding t classifier utput types: Label utput the class which the sample belngs t E.g. is Class Cntinuus-valued utput Output a value (prbability) fr each class E.g. is Class 0.7 Class

21 MCS: Fusin Label Output: Demcracy Methd Unanimity (all agree) Many unknwn cases Simple Majrity (50% + ) Many unknwn cases Majrity Vte (mst vtes) Usually can make a decisin Als called the Plurality MCS: Fusin Cntinuus-valued utput : Average µ ( ) = j i= Simple Average: weight (w) is the same fr a classifier L w d i, j( ) Weighted Average a mre accurate classifier has a larger weight (w) i where: w i : weight f the ithclassifier d i,j : utput f the ithclassifier n the class j 8 8 Multiple System (MCS) MCS must be better than Single All cases: NO! But in many cases it is true 83