Data Mining with Linear Discriminants. Exercise: Business Intelligence (Part 6) Summer Term 2014 Stefan Feuerriegel

Transcription

1 Data Mining with Linear Discriminants Exercise: Business Intelligence (Part 6) Summer Term 2014 Stefan Feuerriegel

2 Tday s Lecture Objectives 1 Recgnizing the ideas f artificial neural netwrks and their use in R 2 Understanding the cncept and the usage f supprt vectr machines 3 Being able t evaluate the predictive perfrmance in terms f bth metrics and the receiver perating characteristic curve 4 Distinguishing predictive and explanatry pwer Data Mining, with Linear Discriminants 2

3 Outline 1 Recap 2 Linear Discriminants 3 Artificial Neural Netwrks 4 Supprt Vectr Machines 5 Predictin Perfrmance 6 Wrap-Up Data Mining, with Linear Discriminants 3

4 Outline 1 Recap 2 Linear Discriminants 3 Artificial Neural Netwrks 4 Supprt Vectr Machines 5 Predictin Perfrmance 6 Wrap-Up Data Mining, with Linear Discriminants: Recap 4

5 Supervised vs. Unsupervised Learning Supervised learning Machine learning task f inferring a functin frm labeled training data Training data includes bth the input and the desired results crrect results (target values) are given Unsupervised learning Methds try t find hidden structure in unlabeled data The mdel is nt prvided with the crrect results during the training N errr r reward signal t evaluate a ptential slutin Examples: Hidden Markv mdels Dimensin reductin (e. g. by principal cmpnent analysis) Clustering (e. g. by k-means algrithm) grup int classes n the basis f their statistical prperties nly Data Mining, with Linear Discriminants: Recap 5

6 Feature B Feature B Taxnmy f Machine Learning Machine learning estimates functin and parameter in y = f(x,w) Type f methd varies depending n the nature f what is predicted Regressin Predicted value refers t a real number Cntinuus y Classificatin Clustering Predicted value refers t a class label Discrete y (e. g. class membership) Grup pints int clusters based n hw "near" they are t ne anther Identify structure in data Feature A Data Mining, with Linear Discriminants: Recap Feature A 6

7 K -Nearest Neighbr Classificatin Input: training examples as vectrs in a multidimensinal feature space, each with a class label N training phase t calculate internal parameters Testing: Assign t class accrding t k-nearest neighbrs Classificatin as majrity vte Prblematic Skewed data Unequal frequency f classes What label t assign t the circle?? Data Mining, with Linear Discriminants: Recap 7

8 Decisin Trees Flwchart-like structure in which ndes represent tests n attributes End ndes (leaves) f each branch represent class labels Example: Decisin tree fr playing tennis Outlk Sunny Overcast Rain Humidity Yes Wind High Nrmal Strng Weak N Yes N Yes Data Mining, with Linear Discriminants: Recap 8

9 Decisin Trees Issues Hw deep t grw? Hw t handle cntinuus attributes? Hw t chse an apprpriate attributes selectin measure? Hw t handle data with missing attributes values? Advantages Simple t understand and interpret Requires nly few bservatins Wrds, best and expected values can be determined fr different scenaris Disadvantages Infrmatin Gain criterin is biased in favr f attributes with mre levels Calculatins becme cmplex if values are uncertain and/r utcmes are linked Data Mining, with Linear Discriminants: Recap 9

10 Feature B k-means Clustering Partitin n bservatins int k clusters in which each bservatin belngs t the cluster with the nearest mean, serving as a prttype f the cluster Feature A Cmputatinally expensive; instead, we use efficient heuristics Default: Euclidean distance as metric and variance as a measure f cluster scatter Data Mining, with Linear Discriminants: Recap 10

11 Outline 1 Recap 2 Linear Discriminants 3 Artificial Neural Netwrks 4 Supprt Vectr Machines 5 Predictin Perfrmance 6 Wrap-Up Data Mining, with Linear Discriminants: Linear Discriminants 11

12 Classificatin Prblems General classificatin prblem Gal: Take a new input x and assign it t ne f K classes C k Given training set X = [x 1... x n ] T with target values T = [t 1,...,t n ] T Number f dimensins D, i. e. x i R D Learn a discriminant functin y(x) t perfrm the classificatin 2-class prblem with binary target values t i {0,1} Decide fr class C 1 if y(x) > 0, else fr class C 2 K -class prblem with 1-f-K cding scheme, i. e. t i [0,1,0,0,0] T Data Mining, with Linear Discriminants: Linear Discriminants 12

13 Feature B Feature B Linear Separability If a data set can be perfectly classified by a linear discriminant, then we call it linearly separable Feature A Linearly separable Feature A Nt linearly separable Data Mining, with Linear Discriminants: Linear Discriminants 13

14 Linear Discriminant Functins Decisin bundary given by y(x) = w T x + b = 0 defines a hyperplane Classes labeled accrding t sign ( w T x + b ) Nrmal vectr w and ffset b w y(x) > 0 y(x) = 0 x 2 y(x) < 0 w x 1 Data Mining, with Linear Discriminants: Linear Discriminants 14

15 Learning Discriminant Functins Linear discriminant functins given by y(x) = w T x + b = = D i=0 D i=1 w i x i + b w i x i with x 0 = 1 Weight vectr w "Bias" b, i. e. threshld Gal: Chse w and b, r w respectively, such that b t 1... w T X +. w T X T is minimal b t n Data Mining, with Linear Discriminants: Linear Discriminants 15

16 Chsing Discriminant Functin Slving w T X T by least-squares has drawbacks Least-squares is very sensitive t utliers Errr functin penalizes predictins that are "t crrect" Wrks nly fr linearly separable prblems Least-squares assumes Gaussian distributin c c 2 0 c c Alternative slutins (e. g. in blue): Generalized linear mdels ( neural netwrks), supprt vectr machines, etc. frm Leibe (2010). Data Mining, with Linear Discriminants: Linear Discriminants 16

17 Outline 1 Recap 2 Linear Discriminants 3 Artificial Neural Netwrks 4 Supprt Vectr Machines 5 Predictin Perfrmance 6 Wrap-Up Data Mining, with Linear Discriminants: Neural Netwrks 17

18 Generalized Linear Mdel Linear mdel y(x) = w T x + b Generalized linear mdel with activatin functin A y(x) = A ( w T x + b ) Other than least-squares, chice f activatin functin shuld limit influence f utliers, e. g. using a threshld as A A(z) Data Mining, with Linear Discriminants: Neural Netwrks 18

19 Relatinship t Neural Netwrks In 2-class case y(x) = with x 0 = 1 D i=0 A(w i x i ) Single-layer perceptrn threshld w0 w1 y(x) wd utput weights In multi-class case y k (x) = with x 0 = 1 D i=0 A(w k,i x i ) Multi-class perceptrn threshlds wk0 y1(x) yk(x) utputs weights wki x0=1 x1 xd inputs X0 = 1 x1 xd inputs Data Mining, with Linear Discriminants: Neural Netwrks 19

20 Artificial Neural Netwrks Artificial neural netwrks (ANN) are cmputatinal mdels t cmpute f : X Y, inspired by the central nervus system Cmpute f by feeding infrmatin thrugh the netwrk Represented as a system f cnnected neurns ANNs are universal apprximatrs amng cntinuus functins (under certain mild assumptins) Input Hidden Output x 1 z 1 x N z M y 1 y 2 Data Mining, with Linear Discriminants: Neural Netwrks 20

21 Layers in Neural Netwrks Neurns are arranged in three (r mre) layers First layer: Input neurns receive the input vectr x X Hidden layer(s): Cnnect input and utput neurns Final layer: Output neurns cmpute a respnse ỹ Y Input Hidden Output x 1 z 1 x N z M y 1 y 2 When neurns are cnnected as a directed graph withut cycles, this is called a feed-frward ANN Data Mining, with Linear Discriminants: Neural Netwrks 21

22 Feeding Infrmatin thrugh Neural Netwrks Input z j f each neurn j = 1,...,M is a weighted sum f all previus neurns calculated as ( where z j = A w 0,j + N i=1 w i,j x i ) = A ( w 0,j + w T j x ) x i are the values frm the input layer suitable cefficients w i,j fr i = 1,...,N and j = 1,...,M Predefined nn-linear functin A is referred t as the activatin functin Frequent chice: Lgistic functin 1 A(z) = 1 + e z Cefficients w i,j learned frm e. g. a back-prpagatin algrithm Data Mining, with Linear Discriminants: Neural Netwrks 22

23 Lgistic Functin Lgistic A(z) z Resembles a threshld functin { 0, z < 0, A(z) 1, z > 0 Data Mining, with Linear Discriminants: Neural Netwrks 23

24 Neural Netwrks in R Lading required library nnet library(nnet) Accessing credit scres library(caret) data(germancredit) Split data int index subset fr training (20%) and testing (80%) instances intrain <- runif(nrw(germancredit)) < 0.2 Data Mining, with Linear Discriminants: Neural Netwrks 24

25 Neural Netwrks in R Train neural netwrk with n ndes in the hidden layer via nnet(frmula, data=d, size=n...) nn <- nnet(class ~., data=germancredit[intrain,], size=15, maxit=100, rang=0.1, decay=5e-4) ## # weights: 946 ## initial value ## iter 10 value ## iter 20 value ## iter 30 value ## iter 40 value ## iter 50 value ## iter 60 value ## iter 70 value ## iter 80 value ## iter 90 value ## iter 100 value ## final value ## stpped after 100 iteratins Data Mining, with Linear Discriminants: Neural Netwrks 25

26 Neural Netwrks in R Predict credit scres via predict(nn, test, type="class") pred <- predict(nn, GermanCredit[-inTrain,], type="class") Cnfusin matrix via table(pred=pred_classes, true=true_classes) cm <- table(pred=pred, true=germancredit$class[-intrain]) Calculate accuracy sum(diag(cm))/sum(sum(cm)) ## [1] Data Mining, with Linear Discriminants: Neural Netwrks 26

27 Outline 1 Recap 2 Linear Discriminants 3 Artificial Neural Netwrks 4 Supprt Vectr Machines 5 Predictin Perfrmance 6 Wrap-Up Data Mining, with Linear Discriminants: Supprt Vectr Machines 27

28 Supprt Vectr Machine (SVM) Which f these linear separatrs is ptimal? Idea: Maximize separating margin (here: A) Data pints n the margin are called supprt vectrs When calculating decisin bundary, nly supprt vectrs matter; ther training data is ignred Frmulatin as cnvex ptimizatin prblem with glbal slutin y B A x Data Mining, with Linear Discriminants: Supprt Vectr Machines 28

29 SVM in R Lading required library e1071 library(e1071) Accessing credit scres library(caret) data(germancredit) Split data int index subset fr training (20%) and testing (80%) instances intrain <- runif(nrw(germancredit)) < 0.2 Data Mining, with Linear Discriminants: Supprt Vectr Machines 29

30 SVM in R Train supprt vectr machine fr classificatin via svm(frmula, data=d, type="c-classificatin") mdel <- svm(class ~., data=germancredit[intrain,], type="c-classificatin") Predict credit scres fr testing instances test via predict(svm, test) pred <- predict(mdel, GermanCredit[-inTrain, ]) head(cbind(pred, GermanCredit$Class[-inTrain])) ## pred ## ## ## ## ## ## First rw gives predicted utcmes, secnd are actual (true) values Data Mining, with Linear Discriminants: Supprt Vectr Machines 30

31 SVM in R Cnfusin matrix via table(pred=pred_classes, true=true_classes) cm <- table(pred=pred, true=germancredit$class[-intrain]) cm ## true ## pred Bad Gd ## Bad 57 1 ## Gd Calculate accuracy sum(diag(cm))/sum(sum(cm)) ## [1] Data Mining, with Linear Discriminants: Supprt Vectr Machines 31

32 Outline 1 Recap 2 Linear Discriminants 3 Artificial Neural Netwrks 4 Supprt Vectr Machines 5 Predictin Perfrmance 6 Wrap-Up Data Mining, with Linear Discriminants: Predictin Perfrmance 32

33 Assessment f Mdels 1 Predictive perfrmance (measured by accuracy, recall, F1, ROC,... ) 2 Cmputatin time fr bth mdel building and predicting 3 Rbustness t nise in predictr values 4 Scalability 5 Interpretability transparency, ease f understanding Mdel Allws n < k Interpret. # Tuning Parameters Rbust t Predictr Nise Cmp. Time Linear Regressin 0 ANN 1 2 SVM/SVR 1 3 k-nn 1 Single Tree 1 Randm Frest 0 1 Bsted Trees 3 frm Kuhn & Jhnsn (2013). Applied Predictive Mdeling. Data Mining, with Linear Discriminants: Predictin Perfrmance 33

34 Accuracy vs. Precisin Accuracy Clseness t the actual (true) value Precisin Similarity f repeated measurements under unchanged cnditins High accuracy, but lw precisin High precisin, but lw accuracy Data Mining, with Linear Discriminants: Predictin Perfrmance 34

35 Cnfusin Matrix Cnfusin matrix (als named cntingency table r errr matrix) displays predictive perfrmance Cnditin (as determined by Gld standard) True False Psitive Outcme True Psitive (TP) False Psitive (FP) Type I Errr False Alarm Precisin r Psitive Predictive Value = TP TP+FP Negative Outcme False Negative (FN) Type II Errr / Miss True Negative (TN) Sensitivity = TP Rate = TP TP+FN Specificity = TN Rate = TN FP+TN Accuracy = TP+TN Ttal Equivalent with hit rate and recall Data Mining, with Linear Discriminants: Predictin Perfrmance 35

36 Cnfusin Matrix Example: Bld prbe t test fr cancer Patient with Cancer True False Psitive Bld Test Outcme TP: Cancer crrectly diagnsed FP: Healthy persn diagnsed cancer Precisin = TP TP+FP Negative Bld Test Outcme FN: Cancer nt diagnsed TN: Healthy persn diagnsed as healthy Sensitivity = TP Rate = TP TP+FN Specificity = TN Rate = TN FP+TN Accuracy = TP+TN Ttal Different lss functins: Redundant, e 1000 check in FP case, cmpared t lethal utcme in FN case Data Mining, with Linear Discriminants: Predictin Perfrmance 36

37 Assessing Predictin Perfrmance Imagine the fllwing cnfusin matrix with an accuracy f 65% Patient with Cancer True False Psitive Bld Test Outcme Negative Bld Test Outcme TP = 60 FP = 5 Precisin = TP TP+FP 0.92 FN = 30 TN = 5 Sensitivity = TP TP+FN 0.67 Specificity = TN FP+TN = 0.50 Accuracy = TP+TN Ttal = 0.65 Is this a "gd" result? N, because f unevenly distributed data, a mdel which always guesses psitive will scre an accuracy f 60% Data Mining, with Linear Discriminants: Predictin Perfrmance 37

38 Predictin Perfrmance in R Cnfusin matrix in variable cm ## True False ## Ps ## Neg Calculating accuracy (cm[1,1]+cm[2,2])/ (cm[1,1]+cm[1,2]+cm[2,1]+cm[2,2]) ## [1] 0.7 # Alternative that wrks als fr multi-class data sum(diag(cm))/sum(sum(cm)) ## [1] 0.7 Calculating precisin cm[1, 1]/(cm[1, 1] + cm[1, 2]) ## [1] 0.75 Data Mining, with Linear Discriminants: Predictin Perfrmance 38

39 Trade-Off: Sensitivity vs. Specificity/Precisin Perfrmance gals frequently place mre emphasis n either sensitivity r specificity/precisin Example: Airprt scanners triggered n lw-risk items like belts (lw precisin), but reduce risk f missing risky bjects (high sensitivity) Trade-Off: F1 scre is the harmnic mean f precisin and sensitivity 2TP F1 = 2TP + FP + FN Visualized by receiver perating characteristic (ROC) curve Data Mining, with Linear Discriminants: Predictin Perfrmance 39

40 Sensitivity Receiver Operating Characteristic (ROC) ROC illustrates perfrmance f binary classifier as its discriminatin threshld y(x) is varied Interpretatin: Curve A is randm guessing (50% crrect guesses) Curve frm mdel B perfrms better than A, but wrse than C Curve C frm perfect predictin Area suth-east f curve is named area under the curve and shuld be maximized C B A Specifity Data Mining, with Linear Discriminants: Predictin Perfrmance 40

41 ROC in R Lad required library proc library(proc) Perfrm predictin and retrieve decisin values dv pred <- predict(mdel, GermanCredit[-inTrain,], decisin.values=true) dv <- attributes(pred)$decisin.values Data Mining, with Linear Discriminants: Predictin Perfrmance 41

42 ROC in R Plt ROC curve via plt.rc(classes, dv) plt.rc(as.numeric(germancredit$class[-intrain]), dv, xlab = "1 - Specificity") Sensitivity Specificity ## ## Call: ## plt.rc.default(x = as.numeric(germancredit$class[-intrain]), predictr = dv, xlab = ## ## Data: dv in 300 cntrls (as.numeric(germancredit$class[-intrain]) 1) > 699 cases (as.num ## Area under the curve: Data Mining, with Linear Discriminants: Predictin Perfrmance 42

43 Predictive vs. Explanatry Pwer Significant difference between predicting and explaining: 1 Empirical Mdels fr Predictin Empirical predictive mdels (e. g. statistical mdels, methds frm data mining) designed t predict new/future bservatins Predictive Analytics describes the evaluatin f the predictive pwer, such as accuracy r precisin 2 Empirical Mdels fr Explanatin Any type f statistical mdel used fr testing causal hypthesis Use methds fr evaluating the explanatry pwer, such as statistical tests r measures like R 2 Data Mining, with Linear Discriminants: Predictin Perfrmance 43

44 Predictive vs. Explanatry Pwer N = 15 N = 100 Explanatry pwer des nt imply predictive pwer Red is the best explanatry mdel; gray the best predictive In particular, dummies d nt translate well t predictive mdes D nt write smething like "the regressin prves the predictive pwer f regressr x i " 44 Data Mining, with Linear Discriminants: Predictin Perfrmance

45 Overfitting When learning algrithm is perfrmed fr t lng, the learner may adjust t very specific randm features nt related t the target functin Overfitting: Perfrmance n training data (in gray) still increases, while the perfrmance n unseen data (in red) becmes wrse y Mean Squared Errr x Flexibility Data Mining, with Linear Discriminants: Predictin Perfrmance 45

46 Overfitting in R Given data x t predict y plt(x, y, pch = 20) Estimate plynmials f rder P = 1,...,6 and cmpare errrs n training and testing data intrain <- runif(length(x)) < 0.2 err.fit <- c() err.pred <- c() y fr (i in 1:6) { # Explanatry mdel xx <- x[intrain] m <- lm(y[intrain] ~ ply(xx, i, raw=true)) err.fit[i] <- sqrt(mean(m$residuals^2)) # Predictive mdel xx <- x[-intrain] err <- predict(m, ply(xx, i, raw=true))-y[-intrain] err.pred[i] <- sqrt(mean(err^2)) } x Data Mining, with Linear Discriminants: Predictin Perfrmance 46

47 Overfitting in R Perfrmance n training data (in gray) still increases, while the perfrmance n unseen data (in red) becmes wrse plt(1:6, err.fit, type='l', ylim=c(3.5, 31), cl="gray", xlab="order f plynmial", ylab="errr n training/testing data") lines(1:6, err.pred, cl="firebrick3") Errr n training/testing data Order f plynmial Data Mining, with Linear Discriminants: Predictin Perfrmance 47

48 Supprt Vectr Regressin in R Supprt vectr regressin (SVR) predicts cntinuus values Accessing weekly stck market returns fr 21 years library(islr) data(weekly) intrain <- runif(nrw(weekly)) < 0.2 When training, use type="eps-regressin" instead mdel <- svm(tday ~ Lag1 + Lag2 + Lag3 + Lag4 + Lag5, data=weekly[intrain,], type="eps-regressin") Cmpute rt mean square errr (RMSE) pred <- predict(mdel, Weekly[-inTrain, ]) sqrt(mean(pred - Weekly$Tday[-inTrain])^2) ## [1] given by RMSE = 1 N N i=1 (pred i true i ) 2 and cmpare t standard deviatin f σ 2 = (N in denminatr, nt N 1) Data Mining, with Linear Discriminants: Predictin Perfrmance 48

49 Outline 1 Recap 2 Linear Discriminants 3 Artificial Neural Netwrks 4 Supprt Vectr Machines 5 Predictin Perfrmance 6 Wrap-Up Data Mining, with Linear Discriminants: Wrap-Up 49

50 Summary: Data Mining with Linear Discriminants Linearly separable Artificial neural netwrk Supprt vectr machine Cnfusin matrix Data can be perfectly classified by linear discriminant Feeding infrmatin thrugh the netwrk nnet(frmula, data=d, size=n...) Maximize separating margin svm(frmula, data=d, type=...) Tabular utline f crrect/incrrect guesses Predictive perfrmance Measured by accuracy, precisin,... ROC curve Overfitting Visualize trade-ff between sensitivity and specificity Perfrmance n training data increases, different frm unseen bservatins Data Mining, with Linear Discriminants: Wrap-Up 50