DISCRIMINANT ANALYSIS: LDA AND QDA

Transcription

1 Stat 427/627 Statistical Machine Learning (Baron) HOMEWORK 6, Solutions DISCRIMINANT ANALYSIS: LDA AND QDA. Chap 4, exercise 5. (a) On a training set, LDA and QDA are both expected to perform well. LDA uses a correct model, and QDA includes the linear model as a special case. QDA, including a wider range of models, will fit the given particular data set better. On a test set, however, LDA should be more accurate because QDA, with additional parameters and an overfit, will have a higher variance. (b) For a nonlinear boundary, QDA should perform better on both the training and test sets because LDA will not capture the nonlinear pattern. (c) As the sample size increases, QDA will perform better because its higher variance will be offset by a large sample size. (d) False. A higher variance of QDA prediction, without any gain in bias, will result in a higher error rate. 2. See the last page. 3. Chap 4, exercise 0[e,f,g,h]. (e) > library(islr) > attach(weekly) > Weekly_training Weekly[ Year < 2008, ] # Split the data by year into > Weekly_testing Weekly[ Year > 2008, ] # the training and testing sets > library(mass) # This library contains LDA > lda.fit lda(direction ~ Lag2, dataweekly_training) # Fit using training data only # Now use this model to predict classification of the testing data > Direction_Predicted_LDA predict(lda.fit, data.frame(weekly_testing))$class > > table( Weekly_testing$Direction, Direction_Predicted_LDA ) # Confusion matrix Direction_Predicted_LDA Direction Down Up Down 9 34 Up 5 56 > mean( Weekly_testing$Direction Direction_Predicted_LDA ) [] # Correct classification rate of LDA is 62.5%. (f) # Similarly with QDA. > qda.fit qda(direction ~ Lag2, dataweekly_training) > Direction_Predicted_QDA predict(qda.fit, data.frame(weekly_testing))$class > table( Weekly_testing$Direction, Direction_Predicted_QDA ) Direction_Predicted_QDA

2 Direction Down Up # Correct classification rate of QDA is only 58.7%. Down 0 43 # However, QDA always predicted that the market goes Up. Up 0 6 # LDA has a higher prediction power. QDA seems to be an overfit. > mean( Weekly_testing$Direction Direction_Predicted_QDA ) [] # Logistic regression and LDA yield the best results so far. (g) # KNN with k is a very flexible method, matching the local features. > X.train matrix( Lag2[ Year < 2008 ], sum(year < 2008), ) > X.test matrix( Lag2[ Year > 2009 ], sum(year > 2009), ) > Y.train Direction[ Year < 2008 ] > Y.test Direction[ Year > 2009 ] > knn.fit knn( X.train, X.test, Y.train, ) > table( Y.test, knn.fit ) knn.fit Y.test Down Up Down 2 22 Up > mean( Y.test knn.fit ) [] # With k4, the KNN method gives a better classification rate for these data: > knn.fit knn( X.train, X.test, Y.train, 4 ) > mean( Y.test knn.fit ) [] (h) Among the methods that we explored, Logistic regression and LDA yield the highest classification rate, KNN with K4 yields the classification rate QDA yields the classification rate KNN with K yields the classification rate Chap 4, exercise 3. > attach(boston) # Attach a new dataset Boston from ISLR package >?Boston # Data description > names(boston) [] "crim" "zn" "indus" "chas" "nox" "rm" "age" "dis" "rad" "tax" "ptratio" "black" "lstat" "medv" > Crime_Rate ( crim > median(crim) ) # Categorical variable Crime_Rate > table(crime_rate) Crime_Rate FALSE TRUE 2

3 # The median splits the data evenly > Boston cbind( Boston, Crime_Rate ) # Logistic regression First, we ll fit the full model. > lreg.fit glm(crime_rate ~ zn + indus + chas + nox + rm + age + dis + rad + tax + ptratio + black + lstat + medv, family"binomial") > summary(lreg.fit) Estimate Std. Error z value Pr(> z ) (Intercept) e-07 *** zn * indus chas nox e- *** rm age dis ** rad e-05 *** tax * ptratio ** black * lstat medv * # Some variables are not significant. We ll fit a reduced model, without them. > lreg.fit glm(crime_rate ~ zn + nox + age + dis + rad + tax + ptratio + black + medv, family"binomial") > summary(lreg.fit) Estimate Std. Error z value Pr(> z ) (Intercept) e-07 *** zn ** nox e- *** age * dis ** rad e-07 *** tax ** ptratio ** black * medv ** # To evaluate the model s predictive performance, we ll use the validation set method. > n length(crim) # Split the data randomly into training and testing subsets > n [] 506 3

4 > Z sample(n,n/2) > Data_training Boston[ Z, ] > Data_testing Boston[ -Z, ] > lreg.fit glm(crime_rate ~ zn + nox + age + dis + rad + tax + ptratio + black + medv, datadata_training, family"binomial") # Fit a logistic regression model on the training data # and use this model to predict the testing data > Predicted_probability predict( lreg.fit, data.frame(data_testing), type"response" ) > Predicted_Crime_Rate *(Predicted_probability > 0.5) # Predicted crime rate: high, 0 low > table( Data_testing$Crime_Rate, Predicted_Crime_Rate ) Predicted_Crime_Rate > mean( Data_testing$Crime_Rate Predicted_Crime_Rate ) [] # Logistic regression shows the correct classification rate of 89.5%, the error rate is 0.5%. # Now, try LDA. For a fair comparison, use the same training data, then predict the testing data. > lda.fit lda(crime_rate ~ zn + nox + age + dis + rad + tax + ptratio + black + medv, datadata_training) > Predicted_Crime_Rate_LDA predict(lda.fit, data.frame(data_testing))$class > table( Data_testing$Crime_Rate, Predicted_Crime_Rate_LDA ) Predicted_Crime_Rate_LDA > mean( Data_testing$Crime_Rate! Predicted_Crime_Rate_LDA ) [] # LDA s error rate is higher, 5.9%. # What about QDA? If logit has a nonlinear relation with predictors, QDA may be a better method. 4

5 > qda.fit qda(crime_rate ~ zn + nox + age + dis + rad + tax + ptratio + black + medv, datadata_training) > Predicted_Crime_Rate_QDA predict(qda.fit, data.frame(data_testing))$class > table( Data_testing$Crime_Rate, Predicted_Crime_Rate_QDA ) Predicted_Crime_Rate_QDA > mean( Data_testing$Crime_Rate! Predicted_Crime_Rate_QDA ) [] # QDA s error rate of 3.4% is between the LDA and the logistic regression. According to these quick results, logistic regression has the best prediction power. # It may be interesting to look at the distribution of predicted probabilities of high crime zones, produced by the three classification methods. All of them mostly discriminate between the two groups pretty well. > hist(predicted_probability, main'predicted PROBABILITY BY LOGISTIC REGRESSION') > hist(predict(lda.fit, data.frame(data_testing))$posterior, main'posterior DISTRIBUTION BY LDA') > hist(predict(qda.fit, data.frame(data_testing))$posterior, main'posterior DISTRIBUTION BY QDA') KNN method # Define training and testing data > names(data_training) [] "crim" "zn" "indus" "chas" "nox" [6] "rm" "age" "dis" "rad" "tax" [] "ptratio" "black" "lstat" "medv" "Crime_Rate" # Variables zn + nox + age + dis + rad + tax + ptratio + black + medv are in columns 2,5,7,8,9,0,,2,4. > X.train Data_training[, c(2,5,7,8,9,0,,2,4) ] 5

6 2. (a) For Class A, based on (4, 2), X A 3, s 2 A (4 3)2 +(2 3) 2 2, and s 2 A.4. For Class B, based on (9, 6, 3), X B 6, s 2 B (9 6)2 +(6 6) 2 +(3 6) 2 9, and s 3 B 3. The pooled variance is s 2 (4 3)2 +(2 3) 2 +(9 6) 2 +(6 6) 2 +(3 6) 2 20, and the pooled standard deviation is s (b) We know that in this situation, when π A π B 0.5, L(A B) L(B A), and σ A σ B, the point is classified to the closest class mean. x 5 is closer to X B 6, so we classify it to Class B. (c) Calculate posterior probabilities P {A x 5} π A f A (5) π A f A (5) + π B f B (5) π A σ exp { (5 µ 2π A) 2 /2σ 2 } π A σ exp { (5 µ 2π A) 2 /2σ 2 } + π B σ exp { (5 µ 2π B) 2 /2σ 2 } estimated by (0.8) exp { (5 3) 2 /(2 20/3)} (0.8) exp { (5 3) 2 /(2 20/3)} + (0.2) exp { (5 6) 2 /(2 20/3)} P {B x 5} P {A x 5} With a zero-one loss function, we classify the new observation according to the higher posterior probability, i.e., to Class A. (d) We already have the posterior probabilities. Then, The risk of classifying the new observation into Class A is Risk(A) (3)P {B x 5} The risk of classifying the new observation into Class B is Risk(B) ()P {A x 5} So, we classify x 5 into Class A because this decision has a lower risk. (e) Now we have to use separate variances instead of the pooled variance. With equal prior probabilities, we have P {A x 5} π A σ A 2π exp { (5 µ A ) 2 /2σA} 2 π A σ A 2π exp { (5 µ A ) 2 /2σA} 2 + π B σ B 2π exp { (5 µ B ) 2 /2σB} 2 estimated by (0.5) exp { (5.4 3)2 /(2 2)} (0.5) exp { (5.4 3)2 /(2 2)} + (0.5) exp { (5 3 6)2 /(2 9)} P {B x 5} P {A x 5} With a zero-one loss function, we classify the new observation according to the higher posterior probability, i.e., to Class B. With unequal probabilities π A 0.8 and π B 0.2, P {A x 5} (0.8) exp { (5.4 3)2 /(2 2)} (0.8) exp { (5.4 3)2 /(2 2)} + (0.2) exp { (5 3 6)2 /(2 9)}

7 P {B x 5} P {A x 5} and we classify x to Class A. With L(A B) 3L(B A), we compute the risks Risk(A) (3)P {B x 5} (3)(0.2326) Risk(B) ()P {A x 5} and classify x 5 into Class A because this decision has a lower risk. (f) In fact, we used LDA in (b-d) and QDA in (e), because LDA assumes σ A σ B whereas QDA does not make this assumption.

8 P {B x 5} P {A x 5} and we classify x to Class A. With L(A B) 3L(B A), we compute the risks Risk(A) (3)P {B x 5} (3)(0.2326) Risk(B) ()P {B x 5} and classify x 5 into Class A because this decision has a lower risk. (f) In fact, we used LDA in (b-d) and QDA in (e), because LDA assumes σ A σ B whereas QDA does not make this assumption.