Discrimination and Classification

Transcription

1 Dirimination and Claifiation Nathaniel E. Helwig Aitant Profeor of Pyhology and Statiti Unierity of Minneota (Twin Citie) Updated 14-Mar-2017 Nathaniel E. Helwig (U of Minneota) Dirimination and Claifiation Updated 14-Mar-2017 : Slide 1

2 Copyright Copyright 2017 by Nathaniel E. Helwig Nathaniel E. Helwig (U of Minneota) Dirimination and Claifiation Updated 14-Mar-2017 : Slide 2

3 Outline of Note 1) Claifying Two Population Oeriew of Problem Cot of Milaifiation 2) Two Multiariate Normal Equal Coariane Unequal Coariane 3) Ealuating Claifiation Milaifiation Meaure Quality in LDA 4) Claifying g 2 Population Oeriew of Problem Cot of Milaifiation Diriminant Analyi 5) Iri Data Example Data Oeriew LDA Example QDA Example Nathaniel E. Helwig (U of Minneota) Dirimination and Claifiation Updated 14-Mar-2017 : Slide 3

4 Purpoe of Dirimination and Claifiation Dirimination attempt to eparate ditint et of objet, and laifiation attempt to alloate new objet to predefined group. There are two typial goal of dirimination and laifiation: 1 Data deription: find diriminant that bet eparate group 2 Data alloation: put new objet in group ia the diriminant Note that goal 1 i dirimination, and goal 2 i laifiation/alloation. Nathaniel E. Helwig (U of Minneota) Dirimination and Claifiation Updated 14-Mar-2017 : Slide 4

5 Claifying Two Population Claifying Two Population Nathaniel E. Helwig (U of Minneota) Dirimination and Claifiation Updated 14-Mar-2017 : Slide 5

6 Claifying Two Population Oeriew of Problem The Two Population Claifiation Problem Let X = (X 1,..., X p ) denote a random etor and let f 1 (x) denote the probability denity funtion (pdf) for population π 1 f 2 (x) denote the probability denity funtion (pdf) for population π 2 Problem: Gien a realization X = x, we want to aign x to π 1 or π 2. We want to find ome laifiation rule to determine whether a realization X = x hould be aigned to population π 1 or π 2. Nathaniel E. Helwig (U of Minneota) Dirimination and Claifiation Updated 14-Mar-2017 : Slide 6

7 Claifying Two Population Viualizing a Claifiation Rule Oeriew of Problem Let Ω denote the ample pae, i.e., all poible alue of x, and R 1 Ω i the ubet of Ω for whih we laify x a π 1 R 2 = Ω R 1 i the ubet of Ω for whih we laify x a π 2 Figure: Figure 11.2 from Applied Multiariate Statitial Analyi, 6th Ed (Johnon & Wihern). Viualization i for p = 2 ariable. Nathaniel E. Helwig (U of Minneota) Dirimination and Claifiation Updated 14-Mar-2017 : Slide 7

8 Claifying Two Population Probability of Milaifiation Cot of Milaifiation The onditional probability P(2 1) of laifying an objet a π 2 when the objet really belong to π 1 i gien by P(2 1) = P(X R 2 π 1 ) = f 1 (x)dx R 2 The onditional probability P(1 2) of laifying an objet a π 1 when the objet really belong to π 2 i gien by P(1 2) = P(X R 1 π 2 ) = f 2 (x)dx R 1 Nathaniel E. Helwig (U of Minneota) Dirimination and Claifiation Updated 14-Mar-2017 : Slide 8

9 Claifying Two Population Cot of Milaifiation Viualizing the Probability of Milaifiation Figure: Figure 11.3 from Applied Multiariate Statitial Analyi, 6th Ed (Johnon & Wihern). Viualization i for p = 1 ariable. Nathaniel E. Helwig (U of Minneota) Dirimination and Claifiation Updated 14-Mar-2017 : Slide 9

10 Claifying Two Population Inorporating Prior Probabilitie Cot of Milaifiation Let p 1 and p 2 denote the prior probabilitie that an objet belong to π 1 and π 2, repetiely, with the ontraint that p 1 + p 2 = 1. The oerall probabilitie of the four outome hae the form P(orretly laify a π 1 ) = P(X R 1 π 1 )P(π 1 ) = P(1 1)p 1 P(orretly laify a π 2 ) = P(X R 2 π 2 )P(π 2 ) = P(2 2)p 2 P(milaify π 1 a π 2 ) = P(X R 2 π 1 )P(π 1 ) = P(2 1)p 1 P(milaify π 2 a π 1 ) = P(X R 1 π 2 )P(π 2 ) = P(1 2)p 2 Nathaniel E. Helwig (U of Minneota) Dirimination and Claifiation Updated 14-Mar-2017 : Slide 10

11 Claifying Two Population Cot of Milaifiation Claifiation Table and Milaifiation Cot In many real world ae, ot of milaifiation are not equal: π 1 and π 2 are dieaed and healthy π 1 and π 2 are guilty and not guilty π 1 and π 2 are buy and not buy tok We an make a ot matrix to tabulate our milaifiation ot: Truth: Claify a: π 1 π 2 π 1 0 (2 1) π 2 (1 2) 0 The expeted ot of milaifiation (ECM) i defined a ECM = (2 1)P(2 1)p 1 + (1 2)P(1 2)p 2 Nathaniel E. Helwig (U of Minneota) Dirimination and Claifiation Updated 14-Mar-2017 : Slide 11

12 Claifying Two Population Cot of Milaifiation Claifiation Rule (Region) Minimizing ECM The R 1 and R 2 that minimize the ECM are defined ia the inequalitie: ( ) ( ) f 1 (x) (1 2) R 1 : f 2 (x) p2 (2 1) p ( ) ( 1 ) f 1 (x) (1 2) R 2 : f 2 (x) < p2 (2 1) p 1 If (1 2) = (2 1), then we are laifying ia poterior probabilitie. If (1 2) = (2 1) and p 1 = p 2, then the laifiation rule redue to R 1 : R 2 : f 1 (x) f 2 (x) 1 f 1 (x) f 2 (x) < 1 Nathaniel E. Helwig (U of Minneota) Dirimination and Claifiation Updated 14-Mar-2017 : Slide 12

13 Claifiation with Two Multiariate Normal Population Two Multiariate Normal Population Nathaniel E. Helwig (U of Minneota) Dirimination and Claifiation Updated 14-Mar-2017 : Slide 13

14 Claifiation with Two Multiariate Normal Population Equal Coariane Matrie MVN Two Population Claifiation Problem Let X = (X 1,..., X p ) denote a random etor and let f 1 (x) N(µ 1, Σ) denote the pdf for population π 1 f 2 (x) N(µ 2, Σ) denote the pdf for population π 2 Problem: Gien a realization X = x, we want to aign x to π 1 or π 2. We want to find ome laifiation rule to determine whether a realization X = x hould be aigned to population π 1 or π 2. Nathaniel E. Helwig (U of Minneota) Dirimination and Claifiation Updated 14-Mar-2017 : Slide 14

15 Claifiation with Two Multiariate Normal Population Equal Coariane Matrie Claifiation Rule Minimizing ECM The multiariate normal denitie hae the form f k (x) = (2π) p/2 Σ 1/2 exp{ (1/2)(x µ k ) Σ 1 (x µ k )} for k {1, 2}, whih implie that f = f { 1(x) f 2 (x) = exp 1 2 (x µ 1) Σ 1 (x µ 1 ) + 1 } 2 (x µ 2) Σ 1 (x µ 2 ) { = exp (µ 1 µ 2 ) Σ 1 x 1 } 2 (µ 1 µ 2 ) Σ 1 (µ 1 + µ 2 ) The R 1 and R 2 that minimize the ECM are defined ia the inequalitie: [( ) ( )] (1 2) R 1 : log(f p2 ) log (2 1) p [( ) ( 1 )] (1 2) R 2 : log(f p2 ) < log (2 1) Nathaniel E. Helwig (U of Minneota) Dirimination and Claifiation Updated 14-Mar-2017 : Slide 15 p 1

16 Claifiation with Two Multiariate Normal Population Claifiation Rule in Pratie Equal Coariane Matrie The rule on the preiou lide depend on the population parameter µ 1, µ 2, and Σ, whih are often unknown in pratie. Gien n 1 independent oberation from π 1 and n 2 independent oberation from π 2, we an etimate the needed parameter: ˆµ 1 = x 1 = 1 n 1 x i(1) and ˆµ n 2 = x 2 = 1 n 2 1 n 2 ˆΣ = S p = i=1 1 n 1 + n 2 2 [ n1 i=1 x i(2) ] n 2 (x i(1) x 1 )(x i(1) x 1 ) + (x i(2) x 2 )(x i(2) x 2 ) i=1 i=1 The etimated laifiation rule replae f with it ample etimate: { ˆf = exp ( x 1 x 2 ) S 1 p x 1 } 2 ( x 1 x 2 ) S 1 p ( x 1 + x 2 ) Nathaniel E. Helwig (U of Minneota) Dirimination and Claifiation Updated 14-Mar-2017 : Slide 16

17 Claifiation with Two Multiariate Normal Population Equal Coariane Matrie Claifiation Rule in Pratie (ontinued) If ν = ( ) ( ) (1 2) p2 (2 1) p 1 = 1, then the rule beome R 1 : R 2 : ŷ ˆm ŷ < ˆm where ŷ = â x and ˆm = 1 2 (ȳ 1 + ȳ 2 ) with â = ( x 1 x 2 ) S 1 p, ȳ 1 = â x 1, and ȳ 2 = â x 2 Sale of â i not uniquely determined, o normalize â uing either: 1 â = â/ â (unit length) 2 â = â/â 1 (firt element 1) Nathaniel E. Helwig (U of Minneota) Dirimination and Claifiation Updated 14-Mar-2017 : Slide 17

18 Claifiation with Two Multiariate Normal Population Equal Coariane Matrie Fiher Linear Diriminant Funtion R. A. Fiher arried at the deiion rule on the preiou lide uing an entirely different argument. Fiher onidered finding the linear ombination Y = a X that bet eparate the group: where eparation = ȳ 1 ȳ 2 y ȳ 1 i the mean of the Y ore for the oberation from π 1 ȳ 2 i the mean of the Y ore for the oberation from π 2 2 y = n1 i=1 (y i(1) ȳ 1 ) 2 + n 2 i=1 (y i(2) ȳ 2 ) 2 n 1 +n 2 2 i the pooled ariane Nathaniel E. Helwig (U of Minneota) Dirimination and Claifiation Updated 14-Mar-2017 : Slide 18

19 Claifiation with Two Multiariate Normal Population Equal Coariane Matrie Fiher Linear Diriminant Funtion (ontinued) Setting â = ( x 1 x 2 ) S 1 p maximize the eparation eparation 2 = (ȳ 1 ȳ 2 ) 2 2 y = (â x 1 â x 2 ) 2 â S p â = (â d) 2 â S p â = d S 1 p d = D 2 oerall all poible a etor, where d = x 1 x 2. Nathaniel E. Helwig (U of Minneota) Dirimination and Claifiation Updated 14-Mar-2017 : Slide 19

20 Claifiation with Two Multiariate Normal Population Equal Coariane Matrie Viualizing Fiher Linear Diriminant Funtion Figure: Figure 11.5 from Applied Multiariate Statitial Analyi, 6th Ed (Johnon & Wihern). Viualization i for p = 2 ariable. Nathaniel E. Helwig (U of Minneota) Dirimination and Claifiation Updated 14-Mar-2017 : Slide 20

21 Claifiation with Two Multiariate Normal Population Unequal Coariane Matrie MVN Two Population Claifiation Problem (Σ 1 Σ 2 ) Let X = (X 1,..., X p ) denote a random etor and let f 1 (x) N(µ 1, Σ 1 ) denote the pdf for population π 1 f 2 (x) N(µ 2, Σ 2 ) denote the pdf for population π 2 Problem: Gien a realization X = x, we want to aign x to π 1 or π 2. We want to find ome laifiation rule to determine whether a realization X = x hould be aigned to population π 1 or π 2. Nathaniel E. Helwig (U of Minneota) Dirimination and Claifiation Updated 14-Mar-2017 : Slide 21

22 Claifiation with Two Multiariate Normal Population Unequal Coariane Matrie Claifiation Rule Minimizing ECM (Σ 1 Σ 2 ) The multiariate normal denitie hae the form f k (x) = (2π) p/2 Σ k 1/2 exp{ (1/2)(x µ k ) Σ 1 k (x µ k)} for k {1, 2}, whih implie that f = f ( ) 1/2 { 1(x) f 2 (x) = Σ1 exp 1 Σ 2 2 (x µ 1 ) Σ 1 1 (x µ 1 ) + 1 } 2 (x µ 2 ) Σ 1 2 (x µ 2 ) The R 1 and R 2 that minimize the ECM are defined ia the inequalitie: [( ) ( )] (1 2) R 1 : log(f p2 ) log (2 1) p [( ) ( 1 )] (1 2) R 2 : log(f p2 ) < log (2 1) p 1 Nathaniel E. Helwig (U of Minneota) Dirimination and Claifiation Updated 14-Mar-2017 : Slide 22

23 Claifiation with Two Multiariate Normal Population Unequal Coariane Matrie Claifiation Rule in Pratie (Σ 1 Σ 2 ) The rule on the preiou lide depend on the population parameter µ 1, µ 2, Σ 1, and Σ 2, whih are often unknown in pratie. Gien n 1 independent oberation from π 1 and n 2 independent oberation from π 2, we an etimate the needed parameter: ˆµ 1 = x 1 = 1 n 1 x i(1) and ˆΣ1 = S 1 = 1 n 1 n 1 1 i=1 ˆµ 2 = x 2 = 1 n 2 n 2 i=1 x i(2) and ˆΣ2 = S 2 = 1 n 2 1 n 1 i=1 n 2 (x i(1) x 1 )(x i(1) x 1 ) (x i(2) x 2 )(x i(2) x 2 ) i=1 The etimated laifiation rule replae f with it ample etimate: ( ) 1/2 { ˆf S1 = exp 1 S 2 2 (x x 1) S 1 1 (x x 1) + 1 } 2 (x x 2) S 1 2 (x x 2) Nathaniel E. Helwig (U of Minneota) Dirimination and Claifiation Updated 14-Mar-2017 : Slide 23

24 Claifiation with Two Multiariate Normal Population Unequal Coariane Matrie Claifiation Rule in Pratie (Σ 1 Σ 2 ), ontinued Note that we an write [ ( S1 ) ] log(ˆf 1/2 ) = log e 1 2 (x x 1) S 1 1 (x x 1)+ 1 2 (x x 2) S 1 2 (x x 2) S 2 where = ŷ ˆm ŷ = 1 2 x (S 1 1 S 1 2 )x + ( x 1 S 1 1 x 2 S 1 2 )x ˆm = 1 ( ) 2 log S1 + 1 S 2 2 ( x 1 S 1 1 x 1 x 2 S 1 2 x 2 ) ŷ i a quadrati funtion of x, o thi a quadrati laifiation rule. Nathaniel E. Helwig (U of Minneota) Dirimination and Claifiation Updated 14-Mar-2017 : Slide 24

25 Claifiation with Two Multiariate Normal Population Unequal Coariane Matrie Caution: Quadrati Claifiation of Non-Normal Data Figure: Figure 11.6 from Applied Multiariate Statitial Analyi, 6th Ed (Johnon & Wihern). Viualization i for p = 1 ariable. Nathaniel E. Helwig (U of Minneota) Dirimination and Claifiation Updated 14-Mar-2017 : Slide 25

26 Ealuating Claifiation Funtion Ealuating Claifiation Funtion Nathaniel E. Helwig (U of Minneota) Dirimination and Claifiation Updated 14-Mar-2017 : Slide 26

27 Ealuating Claifiation Funtion Milaifiation Meaure Quantifying the Quality of a Claifiation Rule To determine if a laifiation rule i good we an examine the error rate, i.e., milaifiation probabilitie. The population parameter are unknown in pratie, o we fou on approahe that an etimate the error rate from the obered data. We want our laifiation rule to ro-alidate to new data, o we onider ro-alidation proedure. Nathaniel E. Helwig (U of Minneota) Dirimination and Claifiation Updated 14-Mar-2017 : Slide 27

28 Ealuating Claifiation Funtion Milaifiation Meaure Total Probability of Milaifiation The Total Probability of Milaifiation (TPM) i defined a TPM(R 1, R 2 ) = p 1 f 1 (x)dx + p 2 f 2 (x)dx R 2 R 1 for any laifiation rule (region) that partition Ω = R 1 R 2. The Optimum Error Rate (OER) i the minimum poible alue of TPM OER = min R 1,R 2 TPM(R 1, R 2 ) ubjet to Ω = R 1 R 2 whih i obtained when R 1 : f 1(x) f 2 (x) p 2 p 1 and R 2 : f 1(x) f 2 (x) < p 2 p 1. If (1 2) = (2 1), minimizing TPM i ame a minimizing ECM Nathaniel E. Helwig (U of Minneota) Dirimination and Claifiation Updated 14-Mar-2017 : Slide 28

29 Atual Error Rate Ealuating Claifiation Funtion Milaifiation Meaure The error rate on the preiou lide require knowledge of the (typially unknown) parameter that define the denitie f 1 ( ) and f 2 ( ). Example: For LDA, alulating OER require µ 1, µ 2, and Σ The Atual Error Rate (AER) i defined uing the ample etimate AER( ˆR 1, ˆR 2 ) = p 1 f 1 (x)dx + p 2 f 2 (x)dx ˆR 2 ˆR 1 where ˆR 1 and ˆR 2 denote etimate from ample ize n 1 and n 2. Nathaniel E. Helwig (U of Minneota) Dirimination and Claifiation Updated 14-Mar-2017 : Slide 29

30 Ealuating Claifiation Funtion Apparent Error Rate Milaifiation Meaure The Apparent Error Rate (APER) i an optimiti etimate of AER. Etimate the AER uing the obered (training) ample of data The onfuion matrix for a ample of data i Truth: Claified a: π 1 π 2 π 1 n C1 n M1 n 1 π 2 n M2 n C2 n 2 where n Ck i the number orretly laified in population k {1, 2} n M1 = n 1 n C1 i the number from π 1 that are milaified n M2 = n 2 n C2 i the number from π 2 that are milaified Nathaniel E. Helwig (U of Minneota) Dirimination and Claifiation Updated 14-Mar-2017 : Slide 30

31 Ealuating Claifiation Funtion Apparent Error Rate (ontinued) Milaifiation Meaure Gien a ample of data with onfuion matrix Truth: Claified a: π 1 π 2 π 1 n C1 n M1 n 1 π 2 n M2 n C2 n 2 the APER i alulated a APER = n M1 + n M2 n 1 + n 2 whih i the total proportion of milaified ample oberation. Nathaniel E. Helwig (U of Minneota) Dirimination and Claifiation Updated 14-Mar-2017 : Slide 31

32 Ealuating Claifiation Funtion Milaifiation Meaure Leae-One-Out (Ordinary) Cro-Validation Lahenbruh propoed a better approah to etimate the AER: 1. Population 1 (for i = 1,..., n 1 ) (a) Hold out the i-th oberation from π 1 and build laifiation rule (b) Ue laifiation rule from Step 1(a) to laify the i-th oberation 2. Population 2 (for i = 1,..., n 2 ) (a) Hold out the i-th oberation from π 2 and build laifiation rule (b) Ue laifiation rule from Step 2(a) to laify the i-th oberation An (almot) unbiaed etimate of the expeted AER i gien by Ê(AER) = n M1 + n M2 n 1 + n 2 where nm1 and n M2 are the number of milaified oberation uing the aboe leae-one-out proedure. Nathaniel E. Helwig (U of Minneota) Dirimination and Claifiation Updated 14-Mar-2017 : Slide 32

33 Ealuating Claifiation Funtion Reiiting Linear Diriminant Analyi Claifiation Quality in Linear Diriminant Analyi Let X = (X 1,..., X p ) denote a random etor and let f 1 (x) N(µ 1, Σ) denote the pdf for population π 1 f 2 (x) N(µ 2, Σ) denote the pdf for population π 2 Reminder: auming that ( ) ( ) (1 2) p2 (2 1) p 1 = 1, the laifiation rule i R 1 : R 2 : Y m Y < m where Y = a X and m = 1 2 (µ Y 1 + µ Y2 ) with a = (µ 1 µ 2 ) Σ 1, µ Y1 = a µ 1, and µ Y2 = a µ 2 Nathaniel E. Helwig (U of Minneota) Dirimination and Claifiation Updated 14-Mar-2017 : Slide 33

34 Ealuating Claifiation Funtion Claifiation Quality in Linear Diriminant Analyi Reiiting Linear Diriminant Analyi (ontinued) Y = a X = (µ 1 µ 2 ) Σ 1 X i a linear funtion of X, o... µ Y1 = a µ 1 = (µ 1 µ 2 ) Σ 1 µ 1 µ Y2 = a µ 2 = (µ 1 µ 2 ) Σ 1 µ 2 σ 2 Y = a Σa = (µ 1 µ 2 ) Σ 1 (µ 1 µ 2 ) = 2 And ine X i multiariate normal, we hae that { N(µY1, Y 2 ) if from π 1 N(µ Y2, 2 ) if from π 2 i.e., Y i uniariate normal with population dependent mean. Nathaniel E. Helwig (U of Minneota) Dirimination and Claifiation Updated 14-Mar-2017 : Slide 34

35 Ealuating Claifiation Funtion Viualizing Milaifiation in LDA Claifiation Quality in Linear Diriminant Analyi Figure: Figure 11.7 from Applied Multiariate Statitial Analyi, 6th Ed (Johnon & Wihern). Nathaniel E. Helwig (U of Minneota) Dirimination and Claifiation Updated 14-Mar-2017 : Slide 35

36 Ealuating Claifiation Funtion Claifiation Quality in Linear Diriminant Analyi Calulating Milaifiation in LDA (laify π 1 a π 2 ) Defining m = (1/2)(µ 1 µ 2 ) Σ 1 (µ 1 + µ 2 ), we hae that P(milaify π 1 a π 2 ) = P(X R 2 π 1 ) = P(2 1) = P (Y < m) ( ) Y µ Y1 = P < m (µ 1 µ 2 )Σ 1 µ 1 σ Y ) = P (Z < (1/2) 2 = Φ( /2) where Φ( ) denote the CDF of the tandard normal ditribution. Nathaniel E. Helwig (U of Minneota) Dirimination and Claifiation Updated 14-Mar-2017 : Slide 36

37 Ealuating Claifiation Funtion Claifiation Quality in Linear Diriminant Analyi Calulating Milaifiation in LDA (laify π 2 a π 1 ) Defining m = (1/2)(µ 1 µ 2 ) Σ 1 (µ 1 + µ 2 ), we hae that P(milaify π 2 a π 1 ) = P(X R 1 π 2 ) = P(1 2) = P (Y m) ( ) Y µ Y2 = P m (µ 1 µ 2 )Σ 1 µ 2 σ Y ) = P (Z (1/2) 2 = 1 Φ( /2) = Φ( /2) where Φ( ) denote the CDF of the tandard normal ditribution. Nathaniel E. Helwig (U of Minneota) Dirimination and Claifiation Updated 14-Mar-2017 : Slide 37

38 Ealuating Claifiation Funtion Claifiation Quality in Linear Diriminant Analyi Optimum Error Rate for Linear Diriminant Analyi For the LDA laifiation rule, we hae that OER = min R 1,R 2 TPM(R 1, R 2 ) = 1 2 P(milaify π 1 a π 2 ) P(milaify π 2 a π 1 ) = 1 2 Φ( /2) + 1 [1 Φ( /2)] 2 = Φ( /2) o the OER i a funtion of the effet ize = (µ 1 µ 2 ) Σ 1 (µ 1 µ 2 ) whih i ditane meaure between µ 1 and µ 2. Nathaniel E. Helwig (U of Minneota) Dirimination and Claifiation Updated 14-Mar-2017 : Slide 38

39 Claifying g 2 Population Claifying g 2 Population Nathaniel E. Helwig (U of Minneota) Dirimination and Claifiation Updated 14-Mar-2017 : Slide 39

40 Claifying g 2 Population Oeriew of Problem The g Population Claifiation Problem Let X = (X 1,..., X p ) denote a random etor and let f k (x) denote the probability denity funtion (pdf) for population π k for k {1,..., g}. Problem: Gien a realization X = x, we want to aign x to a π k. We want to find ome laifiation rule to determine whether a realization X = x hould be aigned to population π 1, π 2,..., or π g. Nathaniel E. Helwig (U of Minneota) Dirimination and Claifiation Updated 14-Mar-2017 : Slide 40

41 Claifying g 2 Population Oeriew of Problem Claifiation Rule with g 2 Population Let Ω denote the ample pae, i.e., all poible alue of x, and R 1 Ω i the ubet of Ω for whih we laify x a π 1 R 2 Ω i the ubet of Ω for whih we laify x a π 2. R g Ω i the ubet of Ω for whih we laify x a π g Ω = R 1 R 2 R g and R k R l = for all k l. The laifiation rule partition the ample pae The laifiation region are mutually exluie Nathaniel E. Helwig (U of Minneota) Dirimination and Claifiation Updated 14-Mar-2017 : Slide 41

42 Claifying g 2 Population Oeriew of Problem Viualizing a Claifiation Rule: g = 3 Population Figure: Figure from Applied Multiariate Statitial Analyi, 6th Ed (Johnon & Wihern). Viualization i for p = 2 ariable. Nathaniel E. Helwig (U of Minneota) Dirimination and Claifiation Updated 14-Mar-2017 : Slide 42

43 Claifying g 2 Population Cot of Milaifiation Probability and Cot of Milaifiation The onditional probability P(l k) of laifying an objet a π l when the objet really belong to π k i gien by P(l k) = P(X R l π k ) = f k (x)dx R l for all k l with k, l {1,..., g}. Note that P(k k) = 1 l k P(l k) by definition. Let (l k) denote the ot of alloating an objet to π l when the objet really belong to π k, and let p k denote the prior probability of π k. Nathaniel E. Helwig (U of Minneota) Dirimination and Claifiation Updated 14-Mar-2017 : Slide 43

44 Claifying g 2 Population Cot of Milaifiation Expeted Cot of Milaifiation (reiited) The onditional expeted ot of milaifying an objet from π k i ECM(k) = l k P(l k)(l k) Inorporating the prior probabilitie, the oerall ECM i gien by g g ECM = p k ECM(k) = P(l k)(l k) l k k=1 k=1 p k Nathaniel E. Helwig (U of Minneota) Dirimination and Claifiation Updated 14-Mar-2017 : Slide 44

45 Claifying g 2 Population Cot of Milaifiation Minimum ECM Claifiation Rule The laifiation region {R 1, R 2,..., R g } that minimize the ECM are defined by alloating X = x to the population π k that minimize p l f l (x)(k l) l k To undertand the logi of the laifiation rule, uppoe that we hae equal ot, i.e., (l k) = (k l) = 1 for all k, l {1,..., g} We alloate x to the population π k that minimize l k p lf l (x) Minimizing l k p lf l (x) i the ame a maximizing p k f k (x) Alloate x to population π k if p k f k (x) > p l f l (x) for all l k Thi i equialent to maximizing the poterior probability P(π k x) Nathaniel E. Helwig (U of Minneota) Dirimination and Claifiation Updated 14-Mar-2017 : Slide 45

46 Claifying g 2 Population Fiher Diriminant Analyi Oeriew of Fiher Approah Fiher deeloped hi diriminant analyi for g > 2 population. Idea: find a mall number of linear ombination (e.g., a 1 x, a 2 x, a 3 x) that bet eparate the group. Offer a imple and ueful proedure for laifiation, whih alo proide nie iualization. Plot the linear ombination to iualize the diriminant Nathaniel E. Helwig (U of Minneota) Dirimination and Claifiation Updated 14-Mar-2017 : Slide 46

47 Claifying g 2 Population Fiher Diriminant Analyi Aumption of Fiher Diriminant Analyi Let X = (X 1,..., X p ) denote a random etor and let f k (x) (µ k, Σ) denote the pdf for population π k. Note the homogeneity of oariane matrix aumption Do not need the multiariate normality aumption Let µ = 1 g g k=1 µ k denote the mean of the ombined population, and g B µ = (µ k µ)(µ k µ) k=1 denote Between um-of-quare and roprodut (SSCP) matrix. Nathaniel E. Helwig (U of Minneota) Dirimination and Claifiation Updated 14-Mar-2017 : Slide 47

48 Claifying g 2 Population Fiher Diriminant Analyi Propertie of a Linear Combination Define new ariable Y = a X whih ha propertie E(Y π k ) = a E(X π k ) = a µ k V (Y π k ) = a V (X π k )a = a Σa and note that the oerall mean of Y ha the form µ Y = 1 g g µ Yk = 1 g k=1 g a µ k = a µ k=1 Nathaniel E. Helwig (U of Minneota) Dirimination and Claifiation Updated 14-Mar-2017 : Slide 48

49 Claifying g 2 Population Fiher Diriminant Analyi Between eru Within Group Variability Form the ratio of the between group eparation oer the ariane of Y : F = = g k=1 (µ Y k µ Y ) 2 σ 2 Y g k=1 (a µ k a µ) 2 a Σa = a [ g k=1 (µ k µ)(µ k µ) ] a a Σa = a B µ a a Σa Note that higher F alue relate to more eparation between group. Nathaniel E. Helwig (U of Minneota) Dirimination and Claifiation Updated 14-Mar-2017 : Slide 49

50 Claifying g 2 Population Fiher Diriminant Analyi Population Diriminant The population k-th diriminant i the linear ombination Y k = a k X where a k i proportional to the k-th eigenetor of Σ 1 B µ. k = 1,..., where = min(g 1, p) The a k are aled to make the Y k hae unit ariane, i.e., a k Σa k = 1. a k Σa l = 0 for k l Note that thi i only ueful if we omehow know the true population parameter µ 1,..., µ g and Σ. Nathaniel E. Helwig (U of Minneota) Dirimination and Claifiation Updated 14-Mar-2017 : Slide 50

51 Claifying g 2 Population Sample Diriminant Fiher Diriminant Analyi The ample etimated Between and Within SSCP matrie are B = g ( x k x)( x k x) and W = k=1 g n k (x i(k) x k )(x i(k) x k ) k=1 i=1 where x k = 1 n k nk i=1 x i(k) and x = 1 g g k=1 x k. The ample k-th diriminant i the linear ombination Ŷ k = â k X where â k i proportional to the k-th eigenetor of W 1 B. The â k are aled to make the Ŷk hae unit ariane, i.e., â ˆΣâ k k = 1, where ˆΣ = S p = 1 n g W with n = g k=1 n k. Nathaniel E. Helwig (U of Minneota) Dirimination and Claifiation Updated 14-Mar-2017 : Slide 51

52 Claifying g 2 Population Fiher Diriminant Analyi Propertie of Population Diriminant Let Y = A X where A = [a 1,..., a ]. Y = (Y 1,..., Y ) ontain the diriminant Column of A ontain the linear ombination weight The mean of Y i gien by and the oariane matrix for Y i E(Y π k ) = A E(X π k ) = A µ k = µ ky Co(Y ) = A Co(X π k )A = A ΣA = I beaue the diriminant hae unit ariane and are unorrelated. Remember: a k Σa l = δ kl where δ kl i Kroneker δ Nathaniel E. Helwig (U of Minneota) Dirimination and Claifiation Updated 14-Mar-2017 : Slide 52

53 Claifying g 2 Population Fiher Diriminant Analyi Claifying New Objet with Diriminant Gien a realization X = x, define y = A x and alulate the ditane between the obered y = (y 1,..., y ) and the k-th population mean: D k = (y µ ky ) (y µ ky ) = (y l µ kyl ) 2 = l=1 where µ ky = A µ k and y l = a l x and µ ky l = a l µ k. [a l (x µ k)] 2 l=1 To build a ditane uing r diriminant, ue D (r) k = r (y l µ kyl ) 2 = l=1 r [a l (x µ k)] 2 and laify x to the population π k that minimize the ditane D (r) k. Nathaniel E. Helwig (U of Minneota) Dirimination and Claifiation Updated 14-Mar-2017 : Slide 53 l=1

54 Claifying g 2 Population Fiher Diriminant Analyi Claifying New Objet with Sample Diriminant Gien a realization X = x, define ŷ =  x and alulate the ditane between the obered ŷ = (ŷ 1,..., ŷ ) and the k-th ample mean: ˆD k = (ŷ ˆµ ky ) (ŷ ˆµ ky ) = (ŷ l ˆµ kyl ) 2 = l=1 where ˆµ ky =  x k and ŷ l = â l x and ˆµ ky l = â l x k. [â l (x x k )] 2 l=1 To build a ditane uing r diriminant, ue ˆD (r) k = r (ŷ l ˆµ kyl ) 2 = l=1 r [â l (x x k )] 2 l=1 and laify x to the population π k that minimize the ditane ˆD (r) k. Nathaniel E. Helwig (U of Minneota) Dirimination and Claifiation Updated 14-Mar-2017 : Slide 54

55 Claifying g 2 Population Fiher Diriminant Analyi Relation to MVN Claifiation Problem Let X = (X 1,..., X p ) be a random etor and let f k (x) N(µ k, Σ k ) denote the pdf for population π k. Auming equal milaifiation ot, we alloate X = x to the population π k that minimize l k p lf l (x) maximize p k f k (x). Equialent to alloating X = x to the population π k that maximize d Q k (x) = Quadrati diriminant ore = 1 2 ln( Σ k ) 1 2 (x µ k) Σ 1 k (x µ k) + ln(p k ) dk L (x) = Linear diriminant ore = µ k Σ 1 x 1 2 µ k Σ 1 µ k + ln(p k ) where d L k i ued when Σ k = Σ for all k {1,..., g}. Nathaniel E. Helwig (U of Minneota) Dirimination and Claifiation Updated 14-Mar-2017 : Slide 55

56 Claifying g 2 Population Fiher Diriminant Analyi Relation to MVN Claifiation Problem (ontinued) If we aume that p k = 1/g for all k {1,..., g}, then d L k (x) = µ k Σ 1 x 1 2 µ k Σ 1 µ k Define the linear ombination y j = a j x, where a j = Σ 1/2 j with j denoting the j-th eigenetor of B µ = Σ 1/2 B µ Σ 1/2. Then p p D k = (y j µ kyj ) 2 = [a j (x µ k)] 2 = (x µ k ) Σ 1 (x µ k ) j=1 = 2d L k (x) + α j=1 where α = x Σ 1 x i ontant aro population. If rank( B µ ) = r, alloating to the population π k that maximize dk L (x) i equialent to alloating to the population π k that minimize D (r) k. Nathaniel E. Helwig (U of Minneota) Dirimination and Claifiation Updated 14-Mar-2017 : Slide 56

57 Fiher Iri Data Example Fiher Iri Data Example Nathaniel E. Helwig (U of Minneota) Dirimination and Claifiation Updated 14-Mar-2017 : Slide 57

58 Fiher Iri Data Example Data Oeriew Fiher (or Anderon ) Famou Iri Data R. A. Fiher publihed the LDA approah in 1936 and ued Edgar Anderon iri flower dataet a an example. The dataet onit of meaurement of p = 4 ariable taken from n k = 50 flower randomly ampled from eah of g = 3 peie. Variable: Sepal Length, Sepal Width, Petal Length, Petal Width Speie: etoa, eriolor, irginia The goal wa/i to build a linear diriminant funtion that bet laifie a new flower into one of the three peie. Nathaniel E. Helwig (U of Minneota) Dirimination and Claifiation Updated 14-Mar-2017 : Slide 58

59 Fiher Iri Data Example Fiher Famou Iri Data in R Data Oeriew > head(iri) Sepal.Length Sepal.Width Petal.Length Petal.Width Speie etoa etoa etoa etoa etoa etoa > olmean(iri[iri$speie=="etoa",1:4]) Sepal.Length Sepal.Width Petal.Length Petal.Width > olmean(iri[iri$speie=="eriolor",1:4]) Sepal.Length Sepal.Width Petal.Length Petal.Width > olmean(iri[iri$speie=="irginia",1:4]) Sepal.Length Sepal.Width Petal.Length Petal.Width > p <- 4L > g <- 3L Nathaniel E. Helwig (U of Minneota) Dirimination and Claifiation Updated 14-Mar-2017 : Slide 59

60 Fiher Iri Data Example Data Oeriew Make Pooled Coariane Matrix # make pooled oariane matrix > Sp <- matrix(0, p, p) > nx <- rep(0, g) > le <- leel(iri$speie) > for(k in 1:g){ + x <- iri[iri$speie==le[k],1:p] + nx[k] <- nrow(x) + Sp <- Sp + o(x) * (nx[k] - 1) + } > Sp <- Sp / (um(nx) - g) > round(sp, 3) Sepal.Length Sepal.Width Petal.Length Petal.Width Sepal.Length Sepal.Width Petal.Length Petal.Width Nathaniel E. Helwig (U of Minneota) Dirimination and Claifiation Updated 14-Mar-2017 : Slide 60

61 Fiher Iri Data Example Linear Diriminant Analyi LDA in R ia the lda Funtion (MASS Pakage) # fit lda model > library(mass) > ldamod <- lda(speie ~., data=iri, prior=rep(1/3, 3)) # hek the LDA oeffiient/aling > ldamod$aling LD1 LD2 Sepal.Length Sepal.Width Petal.Length Petal.Width > roprod(ldamod$aling, Sp) %*% ldamod$aling LD1 LD2 LD e e-16 LD e e+00 # reate the (entered) diriminant ore > mu.k <- ldamod$mean > mu <- olmean(mu.k) > dore <- ale(iri[,1:p], enter=mu, ale=f) %*% ldamod$aling > um((dore - predit(ldamod)$x)^2) [1] e-28 Nathaniel E. Helwig (U of Minneota) Dirimination and Claifiation Updated 14-Mar-2017 : Slide 61

62 Fiher Iri Data Example Linear Diriminant Analyi Plot LDA Reult: Sore and Coeffiient Diriminant Sore Diriminant Coeffiient LD etoa eriolor irginia LD Petal.Width Petal.Length Sepal.Width Sepal.Length R ode for left plot: LD1 plot(dore, xlab="ld1", ylab="ld2", ph=pid, ol=pid, main="diriminant Sore", xlim=(-10, 10), ylim=(-3, 3)) legend("top",le,ph=1:3,ol=1:3,bty="n") Nathaniel E. Helwig (U of Minneota) Dirimination and Claifiation Updated 14-Mar-2017 : Slide 62 LD1

63 Fiher Iri Data Example Linear Diriminant Analyi Plot LDA Reult: Diriminant Partition LD1 LD2 app. error rate: 0.02 Partition Plot library(klar) peie <- fator(rep(("","",""), eah=50)) partimat(x=dore[,2:1], grouping=peie, method="lda") Nathaniel E. Helwig (U of Minneota) Dirimination and Claifiation Updated 14-Mar-2017 : Slide 63

64 Fiher Iri Data Example Linear Diriminant Analyi Plot LDA Reult: All Pairwie Partition Sepal.Width Sepal.Length app. error rate: Petal.Length Sepal.Length app. error rate: Petal.Length Sepal.Width app. error rate: Petal.Width Sepal.Length app. error rate: Petal.Width Sepal.Width app. error rate: Petal.Width Petal.Length app. error rate: 0.04 Partition Plot library(klar) peie <- fator(rep(("","",""), eah=50)) partimat(x=iri[,1:4], grouping=peie, method="lda") Nathaniel E. Helwig (U of Minneota) Dirimination and Claifiation Updated 14-Mar-2017 : Slide 64

65 Fiher Iri Data Example APER and Expeted AER Linear Diriminant Analyi # make onfuion matrix (and APER) > onfuion <- table(iri$speie, predit(ldamod)$la) > onfuion etoa eriolor irginia etoa eriolor irginia > n <- um(onfuion) > aper <- (n - um(diag(onfuion))) / n > aper [1] 0.02 # ue CV to get expeted AER > ldamodcv <- lda(speie ~., data=iri, prior=rep(1/3, 3), CV=TRUE) > onfuioncv <- table(iri$speie, ldamodcv$la) > onfuioncv etoa eriolor irginia etoa eriolor irginia > eaer <- (n - um(diag(onfuioncv))) / n > eaer [1] 0.02 Nathaniel E. Helwig (U of Minneota) Dirimination and Claifiation Updated 14-Mar-2017 : Slide 65

66 Fiher Iri Data Example Linear Diriminant Analyi Split Data into Training (70%) and Teting (30%) Set > # plit into eparate matrie for eah flower > X <- ubet(iri, Speie=="etoa") > X <- ubet(iri, Speie=="eriolor") > X <- ubet(iri, Speie=="irginia") # plit into training and teting > et.eed(1) > id <- ample.int(n=50, ize=35) > id <- ample.int(n=50, ize=35) > id <- ample.int(n=50, ize=35) > Xtrain <- rbind(x[id,], X[id,], X[id,]) > Xtet <- rbind(x[-id,], X[-id,], X[-id,]) # fit lda to training and ealuate on teting > ldatrain <- lda(speie ~., data=xtrain, prior=rep(1/3, 3)) > onfuiontet <- table(xtet$speie, predit(ldatrain, newdata=xtet)$la) > onfuiontet etoa eriolor irginia etoa eriolor irginia > n <- um(onfuiontet) > aer <- (n - um(diag(onfuiontet))) / n > aer [1] Nathaniel E. Helwig (U of Minneota) Dirimination and Claifiation Updated 14-Mar-2017 : Slide 66

67 Fiher Iri Data Example Linear Diriminant Analyi Two-Fold CV with 100 Random 70/30 Split > nrep <- 100 > aer <- rep(0, nrep) > et.eed(1) > for(k in 1:nrep){ + id <- ample.int(n=50, ize=35) + id <- ample.int(n=50, ize=35) + id <- ample.int(n=50, ize=35) + Xtrain <- rbind(x[id,], X[id,], X[id,]) + Xtet <- rbind(x[-id,], X[-id,], X[-id,]) + ldatrain <- lda(speie ~., data=xtrain, prior=rep(1/3, 3)) + onfuiontet <- table(xtet$speie, predit(ldatrain, newdata=xtet)$la) + onfuiontet + n <- um(onfuiontet) + aer[k] <- (n - um(diag(onfuiontet))) / n + } > mean(aer) [1] Nathaniel E. Helwig (U of Minneota) Dirimination and Claifiation Updated 14-Mar-2017 : Slide 67

68 Fiher Iri Data Example Quadrati Diriminant Analyi QDA in R ia the qda Funtion (MASS Pakage) # fit qda model > library(mass) > qdamod <- qda(speie ~., data=iri, prior=rep(1/3, 3)) > name(qdamod) [1] "prior" "ount" "mean" "aling" "ldet" "le" "N" [8] "all" "term" "xleel" # hek the QDA oeffiient/aling > dim(qdamod$aling) [1] > round(roprod(qdamod$aling[,,1], o(x[,1:p])) %*% qdamod$aling[,,1], 4) > round(roprod(qdamod$aling[,,2], o(x[,1:p])) %*% qdamod$aling[,,2], 4) > round(roprod(qdamod$aling[,,3], o(x[,1:p])) %*% qdamod$aling[,,3], 4) Nathaniel E. Helwig (U of Minneota) Dirimination and Claifiation Updated 14-Mar-2017 : Slide 68

69 Fiher Iri Data Example Quadrati Diriminant Analyi Plot QDA Reult: All Pairwie Partition Sepal.Width Sepal.Length app. error rate: Petal.Length Sepal.Length app. error rate: Petal.Length Sepal.Width app. error rate: Petal.Width Sepal.Length app. error rate: Petal.Width Sepal.Width app. error rate: Petal.Width Petal.Length app. error rate: 0.02 Partition Plot library(klar) peie <- fator(rep(("","",""), eah=50)) partimat(x=iri[,1:4], grouping=peie, method="qda") Nathaniel E. Helwig (U of Minneota) Dirimination and Claifiation Updated 14-Mar-2017 : Slide 69

70 Fiher Iri Data Example APER and Expeted AER Quadrati Diriminant Analyi # make onfuion matrix (and APER) > onfuion <- table(iri$speie, predit(qdamod)$la) > onfuion etoa eriolor irginia etoa eriolor irginia > n <- um(onfuion) > aper <- (n - um(diag(onfuion))) / n > aper [1] 0.02 # ue CV to get expeted AER > qdamodcv <- qda(speie ~., data=iri, prior=rep(1/3, 3), CV=TRUE) > onfuioncv <- table(iri$speie, qdamodcv$la) > onfuioncv etoa eriolor irginia etoa eriolor irginia > eaer <- (n - um(diag(onfuioncv))) / n > eaer [1] Nathaniel E. Helwig (U of Minneota) Dirimination and Claifiation Updated 14-Mar-2017 : Slide 70

71 Fiher Iri Data Example Quadrati Diriminant Analyi Split Data into Training (70%) and Teting (30%) Set > # plit into eparate matrie for eah flower > X <- ubet(iri, Speie=="etoa") > X <- ubet(iri, Speie=="eriolor") > X <- ubet(iri, Speie=="irginia") > # plit into training and teting > et.eed(1) > id <- ample.int(n=50, ize=35) > id <- ample.int(n=50, ize=35) > id <- ample.int(n=50, ize=35) > Xtrain <- rbind(x[id,], X[id,], X[id,]) > Xtet <- rbind(x[-id,], X[-id,], X[-id,]) # fit qda to training and ealuate on teting > qdatrain <- qda(speie ~., data=xtrain, prior=rep(1/3, 3)) > onfuiontet <- table(xtet$speie, predit(qdatrain, newdata=xtet)$la) > onfuiontet etoa eriolor irginia etoa eriolor irginia > n <- um(onfuiontet) > aer <- (n - um(diag(onfuiontet))) / n > aer [1] Nathaniel E. Helwig (U of Minneota) Dirimination and Claifiation Updated 14-Mar-2017 : Slide 71

72 Fiher Iri Data Example Quadrati Diriminant Analyi Two-Fold CV with 100 Random 70/30 Split > nrep <- 100 > aer <- rep(0, nrep) > et.eed(1) > for(k in 1:nrep){ + id <- ample.int(n=50, ize=35) + id <- ample.int(n=50, ize=35) + id <- ample.int(n=50, ize=35) + Xtrain <- rbind(x[id,], X[id,], X[id,]) + Xtet <- rbind(x[-id,], X[-id,], X[-id,]) + qdatrain <- qda(speie ~., data=xtrain, prior=rep(1/3, 3)) + onfuiontet <- table(xtet$speie, predit(qdatrain, newdata=xtet)$la) + onfuiontet + n <- um(onfuiontet) + aer[k] <- (n - um(diag(onfuiontet))) / n + } > mean(aer) [1] Nathaniel E. Helwig (U of Minneota) Dirimination and Claifiation Updated 14-Mar-2017 : Slide 72

73 Fiher Iri Data Example Quadrati Diriminant Analyi Plot LDA and QDA Reult uing PCA LDA Reult QDA Reult PC etoa eriolor irginia PC etoa eriolor irginia PC PC1 Nathaniel E. Helwig (U of Minneota) Dirimination and Claifiation Updated 14-Mar-2017 : Slide 73

74 Fiher Iri Data Example Quadrati Diriminant Analyi Plot LDA and QDA Reult uing PCA (R ode) R ode for plot on preiou lide: # iualize LDA and QDA reult ia PCA ldaid <- a.integer(predit(ldamod)$la) qdaid <- a.integer(predit(qdamod)$la) pamod <- prinomp(iri[,1:4]) de.new(width=10, height=5, norstudiogd=true) par(mfrow=(1,2)) plot(pamod$ore[,1:2], xlab="pc1", ylab="pc2", ph=ldaid, ol=ldaid, main="lda Reult", xlim=(-4, 4), ylim=(-2, 2)) legend("topright",le,ph=1:3,ol=1:3,bty="n") abline(h=0,lty=3) abline(=0,lty=3) plot(pamod$ore[,1:2], xlab="pc1", ylab="pc2", ph=qdaid, ol=qdaid, main="qda Reult", xlim=(-4, 4), ylim=(-2, 2)) legend("topright",le,ph=1:3,ol=1:3,bty="n") abline(h=0,lty=3) abline(=0,lty=3) Nathaniel E. Helwig (U of Minneota) Dirimination and Claifiation Updated 14-Mar-2017 : Slide 74