Lecture 7: Lear ad quadratc classfers Bayes classfers for ormally dstrbuted classes Case : Σ σ I Case : Σ Σ (Σ daoal Case : Σ Σ (Σ o-daoal Case 4: Σ σ I Case 5: Σ Σ j eeral case Lear ad quadratc classfers: coclusos Itroducto to Patter Recoto Rcardo Guterrez-Osua Wrht State Uversty
Bayes classfers for ormally dstrbuted classes O Lecture 4 we showed that the decso rule (MAP that mmzed the probablty of error could be formulated terms of a famly of dscrmat fuctos choose f (x > (x j where (x P( j x As we wll show, for classes that are ormally dstrbuted, ths famly of dscrmat fuctos ca be reduced to very smple expressos Geeral expresso for Gaussa destes he multvarate ormal desty fucto was defed as Dscrmat fuctos Features -/ (x exp P( We take atural los sce the loarthm s a mootocally creas fucto x (x fx(x exp / / ( Wth ths md, ad utlz Bayes rule, the MAP dscrmat fucto becomes P(x P( (x P( x exp (x (x P( / P(x / ( Elmat costat terms (x - lo + ( lo( P( Class assmet Select max (x x x P(x C (x x d Costs hs expresso s called a quadratc dscrmat fucto Itroducto to Patter Recoto Rcardo Guterrez-Osua Wrht State Uversty
Case : Σ σ I hs stuato occurs whe the features are statstcally depedet wth the same varace for all classes I ths case, the quadratc dscrmat fucto becomes ( ( ( - lo( + lo( P( - (x I - lo I + lo P( + lo( P( Expad ths expresso (x x - x - + lo P( x x x + Elmat the term x x, whch s costat for all classes ( ( ( ( lo( P( + secod dropp term the (x x + + lo P( w x + w w where w + lo( P( Sce the dscrmat s lear, the decso boudares (x j (x, wll be hyper-plaes If we assume equal prors (x ( ( x σ µ µ µ C Dstace Dstace Dstace Mmum Selector class hs s called a mmum-dstace or earest mea classfer he loc of costat probablty for each class are hyper-spheres For ut varace (σ, the dstace becomes the Eucldea dstace Itroducto to Patter Recoto Rcardo Guterrez-Osua Wrht State Uversty
Itroducto to Patter Recoto Rcardo Guterrez-Osua Wrht State Uversty 4 Case : Σ σ I, example o llustrate the prevous result, we wll compute the decso boudares for a - class, -dmesoal problem wth the follow class mea vectors ad covarace matrces ad equal prors [ ] [ ] [ ] 5 4 7
Case : Σ Σ (Σ daoal he classes stll have the same covarace matrx, but the features are allowed to have dfferet varaces I ths case, the quadratc dscrmat fucto becomes (x (x (x k k (x[k] x[k] k [k] x[k] O k - lo lo [k] + [k] ( + lo( P( (x k k - lo + lo P( lo ( k k O + lo P( ( + lo P( ( Elmat the term x[k], whch s costat for all classes x[k] [k] + [k] (x lo k + k k k ( lo P( hs dscrmat s lear, so the decso boudares (x j (x, wll be also be hyper-plaes he loc of costat probablty are hyper-ellpses aled wth the feature axs ote that the oly dfferece wth the prevous classfer s that the dstace of each axs s ormalzed by the varace of the axs Itroducto to Patter Recoto Rcardo Guterrez-Osua Wrht State Uversty 5
Itroducto to Patter Recoto Rcardo Guterrez-Osua Wrht State Uversty 6 o llustrate the prevous result, we wll compute the decso boudares for a -class, -dmesoal problem wth the follow class mea vectors ad covarace matrces ad equal prors [ ] [ ] [ ] 5 4 5 Case : Σ Σ (Σ daoal, example
Case : Σ Σ (Σ o-daoal I ths case, all the classes have the same covarace matrx, but ths s o loer daoal he quadratc dscrmat becomes (x - lo - lo ( + lo( P( ( + lo( P( Elmat the term lo, whch s costat for all classes (x + lo P( ( he quadratc term s called the Mahalaobs dstace, a very mportat dstace Statstcal PR Mahalaobs Dstace x - y y y x he Mahalaobs dstace s a vector dstace that uses a - orm - ca be thouht of as a stretch factor o the space ote that for a detty covarace matrx (I, the Mahalaobs dstace becomes the famlar Eucldea dstace µ x x - K x - Itroducto to Patter Recoto Rcardo Guterrez-Osua Wrht State Uversty 7
Case : Σ Σ (Σ o-daoal Expaso of the quadratc term the dscrmat yelds (x + lo P( + Remov the term x - x, whch s costat for all classes (x ( x + lo( P( + ( ( x x x + lo( P( Reoraz terms we obta (x w where x + w w w + lop( x µ µ µ C Dstace Dstace Dstace Mmum Selector class hs dscrmat s lear, so the decso boudares wll also be hyper-plaes he costat probablty loc are hyper-ellpses aled wth the eevectors of If we ca assume equal prors (x (x he classfer becomes a mmum (Mahalaobs dstace classfer Itroducto to Patter Recoto Rcardo Guterrez-Osua Wrht State Uversty 8
Case : Σ Σ (Σ o-daoal, example o llustrate the prevous result, we wll compute the decso boudares for a - class, -dmesoal problem wth the follow class mea vectors ad covarace matrces ad equal prors [ ] [ 5 4] [ 5].7.7.7.7.7.7 Itroducto to Patter Recoto Rcardo Guterrez-Osua Wrht State Uversty 9
Case 4: Σ σ I I ths case, each class has a dfferet covarace matrx, whch s proportoal to the detty matrx he quadratc dscrmat becomes (x - lo - lo ( + lo( P( ( + lo( P( hs expresso caot be reduced further so he decso boudares are quadratc: hyper-ellpses he loc of costat probablty are hyper-spheres aled wth the feature axs Itroducto to Patter Recoto Rcardo Guterrez-Osua Wrht State Uversty
Case 4: Σ σ I, example o llustrate the prevous result, we wll compute the decso boudares for a - class, -dmesoal problem wth the follow class mea vectors ad covarace matrces ad equal prors [ ] [ 5 4] [ 5].5.5 Zoom out Itroducto to Patter Recoto Rcardo Guterrez-Osua Wrht State Uversty
Case 5: Σ Σ j eeral case We already derved the expresso for the eeral case at the be of ths dscusso (x - lo + ( lo( P( Reoraz terms a quadratc form yelds (x x Wx + w x + w W where w w - lo ( + lo( P( he loc of costat probablty for each class are hyper-ellpses, oreted wth the eevectors of Σ for that class he decso boudares are aa quadratc: hyper-ellpses or hyper-parabollods otce that the quadratc expresso the dscrmat s proportoal to the Mahalaobs dstace us the class-codtoal covarace Σ Itroducto to Patter Recoto Rcardo Guterrez-Osua Wrht State Uversty
Case 5: Σ Σ j eeral case, example o llustrate the prevous result, we wll compute the decso boudares for a - class, -dmesoal problem wth the follow class mea vectors ad covarace matrces ad equal prors [ ] [ 5 4] [ 5] 7.5.5.5 Zoom out Itroducto to Patter Recoto Rcardo Guterrez-Osua Wrht State Uversty
Coclusos From the prevous examples we ca extract the follow coclusos he Bayes classfer for ormally dstrbuted classes (eeral case s a quadratc classfer he Bayes classfer for ormally dstrbuted classes wth equal covarace matrces s a lear classfer he mmum Mahalaobs dstace classfer s optmum for ormally dstrbuted classes ad equal covarace matrces ad equal prors he mmum Eucldea dstace classfer s optmum for ormally dstrbuted classes ad equal covarace matrces proportoal to the detty matrx ad equal prors Both Eucldea ad Mahalaobs dstace classfers are lear classfers he oal of ths dscusso was to show that some of the most popular classfers ca be derved from decso-theoretc prcples ad some smplfy assumptos It s mportat to realze that us a specfc (Eucldea or Mahalaobs mmum dstace classfer mplctly correspods to certa statstcal assumptos he questo whether these assumptos hold or do t ca rarely be aswered practce; most cases we are lmted to post ad aswer the questo does ths classfer solve our problem or ot? Itroducto to Patter Recoto Rcardo Guterrez-Osua Wrht State Uversty 4