Linear Discriminant Functions

Size: px

Start display at page:

Download "Linear Discriminant Functions"

Charity Webster
6 years ago
Views:

1 Lear Dscrmat Fuctos

2 Lear Dscrmat Fuctos So far, cocetrate o est fuctos th a ko parametrc form shape of the fucto rectl Here, lear the scrmat fuctos surface separatg fferet clusters hat tpe of surfaces? lear easest! fuctos hperplaes R, ANN, & ML

3 I. populato II. populato III. populato sze IV. populato sze sze sze e R, ANN, & ML 3

4 4 R, ANN, & ML Case I: same pror, same evato Decso bouar s plaar I the mle of the to cluster / / e e p p

5 Case.A he partto plae s perpecular to the le coectg to meas Scalar case Covarace matrces are the same a are agoal th the same varace all features Σ I R, ANN, & ML 5

6 Case I.A: same pror, same evato Eve th multple classes, f the all have the same pror a the same evato, the the ecso bouares form a Vooro agram, or Baes rule s a mmum Euclea stace classfer R, ANN, & ML 6

7 Case.B he partto plae s ot perpecular to the le coectg to meas Same but geeral covarace matrces R, ANN, & ML 7

8 8 R, ANN, & ML Case II: fferet pror, same evato Decso bouar s stll plaar At log log log log / / e e log log log log

9 R, ANN, & ML 9

10 Graphcal Iterpretato D opulato lkelhoo Class Class feature Class msclassfe as Class Class msclassfe as Class 0

11 probablt < + More class msclassfcato Class Class feature probablt < + More class msclassfcato Class Class feature 3/7/003

12 R, ANN, & ML Case III & IV: same or fferet pror, fferet evato Decso bouar s o loger plaar / / log log e e

13 R, ANN, & ML 3

14 Lessos he ecso bouares geeral are NO lear or plaar Eve th a sgle feature a a Gaussa strbuto the bouar ca be complcate hat sa, plaar bouares ca be use to appromate curve, sjot bouares a lot more o ths later, massage the classfer Features ca also be massage he are mathematcall more tractable R, ANN, & ML 4

15 o-categor case g 0 t g 0 g 0 eghtvector threshol eght 0 R, ANN, & ML 5

16 Decso surface Hperplae g, 0 0 g 0 g 0 g 0, 0, g o R, ANN, & ML 6

17 7 R, ANN, & ML Decso surface Hperplae,, g 0 g 0 g 0 0 o g 0 0, g

18 8 R, ANN, & ML rag roceure o-categor case Use tagge samples to eterme the scrmat fucto },...,, { 0 0 j o j o t t j j t t 0 0 0,] [, 0,, j j t t t t Augmete feature vectors

19 rag roceure cot. Each trag sample costras to le o a half plae f 0 0 j R, ANN, & ML 9

20 rag roceure cot. t o 0 t o 0 0 j j 0 R, ANN, & ML 0 0

21 rag roceure cot. 0 Each trag sample costras to le o a pe f, 0 0 th marg j 0 R, ANN, & ML

22 Usg Graet Descet A search mechasm Start at a arbtrarl chose startg pot Move a recto graet to mmze the cost fucto Basc calculus, to be epecte of ever egeer after 5 mute thought R, ANN, & ML

23 3 R, ANN, & ML Cost fucto terms of augmete feature vector [,] pealze for all samples msclassfe Graet recto Upate :msclassf e samples c t.... c c c c k k k k k k k c Usg Graet Descet

24 Graphcal Iterpretato k k k k R, ANN, & ML 4

25 Graphcal Iterpretato cot Weght s the sge sum of samples he more ffcult a sample s to be classfe, the more ts eght Durg classfcato, e have a Ol er prouct of troublesome trag samples a test samples are eee a:eght, : class hese to cocepts are ver mportat, the appear aga a aga later erceptro SVM Kerel methos a R, ANN, & ML 5

26 Number of rog samples t t t b A bas term makes sure that soluto s more the ceter of the feasble rego R, ANN, & ML 6

27 Ho Goo ca a -Categor Classfer be? A B Class A Class B Feature Class B msclassfe as Class A Class A msclassfe as Class B As goo as that b Baesa rule

28 < + Class A Class B feature less Class A msclassfe < more Class B msclassfe < + Class A Class B feature less Class B msclassfe < more Class A msclassfe R, ANN, & ML 8

29 Implemetato Detals Dffcult: features ca be correlate U-correlate features usg SVD A regularzato t t c :msclassf e samples Numercall, the sstem s stll quaratc so GD stll orks R, ANN, & ML 9

30 Solvg A = B Ro terpretato Each ro s a le Itersecto of multple les Or Each ro s a plae Multple plaes efe a feasble rego Colum terpretato Each colum s a vector Combato of these vectors to appromate B R, ANN, & ML 30

31 3 No-teratve Metho W W j j j o egatve postve, 0 0 arg m 0 arg m o ],,, [, ],, [,, ],,, [ Val ol for regresso problem F t later logstc regresso

32 Graphcal Iterpretato H = FICA Icome : classf trag set b leare parameter s a sample sze b meso of ata matr combes the colums of to best appromate Combe features FICA, come, etc. to ecsos loa ^hat s a combato of colums of What s ^hat? Ho close s ^hat to G? 3

33 Graphcal Iterpretato H = FICA Icome H projects oto the space spae b colums of Smplf the ecsos to ft the features arg m W W arg m arg m W H H 33

34 34 Ugl Math UU U VΣ V Σ Σ V UΣ U VΣ VΣΣ UΣ U VΣ UΣ U VΣ UΣ V V V V V UΣV UU s the staar form of a projecto operator U : er prouct th the bass vector U: epa o the bass vector a U has the same colum space

35 =, eact soluto roblem # >, least square, most lkel scearos Whe <, there are ot eough costrats to eterme coeffcets uquel = W 35

36 roblem # If fferet attrbutes are hghl correlate come a FICA he colums become epeet Coeffcets are the poorl eterme th hgh varace E.g., large postve coeffcet o oe ca be cacele b a smlarl large egatve coeffcet o ts correlate cous Sze costrat s helpful Caveat: costrat s problem epeet 36

37 37 Rge Regresso regularzato I I I W W rge rge j j j j j o rge λ λ, ˆ 0 0 arg m arg m W j j j o egatve postve, 0 arg m

38 38 Ugl Math u u U Σ I Σ Σ UΣ U Σ I V Σ VΣ V UΣ U Σ V I Σ VΣ UΣ U VΣ I UΣ U VΣ UΣ I I rge rge λ λ V λ V V λ V V λ V V λ λ UΣV

39 Ho to Decpher hs u λ u Re: best estmate hat s compose of colums of U bass features, recall U a have the same colum space Gree: ho these bass colums are eghe Blue: projecto of target oto these colums ogether: represetg a bo-ftte coorate sstem u 39

40 Sebar Recall that race sum of the agoals of a matr s the same as the sum of the egevalues roof: ever matr has a staar Jora form a upper tragular matr here the egevalues appear o the agoal trace=sum of egevalues Jora form results from a smlart trasform A - hch oes ot chage egevalues A A A A J 40

41 hscal Iterpretato Sgular values of represets the sprea of ata alog fferet bo-fttg mesos orthoormal colums o estmate =<, rge > regularzato mmzes the cotrbuto from less sprea-out mesos Less sprea-out mesos usuall have much larger varace hgh meso ege moes harer to estmate graets relabl race +I - s calle effectve egrees of freeom 4

42 More Detals H race +I - s calle effectve egrees of freeom Cotrols ho ma ege moes are actuall use or actve f, 0, f 0, Dfferet methos are possble Shrkg smoother: cotrbutos are scale rojecto smoother: cotrbutos are use or ot use 0 4

43 43 Dual Formulato teratve Weght vector ca be epresse as a sum of the trag feature vectors α a α a Regular Rge regresso W W rge j j j j j o rge 0 arg m arg m H

44 44 Dual Formulato cot. α a I G α α I G I α α I α α α α I I,,,,,,, g a a

45 45 I More Detals..... I Gram matr

46 Observatos rmar s b rag: Slo for hgh feature meso Use: fast O g, λi Dual Ol er proucts are volve s b rag: Fast for hgh feature meso Use: Slo O N er prouct to evaluate, each requres multplcatos, g I,, 46

47 47 Graphcal Iterpretato = I,,,,,, g a a j j j,

48 48 Oe Etreme erfect Ucorrelate g I,,,,, =, j j, Orthogoal projecto o geeralzato

49 49 Geeral Case V V V V V V V V U Σ I Σ U ΣU Σ I Σ U Σ I Σ U Σ I Σ U UΣ U I Σ U UΣ U I Σ U UΣ I U UΣ Σ U I ˆ Ho to terpret ths? Does ths stll make sese?

50 hscal Meag of SVD Assume that > s of rak at most U are the bo ata-ftte aes U s a projecto from to space S s the mportace of the mesos V s the represetato of the the space U Σ V 50

51 ˆ U Iterpretato Σ I u λ I the e, ucorrelate space, there are ol trag vectors a ecsos Re: ucorrelate ecso vector Gree: eghtg of the sgfcace of the compoets the ucorrelate ecso vector Blue: trasforme ucorrelate trag samples Stll the same terpretato: smlart measuremet a e space b Gram matr Ier prouct of trag samples a e sample Σ U u 5

52 Frst Importat Cocept he computato volves ol er prouct For trag samples computg the Gram matr For e sample computg regresso or classfcato results Smlart s measure terms of agle, stea of stace 5

53 Seco Importat Cocept Usg agle or stace for smlart measuremet oes t make problems easer or harer If ou caot separate ata, t oes t matter hat smlart measures ou use Massage ata rasform ata to hgher eve fte - mesoal space Data become more lkel to be learl separable caveat: choce of the kerel fucto s mportat Caot perform er prouct effcetl Kerel trck o ot have to 53

54 I realt Calculatg verse of ^t s ver epesve he soluto s b terato Furthermore, features are ofte ot use rectl, but certa olear trasformato of features are use Furthermore, such olear trasformato s ot calculate eplctl b Kerel trck R, ANN, & ML 54

55 Math Detal * * * Nolear trasform * * * R, ANN, & ML 55

56 Math Detal cot R, ANN, & ML 56

57 Math Detals cot. = 0 R, ANN, & ML 57

58 Math Detals hs represets a Gauss-Seal teratve soluto to the problem R, ANN, & ML 58

59 Mult-categor case Both a 3 3 or or c- to-categor cc-/ to-categor agast all for all R, ANN, & ML 59

60 Mult-categor Case cot. R, ANN, & ML 60

61 6 R, ANN, & ML Mult-categor case heoretcal Kesler s costructo Assume lear separablt k k j k f j all for c classes c 0,..., ˆ ˆ ˆ ].. [ ˆ c c c c c correctl meso ˆ,..., ˆ, ˆ samples must classf meso ˆ eght oe 3 c c c

62 6 R, ANN, & ML Graphcal Iterpretato c c G j all for g g c g j t 0,..., 0 ma ecso g g g g g g ecso g g g

63 Kesler Costructo rag fake -class Oe bg =[ c ] Ever trag sample s uplcate agast c- to geerate c- postve samples Staar -class teratve graet escet trag Classfcato Break o to c compoets c Evaluate a sample agast all. ake the largest oe as result R, ANN, & ML 63

64 Lear Mache g t 0 g g j,..., c for all j g g g g ma ecso g g R, ANN, & ML 64

65 65 R, ANN, & ML Multple-categores Kesler costructo oes ot etect boues F cluster ceter classes c features samples f f f c c c : : :

66 c 40 4 R, ANN, & ML 66

67 I Realt Lear Mache orks f samples a class tght, compact clusters class statstcs are sgle moe oe sgle peak the, a class ca be represete b a tpcal sample class mea a case of earest cetro classfer otherse... R, ANN, & ML 67

68 Lear Mache Eample et Classfcato Use staar F/IDF eghte vectors to represet tet ocumets ormalze b mamum term frequec. For each categor, compute a prototpe vector b summg the vectors of the trag ocumets the categor. Assg test ocumets to the categor th the closest prototpe vector base o cose smlart R, ANN, & ML 68

69 erm Frequec erm frequecterm, ocumet: tft, t: term, : ocumet Ra frequec ft,: # of occurreces Boolea frequec: or 0 Log-scale frequec: log ft,+ Augmete: ajuste for ocumet legth / b ma ra freq of a term ocumet R, ANN, & ML 69

70 Iverse Documet Frequec N: total umber of ocumets corpus umber of ocumets here t appears ealze commo terms corpus R, ANN, & ML 70

71 F/IDF hs s usuall a ver log vector, th keors Each ocumet s escrbe b such a log vector, recorg occurrece of all keors Aga, the scheme s aïve Baesa, correlato amog terms b-grams, trgrams, etc. s gore R, ANN, & ML 7

72 et Categorzato, Roccho rag Assume the set of categores s {c, c, c } For from to let p = <0, 0,,0> t. prototpe vectors For each trag eample <, c> D Let be the frequec ormalze F/IDF term vector for oc Let = j: c j = c sum all the ocumet vectors c to get p Let p = p + 7 7

73 Roccho et Categorzato est Gve test ocumet Let be the F/IDF eghte term vector for Let m = t. mamum cossm For from to : compute smlart to prototpe vector Let s = cossm, p f s > m let m = s let r = c upate most smlar class prototpe Retur class r 73 73

74 Illustrato of Roccho et Categorzato 74 74

75 Roccho ropertes Does ot guaratee a cosstet hpothess. Forms a smple geeralzato of the eamples each class a prototpe. rototpe vector oes ot ee to be average or otherse ormalze for legth sce cose smlart s sestve to vector legth. Classfcato s base o smlart to class prototpes

76 Other More ractcal Classfers Applcable for multple classes Applcable for hgh feature mesos Applcable for classes th multple moes peaks R, ANN, & ML 76

77 o phases hase I trag: collect tagge tpcal samples from all classes, measure a recor ther features the feature space some statstcs mght be compute as ell hase II classfcato: gve a uko sample, classf that base o smlart or oershp the feature space R, ANN, & ML 77

78 Nearest Cetro Classfer s class, f, j,..., j Nee to recor class cetros A sgle cetro -> lear mache moel Multple cetros possble e.g. perform EM o mture of Gaussa, but ho o ou f them f >3? R, ANN, & ML 78

79 Nearest Neghbor Classfer s class, f k,,...,, l, k j, l j,..., m j Do ot ee to recor class cetros No aalss ecessar Multple moes/classes ok Nee to remember all trag ata Computato efforts stace checkg Ho about outlers? Ho about overfttg? R, ANN, & ML 79

80 Geometrc Iterpretato Nearest eghbor classfer performs Voroo partto of the feature space I that sese, t s smlar to assumg that fferet class strbutos have the same pror a varace R, ANN, & ML 80

81 K-Nearest Neghbor k-nn Nearest eghbor ca be susceptble to ose a outlers Ho about use more tha? I.e. assg a sample to the class hch has the most represetatves amog k earest eghbors of the sample Itutvel appealg a folloe from arse Wos & k-nn est estmato A compromse betee earest eghbor too much ata a erratc behavors a earest cetro global est ft R, ANN, & ML 8

82 8 R, ANN, & ML k-nn classfer arse o varat From est estmato to classfer the same prcple labele trag samples Gve a quer sample, f k earest samples from the trag set Collect k total samples for all classes, hchever class has the largest represetato the k samples s / / / /,, /, k k k k V k V k V k V k p p p V k p c j c j j

83 83 R, ANN, & ML k-nn Classfer pool varat We ee at least k samples to mata goo resoluto Assume the umber of samples collecte reflects the pror probablt Collect the same # of samples sa, k, hchever class ees a smaller eghborhoo to o that s / /,, /, V V V V V p V V V p V k V k p p p V k p c j j c j j

84 3 Nearest Neghbor Illustrato Eucla Dstace

85 K Nearest Neghbor for et rag: For each each trag eample <, c> D Compute the correspog F-IDF vector,, for ocumet est stace : Compute F-IDF vector for ocumet For each <, c> D Let s = cossm, Sort eamples,, D b ecreasg value of s Let N be the frst k eamples D. get most smlar eghbors Retur the majort class of eamples N 85 85

86 Illustrato of 3 Nearest Neghbor for et 86 86

87 Roccho Aomol rototpe moels have problems th polmorphc sjuctve categores

88 3 Nearest Neghbor Comparso Nearest Neghbor tes to hale polmorphc categores better

89 89 R, ANN, & ML Ho Goo Are the knn? Ho goo ca t be? Aga, the best case scearo s the oe ctate b Baes rule: assg to the class that most lkel prouces t base o a posteror probablt p e e m m ma * * * Hol Gral

90 Hoever,k s are ot ba ether Surprsgl, earest s ot more tha tce as ba as Baesa a k approaches Baesa for large k R, ANN, & ML 90

91 Itereste the roof? As promse, e o t o proof Istea, e rel o tuto : sample, : earest eghbor to q: sample s class, q : class Q: hat s q? A: arg ma ' m Wth a large umber of samples, t s reasoable to assume that s close to ' ' If m ~ Baes a lkel prouce the same results If m ~ /c Baes a lkel prouce fferet results, but both error rates are -/c R, ANN, & ML 9

92 roof Sketch We are lookg for scearos here a ts earest eghbor belog to fferet classes q a q I fact, e have to look at cases here the umber of trag samples are ver ver large Because epes o the samples use trag a proof ca ot be base o the partcular trag set use epes o samples use, e ll rte as stea R, ANN, & ML 9

93 93 R, ANN, & ML roof Sketch cot. Error s he a are fferet classes Average error caot epe o hch epes o the partcular trag sample set c c e ' ' ' ', ',, q q q q ' ' p e e ', Because all the trag samples a test samples are ra epeetl

94 94 R, ANN, & ML roof Sketch cot. Combe them together, e have c p e ' ' '] [ Whe s large, t s reasoable to epect a are close c c c p e p ' ' '] [ ' ' '] [ lm lm ' ' lm Correct f s Nearest sample s also ca be a class

95 95 R, ANN, & ML roof Sketch cot. he over all possble s p p e e c ] [ lm lm A quck check, f m ~ m m c Hol Gral ] [

96 96 R, ANN, & ML roof Sketch cot. Otherse c m m c, he seco term s mmze f all of the other classes are equall lkel m e m c e * *

97 97 R, ANN, & ML roof Sketch cot. * * * * * * * * * * * * e e c c e c e e e c e e c e e c e e c c : er bou A eve tght * * * c c

98 Other Varatos Dstace eghte: vote s eghe b ho close a trag sample s to the test sample Dmeso eghte: stace s calculate b eghg features uequall Weghts ca be leare b cross-valato R, ANN, & ML 98

99 Aaptve Nearest Neghbors Importat for hgh-mesoal feature space here eghbors are far apart Iea: f local regos a compute feature mesos Where class labels chage a lot arroer focus Where class labels oes t chage a lot er focus R, ANN, & ML 99

100 Aaptve Nearest Neghbors cot. o classes a to features Uform strbuto but label chages ol Etet to capture more features R, ANN, & ML 00

101 Aaptve Nearest Neghbors cot. he same ea as meso reucto Use k to f some eghborg pots frst he recompute the stace measuremets W -/ W -/ spheres the ata th class var Legthe the meso th small ege values B* betee class var R, ANN, & ML 0

102 Local Weghte Regresso k s a local appromato metho thout eplctl bulg the local ecso surface Appromato b eplctl bulg such a surface s possble Dfferece from parametrc techques Local samples are use Weghte b stace Multple local appromatos stea of oe global oe R, ANN, & ML 0

103 03 R, ANN, & ML Eample Assume that locall the ecso surface s a lear fucto of the attrbutes a, ˆ ˆ q k o K f f E a a a f q K s a ocreasg fucto, e.g.,,, q q q e k

104 Learg Rule Startg from a arbtrar set of eghts If f true a f^hat estmate are the same, o chage Otherse, chage fˆ o a k f q a a fˆ K q, a :learg rate R, ANN, & ML 04

105 Learg Rule cot. We ll see later that ths rule s the perceptro learg rule use perceptro learg ANN he locall eghte appromato s ver smlar to the raal bass fucto learg ANN R, ANN, & ML 05

CS 1675 Introduction to Machine Learning Lecture 12 Support vector machines

CS 1675 Introduction to Machine Learning Lecture 12 Support vector machines CS 675 Itroducto to Mache Learg Lecture Support vector maches Mlos Hauskrecht mlos@cs.ptt.edu 539 Seott Square Mdterm eam October 9, 7 I-class eam Closed book Stud materal: Lecture otes Correspodg chapters