Image Processing 1 (IP1) Bildverarbeitung 1

Transcription

1 MIN-Fakultät Fachbereich Infrmatik Arbeitsbereich SAV/BV (KOGS) Image Prcessing 1 (IP1) Bildverarbeitung 1 Lecture 15 Pa;ern Recgni=n Winter Semester 2014/15 Dr. Benjamin Seppke Prf. Siegfried S=ehl

2 What is "Pa*ern Recgni2n"? The term "Pa;ern Recgni=n" ("Mustererkennung") is used fr Methds fr classifying unknwn bjects based n feature vectrs (narrw sense meaning f Pa*ern Recgni2n) Methds r analyzing signals and recgnizing interes2ng pa*erns (wide sense meaning f Pa*ern Recgni2n) Pa;ern recgni=n can be applied t all kinds f signals, e.g. images acus=c signals seismgraphic signals tmgraphic data etc. The fllwing sec=n deals with Pa;ern Recgni=n in the narrw sense. (see Duda and Hart, Pa;ern Classifica=n and Scene Analysis, Wiley 73) 2

3 Intrductry Eample: Where is Wally? (c) Martin Handfrd, Walker Bks 3

4 (c) Martin Handfrd, Walker Bks 4

5 Basic Terminlgy fr Pa*ern Recgni2n bject feature etrac2n feature vectr classifica2n in feature space bject class K N T = 1... N y T = y 1 y 2... y N (k) y i M k g k ( ) classes ω 1... ω K dimensin f feature space ( ) ( ) feature vectr i- th prttyp f class k Prblem: Determine prttype (feature vectr with knwn class membership) number f prttypes fr class k discriminant func=n fr class k ω k k j g k g k ( ) ( ) > g j ( ) such that 5

6 Eample: Animal Ftprints Bear Hare Wlf What features can be used t dis2nguish the 3 ftprint classes? 6

7 w h A Feature Space fr Ftprints ω 1 = wlf ω 2 = bear c = circumference = 2(w+h) a = area = w h pa = print area = card({ g(,y) > 0}) 1 = "squareness" = = "slidness" = a c 2 pa c 1 ω 3 = hare * * * * * ω 3 ω 2 ω

8 Discriminant Func2ns fr Ftprints 1 1 * * * * ω * 3 ω 1 ω 2 Quadra2c discriminant func2ns: g 1 = g 2 = g 3 = * * * * ω * 3 ω 1 ω = 0 Piecewise linear discriminant func2ns: g 1 = 1 if ( > 0) ( < 0) else 0 g 2 = 1 if ( > 0) ( > 0) else 0 g 3 = 1 if ( < 0) ( < 0) else

9 Linear Discriminant Func2ns Linear discriminant func2ns are a*rac2ve because they can be easily determined frm prttypes easily analyzed easily evaluated Basic frm f linear discriminant func=n: g k ( ) = ( w k ) T + w k0 g k Fr N=2 the discriminant func=n is a 3D plane bundary line: g k ( ) = ( w k ) T + w k0 = 0 1 9

10 Class Average Minimal Distance Classifica2n Represent prttypes by class averages Assign bject t class with minimum distance between bject and class average Fr a 2- class prblem, the minimal distance criterin always results in a linear discriminant func2n 1 Class average minimal distance classifica2n may nt separate prttypes even if they are linearly separable 10

11 Nearest Neighbur Classifica2n Assign bject t class with nearest prttype Piece- wise linear discriminant func2n 1 The nearest neighbur criterin classifies all prttypes crrectly (ecept equal prttypes f different classes). The decisin regins are nt necessarily cherent. 11

12 Generalized Linear Discriminant Func2ns 1 Eample: Prttypes are nt linearly separable A quadra=c discriminant func=n may wrk: g k ( ) = a a b 11 ( 1 ) 2 + b 22 ( ) 2 + b c ( ) with T = 1 Transfrma2n f prttypes int higher- dimensinal feature space may allw linear discriminant func2ns. Transfrma=n fr the eample: Linear discriminant func=n in z- space: ( ) 2, z 4 = ( ) 2, z 5 = 1 ( ) = a 1 z 1 + a 2 z 2 + a 3 z 3 + a 4 z 4 + a 5 z 5 + c z 1 = 1, z 2 =, z 3 = 1 z g k Advantage: Linear separa=n algrithms may be applied Disadvantage: Dimensinality f feature space is dras=cally increased 12

13 Linear Discriminant Func2ns fr 2- Class Prblems Nrmalize prttypes such that y T = 1 y 1 y 2... y N Discriminant func=n g can be epressed as ( ) = a T with a T = ( a 0 a 1... a N ) g Prttypes f class ω 2 are negated such that à crrect classifica=n f bth classes a T y > 0 ( ) Slu=n regin in weight space (if it eists) is the space at the psi=ve side f all hyperplanes a T y = 0. Any weight vectr a in this slu=n regin gives a crrect discriminant func=n. Pssible further cnstraints n slu=n vectr a: a =1 y a T y > b weight space b is "margin", i.e. minimal distance f a crrectly classified pint frm the hyperplanes defined by the prttypes. y (1) a 1 slu=n regin fr weight vectrs y (2) a 0 13

14 Perceptrn Learning Rule A slu=n vectr a can be determined itera=vely by minimizing a criterin func=n J( a) by gradient descent. a 1 Perceptrn criterin func2n: J p ( a) = a ( T y ) with B = {all misclassified prttypes} Basic gradient descent algrithm: Gradient: Step: Weight vectr y B gradient direc=n ( ) J p ( a) = y y B a k+1 = a k + ρ k a is mdified in nega=ve Eample (see illustra2n) with: y 1 = 1 2 y ( ) y B ( ) T, y 2 = ( 1 1) T, ρ = 2 y (1) y (2) itera=ns viewed in weight space k a k a 0 slu=n

15 Minimizing the Discriminant Criterin General frm f gradient descent: a k+1 = a k ρ k J( a k ) with J( a $ k ) T = & J % a 0 One can determine the p=mal ρ k which achieves the minimal J( a k+1 ) at the kth step by apprima=ng J( a) with a secnd- rder Taylr series epansin: where D( a k ) is the matri f secnd deriva=ves 2 J evaluated at. Using the itera=n rule: The minimizing ρ k is: J... J a 1 a N J( a k ) J( a k+1 )+ T J( a k )( a a k )+ 1 2 ( a a k ) T D( a k )( a a k ) J( a k+1 ) J( a k ) ρ k T J( a k ) ρ k Newtn s algrithm is an alterna=ve: Chse a k+1 which minimizes J( a) in the Taylr series apprima=n. a k+1 = a k D 1 J ( a k ) ρ k = a i a j ' ) ( ( ) 2 J( a k ) T D( a k ) J( a k ) T J( a k ) 2 J( a k ) T D( a k ) J( a k ) ak 15

16 Prblems: Quadra2c Criterin Func2n Quadra2c criterin func2n: J q ( a ) = a T y slw cnvergence clse t bundaries dminated by lng sample vectrs y Nrmalized quadra2c criterin func2n: J r Gradient: ( ) 2 y B a ( ) = 1 2 J r y B a ( ) = a T y b ( ) 2 y B y 2 a T y b y 2 y with B = {all samples where a T y 0 } a T y 0 with B = {all samples where } a T y < b Itera=n rule: a k+1 = a k + ρ k y B b a T y y 2 y 16

17 Relaa2n Rule If crrec=ns based n the nrmalized quadra=c criterin are perfrmed fr each single sample, ne gets the "relaa=n rule": a k+1 = a k + ρ b a T k y (k) y (k) T where ak y (k) < b a k y (k) 2 T a k y (k) = b Distance frm t hyperplane is: b a k T y (k) y (k) 2 Fr ρ = 1, the itera=n rule calls fr mving directly t the hyperplane T à "relaa=n" f tensin in inequality a k y (k) < b Typical values: 0 < ρ < 2 ρ < 1 "underrelaa=n" a k k ρ > 1 "verrelaa=n a 0 y (k) a 1 a k a k T y (k) = b 17

18 Minimum Squared Errr New criterin func=n fr all samples: Find a such that a T yi = b i with b i = sme psi=ve cnstant T $ y 1 y $ In matri nta=n: Y a = # & # i1 & # T y & b with Y = 2 # y # & and y i = i2 & # & # " & # " & # T " y & # M y & % " inn % In general, M >> N and Y -1 des nt eist, hence a = Y 1 b is n slu=n. Classical slu=n technique: Minimize squared errr criterin: J s a T yi b i a ( ) = Y a b 2 = Clsed- frm slu=n by sejng the gradient equal t 0. ( ) 2 J s a ( ) = 2Y T Y a b ( ) = 0 a = ( Y T Y ) 1 Y T b if (Y T Y) -1 Y T is nnsingular pseudinverse f Y 18

19 H- Kashyap Prcedure The MSE slu=n a = ( Y T Y ) 1 Y T b des nt necessarily prvide a separa=ng hyperplane if the classes are linearly separable, because b is chsen arbitrarily. H- Kashyap algrithm searches fr a and b such that Y a = b > 0 by minimizing J s w.r.t. a and b: 1. Iterate ver a by chsing a k = ( Y T Y ) 1 Y T b k 2. Iterate ver b by chsing b 1 > 0 : b k+1 = b k + 2ρ + e k 0 < ρ < 1 with e k = Y a k b k errr vectr e + k = 1 2 e k + ( e k ) psi=ve part f e k H- Kashyap itera=n ver b generates sequence f margin vectrs bwhich - minimizes squared errr criterin - gives nly psi=ve margins b > 0 Fr linearly separable classes and 0 < ρ < 1, the H- Kashyap algrithm will cnverge in a finite number f steps. 19

20 Discrimina2n with Pten2al Func2ns Idea: Electrsta=c pten=al centered at each prttype may sum up t a useful discriminant func=n Eample: pten=al func=n K, i discriminant func=n g ( ) = 1 2 i ( ) = q i K (, i ) i "charges" q i may be adjusted in learning prcedure 20

21 Cnstruc2n f Discriminant Func2ns Based n Pten2al Func2ns Different chices fr pten2al func2ns are pssible, fr eample: K(, k ) = σ 2 σ 2 + k 2 K(, k ) = e 1 2σ 2 k 2 Pten2al func2ns must be tuned t prvide the right kind f interpla2n between samples Itera2ve cnstruc2n: g ( ) = % ' & ' (' g( )+ K(, k ) if k is f class 1 and g( k ) 0 g( ) K(, k ) if k is f class 2 and g( k ) 0 g( ) therwise 21