Subspace Methods for Visual Learning and Recognition

Transcription

1 This is a shortened version of the tutorial given at the ECCV 2002, Copenhagen, and ICPR 2002, Quebec City. Copyright 2002 by Aleš Leonardis, University of Ljubljana, and Horst Bischof, Graz University of Technology Subspace Methods for Visual Learning and Recognition Aleš Leonardis Faculty of Computer and Information Science University of Ljubljana Slovenia Ales.Leonardis@fri.uni-lj.si

2 Outline Part 1 Motivation Appearance based learning and recognition Subspace methods for visual object recognition Principal Components Analysis (PCA) Linear Discriminant Analysis (LDA) Canonical Correlation Analysis (CCA) Independent Component Analysis (ICA) Non-negative Matrix Factorization (NMF) Kernel methods for non-linear subspaces 2

3 Outline Part 2 Robot localization Robust representations and recognition Robust PCA recognition Scale invariant recognition using PCA Illumination insensitive recognition Representations for panoramic images Incremental building of eigenspaces Multiple eigenspaces for efficient representation Robust building of eigenspaces Research issues 3

4 Learning and recognition 3D reconstruction matching learning matching scene training images input image 4

5 Appearance-based approaches Attention in the appearance-based approaches Encompass combined effects of: shape, reflectance properties, pose in the scene, illumination conditions. Acquired through an automatic learning phase. 5

6 Appearance-based approaches Objects are represented by a large number of views: 6

7 Subspace Methods Images are represented as points in the N-dimensional vector space Set of images populate only a small fraction of the space Characterize subspace spanned by images Image set Basis images Representation 7

8 Subspace Methods Properties of the representation: Optimal Reconstruction PCA Optimal Separation LDA Optimal Correlation CCA Independent Factors ICA Non-negative Factors NMF Non-linear Extension Kernel Methods 8

9 9 Eigenspace representation Image set (normalised, zero-mean) We are looking for orthonormal basis functions: Individual image is a linear combination of basis functions ) ( ) ( )) ( ) ( ( ) ( ) ( y x u y x u y u x y x j j k j j j j k j j j k j j j q q q q q q = = = = =

10 Optimisation problem Best basis functions n? Taking the k eigenvectors with the largest eigenvalues of PCA or Karhunen-Loéve Transform (KLT) 10

11 n << m Efficient eigenspace computation Compute the eigenvectors u' i, i = 0,...,n-1, of the inner product matrix The eigenvectors of XX T can be obtained by using XX T Xv i '=l' i Xv i ': 11

12 Principal Component Analysis 12

13 Principal Component Analysis = q 1 + q 2 + q

14 Image representation with PCA u 1 u 2 u 3 14

15 Image presentation with PCA 15

16 Properties PCA It can be shown that the mean square error between x i and its reconstruction using only m principle eigenvectors is given by the expression : j= 1 j= 1 PCA minimizes reconstruction error N λ j m λ j = N λ j= m+ 1 j PCA maximizes variance of projection Finds a more natural coordinate system for the sample data.

17 PCA for visual recognition and pose estimation Objects are represented as coordinates in an n-dimensional eigenspace. An example: 3-D space with points representing individual objects or a manifold representing parametric eigenspace (e.g., orientation, pose, illumination). u 1 u 0 u 2 17

18 PCA for visual recognition and pose estimation Calculate coefficients Search for the nearest point (individual or on the curve) Point determines object and/or pose 18

19 Calculation of coefficients To recover q i the image is projected onto the eigenspace q ( x) i =< x,u i >= n 1 j= 1 x j u ij 1 i k < > = q 1 < > + q 2 < > +... =q 1 < > = q 1 < > + q 2 < > +... =q 2 Complete image x is required to calculate q i. Corresponds to Least-Squares Solution 19

20 Linear Discriminant Analysis (LDA) PCA minimizes projection error Best discriminating Projection PCA-Projection PCA is unsupervised no information on classes is used Discriminating information might be lost 20

21 LDA Linear Discriminance Analysis (LDA) Maximize distance between classes Minimize distance within a class Fisher Linear Discriminance ρ( w) = w w T T S S B W w w 21

22 LDA: Problem formulation n Sample images: c classes: { x,, } 1 Λ x n { c,, } 1 Λ c c Average of each class: Total average: µ i µ = = 1 n i 1 x k χ N n k= 1 i x k x k 22

23 LDA: Practice Scatter of class i: ( )( ) T S i = x k c i x k - m i x k - m i Within class scatter: c S W = S i i= 1 Between class scatter: S B = c i= 1 c i ( m - m )( m - m ) i i T Total scatter: S = S + T W S B 23

24 LDA S 1 S B S W = S 1 + S 2 S 2 Good separation 24

25 Maximization of ρ( w) = w w T T S S B W w w LDA is given by solution of generalized eigenvalue problem S B w = λs w w For the c-class case we obtain (at most) c-1 projections as the largest eigenvalues of S B w = λs i ww i 25

26 LDA Example Fisherface of recognition Glasses/NoGlasses (Belhumeur et.al. 1997) 26

27 Canonical Correlation Analysis (CCA) Also supervised method but motivated by regression tasks, e.g. pose estimation. Canonical Correlation Analysis relates two sets of observations by determining pairs of directions that yield maximum correlation between these sets. Find a pair of directions (canonical factors) w x R p, w y R q, so that the correlation of the projections c = w xt x and d = w yt y becomes maximal. 27

28 What is CCA? Canonical Correlation 0 r 1 Between Set Covariance Matrix ρ = E[ w w T x C E[ c T x w xx E[ cd] E[ w xx T x w 2 T C x ] E[ d xy T x w w x w xy T y y T ] E[ w C 2 ] yy w = y w T y ] y yy T w y ] = 28

29 What is CCA? Finding solutions w = w w x y A = 0 C yx C 0 xy, B = C 0 xx C 0 yy Rayleigh Quotient Generalized Eigenproblem r = w w T T Aw Bw Aw = µbw 29

30 CCA Example Parametric eigenspace obtained by PCA for 2DoF in pose

31 CCA Example CCA representation (projections of training images onto w x1, w x2 )

32 Independent Component Analysis (ICA) ICA is a powerful technique from signal processing (Blind Source Separation) Can be seen as an extension of PCA PCA takes into account only statistics up to 2 nd order ICA finds components that are statistically independent (or as independent as possible) 33

33 Independent Component Analysis (ICA) m scalar variables X=(x 1... x m ) T They are assumed to be obtained as linear mixtures of n sources S=(s 1... s n ) T X = AS Task: Given X find A, S (under the assumption that S are independent) 34

34 ICA Example Original Sources Mixtures Recovered Sources 35

35 ICA Example ICA basis obtained from 16x16 patches of natural images (Bell&Sejnowski 96) 36

36 Face Recognition using ICA PCA vs. ICA on Ferret DB (Baek et.al. 02) PCA ICA 37

37 Non-Negative Matrix Factorization (NMF) How can we obtain part-based representation? Local representation where parts are added E.g. learn from a set of faces the parts a face consists of, i.e. eyes, nose, mouth, etc. Non-Negative Matrix Factorization (Lee & Seung 1999) lead to part based representation 38

38 Matrix Factorization - Constraints V» WH PCA: W are orthonormal basis vectors W = [ w, w2, 1 Λ, w n ], w i w j = δ ij VQ : H are unity vectors H = [ 2 h1, h, Λ, hn], h j = [0,0,1,0, Λ T,0] NMF: V,W,H are non-negative Vij, Wij, Hij 0 i, j 39

39 Learning Find basis images from the training set Training images Basis images 41

40 Face features Basis images Encoding (Coefficients) Reconstructed image 42

41 Kernel Methods All presented methods are linear Can we generalize to non-linear methods in a computational efficient manner? 43

42 Kernel Methods Kernel Methods are powerful methods (introduced with Support Vector Machines) to generalize linear methods BASIC IDEA: 1. Non-linear mapping of data in high dimensional space 2. Perform linear method in high-dimensional space Non-linear method in original space 44

43 Outline Part 2 Robot localization Robust representations and recognition Robust recognition using PCA Scale invariant recognition using PCA Illumination insensitive recognition Representations for panoramic images Incremental building of eigenspaces Multiple eigenspaces for efficient representation Robust building of eigenspaces Research issues 45

44 A variety of successful applications: Appearance-based approaches Human face recognition e.g. [Turk & Pentland] Visual inspection e.g. [Yoshimura & Kanade] Visual positioning and tracking of robot manipulators, e.g. [Nayar & Murase] Tracking e.g., [Black & Jepson] Illumination planning e.g., [Murase & Nayar] Image spotting e.g., [Murase & Nayar] Mobile robot localization e.g., [Jogan & Leonardis] Background modeling e.g., [Oliver, Rosario & Pentland] 46

45 Mobile Robot 47

46 Panoramic image 48

47 Environment map environments are represented by a large number of views localisation = recognition 49

48 Compression with PCA 50

49 Image representation with PCA 51

50 Localisation 52

51 Distance vs. similarity 53

52 Robot localisation Interpolated hyper-surface represents the memorized environment. The parameters to be retrieved are related to position and orientation. Parameters of an input image are obtained by scalar product. 54

53 Localisation 55

54 Enhancing recognition and representations Occlusions, varying background, outliers Robust recognition using PCA Scale variance Multiresolution coefficient estimation Scale invariant recognition using PCA Illumination variations Illumination insensitive recognition Rotated panoramic images Spinning eigenimages Incremental building of eigenspaces Multiple eigenspaces for efficient representations Robust building of eigenspaces 56

55 Occlusions 57

56 Calculation of coefficients To recover q i the image is projected onto the eigenspace q ( x) i =< x,u i >= n 1 j= 1 x j u ij 1 i k < > = q 1 < > + q 2 < > +... =q 1 < > = q 1 < > + q 2 < > +... =q 2 Complete image x is required to calculate q i. Corresponds to Least-Squares Solution 58

57 Non-robustness 59

58 Robust method 60

59 Robust algorithm 61

60 Selection 62

61 Robust recovery of coefficients 63

62 Robust localisation under occlusions 66

63 Robust localisation at 60% occlusion Standard approach Robust approach 67

64 Mean error of localisation Mean error of localisation with respect to % of occlusion 68

65 Illumination insensitive recognition Recognition of objects under varying illumination global illumination changes highlights shadows Dramatic effects of illumination on objects appearance Training set under a single ambient illumination 69

66 Illumination insensitive recognition Our Approach Global eigenspace representation Local gradient based filters Efficient combination of global and local representations Robust coefficient recovery in eigenspaces 70

67 Eigenspaces and filtering 71

68 Filtered eigenspaces 72

69 Gradient-based filters Global illumination Gradient-based filters Steerable filters [Simoncelli] 73

70 Robust coefficient recovery Highlights and shadows Robust coefficient recovery Robust solution of linear equations = a + a + a +Λ = a + a + a +Λ = a + a + a +Λ Μ Μ a + a + + Λ = a Hypothesize & Select 74

71 Experimental results Test images Our approach Standard method Demo 75

72 Experimental results Robust filtered method - all eigenvectors used obj % ang avg Standard method - all eigenvectors used obj % ang avg

73 Research issues Comparative studies (e.g., LDA versus PCA, PCA versus ICA) Robust learning of other representations (e.g. LDA, CCA) Integration of robust learning with modular eigenspaces Local versus Global subspace represenations Combination of subspace representations in a hierarchical framework 77

74 Further readings Recognizing objects by their appearance using eigenimages (SOFSEM 2000, LNCS 1963) Robust recognition using eigenimages (CVIU 2000, Special Issue on Robust Methods in CV) Illumination insensitive eigenspaces (ICCV 2001) Mobile robot localization under varying illumination (ICPR 2002) Eigenspace of spinning images (OMNI 2000, ICPR 2000, ICAR 2001) Incremental building of eigenspaces (ICRA 2002, ICPR 2002, CogVis 2002) Multiple eigenspaces (Pattern Recognition 2002) Robust building of eigenspaces (ECCV 2002) Special issue of Pattern Recognition on Kernel and Subspace Methods in Computer Vision (Guest Editors A. Leonardis and H. Bischof), to appear in