CS9840 Learning and Computer Vision Prof. Olga Veksler. Lecture 2. Some Concepts from Computer Vision Curse of Dimensionality PCA

Transcription

1 CS9840 Learning an Computer Vision Prof. Olga Veksler Lecture Some Concepts from Computer Vision Curse of Dimensionality PCA Some Slies are from Cornelia, Fermüller, Mubarak Shah, Gary Braski, Sebastian Thrun Outline Some Concepts in Image Processing/Vision Optical Flow Fiel (relate to motion fiel) Correlation Curse of Dimensionality an Dimensionality reuction with PCA Net time: Recognizing Action at a Distance by A. Efros, A.Berg, G. Mori, Jitenra Malik Also: "80 million tiny images: a large ataset for non-parametric object an scene recognition", A. Torralba, R. Fergus, W. Freeman there shoul be a link to PDF file on our web site

2 Optical flow first image I secon image I How to estimate piel motion from image I to image I? Solve piel corresponence problem given a piel in I, look for nearby piels of the same color in I Key assumptions color constancy: a point in I looks the same in I For grayscale images, this is brightness constancy small motion: points o not move very far This is calle the optical flow problem Optical Flow Fiel

3 Optical Flow an Motion Fiel Optical flow fiel is the apparent motion of brightness patterns between (or several) frames in an image sequence Why oes brightness change between frames? Assuming that illumination oes not change: changes are ue to the RELATIVE MOTION between the scene an the camera There are 3 possibilities: Camera still, moving scene Moving camera, still scene Moving camera, moving scene Motion Fiel (MF) The MF assigns a velocity vector to each piel in the image These velocities are INDUCED by the RELATIVE MOTION between the camera an the 3D scene The MF is the projection of the 3D velocities on the image plane 3

4 Eamples of Motion Fiels (a) (b) (c) (a) Translation perpenicular to a surface. (b) Rotation about ais perpenicular to image plane. (c) Translation parallel to a surface at a constant istance. () Translation parallel to an obstacle in front of a more istant backgroun. () Optical Flow vs. Motion Fiel Recall that Optical Flow is the apparent motion of brightness patterns We equate Optical Flow Fiel with Motion Fiel Frequently works, but now always: (a) (b) (a) (b) A smooth sphere is rotating uner constant illumination. Thus the optical flow fiel is zero, but the motion fiel is not A fie sphere is illuminate by a moving source the shaing of the image changes. Thus the motion fiel is zero, but the optical flow fiel is not 4

5 Optical Flow vs. Motion Fiel Often (but not always) optical flow correspons to the true motion of the scene Human Motion System Illusory Snakes from Gary Braski an Sebastian Thrun 5

6 Computing Optical Flow: Brightness Constancy Equation Let P be a moving point in 3D: At time t, P has coorinates (X(t),Y(t),Z(t)) Let p=((t),y(t)) be the coorinates of its image at time t Let E((t),y(t),t) be the brightness at p at time t. Brightness Constancy Assumption: As P moves over time, E((t),y(t),t) remains constant Computing Optical Flow: Brightness Constancy Equation Taking erivative wrt time: 6

7 Computing Optical Flow: Brightness Constancy Equation equation with unknowns Let (Frame spatial graient) (optical flow) an (erivative across frames) Computing Optical Flow: Brightness Constancy Equation How to get more equations for a piel? Basic iea: impose aitional constraints most common is to assume that the flow fiel is smooth locally one metho: preten the piel s neighbors have the same (u,v) If we use a 55 winow, that gives us 5 equations per piel! E E E E t p Ep u v 0 i p Ey p p E p p E p 5 y y matri E 5 5 i u v vector Et p Et p E t p5 vector b 5 7

8 Vieo Sequence * * Picture from Khurram Hassan-Shafique CAP545 Computer Vision 003 Optical Flow Results * From Khurram Hassan-Shafique CAP545 Computer Vision 003 8

9 Revisiting the small motion assumption Is this motion small enough? Probably not it s much larger than one piel ( n orer terms ominate) How might we solve this problem? Reuce the resolution! 9

10 Coarse-to-fine optical flow estimation u=.5 piels u=.5 piels u=5 piels image H u=0 piels image I Gaussian pyrami of image H Gaussian pyrami of image I Iterative Refinement Iterative Lukas-Kanae Algorithm. Estimate velocity at each piel by solving Lucas- Kanae equations. Warp H towars I using the estimate flow fiel - use image warping techniques 3. Repeat until convergence 0

11 Coarse-to-fine optical flow estimation run iterative L-K run iterative L-K warp & upsample... image HJ Gaussian pyrami of image H image I Gaussian pyrami of image I Optical Flow Results * From Khurram Hassan-Shafique CAP545 Computer Vision 003

12 Other Concepts to Review Convolution is the operation of applying a kernel to each piel of an image image kernel Result of convolution has the same imension as the image For eample: Convolution is frequently enote by *, for eample I*K Other Concepts to Review Gaussian smoothing (blurring): convolution operator that is use to `blur' images an removes small etail an noise from an image *

13 Gaussian Smoothing vs. Averaging Gaussian Smoothing Smoothing by Averaging 5 Other Concepts to Review Image graient: points in the irection of the most rapi increase in intensity of image f Sobel operator to compute graient: f f y Results: 3

14 Other Concepts to Review Cross-correlation c f,g f i gi measures similarity between images (or image regions) f an g works OK if there is no change in intensity NCC f,g f i f g i i f i f g i i k g g i Normalize cross correlation, more popular in image processing Insensitive to linear intensity changes between image patches f an g / large similarity small similarity Curse of Dimensionality Problems of high imensional ata, the curse of imensionality running time overfitting number of samples require Dimensionality Reuction Methos Principle Component Analysis 4

15 Curse of Dimensionality: Compleity Compleity (running time) increases with imension A lot of methos have at least O(n ) compleity, where n is the number of samples For eample if we nee to estimate covariance matri So as becomes large, O(n ) compleity may be too costly Curse of Dimensionality: Number of Samples Suppose we want to use the nearest neighbor approach with k = (NN) Suppose we start with only one feature 0 This feature is not iscriminative, i.e. it oes not separate the classes well We ecie to use features. For the NN metho to work well, nee a lot of samples, i.e. samples have to be ense To maintain the same ensity as in D (9 samples per unit length), how many samples o we nee? 5

16 Curse of Dimensionality: Number of Samples We nee 9 samples to maintain the same ensity as in D 0 Curse of Dimensionality: Number of Samples Of course, when we go from feature to, no one gives us more samples, we still have 9 0 This is way too sparse for NN to work well 6

17 Curse of Dimensionality: Number of Samples Things go from ba to worse if we ecie to use 3 features: 0 If 9 was ense enough in D, in 3D we nee 9 3 =79 samples! Curse of Dimensionality: Number of Samples In general, if n samples is ense enough in D Then in imensions we nee n samples! An n grows really really fast as a function of Common pitfall: If we can t solve a problem with a few features, aing more features seems like a goo iea However the number of samples usually stays the same The metho with more features is likely to perform worse instea of epecte better 7

18 The Curse of Dimensionality We shoul try to avoi creating lot of features Often no choice, problem starts with many features Eample: Face Detection One sample point is k by m array of piels Feature etraction is not trivial, usually every piel is taken as a feature Typical imension is 0 by 0 = 400 Suppose 0 samples are ense enough for imension. Nee only samples The Curse of Dimensionality Face Detection, imension of one sample point is km The fact that we set up the problem with km imensions (features) oes not mean it is really a km-imensional problem Space of all k by m images has km imensions Space of all k by m faces must be much smaller, since faces form a tiny fraction of all possible images Most likely we are not setting the problem up with the right features If we use better features, we are likely nee much less than km-imensions 8

19 9 Dimensionality Reuction High imensionality is challenging an reunant It is natural to try to reuce imensionality Reuce imensionality by feature combination: combine ol features to create new features y y k with y y y f k For eample, Ieally, the new vector y shoul retain from all information important for classification Dimensionality Reuction The best f() is most likely a non-linear function Linear functions are easier to fin though k with y y w w w w W k k k Thus it can be represente by a matri W: For now, assume that f() is a linear mapping

20 Principle Component Analysis (PCA) Main iea: seek most accurate ata representation in a lower imensional space Eample in -D Project ata to -D subspace (a line) which minimize the projection error imension imension large projection errors, ba line to project to imension imension small projection errors, goo line to project to Notice that the the goo line to use for projection lies in the irection of largest variance PCA After the ata is projecte on the best line, nee to transform the coorinate system to get D representation for vector y y Note that new ata y has the same variance as ol ata in the irection of the green line PCA preserves largest variances in the ata 0

21 PCA: Approimation of Elliptical Clou in 3D best D approimation best D approimation PCA What is the irection of largest variance in ata? Recall that if has multivariate istribution N(,), irection of largest variance is given by eigenvector corresponing to the largest eigenvalue of This is a hint that we shoul be looking at the covariance matri of the ata (note that PCA can be applie to istributions other than Gaussian)

22 PCA: Linear Algebra Review Let V be a imensional linear space, an W be a k imensional linear subspace of V We can always fin a set of imensional vectors {e,e,,e k } which forms an orthonormal basis for W <e i,e j > = 0 if i is not equal to j an <e i,e i > = Thus any vector in W can be written as k e e... kek iei for scalars i,..., k W Let V = R an W be the line -y=0. Then the orthonormal basis for W is / / 5 5 PCA: Linear Algebra Recall that subspace W contains the zero vector, i.e. it goes through the origin this line is not a subspace of R It is convenient to project to subspace W: thus we nee to shift everything this line is a subspace of R

23 PCA Derivation: Shift by the Mean Vector Before PCA, subtract sample mean from the ata n i ˆ n i The new ata has zero mean: E(X-E(X)) = E(X)-E(X) = 0 All we i is change the coorinate system ˆ ˆ Another way to look at it: first step of getting y is to subtract the mean of y f g ˆ PCA: Derivation We want to fin the most accurate representation of ata D={,,, n } in some subspace W which has imension k < Let {e,e,,e k } be the orthonormal basis for W. Any k vector in W can be written as Thus i i e i will be represente by some vector in W k e i i i Error this representation: error k i ie i error i e i W 3

24 PCA: Derivation To fin the total error, we nee to sum over all j s Any j can be written as Thus the total error for representation of all ata D is: sum over all ata points J k i ji e i e ek,,... nk j,..., e unknowns n j k i error at one point ji i PCA: Derivation A lot of math.to finally get: Let S be the scatter matri, it is just n- times the sample covariance matri ˆ n j n j ˆ j ˆ t To minimize J take for the basis of W the k eigenvectors of S corresponing to the k largest eigenvalues 4

25 PCA The larger the eigenvalue of S, the larger is the variance in the irection of corresponing eigenvector This result is eactly what we epecte: project into subspace of imension k which has the largest variance This is very intuitive: restrict attention to irections where the scatter is the greatest PCA Thus PCA can be thought of as fining new orthogonal basis by rotating the ol ais until the irections of maimum variance are foun 5

26 PCA as Data Approimation Let {e,e,,e } be all eigenvectors of the scatter matri S, sorte in orer of ecreasing corresponing eigenvalue Without any approimation, for any sample i : error of approimation i j e j j e e k k k e k... e approimation of i coefficients m = t ie m are calle principle components The larger k, the better is the approimation Components are arrange in orer of importance, more important components come first Thus PCA takes the first k most important components of i as an approimation to i PCA: Last Step Now we know how to project the ata Last step is to change the coorinates to get final k-imensional vector y y Let matri E e Then the coorinate transformation is e k Uner E t, the eigenvectors become the stanar basis: t E e i t y E e e i ei ek 0 0 6

27 Recipe for Dimension Reuction with PCA Data D={,,, n }. Each i is a -imensional vector. Wish to use PCA to reuce imension to k n. Fin the sample mean ˆ i n i. Subtract sample mean from the ata z i ˆ t 3. Compute the scatter matri S z i z i 4. Compute eigenvectors e,e,,e k corresponing to the k largest eigenvalues of S 5. Let e,e,,e k be the columns of matri E e 6. The esire y which is the closest approimation t to is y E z n i i e k PCA Eample Using Matlab Let D = {(,),(,3),(3,),(4,4),(5,4),(6,7),(7,6),(9,7)} Convenient to arrange ata in array X Mean meanx Subtract mean from ata to get new ata array Z Z X X repmat,8, Compute the scatter matri S S 7 cov Z matlab uses unbiase estimate for covariance, so S=(n-)*cov(Z) 7

28 PCA Eample Using Matlab Use [V,D] =eig(s) to get eigenvalues an eigenvectors of S 87 an e an e Projection to D space in the irection of e t t Y e Z y 5. y 8 Drawbacks of PCA PCA was esigne for accurate ata representation, not for ata classification Preserves as much variance in ata as possible If irections of maimum variance is important for classification, will work However the irections of maimum variance may be useless for classification apply PCA to each class 8

29 Net Time Paper: Recognizing Action at a Distance by A. Efros, A.Berg, G. Mori, Jitenra Malik will watch the conference presentation Also: "80 million tiny images: a large ataset for nonparametric object an scene recognition", A. Torralba, R. Fergus, W. Freeman When reaing papers, think about following: What is the problem paper tries to solve What makes this problem ifficult? What is the metho use in the paper to solve the problem What is the contribution of the paper (what new oes it o)? Do the eperimental results look goo to you? 9