ECE 68: Digital Image Processing Lecture 8 Prof. Sinisa Todorovic sinisa@eecs.oregonstate.edu 1
Point Descriptors
Point Descriptors Describe image properties in the neighborhood of a keypoint Descriptors = Vectors that are ideally affine invariant Popular descriptors: Scale invariant feature transform (SIFT) Steerable filters Shape context and geometric blur Gradient location and orientation histogram (GLOH) Histogram of Oriented Gradients (HOGs)
SIFT The key idea: a point can be described by a distribution of intensity gradients in the neighborhood of that point
SIFT Descriptor 18-D vector = (x blocks) x (8 bins of histogram) gradients of a 16x16 patch centered at the point histogram of gradients at certain angles of a x subpatch The figure illustrates only 8x8 pixel neighborhood that is transformed into x blocks, for visibility
MATLAB Code for SIFT 6
Matching Cost of Two Descriptors Euclidean distance: (d 1, d )=kd 1 d k Chi-squared distance: (d 1, d )= X i (d 1i d i ) d 1i + d i 7
Matching Formulation Given two sets of descriptors to be matched V = {d 1, d,...,d N }, and V 0 = {d 0 1, d 0,...,d 0 M } Find the legal mapping f F f := {(d, d 0 ): d V, d 0 V 0 } Which minimizes the total cost of matching ˆf =min ff X (d,d 0 )f (d, d 0 ), (d, d 0 ) 0 8
Total Cost of Matching A = cost matrix 1 0 0 0... M 110 1 0 1 0... 1M... N M X (d,d 0 ) (d, d 0 )=tr(a T 1) matrix of all ones 9
Total Cost of Matching A = cost matrix 1 0 0 0... M 110 1 0 1 0... 1M... N M X (d,d 0 ) (d, d 0 )=tr(a T 1) matrix of all ones X (d,d 0 )f (d, d 0 )=? 10
Linearization X (d,d 0 )f (d, d 0 X ) = (d, d 0 ) 1 (d,d 0 )f X = (d, d 0 ) x(d, d 0 ) (d,d 0 ) x(d, d 0 )=1, if (d, d 0 ) f matched pair x(d, d 0 )=0, if (d, d 0 ) / f unmatched pair 11
Linearization Linearization by introducing an indicator matrix X = 0 0 1... 0 0 0 1 0... 0 0... N M x(d, d 0 )=1, if (d, d 0 ) f matched pair x(d, d 0 )=0, if (d, d 0 ) / f unmatched pair 1
Linearization A = 1 0 0 0... M 110 1 0 1 0... 1M... N M X = 0 0 1... 0 0 0 1 0... 0 0... N M X (d,d 0 )f (d, d 0 )= X (d,d 0 ) d,d 0 x d,d 0 = tr(a T X) 1
Matching Formulation ˆX =min X tr(at X) ˆX = 0 trivial solution we need to constrain the formulation 1
Matching Formulation min X tr(at X) subject to: 8d V, 8d 0 V 0,x dd {0, 1} 0 X 8d, x dd =1 0 d 0 8d 0 X, x dd =1 0 d what is the meaning of this constraint? 1
Matching Formulation min X tr(at X) subject to: 8d V, 8d 0 V 0,x dd 0 {0, 1} 8d, 8d 0, X x dd =1 0 d 0 X x dd =1 0 d one-to-one matching 16
Relaxation min X tr(at X) subject to: 8d V, 8d 0 V 0,x dd 0 [0, 1] 8d, 8d 0, X x dd =1 0 d 0 X x dd =1 0 d one-to-one matching 17
Linear Assignment Problem subject to: min X tr(at X) 8d V, 8d 0 V 0,x dd 0 [0, 1] 8d, 8d 0, X x dd =1 0 d 0 X x dd =1 0 d one-to-one matching Hungarian algorithm for the balanced problem V = V 18
Hungarian Algorithm 19
The Hungarian Algorithm 1. From each row of A, find the row minimum, and subtract it from all elements in that row.. From each column of A, find the column minimum, and subtract it from all elements in that column.. Cross out the minimum number of rows and columns in A to cover all zero elements of A. If all rows of A are crossed out, we are done, and go to step 6.. Otherwise, find the minimum entry of A that is not crossed out. Subtract it from all entries of A that are not crossed out. Also, add it to all elements that are crossed out. Return to step with the new matrix. 6. Solutions are zero elements of A. Go first for the zero element which is unique in its row and column. Then, delete that row and column from A. Repeat until you delete all rows or columns from A. 0
Example -- The Hungarian Algorithm given a cost matrix A = 6 1 8 7 1 6 7 8 9 7 6 10 A = 6 9 0 0 10 0 6 0 8 step 1: find minimums in each row and subtract 7 1
Example -- The Hungarian Algorithm step : find minimums in each column and subtract A = 6 9 0 0 0 10 1 0 7 0 6
Example -- The Hungarian Algorithm step : cross out the zeros with a minimum number of lines A = 6 9 0 0 0 10 1 0 7 0 6 bold means crossed out we found lines < rows
Example -- The Hungarian Algorithm step : find minimum that is not crossed out A = 6 9 0 0 0 10 1 0 7 0 6
Example -- The Hungarian Algorithm step : subtract from non-crossed and add to crossed out elements A = 6 10 1 1 1 9 0 6 1 7 1 1
Example -- The Hungarian Algorithm Return to step 1: find minimums in each row and subtract A = 6 9 0 0 1 9 0 0 7 0 0 6
Example -- The Hungarian Algorithm Repeated step : find minimums in each column and subtract A = 6 9 0 0 1 9 0 0 7 0 0 note: no change from the previous step 7
Example -- The Hungarian Algorithm Repeated step : cross out the zeros with a minimum number of lines A = 6 9 0 0 1 9 0 0 7 0 0 bold means crossed out we found lines = rows 8
Example -- The Hungarian Algorithm -- Solution go for the unique solution first step 6: A = 6 9 0 0 1 9 0 0 7 0 0 f = {(d, d 0 )} 9
Example -- The Hungarian Algorithm -- Solution go for the unique solution first step : A = 6 9 0 0 1 9 0 0 7 0 0 f = {(d, d 0 ), (d, d 0 1)} 0
Example -- The Hungarian Algorithm -- Solution go for the unique solution first step : A = 6 9 0 0 1 9 0 0 7 0 0 f = {(d, d 0 ), (d, d 0 1), (d 1, d 0 )} 1
Example -- The Hungarian Algorithm -- Solution go for the unique solution first step : A = 6 9 0 0 1 9 0 0 7 0 0 f = {(d, d 0 ), (d, d 0 1), (d 1, d 0 ), (d, d 0 )} There is a number of alternative solutions!