PSET 0 Due Today by 11:59pm Any issues with submissions, post on Piazza. Fall 2018: T-R: Lopata 101. Median Filter / Order Statistics

Transcription

1 CSE 559A: Computer Vision OFFICE HOURS This Friday (and this Friday only): Zhihao's Office Hours in Jolley 43 instead of 309. Monday Office Hours: 5:30-6:30pm, Collaboration Jolley 27. PSET 0 Due Today by :59pm Any issues with submissions, post on Piazza. Fall 208: T-R: Lopata 0 Instructor: Ayan Chakrabarti (ayan@wustl.edu). Course Staff: Zhihao Xia, Charlie Wu, Han Liu Sep 6, 208 LAST TIME Convolutions Simplest spatial linear operation Output at each pixel is a function of a limited number of pixels in the input Linear Function Same function for different neighborhoods OTHER NEIGHBORHOOD OPERATIONS Median Filter / Order Statistics Neighborhood function N[ n ] {0, } O en better at removing outliers than convolution. Y[n] = Median{X[n n ]} N[ n ]= Edge & Line Detection: A Stereotypical Vision Algorithm Pipeline Use convolutions to detect local image properties (Gradients) Apply local non-linear processing to get local features (Edges) Aggregate information to find long-range structures (Lines) Source: Wikipedia Other ops: Y[n] = max / min{x[n n ]} N[ n ]>0

2 OTHER NEIGHBORHOOD OPERATIONS Morphological Operations Conducted on binary images (X [n] {0, }) Erosion: Dilation: Y [n] = AND {X [n n ]}N n Y [n] = OR {X [n n ]}N n [ [ ]= ]= ( if all neighbors ) ( if any neighbor ) Opening: Erosion followed by Dilation Closing: Dilation followed by Erosion See Szeliski Sec 3.3.2

3

4 Guided Bilateral Filter: B[ n n 2 based on a separate image Z [n]: depth, infra-red, etc., ] Far less efficient than convolution Filter also has to be computed, normalized, at each output location. Efficient Datastructures Possible Further Reading: Paris et al., SIGGRAPH/CVPR Course on Bilateral Filtering Recent work on using this for inference, best paper runner up at ECCV 206 Barron & Poole, The Fast Bilateral Solver, ECCV 206. Quick Recap: Complex Numbers A complex number f = x j ywhere x and y are scalar numbers. j = (EE convention: we use j instead of i x and y are called the real and imaginary components of f Think of f as a 2-D vector with special definitions of addition, multiplication, etc. ( x j y ) ( x 2 j y 2 ) = ( x x 2 ) j ( y y 2 ) ( x j y ) ( x 2 j y 2 ) = ( x x 2 y y 2 ) j ( x 2 y x y 2 ) ( x j y ) x 2 = x x 2 j y x 2 Conjugate: x j y = x j y = x j( y) Magnitude: (x j y) (x j y) = x 2 y 2

5 Quick Recap: Complex Numbers Euler's Formula exp(jθ) = cos θ j sin θ x j y = M exp(j θ) M = x 2 y 2, θ = tan (y, x) θ is called the "phase" M exp(jθ) = M exp( jθ) (x j y) exp(j ) = M exp(j(θ )) θ 0 θ 0 Preserves magnitude, adds to phase exp(j0) = exp(jnπ) = where N is an even integer, and = where N is an odd integer. Real in both cases The Discrete 2D Fourier Transform [X] = F[u, v] = X[, ] exp ( j u n 2π x n x n y ( )) n =0 x n =0 y Defined for a single-channel / grayscale image X. F is a "complex valued" array indexed by integers u, v. Each F[u, v] depends on the intensities at all pixels. exp(j θ) = cos θ j sin θ The Discrete 2D Fourier Transform [X] = F[u, v] = X[, ] exp ( j u n 2π x n x n y ( )) n =0 x n =0 y exp(j θ) = cos θ j sin θ The Discrete 2D Fourier Transform [X] = F[u, v] = X[, ] exp ( j u n 2π x n x n y ( )) n =0 x n =0 y exp(j θ) = cos θ j sin θ Note that F[u, v] = F[u W, v] = F[u, v H] because of periodicity. exp ( j 2π ( (u W ) n x W H )) = exp ( j u n 2π x ( n x W H )) = exp ( j u n 2π x ( ) j 2 π ) = exp ( j u n 2π x n x ( )) exp( j 2n x π) Note that F[u, v] = F[u W, v] = F[u, v H] because of periodicity. Therefore, we typically store F[u, v] for u {0,,W }, v {0,,H }. Can think of F[u, v] as a complex-valued "image" with the same number of pixels as X. Can be implemented fairly efficiently using the FFT algorithm: O(n log n) (o en, FFT is used to refer to the operation itself). = exp ( j u n 2π x ( ))

6 The Discrete 2D Fourier Transform Pair [X] = F[u, v] = X[, ] exp ( j u n x n x n y 2π ( )) n =0 x n =0 y [F] = X[, ] = F[u, v] exp ( j u n x n x n y 2π ( )) If X is real-valued, F[ u, v] = F[W u, H v] = F [u, v], where F implies complex conjugate. F[0, 0] is o en called the DC component. It is the average intensity of X. It is real if X is real. Only independent "numbers" in F[u, v] (counting real and imaginary separately) if X is real. Parseval's Theorem: (energy preserving upto constant factor) F[u, v] 2 = F[u, v] F [u, v] = X[ n x, n y ] 2 u,v u,v n, x n y The Discrete 2D Fourier Transform Pair for a fixed integers, [X] = F[u, v] = X[, ] exp ( j u n x n x n y 2π ( )) n =0 x n =0 y F [u, v] = F[u, v] exp ( j 2π ( u t x v t y W H [ ] = X[, ] = [u, v] exp ( j u n F n x t x n y t y F x 2π ( )) t x t y A change in the phase of the Fourier coefficients, that is linear in u, v, leads to a translation in the image. )) where each F[u, v] = S uv[ n x, n y ] X[ n x, n y ] =0 =0 can be thought of as a different (complex-valued) image: n x n y [, ] = exp ( j u n x n x n y 2π ( )) where each F[u, v] = can be thought of as a different (complex-valued) image: F[u, v] is the inner-product between X and. (scaled by ), X u n x [ n x, n y ] = exp ( j 2π ( )) For, x x x, y C n x, y = y is the Hermitian of x Transpose Conjugate (transpose the vector, and take conjugate of each entry) x y = i x iy i

7 where each F[u, v] = can be thought of as a different (complex-valued) image: F[u, v] is the inner-product between X and. (scaled by ), X u n x [ n x, n y ] = exp ( j 2π ( )) where each can be thought of as a different (complex-valued) image: F[u, v] is the inner-product between X and. (scaled by ) Property: F[u, v] =, X u n x [ n x, n y ] = exp ( j 2π (, S u = if = u & = v, and 0 otherwise. v v u )) Inverse-DFT: u n x X[ n x, n y ] = F[u, v] exp ( j 2π ( )) where each can be thought of as a different (complex-valued) image: F[u, v] is the inner-product between X and. (scaled by ) Property: F[u, v] =, X u n x [ n x, n y ] = exp ( j 2π (, S u = if = u & = v, and 0 otherwise. v v u )) F[u, v] = S, X uv, X = F[u, v], S u = if = u & = v, and 0 otherwise. v u v Inverse-DFT: X = F[u, v] X is a weighted sum of the images, weights are given by F[u, v].

8 F[u, v] = S, X uv, X = F[u, v], S u = if = u & = v, and 0 otherwise. v u v S is a matrix with each column a different. So, S S = S S = I S = S. This means S is a unitary matrix. Multiplication by S is a co-ordinate transform: F = S X, X = W H S F X are the co-ordinates of a point in a dimensional space. Multiplication by S changes the 'co-ordinate system'. In the new co-ordinate system, each 'dimension' now corresponds to frequency rather than location. S is a length-preserving matrix ( S X 2 = X 2 ). It does rotations or reflections (in dimensional space).

9

10 A k is not square for valid / long convolution. Question: Y = X k Y = Let Y = A k X correspond to Y = X k. Now, let X = A T Y. How is X related to Y by convolution? valid k What operation does represent? A T k X A k k[, ] A: Full convolution with n x n y (flipped version of k) Now if we consider the square A k matrix corresponding to 'same' convolution with circular padding, i.e. padding as X[W n x, n y ] = X[ n x, n y ], X[ n x, n y ] = X[ n x, H n y ], etc. A k Then, is diagonalized by the Fourier Transform! Here, D k is a diagonal matrix. The above equation holds for every A k You get different diagonal matrices. Y = X k Y = But S is the diagonalizing basis for all kernels. A k = S D k S X A k In the Fourier co-ordinate system, convolution is a 'point-wise' operation! D k Y = A k X = S D k S X ( S Y ) = D k ( S X) Why does this happen? X = u,v F[u, v] Y = X k = u,v F[u, v] k(by linearity / distributivity) ( k)[n] = n k[ n ] [n n ] [n n ], assuming circular padding, is also a sinusoid with the same frequency (u, v) and magnitude, but different phase. Multiplying by k[ n ] changes the magnitude, but frequency still the same. Adding different sinusoids of the same frequency gives you another sinusoid of the same frequency. ( k)[ n x, n y ] = d uv:k [ n x, n y ], where d uv:k is some complex scalar. Sinusoids are eigen-functions of convolution Y = X k = F[u, v] S k = ( F[u, v] uv d uv:k ) u,v u,v

11 A k = S D k S What's more, the diagonal elements of D k are the ( W x W y ) Fourier transform of k. This is the convolution theorem. ( D k = diag S Computational advantage for performing (and inverting!) convolution, albeit under circular padding. Good way of analyzing what a kernel is doing by looking at its Fourier transform. Why did we use complex numbers? Like quaternions in Graphics, for convenience! If we used real number co-ordinate transform, convolution would convert to several 2 2 transforms on pairs of co-ordinates. k ) Complex numbers are just a way of grouping these pairs into a single 'number'.