Towards Real Time Pattern Detection

Transcription

1 Towards Real Time attern Detection Yacov Hel-Or The Interdisciplinary Center attern Detection A given pattern is sought in an image/video. The pattern may appear at any location in the image. The pattern may be subject to any transformation (within a given transformation group). joint work with Hagit Hel-Or Haifa University 1 Example in 2D Face detection in images Example in 3D Action detection in video streams 1

2 Why is this a hard problem? The search in Spatial Domain Two types of search: 1. Search in the Spatial Domain. 2. Search in the Transformation Domain. Searching for faces in a 1000x1000 image, is applied 1e6 times, for each pixel location. A very expensive search problem The Search in Transformation Domain rotation of 2D signal projected into its 3 highest eigen-vectors A pattern under transformations draws a manifold in pattern space : R kxk Q T(α) A rotation manifold of a pattern drawn in pattern-space The manifold was projected into its three most significant components. 2

3 The transformation manifold in the pattern space is highly complex: In a very high dimensional space. Non convex. Non regular (two similarly perceived patterns may be distant in pattern space). Efficient Search in the Transformation Domain (In ICI, Barcelona Sept. 2003) A complex search problem Transformation Manifold A pattern can be represented as a point in R kxk T(α) is a transformation T(α) applied to pattern. T(α) for all α forms an orbit in R kxk Fast Search in attern Orbit Assume d(q,) is a distance metric. We would like to find (Q,)=min α d(q, T(α)) T(α 0 ) T(α 1 ) Q (Q,) T(α) T(α 2 ) 3

4 Fast Search in attern Orbit (Cont.) Fast Search in attern Orbit (Cont.) In the general case (Q,) is not a metric. Q Observation: if d(q,)= d(t(α)q, T(α)) R The metric property of (Q,) implies triangular inequality on the distances. Q S (Q,)- (,S) (Q,S) (Q,) is a metric oint-to-orbit dist. = Orbit-to-Orbit dist. Orbit Decomposition In practice T(α) is sampled into T ε (i) = T (εi), i=1,2, We can divide T ε (i) into two sub-orbits: T 2ε (i) and T 2ε (i) where = T ε (1) Orbit Decomposition (Cont.) Q 2ε (,Q) ε (,Q) 2ε (,Q) 2ε (, ) T 2ε (i) 2ε T ε (i) ε T 2ε (i) 2ε T 2ε (i) T ε (i) T 2ε (i) ε (,Q) = Min k { { 2ε d(q,t (,Q) ε (k)), 2ε (,Q) } Since 2ε is a metric and 2ε (, ) can be calculated in advance we may save calculations using the triangle inequality constraint. 4

5 The Orbit Tree The sub-group subdivision can be applied recursively. Threshold = 100 Orbit Tree 5

6 6

7 % Image ixels Remaining Rejection Rate # Distance Calculations Average number of distance computations per pixel is Fast Search in Group Orbit: Summary Observation 1: Orbit distance is a metric when the point distance is transformation invariant. Observation 2: Fast search in orbit distance space can be applied using the triangular inequality. Distant patterns are rejected fast. Important: Can be applied to metric spaces (Non Euclidean). Manifold simplification can be achieved using metric spaces that are NOT Euclidean. Fast search methods in metric spaces are required. 7

8 Efficient Search in the Spatial Domain (In ICCV, France, Oct. 2003) The Euclidean Distance d E ( u, v) = [ I( x u, y v) ( x, y) ] x, y N 2 d E ( u, v, t) = [ I( x u, y v, t w) ( x, y, t) ] x, y, t N 2 The Euclidean Distance Euclidean Distance (Cont.) W de (,W) d E (, W ) = W = ( W ) i T + W W 2 T i i i T Wi i T ( W) = T is constant. W T W can be calculated using a convolution of a window of ones with the squared image. T W can be calculated using a convolution of with the image. i 8

9 The Fourier Transform The Convolution Theorem W i Jean Baptiste Joseph Fourier R(i)= T W i projection of onto W i R = I= FFT { FFT{ I} FFT{ } Complexity (2D case) Norm Distance in Sub-space Representing an image window and the pattern as vectors in R kxk : Naive Fourier Average # Operations per ixel +: 2k 2 *: k 2 +: 36 log n *: 24 log n Space n 2 n 2 Integer Arithm. Yes No Run Time for 1Kx1K Image 32x32 pattern III, 1.8 Ghz 5.14 seconds 4.3 seconds d E (p,q)= w-p 2 = - 2 If p and w were projected onto a unit kernel u it follows from the Cauchy-Schwarz Inequality: d E (p,w) b where b=u T p-u T w w u b p 9

10 Distance Measure in Sub-space (Cont.) If w and p were projected onto a set of kernels: U T (p-w)=b w How can we Expedite the Distance Calculations? Two necessary requirements: 1. Choose projecting kernels [U] having high probability to be parallel to the vector p-w. 2. Choose projecting kernels that are fast to apply. p u 2 Natural Images It can be shown that: u 1 d E T T 1 ( p, w ) b ( U U ) b u 1 V u=-2, v=2 u=-1, v=2 u=0, v=2 u=1, v=2 u=2, v=2 u=-2, v=1 u=-1, v=1 u=0, v=1 u=1, v=1 u=2, v=1 U u=-2, v=0 u=-1, v=0 u=0, v=0 u=1, v=0 u=2, v=0 u=-2, v=-1 u=-1, v=-1 u=0, v=-1 u=1, v=-1 u=2, v=-1 u=-2, v=-2 u=-1, v=-2 u=0, v=-2 u=1, v=-2 u=2, v=-2 10

11 rojecting Kernels: Walsh-Hadamard The Walsh-Hadamard Kernels: Following the above requirement we use the kxk Walsh-Hadamard kernels Each window in a natural image is closely spanned by the first few kernel vectors. Can be applied very fast in a recursive manner. Walsh-Hadamard v.s. Standard Basis: The Walsh-Hadamard Tree (1D case) The lower bound for distance value in % v.s. number of Walsh-Hadamard projections, averaged over 100 patternimage pairs of size 256x256. The lower bound for distance value in % v.s. number of standard basis projections, Averaged over 100 pattern-image pairs of size 256x

12 The Walsh-Hadamard Tree - Example roperties: Descending from a node to its child requires one addition operation per pixel. A projection of the entire image onto one kernel is performed in a top-down traversal. A projection of a particular window in the image onto one kernel is performed in a bottom-up traversal. All operations are performed in integers Complexity (1D): rojecting all windows in the image onto a single kernel requires log k additions per pixel. rojecting all windows in the image onto l<k kernels requires m additions per pixel, where m is the number of nodes preceding the l leaf. rojecting all windows in the image onto k kernels requires 2k additions per pixel. + rojecting a single window onto a single - + kernel requires k-1 additions Walsh-Hadamard Tree (2D): For the 2D case, the projection is performed in a similar manner where the tree depth is 2log k The complexity is calculated accordingly Construction tree for 2x2 basis 12

13 attern Matching algorithm Iteratively apply Walsh-Hadamard kernels to each window w i in the image. At each iteration and for each w i calculate a lowerbound Lb i for p-w i 2. If the lower-bound Lb i is greater than a pre-defined threshold, reject the window w i and ignore it in further projections. attern Matching algorithm - Complexity All windows are projected onto the first kernel : 2logk ops/pixel Only a few windows are further projected using ~2k operations per active window : ε ops/pixel Total : 2logk + ε ops/pixel Example: Sought attern Initial Image: candidates After the 1 st st projection: 563 candidates 13

14 After the 2 nd projection: 16 candidates After the 3 rd projection: 1 candidate ercentage of windows remaining following each projection, averaged over 100 pattern-image pairs. Image size = 256x256, pattern size = 16x16. Accumulated number of additions after each projection averaged over 100 pattern-image pairs. Image size = 256x256, pattern size = 16x16. Average Number of operations per pixel:

15 Example with Noise Original Noise Level = 40 Detected patterns % Windows Remaining rojection # Number of projections required to find all patterns, as a function of noise level. (Threshold is set to minimum). ercentage of windows remaining following each projection, at various noise levels. Image size = 256x256, pattern size = 16x16. DC-invariant attern Matching Complexity (2D case) Original Illumination gradient added Detected patterns. Average # Operations per ixel Space Integer Arithm. Run Time for 1Kx1K Image 32x32 pattern III, 1.8 Ghz Naive +: 2k 2 *: k 2 n 2 Yes 4.86 seconds Fourier +: 36 log n *: 24 log n n 2 No 3.5 seconds New +: 2 log k + ε n 2 log k Yes 78 msec Five projections are required to find all 10 patterns (Threshold is set to minimum). 15

16 Fast Search in Spatial Domain - Summary: Very fast application of projection kernels. rojections are performed with additions/subtractions only (no multiplications). Integer operations (3 times faster for additions). Fast rejection of windows. ossible to perform pattern matching at video rate. Can implement a normalized correlation as well. A C++ implementation is available on the web. Conclusion attern Detection using 2 complementary processes: 1. Reduce search in Transformation Domain. 2. Reduce search in Spatial Domain. rocesses are based on rejection schemes, and are restricted to a specific domain. The two processes can be combined into a single, highly efficient, search process. --- END The size of WH projection kernels are 2 K x2 k. Show how is it possible to apply the fast pattern matching (using Eulcidean distance) for patterns of size different than 2 K x2 k. rove : if d(q,)= d(t(α)q, T(α)), (Q,)=min α,β d(t(β)q, T(α)) is a metric (Hint: since T(α) is a group, for any α there exists a α such that T(α ) T(α) =I ) 16