Synthetic Sensing - Machine Vision: Tracking I MediaRobotics Lab, March 2010

Transcription

1 Synthetic Sensing - Machine Vision: Tracking I MediaRobotics Lab, March 2010 References: Forsyth / Ponce: Computer Vision Horn: Robot Vision Schunk: Machine Vision University of Edingburgh online image processing reference The Computer Vision Homepage Rice University Eigenface Group OpenCV

2 Tracking The idea is old. Tracking is just keep note of things... General requirements: - a something to detect - a way of representing that object to your system - a way to tally the results - a way to find previous results - a way to recover from mistakes

3 Non-image based tracking Biometrics: fingerprints skin pattern facial thermogram gait dna Governmental: social security numbers tax records credit reports

4 Hydra, University of Notre Dame

5 retrospective surveillance: The goal of these systems is to review the captured scenes from other sites in order to validate whether a hint of threat detected at the local site is part of a larger pattern. Imagine our proposed infrastructure deployed to monitor all important landmarks in the United States... Analyzing the images from multiple cameras peering into the crowds can allow detection algorithms to potentially make more reliable identification of terrorists than single cameras. More importantly, we can develop recognition algorithms that, when triggered by the suspicious activity of one tourist, analyze the stored streams from other landmarks to see if this same tourist exhibited suspicious behavior in those other sites; activity which may be missed by each site locally. Analyzing the streams in concert can also help identify more complex threat behavioral patterns. Hydra, University of Notre Dame

6 ...One can imagine recognition algorithms that identify a threat event that involves multiple actors; identified not only because each of these actors exhibit similar suspicious behavior but also by the fact that they all scoped out the landmark sites without overlapping with each other. While one person was identified as video taping the Empire State building and the Statue of Liberty in NYC and Sears tower in Chicago within a week of each other, another individual was also noticed video taping the Brooklyn bridge in New York and Navy pier in Chicago in the same week. Note that tourists video taping landmark sites itself is not the threat; rather the specific pattern and choice of sites might give clues to suspicious behavior... Hydra, University of Notre Dame

7 Also:

8

9 Image based tracking Direct tracking: -image analysis Indirect tracking: -image differencing -optical flow Visual servoing: -feed results back into a controller to steer a moving vehicle

10 Difference images: pyramid of change > nth derivative Stream of images in time Time is here a variable (function of #frames) Effective for global properties

11 Visual tracking of moving objects (multi-camera) The first step in tracking objects is for the system to distinguish moving objects from stationary ones through feature selection and detection. The next step requires the system to make note of the location, speed, size, and shape of moving objects. Finally, the system must learn to recognize and track the same object as it moves out of the visual field of one camera and into the next. The computer is able to do this as long as the visual fields of the its cameras overlap at least somewhat.

12 Visual Tracking with a single camera: algorithm feature selection while (streaming) { feature extraction feature location }

13 Indoor Line Following: Vision-based Line Tracking and Navigation in Structured Environments G. Reccari Y. Caselli F. Zanichelli A. Calafiore Advanced Outdoor Line Following: First results in vision-based crop line tracking Mark Ollis & Anthony Stentz, Robotics Institute (1996) The color segmentation algorithm has two parts: a discriminant and a segmentor. The discriminant computes a function d(i,j) of individual pixels whose output provides some information about whether that pixel is in the cut region or the uncut region; the segmentor then uses the discriminant to produce a segmentation.

14 Features -color -brightness -texture -location -size -form -other local and global geometry (eigenfaces for face detection) > isolated and combined

15 Outlines and Templates -> cvmatchtemplate (C++ only)

16 image template Occurances of the template in the image In Opencv under C++ cvmatchtemplate(image, template, result, CV_TM_CCOEFF_NORMED);

17 z y x x, y x, y

18 Rotation, Scale, Translation-Invariant Template Matching

19 Texture What is texture? - a feature that repeats with some variation - need to separate the repeating elements from the constant elements - often approached with probabilistic distributions - also: wavelets and neuralnets - example: Anil K. Jain, Kalle Karu, Learning Texture Discrimination Masks (February 1996 (Vol. 18, No. 2) pp

20 Eigenfaces Eigenfaces are a set of eigenvectors used in the computer vision problem of human face recognition. These eigenvectors are derived from the covariance matrix of the probability distribution of the high-dimensional vector space of possible faces of human beings. The technique has been used for handwriting, lip reading, voice recognition, and medical imaging. Eigenfaces can be imagined as a set of "standardized face ingredients", derived from statistical analysis of many pictures of faces. Any human face can be considered to be a combination of these standard faces (everyone has eyes, a nose, a mouth..). One person's face might be made up of 10% from face 1, 24% from face 2 and so on.

21 Eigenfaces Practically, eigenfaces are created by finding feature vectors based on deviations from an averaged training set. a) collect some images b) find the average image (sum and divide) c) find the deviated images (differences between individual images and the ave image) d) calculate the covariance matrix (measure of how much a set of variables vary in the same way) e) calculate the eigenvectors of the covariance matrix (vectors that do not change with scaling) f) construct eigenfaces by combining N eigenvectors check wikipedia for an intuitive description of eigenvectors:

22 Eigenfaces Covariance - reminder: Standard deviation and variance only operate on 1 dimension, so that one can only calculate the standard deviation for each dimension of the data set independently of the other dimensions. However, it is useful to have a similar measure to find out how much the dimensions vary from the mean with respect to each other. Covariance is such a measure. Covariance is always measured between 2 dimensions. Here are the base formulae for variance and covariance: For a 3d data set, the covariance matrix would be: note: cov (x,x) = var (x)

23 Eigenfaces Eigenvectors - reminder: - eigenvectors can only be found for square matrices. And, not every square matrix has eigenvectors. An nxn matrix that does have eigenvectors, has N eigenvectors. -eigenvectors are scale independent even if one scales the vector by some amount before one multiplies it, the result is unchanged. - all the eigenvectors of a matrix are orthogonal, ie. at right angles to each other, no matter how many dimensions the matrix has.

24 Rice University: Eigenface Group has python face recognition code implementing eigenfaces: University of Pittsburgh has an online face detection program:

25 Problems with features -perspective -underdefinition (3d, content) -lighting -occlusion -distance -image resolution -test data without training data

26 General approach feature selection while (streaming) { feature extraction feature location } feature location: choose an invariant property for example: center of mass

27 Center of Mass According to Newton's third law the two internal forces are equal and opposite. Adding the equations then gives: Fi1+F1 = a1m1 Fi2+F2 = a2m2 (internal forces are equal and opposite) xcm The two bodies are combined to one body with mass m 1 + m 2 This body is acted upon by the same external forces as our two bodies. This imaginary body then gets an acceleration which we call the acceleration of the center of mass a CM and given by: F1 + F2 = a1m1 + a2m2 a1m1 + a2m2 = (m1+m2) a cm acceleration is the second derivative of the position: x1m1 + x2m2 = (m1+m2) xcm the center of mass 'balances' the two different weights: xcm = (x1m1 + x2m2) / m1m2

28 Calculating the CG in matlab %find the centers of gravity of image Y(i * j) for i = 1 : j C = [C, ((Y(:,i)'*m') / sum(y(:,i)))]; end for i = 1 : k D = [D, ((Y(i,:)*n') / sum(y(i,:)))]; end G = C(2:end); E = D(2:end); PY = round(median(g)); PX = round(median(e)); %Y coordinate of the COG %X coordinate of the COG

29 Color based tracking xstart, ystart, xnew, ynew, xprevious, yprevious while (streaming) { feature extraction > hue >binarize >area feature location > get CG coordinates > xnew, ynew } xstart = xnew ystart = ynew xstart xprevious ystart yprevious xprevious = xnew yprevious = ynew //set only once //set only once //check distance travelled //check distance travelled //update continuously //update continuously

30 Simple Hack with PIL Calculate the bounding box around a region of interest #binarize the final image mask6 = mask5.convert("1", dither=image.none) #find outer corners of remaining area for i in range(x): for j in range(y): if(mask6.getpixel((i,j))): if (i < xmin): xmin = i if(i > xmax): xmax = i if (j < ymin): ymin = j if(y > ymax): ymax = j box = (xmin, ymin, xmax, ymax) draw.rectangle(box, outline=(255,0,0))

31 Simple tracking based on color xstart, ystart, xnew, ynew, xprevious, yprevious while (streaming) { feature extraction > hue binarize area(s) feature location > get CoM coordinates > xnew, ynew xstart xnew ystart ynew xstart xprevious ystart yprevious xprevious = xnew yprevious = ynew }

32 Advanced form tracker: car finder

33 Additional topics: industrial applications: moving line tracking clustering: k-means decomposition: principle component analysis