8/13/10. Visual perception of human motion. Outline. Niko Troje BioMotionLab. Perception is. Stimulus Sensation Perception. Gestalt psychology

Transcription

1 Visual perception of human motion Outline Niko Troje BioMotionLab! Vision from the psychologist s point of view: A bit of history and a few concepts! Biological motion: perception and analysis Department of Psychology & School of Computing Queen s University wwwbiomotionlabca Perception is to infer a consistent, reliable, predictive model of the world given noisy, incomplete, ambiguous sensory data!? Stimulus Sensation Perception Gestalt psychology Gestalt Laws Gestalt (German): form, shape, figure Law of proximity Wolfgang Köhler ( ) Max Wertheimer ( ) Law of similarity Law of good continuation Kurt Koffka ( ) The whole is more than the sum of its parts! Law of common fate 1

2 Gestalt theory Simplicity Principle! Law of pragnanz Prägnanz: German for good figure, design, carrying meaning A stimulus pattern is organized in such a way that the resulting structure is as simple as possible The Simplicity Principle (eg Pomerantz & Kubovy 1986) Simplicity Stimulus Sensation Perception! Simplicity Occam s razor (William of Ockham, 14 th century): "entities must not be multiplied beyond necessity (entia non sunt multiplicanda praeter necessitatem) Information theory, channel capacity, redundancy (Shannon and Weaver 1949) Coding theory (Hochberg & McAlister 1953, Leeuwenberg 1969, 1971; Restle 1979) Attneave 1954, 1981: What the perceptual system likes is short descriptions Minimum description length (Rissanen 1989, Hinton & Zemel 1994) Hermann von Helmholtz ( ) Hermann von Helmholtz ( )! Measured the speed of a nerve impulse! Invented the ophthalmoscope! Important publications Handbook of Physiological Optics The Sensation of Tone as a Physiological Basis for the Theory of Music On the Conservation of Force! Helmholtz advocated the Likelihood Principle Perception as unconscious inference Sensory input will be organized into the most probable distal object or event consistent with that input Statistics Stimulus Sensation Perception Very good reading is a speech he gave in 1878 The Facts of Perception Likelihood Principle Bayes ian Inference! Likelihood Helmholtz s unconscious inference Oliver Selfridge s (1959) pandemonium model of letter recognition Committee rules: Honor physics, avoid accidents Bayesian statistical decision theory: p(w i s) p(s w i ) * p(w i ) posterior probability likelihood prior probability prior p(w i s) p(s w i ) * p(w i ) posterior probability likelihood prior probability C = arg max [p(w i s)] w i 2

3 Bayes ian Inference More examples for ambiguities in visual perception aspect ratio what we want to know slant (of an ellipse) what we can measure aspect ratio of the retinal image context our example p(w i s) p(s w i ) * p(w i ) reflectance illuminance luminance lightness constancy posterior probability likelihood prior probability size distance retinal size intensity wavelength absorption object motion motion of the retina retinal motion size distance ambiguity principle of univariance efference copy theory speed direction of motion image motion aperture problem R = arg max [u(r,w) p(w s)] r 3D shape direction of light 2D retinal image distance motion of observer relative to object retinal motion light-from-above bias motion parallax Criss-cross Illusion E H Adelson, 2000, www-bcsmitedu/gaz/ More examples for ambiguities in visual perception aspect ratio what we want to know slant (of an ellipse) what we can measure aspect ratio of the retinal image context our example reflectance illuminance luminance lightness constancy size distance retinal size intensity wavelength absorption object motion motion of the retina retinal motion size distance ambiguity principle of univariance efference copy theory speed direction of motion image motion aperture problem luminance = illuminance * reflectance 3D shape direction of light 2D retinal image distance motion of observer relative to object retinal motion light-from-above bias motion parallax 3

4 Light-from-above bias Light-from-above bias Simplicity vs Likelihood! Simplicity or Likelihood as an organizing principle? What is role of redunancy? Simplicity: "entities must not be multiplied beyond necessity (Occam s razor) Likelihood: Correlations in the data are necessary to make predictions Leeuwenberg & Boselie 1988 Simplicity = Likelihood Horace Barlow ( )! It can be shown that there is no conflict and that the principles are in fact equivalent Cover & Thomas 1991, Rissanen 1989, Mumford 1992 Chater 1996, Kolmogorov 1965)! What is efficient coding? Efficient coding should convert hidden redundancy into a manifest, explicit, immediately recognizable form, rather than reduce it or eliminate it 4

5 Part II! Biological motion: perception and analysis Gunnar Johansson Starting point Biological motion contains plenty of information Our visual system is well able to retrieve and interpret this information Johansson, 1973 Question Example Principles How can we retrieve information encoded in animate motion? What makes a walker appear male or female? A more general view Applications in computer vision, animation, gait analysis 5

6 Training 3 2 position time x 2 + x 1 Test Visualization x 2 + x 2 O! + x 1 x 1? x 2 The goal is to find a representation that is loss-less supports linear operations is low-dimensional The approach: morphable model Start with a time series of postures expressed as Cartesian coordinates in a body centred coordinate system Express each trajectory as a discrete, low-dimensional Fourier series x 1 The rest is simple: Reduce dimensionality by means of principle components analysis on the set of walkers Linear regression 6

7 p(t) = p 0 + a 1 sin(ωt + φ 1 ) + a 2 sin(2ωt + φ 2 ) + p(t) = p 0 + p 1 sin(ωt) + q 1 cos(ωt) + p 2 sin(2ωt) + q 2 cos(2ωt) + p(t) = p 0 + a 1 sin(ωt + φ 1 ) + a 2 sin(2ωt + φ 2 ) + p(t) = p 0 + p 1 sin(ωt) + q 1 cos(ωt) + p 2 sin(2ωt) + q 2 cos(2ωt) + k power [%] n 14 w = [ P, ω] P = p0 1,x p0 2,x p0 1,y p0 2,y p0 1,z p0 2,z p1 1,x p1 2,x p1 1,y p1 2,y p1 1,z p1 2,z q1 1,x q1 2,x q1 1,y q1 2,y q1 1,z q1 2,z p2 1,x p2 2,x p2 1,y p2 2,y p2 1,z p2 2,z q2 1,x q2 2,x q2 1,y q2 2,y q2 1,z q2 2,z It s a morphable representation! w = p0 1,x p0 2,x p0 3,x p0 4,x p0 n,z p1 1,x q2 n-1,z q2 n,z ω w 1 + w 2 2 Σ w i n Σ (a i w i ) Σ a i w = p 1 p 2 p 3 p m W = p 1,1 p 2,1 p 3,1 p m,1 p 1,2 p 2,2 p 3,2 p m,2 p 1,k p 2,k p 3,k p m,k Principle component analysis training set test data 1 p 1,1 p 2,1 p 3,1 p 1,2 p 2,2 p 3,2 p 1,k p 2,k p 3,k t 1 t 2 t 3 cumulative relative variance 5 W = p m,1 p m,2 W d = M W p p m,k t =? t d = M t p number of principle components W p : predictors W d : dependent variables 7

8 training set test data Example: gender classification W = p 1,1 p 2,1 p 3,1 p 1,2 p 2,2 p 3,2 p 1,k p 2,k p 3,k t = t 1 t 2 t 3 Data: 50 male and 50 female individuals Over-ground walking, 15 landmarks (mostly joints) Fourier-based morphable model p m,1 p m,2 p m,k? Take one pattern out and use the rest to learn the model Apply the model to the remaining pattern Evaluate the outcome Repeat the procedure for every pattern W d = M W p t d = M t p Result: 925% correct classification! W p : predictors W d : dependent variables Troje JOV 2002 use all information (3D, kinematics & shape) to predict gender use only parts (2D) info to predict gender Classification performance of the model Error rate [%] Full Info Structure-only Kinematics-only?? Viewpoint predictors dependent variables Classification performance of the model and of human observers use all information (3D, kinematics & shape) to predict gender use only parts (eg 2D) info to predict gender use gender to predict kinematics Model Psychophysics Error rate [%] Full Info Structure-only Kinematics-only Viewpoint Viewpoint 24 observers, 3 s presentations (PLD) View: 0, 30, 90 deg (between subject) Info: full info, structure-only, dynamic-only (within subject) Troje JOV 2002 predictors Analysis dependent variables Synthesis 8

9 Gender was just an example Use the same idea for other attributes: biological: weight, size, age emotional personality traits clinical: psychology, neurology, biomechanics Analysis Synthesis 3D Reconstruction Zhang, Z, and Troje, N F (2007) 3D Periodic Human Motion Reconstruction from 2D Motion Sequences Neural Computation 19: Zhang, Z, and Troje, N F (2005) View-independent person identification from human gait Neurocomputing 69: biological emotional clinical attributes kinematics (15 markers, 3D) 2D Sigal, L, Troje, N F, Fleet, D (2010) Human Attributes from 3D Pose Tracking Search Results 11th European Conference on Computer Vision (ECCV) mocap X biological emotional clinical computer animation more landmarks, more DoF W = mocap Y mocap Z attributes attributes kinematics (15 markers, 3D) 2D video video IMU GRF other sensors predictors dependent variables 9

10 The goal is to find a representation that supports linear operations is loss-less is low-dimensional morphable model The approach: Start with a time series of postures expressed as Cartesian coordinates in a body centred coordinate system Express each trajectory as a discrete, low-dimensional Fourier series The human visual system is a proof of concept and a source of inspiration for how to infer information from noisy, incomplete, ambiguous sensory data! The rest is simple: Reduce dimensionality by means of principle components analysis on the set of walkers Linear pattern classification Journal of Cognitive Neuroscience, 1991 FIAS 09 Vetter & Troje,

11 Pixel-based representation Correspondence-based representation registration Vetter & Troje, 1997 Correspondence-based representation morphable model texture T = I 1,1 I 1,2 I n,m Vetter & Troje, 1997 S = shape dx 1,1 dy 1,1 dx 1,2 dx n,m dy n,m Vetter & Troje, 1997 Vetter & Troje, 1997 Vetter & Troje, 1997 Blanz & Vetter,