Lecture 12 Body Pose and Dynamics

Transcription

1 Lecture 12 Body Pose and Dynamics Yaser Sheikh Human Motion Modeling and Analysis Fall 2012

2 Debate Advice Affirmative Team Thesis Statement: Every (good) paper has a thesis. What is the most provocative statement the paper is trying to make? Spill the beans early: Explain the key insight of the paper as soon as possible. Like on Slide 1. Teaser: Show an example result first so that it s clear to everyone what the goal is. Also mention the input, clearly. Equations: Explain them carefully or don t include them.

3 Debate Advice Improvement Team Examine the Thesis: Is it sound? Does it overreach? Is it practically useful? State a Core Objection: State a Core Objection early in the process. Explain limitations: Usually in the paper. Assumptions, model restrictions, computation, etc.

4 Human Motion How can we model human motion?

5 Why Model Motion? Keyframing X t X t+1 Xt+2 Xt+3 X t+4 Xt+5 t X t+6

6 Why Model Motion? Labeling X i t : Point Position at time t Z i t : Point Label at time t

7 Why Model Motion? 3D Reconstruction X t = 2 4 X 1 X 2 X 3 X 4 X P 3 Y 1 Y 2 Y 3 Y 4 Y P 5 Z 1 Z 2 Z 3 Z 4 Z P

8 Why Model Motion? Action Recognition

9 Knowns and Unknowns There is structure in human pose and motion Keyframing t Reconstruction Labeling Recognition

10 Lecture in One Slide Notation = = Y t X t+1 D X t X t B Dynamics Latent Variable Model = Y t+1 G Y t Latent Dynamics

11 Representing Pose Configuration Space D Features D Points D Joint Angles X t 2 R D Observation

12 The Space of Actions Subtitle p(x 1,, X F )

13 Human Motion Frame-wise Independent Data Space X t X t+1 X t+2 X t+3 p(x 1,, X F ) = p(x 1 )p(x 2 ) p(x F )

14 p(x 1 )p(x 2 ) p(x F ) p(x t )=N (X t µ, )

15 Autoregressive Models First-order Markov Model Data Space X t X t+1 X t+2 X t+3 p(x 1,, X F ) = p(x 1 ) FY t=2 p(x t X t 1 ) p(x t X 1,, X t 1 )=p(x t X t 1 )

16 First-Order Markov Models Linear Dynamics Data Space X t X t+1 X t+2 X t+3 p(x t X t 1 ) X t = f(x t 1 ) Dynamical Function X t+1 = DX t Linear Dynamics

17 Markov Models First Order X t+1 = DX t D 2 R D D = X t+1 D X t X t+n = D n X t

18 Markov Model Reconstruction X t+1 X t+2 X t = D X t X t+1 X t X t+f +1 X t+f

19 First-order AR Reconstruction Ground truth Reconstructed

20 Autoregressive Models First-order Markov Model Data Space X t X t+1 X t+2 X t+3 p(x 1,, X F ) = p(x 1 ) FY t=2 p(x t X t 1 ) p(x t X t 1 )=N (X t DX t 1, )

21 First-Order AR Prediction X t+1 X t+2 X t+3... X t+f = D D 2 D 3... D F X t Observability Matrix

22 First-Order AR Linear Prediction Ground truth Predicted X t+1 X t+2 X t+3. X t+f = D D 2 D 3. D F X t

23 First-Order AR Prediction Generalization X t+k X t+1+k X t+2+k... X t+f +k = D D 2 D 3... D F X t+k

24 First Order AR Prediction Generalization Ground truth Predicted X t+k X t+1+k X t+2+k. X t+f +k = D D 2 D 3. D F X t+k

25 Autoregressive Models First-order Markov Model Data Space X t X t+1 X t+2 X t+3 p(x t X t 1 )=N (X t DX t 1, )

26 Ideas?

27 Autoregressive Models Second-Order Markov Model Data Space X t X t+1 X t+2 X t+3 p(x 1,, X F ) = p(x 1 )p(x 2 X 1 ) FY t=1 p(x t X t 1, X t 2 )

28 AR Models Second-Order Systems X t+1 = D apple Xt 1 X t D 2 R D 2D = X t 1 X t+1 D X t

29 Considerations AR systems Complexity: Curse of dimensionality, computation, compaction? Predictive Precision: How accurately does the model predict observations? Generalization Ability: How well does the model generalize to new data?

30 Considerations Trade-offs Memory: How far back should you look? How much is it worth in extra dimensionality? Linearity: How much of a limitation is the linearity of the dynamical system?

31 Singular Values of D Compaction across Actions (walk, sit, fall, etc.) magnitude singular value

32 Ideas?

33 Pose Correlations Latent Variable Models How many degrees of freedom are there really? X t N (µ, )

34 Latent Variable Models Linear Models = w 1 +w 2 +w 3

35 Linear Projection Principal Component Analysis = Y t X t B Latent Variable Model {X n } : Training Data Y t 2 R M X t 2 R D M<D

36 Probabilistic PCA Distribution p(x t Y t ) X t = BY t + µ + p(y t )=N (z 0, I) p(x t Y t )=N (X t BY t + µ, 2 I)

37 Probabilistic PCA Generative View of PCA x 2 w x 2 p(x bz) µ } b z w µ p(z) p(x) bz z x 1 x 1

38 Probabilistic PCA Sampling Standing Up X t N (µ, )

39 Probabilistic PCA Sampling Standing Up p(x t Y t )=N (X t BY t + µ, 2 I)

40 Graphical Model Component Analysis X t = BY t + µ + Y 1 Y 2 Y M X 1 X 2 X 3 X D

41 Graphical Model Component Analysis X t = BY t + µ + Latent Space Y t Data Space X t

42 PCA Works! Less than 5cm max-error with <30 components Max. reconstruction error (cm) Max. PCA Recon. Error per Action armmovement bend boxing climb jump run sit walk wave Num. of PCA components (K)

43 Well...Somewhat... Max. Reconstruction error (cm) PCA works when Action is Known Performance of PCA with Data Diversity 1actions. 3actions. 7actions. 10 actions Num. of PCA components (K)

44 Overcomplete Dictionaries Ramakrishna et al arg min Y t kx t BY t k 2 + ky t k 1 X t = BY t X t = BY t = = L1-norm encourages sparsity in Y

45 3D Reconstruction Reprojection Error Decreases at Each Iteration

46 3D Reconstruction 3D Pose and Camera

47

48 Dynamical Models Graphical Models Latent Space Y t Y t+1 Y t+2 Y t+f Data Space X t X t+1 X t+2 X t+f

49 Linear Dynamical System Graphical Summary Y t Y t+1 Y t+2 Y t+f X t X t+1 X t+2 X t+f X t+1 = DX t X t = BY t = = Linear Dynamics Latent Variable Model

50 Model Reduction Dynamics in the Latent Space X t+1 = DX t X t = BY t = = BY t = DBY t Y t = B T DBY t Y t = GY t = G = B T DB

51 Model Reduction Projected Dynamics G = B T DB = = = =

52 Dynamical Models Graphical Models Latent Space Y t Y t+1 Y t+2 Y t+f Data Space X t X t+1 X t+2 X t+f

53 Latent Space Y t Y t+1 Y t+2 Y t+f Data Space X t X t+1 X t+2 X t+f p(x 1,, X F )= Z p(x 1,, X F, Y 1, Y F )dy p(y t Y t 1 )=N (Y t GY t 1, ) p(x t Y t )=N (X t BY t, ) = = p(y 1 )=N (Y 1 µ, V)

54 Nonlinear Dynamical Models? Linear-Gaussian Models work for individual activities Nonlinear Latent Variable Models: Density Networks Generative Topographic Mapping Kernel PCA Gaussian Process Latent Variable Models Nonlinear Dynamics: Switching Linear Dynamical Models Gaussian Process Dynamical Models Sampling-based methods (i.e., Particle Filters)

55 Reading List Pavlovic et al. Learning Switching Linear Models of Human Motion Lawrewnce et al. Gaussian Process Latent Variable Models for Visualisation of High Dimensional Data Fleet, Motion Models for People Tracking. Ramakrishna et al., Reconstructing 3D Human Pose from 2D Image Landmarks, 2012.