Pictorial Structures Revisited: People Detection and Articulated Pose Estimation. Department of Computer Science TU Darmstadt

Transcription

1 Pictorial Structures Revisited: People Detection and Articulated Pose Estimation Mykhaylo Andriluka Stefan Roth Bernt Schiele Department of Computer Science TU Darmstadt

2 Generic model for human detection and pose estimation Human pose estimation [Felzenszwalb&Huttenlocher, ICCV 5], [Ren et al., ICCV 5], [Sigal&Black, CVPR 6], [Zhang et al., CVPR 6], [Jiang&Marin, CVPR 8], [Ramanan, NIPS 6], [Ferrari et al., CVPR 8], [Ferrari et al., CVPR 9] often rather simple appearance model focus on finding optimal assembly of parts People Detection [Viola et al., ICCV 3], [Dalal&Triggs, CVPR 5], [Leibe et al., CVPR 5], [Andriluka et al., CVPR 8] complex appearance model no pose model or limited to walking motion 2

3 Generic model for human detection and pose estimation Human pose estimation [Felzenszwalb&Huttenlocher, ICCV 5], [Ren et al., ICCV 5], [Sigal&Black, CVPR 6], [Zhang et al., CVPR 6], [Jiang&Marin, CVPR 8], [Ramanan, NIPS 6], [Ferrari et al., CVPR 8], [Ferrari et al., CVPR 9] often rather simple appearance model focus on finding optimal assembly of parts People Detection [Viola et al., ICCV 3], [Dalal&Triggs, CVPR 5], [Leibe et al., CVPR 5], [Andriluka et al., CVPR 8] complex appearance model no pose model or limited to walking motion 3

4 Can we make pictorial structures model effective for these tasks? [Fischler&Elschlager, 1973] 4

5 Can we make pictorial structures model effective for these tasks? Yes... if the model components are chosen right. 5

6 Pictorial Structures Model Body is represented as flexible configuration of body parts L L = {l, l 1,..., l N } - configuration of parts D = {d, d 1,..., d N } - part evidence posterior over body poses d i p(l D) p(d L)p(L) likelihood of observations prior on body poses 6

7 Pictorial Structures Model Pictorial structures allow exact and efficient inference. - tree-structured prior - independent part appearance model - discretized part locations - Gaussian pairwise part relationships posterior marginals sumproduct BP p(l i D) L\l i p(l D) l 9 l 5 l 6 l 8 l 1 l 7 l 2 l 3 l 1 l 4 7

8 Can we make pictorial structures model effective for these tasks? So... what are the right components? 8

9 Model Components Appearance Model: orientation 1 likelihood of part 1 Prior and Inference: estimated pose. Local Features... AdaBoost orientation K... likelihood of part N part posteriors.... 9

10 Model Components Appearance Model: orientation 1 likelihood of part 1 Prior and Inference: estimated pose. Local Features... AdaBoost orientation K... likelihood of part N part posteriors.... 1

11 Likelihood Model Build on recent advances in object detection: state-of-the-art image descriptor: Shape Context [Belongie et al., PAMI 2; Mikolajczyk&Schmid, PAMI 5] dense representation discriminative model: AdaBoost classifier for each body part - Shape Context: 96 dimensions (4 angular, 3 radial, 8 gradient orientations) - Feature Vector: concatenate the descriptors inside part bounding box - head: 432 dimensions - torso: 8448 dimensions 11

12 Likelihood Model Part likelihood derived from the boosting score: decision stump weight p(d i l i ) = max part location ( decision stump output t α i,th t (x(l i )) t α i,t, ε ) small constant to deal with part occlusions 12

13 Likelihood Model Input image Head Torso Upper leg Our part likelihoods.... [Ramanan, NIPS 6] 13

14 Likelihood Model Input image Head Torso Upper leg Our part likelihoods.... [Ramanan, NIPS 6] 14

15 Likelihood Model Input image Head Torso Upper leg Our part likelihoods.... [Ramanan, NIPS 6] 15

16 Model Components Appearance Model: orientation 1 likelihood of part 1 Prior and Inference: estimated pose. Local Features... AdaBoost orientation K... likelihood of part N part posteriors

17 Kinematic Tree Prior Represent pairwise part relations [Felzenszwalb & Huttenlocher, IJCV 5] 5 5 p(l) =p(l ) p(l i l j ), (i,j) E p(l 2 l 1 )=N(T 12 (l 2 ) T 21 (l 1 ), Σ 12 1 ) part locations relative to the joint transformed part locations l 2 l l l

18 Kinematic Tree Prior Prior parameters: {T ij, Σ ij } Parameters of the prior are estimated with maximum likelihood mean pose several independent samples igure 2. (left) Kinematic prior learned on the multi-view and 18

19 Evaluation Scenarios 1. Human Pose Estimation People dataset [Ramanan, NIPS 6] 2. Upper-body Pose Estimation Buffy dataset [Ferrari et al., CVPR 8] 3. Pedestrian Detection TUD Pedestrians dataset [Andriluka et al., CVPR 8] 19

20 Evaluation Scenarios 1. Human Pose Estimation People dataset [Ramanan, NIPS 6] 2. Upper-body Pose Estimation Buffy dataset [Ferrari et al., CVPR 8] 3. Pedestrian Detection TUD Pedestrians dataset [Andriluka et al., CVPR 8] 2

21 Scenario 1: Qualitative Results Our model (g) 8/1 (a) 8/1 (d) 7/1 [Ramanan, NIPS 6] 7/1 /1 3/1 Our model (i) 8/1 (l) 6/1 (k) 7/1 [Ramanan, NIPS 6] 4/1 3/1 The numbers on the left of 3/1 21

22 Scenario 1: Quantitative Results Method [Ramanan, NIPS 6] 2nd parse Our inference, edge features from [Ramanan, NIPS 6] Our part detectors (SC) Our prior, our part detectors (SC) Our prior, our part detectors (SIFT) Torso Upper legs Lower legs Upper arm Forearm Head Total

23 Scenario 1: Quantitative Results Method [Ramanan, NIPS 6] 2nd parse Our prior, edge features from [Ramanan, NIPS 6] Our part detectors (SC) Our prior, our part detectors (SC) Our prior, our part detectors (SIFT) Torso Upper legs Lower legs Upper arm Forearm Head Total

24 Scenario 1: Quantitative Results Method [Ramanan, NIPS 6] 2nd parse Our inference, edge features from [Ramanan, NIPS 6] Our part detectors (SC) Our prior, our part detectors (SC) Our prior, our part detectors (SIFT) Torso Upper legs Lower legs Upper arm Forearm Head Total SC = Shape Context 24

25 Scenario 1: Quantitative Results Method [Ramanan, NIPS 6] 2nd parse Our inference, edge features from [Ramanan, NIPS 6] Our part detectors (SC) Our prior, our part detectors (SC) Our prior, our part detectors (SIFT) Torso Upper legs Lower legs Upper arm Forearm Head Total SC = Shape Context 25

26 Scenario 1: Quantitative Results Method [Ramanan, NIPS 6] 2nd parse Our inference, edge features from [Ramanan, NIPS 6] Our part detectors (SC) Our prior, our part detectors (SC) Our prior, our part detectors (SIFT) Torso Upper legs Lower legs Upper arm Forearm Head Total SC = Shape Context 26

27 Evaluation Scenarios 1. Human Pose Estimation People dataset [Ramanan, NIPS 6] 2. Upper-body Pose Estimation Buffy dataset [Ferrari et al., CVPR 8] 3. Pedestrian Detection TUD Pedestrians dataset [Andriluka et al., CVPR 8] 27

28 Estimated upper-body poses 28

29 Quantitative Results Method Torso Upper arm Lower arm Head Total [Ferrari et al. CVPR 8] detectors only full model generic model - prior and appearance learned on the People dataset

30 Quantitative Results Method Torso Upper arm Lower arm Head Total [Ferrari et al. CVPR 8] detectors only full model [Ferrari et al. CVPR 9] generic model - prior and appearance learned on the People dataset

31 Quantitative Results Method Torso Upper arm Lower arm Head Total [Ferrari et al. CVPR 8] detectors only full model [Ferrari et al. CVPR 9] full model, Buffy pose prior specialized upper body prior - appearance learned on the People dataset

32 Typical Failure Cases Foreshortening Part occlusion Detections on other body parts 32

33 Evaluation Scenarios 1. Human Pose Estimation People dataset [Ramanan, NIPS 6] 2. Upper-body Pose Estimation Buffy dataset [Ferrari et al., CVPR 8] 3. Pedestrian Detection TUD Pedestrians dataset [Andriluka et al., CVPR 8] 33

34 People Detection: Results Comparison with state-of-the art in people detection [Andriluka et al., CVPR 8] This work recall our model, 8 parts, tree prior our model, 8 parts, tree prior our model, 8 parts, star prior our model, 8 parts, star prior partism [Andriluka etdetector al., CVPR 8] HOG (INRIA Training HOG (INRIA Training Set) Set) precision

35 Conclusion & Future Work Success of pose estimation by body part detection use well understood pose estimation framework (Pictorial Structures) use appropriate representation for kinematic dependencies use state of the art appearance representation (SIFT, SC) and classification (AdaBoost) Next steps: estimate poses in 3D part occlusions appearance constraints between parts 35

36 Thanks! Acknowledgements: Thanks to Krystian Mikolajczyk for image descriptors code Thanks to Christian Wojek for AdaBoost code and helpful suggestion Thanks to Deva Ramanan and Vittorio Ferrari for making their code and datasets publicly available This work is partially funded by German Research Foundation (DFG) through GRK Code and pre-trained models will be available at: 36