You Are How You Walk: Gait Recognition from Motion Capture Data

Transcription

1 You Are How You Walk: Gait Recognition from Motion Capture Data Michal Balazia Faculty of Informatics, Masaryk University, Brno, Czech Republic Michal Balazia FI MU) Gait Recognition from MoCap Data May 2nd, / 26

2 Human Identification by Gait Data captured by a system of multiple cameras or a depth camera Large tracking space Multiple samples of a single walker High variance in encounter conditions Database of large amount of biometric samples Identification in real time Michal Balazia FI MU) Gait Recognition from MoCap Data May 2nd, / 26

3 Motion Capture Data MoCap) Frame: y z x Structural motion data Skeleton of joints and bones Data = 3D positions of joints in time. Can be collected by a system of multiple cameras Vicon) or a depth camera Microsoft Kinect) Michal Balazia FI MU) Gait Recognition from MoCap Data May 2nd, / 26

4 Raw MoCap Gait Data Model of human body has J joints Measured gait cycle has length of T video frames Raw MoCap gait sample is a tensor γ 1 ) γ 1 T ) g =..... γ J T ) γ J T ) γ j t) R 3 are 3D coordinates of j {1,..., J} at t {1,..., T } Dimensionality 3JT Sample space {g} Michal Balazia FI MU) Gait Recognition from MoCap Data May 2nd, / 26

5 Geometric Features feature signals Examples of geometric gait features: joint angles angle in shoulder-elbow-wrist) video inter-joint frames distances feet distance) joint velocity or acceleration areas of joint polygons upper body span)... features video frames Examples of distance functions: Dynamic Time Warping Minkowski distances... Michal Balazia FI MU) Gait Recognition from MoCap Data May 2nd, / 26

6 Linearly Learned Latent Features Labeled learning sample space {g n, l n )} N L n=1 l n is a label of one of the identity classes {I c } C L c=1 I c has a priori probability p c Consider an optimization criterion J Feature extraction is given by a feature matrix Φ RD D D-dimensional sample space {g n } N n=1 D-dimensional feature space {ĝ n } N n=1 Transform gait samples g n into gait templates ĝ n = Φ g n Examples of distance functions: Mahalanobis distance Minkowski distances... Michal Balazia FI MU) Gait Recognition from MoCap Data May 2nd, / 26

7 Learning by MMC Optimize class separability of the feature space Margin of two classes is the Euclidean distance of their means µ c minus both their variances Σ c Maximum Margin Criterion used by the Support Vector Machines J = 1 2 C L c,c =1 p c p c =... = tr Σ B Σ W ) ) µ c µ c ) µ c µ c ) tr Σ c + Σ c ) Between-class scatter matrix Σ B, within-class scatter matrix Σ W Criterion for a feature matrix Φ ) J Φ) = tr Φ Σ B Σ W ) Φ Solution: solve the generalized eigenvalue problem Σ B Σ W ) Φ = ΛΦ Mahalanobis distance function on templates Michal Balazia FI MU) Gait Recognition from MoCap Data May 2nd, / 26

8 Learning by PCA+LDA 2-stage feature extraction technique Principal Component Analysis and Linear Discriminant Analysis Total scatter matrix Σ T = Σ B + Σ W Criterion for a feature matrix Φ LDA ) J Φ PCA ) = tr Φ PCAΣ T Φ PCA Φ J Φ LDA ) = tr LDA Φ PCA Σ ) BΦ PCA Φ LDA Φ LDA Φ PCA Σ WΦ PCA Φ LDA Solution: solve the generalized eigenvalue problems Σ T Φ PCA = ΛΦ PCA Φ PCAΣ W Φ PCA ) 1 Φ PCAΣ B Φ PCA ) Φ LDA = ΛΦ LDA Mahalanobis distance function on templates Michal Balazia FI MU) Gait Recognition from MoCap Data May 2nd, / 26

9 OPENING HOURS Identity Classification Pipeline BANK Phase I Phase II Spo ed walker Acquiring MoCap data MoCap data Detec ng gait BANK DEPARTURES Phase IV Iden fying walkers Phase III Extrac ng gait features Gait sample Iden fied walker Gait template Michal Balazia FI MU) Gait Recognition from MoCap Data May 2nd, / 26

10 Identity Classification Pipeline DEPARTURES 39 51' 05'' N ' 34'' W 2017/06/21 07:55: ' 06'' N ' 10'' W 2017/06/23 13:24:19 BANK 39 43' 04'' N ' 50'' W 2017/06/24 22:49:38 Michal Balazia FI MU) Gait Recognition from MoCap Data May 2nd, / 26

11 The Classification Problem Identity: Label of a registered identity class. Michal Balazia FI MU) Gait Recognition from MoCap Data May 2nd, / 26

12 But In Video Surveillance Environment... How can we represent the identity in video surveillance environment? Michal Balazia FI MU) Gait Recognition from MoCap Data May 2nd, / 26

13 The Class Discovery Problem Identity: Content of a discovered identity class = movement history. You are how you walk. crime present time Michal Balazia FI MU) Gait Recognition from MoCap Data May 2nd, / 26

14 Universal Gait Features Data need to be acquired without walker s consent New identities can appear on the fly Labels for all encountered people may not always be available Universal gait features features of a high power in recognizing all people and not only those they were learned on We learn the universal gait features by MMC or by PCA+LDA on an auxiliary dataset The dataset is assumed to be rich on covariate conditions aspects of walk people differ in These features create an unsupervised environment particularly suitable for uncooperative person identification Michal Balazia FI MU) Gait Recognition from MoCap Data May 2nd, / 26

15 Evaluation: Database ASF/AMC format of MoCap data CMU MoCap database obtained from the CMU Graphics Lab Extracted database contains only gait cycles motions of two steps) Normalization: position, walk direction and skeleton 7 extracted databases: # identities # gait cycles , , ,923 Michal Balazia FI MU) Gait Recognition from MoCap Data May 2nd, / 26

16 Evaluation: Data Separation Data separation Samples Samples Learning Identities Learning Evaluation Identities Evaluation Normal evaluation Cross-identity evaluation Evaluation of classification estimated by nested cross-validation Michal Balazia FI MU) Gait Recognition from MoCap Data May 2nd, / 26

17 Evaluation: Metrics Class separability coefficients: Davies-Bouldin Index DBI) Dunn Index DI) Silhouette Coefficient SC) Fisher s Discriminant Ratio FDR) Classification based metrics: Cumulative Match Characteristic False Accept / Reject Rate Receiver Operating Characteristic ROC) Recall / Precision Rate Correct Classification Rate CCR) Equal Error Rate EER) Area Under ROC Curve AUC) Mean Average Precision MAP) Michal Balazia FI MU) Gait Recognition from MoCap Data May 2nd, / 26

18 Evaluation: Results method class separability coefficients classification based metrics scalability DBI DI SC FDR CCR EER AUC MAP DCT TD Ahmed Ali Andersson Ball Dikovski Gavrilova ,254 Jiang Krzeszowski ,795 Kumar ,950 Kwolek Preis 1, Sedmidubsky Sinha MMC BR MMC JC PCALDA BR PCALDA JC Random Raw BR ,229 Raw JC ,574 Michal Balazia FI MU) Gait Recognition from MoCap Data May 2nd, / 26

19 Evaluation: Results Michal Balazia FI MU) Gait Recognition from MoCap Data May 2nd, / 26

20 Evaluation: Results Homogeneous set-up with C L = C E {2,..., 27} Heterogeneous set-up with C L = C E {2,..., 27} % 100% 80%,2) 3,3) 4,4) 5,5) 6,6) 7,7) 8,8) 9,9) 0,10) 1,11) 2,12) 3,13) 4,14) 5,15) 6,16) 7,17) 8,18) 9,19) 0,20) 1,21) 2,22) 3,23) 4,24) 5,25) 6,26) 7,27) DBI homogeneous DBI heterogeneous CCR homogenous CCR heterogeneous 60% 40% DBI CCR 20% 0%,2) 3,3) 4,4) 5,5) 6,6) 7,7) 8,8) 9,9) 0,10) 1,11) 2,12) 3,13) 4,14) 5,15) 6,16) 7,17) 8,18) 9,19) 0,20) 1,21) 2,22) 3,23) 4,24) 5,25) 6,26) 7,27) Michal Balazia FI MU) Gait Recognition from MoCap Data May 2nd, / 26

21 Evaluation: Results Heterogeneous set-up with C L {2,..., 27} and C E = 27 Heterogeneous set-up with C L {2,..., 52} and C E = 54 C L ,27) 3,27) 4,27) 5,27) 6,27) 7,27) 8,27) 9,27) 0,27) 1,27) 2,27) 3,27) 4,27) 5,27) 6,27) 7,27) 8,27) 9,27) 0,27) 1,27) 2,27) 3,27) 4,27) 5,27) 6,27) 7,27) DBI CCR,52) 3,51) 4,50) 5,49) 6,48) 7,47) 8,46) 9,45) 0,44) 1,43) 2,42) 3,41) 4,40) 5,39) 6,38) 7,37) 8,36) 9,35) 0,34) 1,33) 2,32) 3,31) 4,30) 5,29) 6,28) 7,27) 8,26) 9,25) 30,24) 31,23) 32,22) 33,21) 34,20) 35,19) 36,18) 37,17) 38,16) 39,15) 40,14) 41,13) 42,12) 43,11) 44,10) 45,9) 46,8) 47,7) 48,6) 49,5) 50,4) 51,3) 52,2) Michal Balazia FI MU) Gait Recognition from MoCap Data May 2nd, / 26

22 Table 2. Classification metrics for all implemented methods. Homogeneous setup, evaluated on the full Evaluation: database with 10-foldResults cross-validation. Methods are ordered by their CCR score. method CCR EER AUC MAP CMC Raw JC RawJC) 1.00 MMC 0.95 MMCBR) 0.95 BR Raw BR RawBR) MMC JC MMCJC) PCA+LDA BR PCA+LDABR) KwolekB KwolekB 0.70 KrzeszowskiT KrzeszowskiT 0.65 PCA+LDA JC PCA+LDAJC) 0.60 DikovskiB DikovskiB 0.55 AhmedF AhmedF AnderssonVO AnderssonVO 0.40 NareshKumarMS NareshKumarMS 0.35 JiangS JiangS 0.30 BallA BallA0.25 SinhaA SinhaA AhmedM AhmedM 0.15 SedmidubskyJ SedmidubskyJ 0.10 AliS AliS 0.05 PreisJ PreisJ Random Random Rank Michal Balazia FI MU) Gait Recognition from MoCap Data May 2nd, / 26 rect Classification Rate Cumulative Corr

23 Evaluation: Results DCT 0.5 CONFIDENTIAL REVIEW COPY. DO NOT DISTRIBUTE. 0 IJCB DBI ROC ,62),62) 3,61) 3,61) 4,60) 4,60) 5,59) 5,59) 6,58) 6,58) 7,57) 7,57) 8,56) 8,56) 9,55) 9,55) 0,54) 0,54) 1,53) 1,53) Ahmed Andersson Dikovski Preis MMC Ali Ball Kwolek Sinha PCA+LDA 2,52) 2,52) 3,51) 3,51) 4,50) 4,50) 5,49) 6,48) 7,47) a) DBI 5,49) 6,48) 7,47) 8,46) 8,46) 9,45) 9,45) 0,44) 0,44) 1,43) 1,43) 2,42) 2,42) 3,41) 3,41) 4,40) 4,40) 5,39) 5,39) 6,38) 6,38) 7,37) 7,37) 8,36) 8,36) 9,35) 9,35) 30,34) 30,34) 31,33) 31,33) 32,32) 32,32) SC PR 0.1 Ahmed 0.0 Ali -0.1 Andersson -0.2 Ball -0.3 Dikovski -0.4 Kwolek Preis -0.5 Sinha -0.6 MMC 0.40 PCA+LDA Ahmed 48 Ali 49 Andersson 49 Ball Dikovski 49 Kwolek 49 Preis 49 Sinha 49 MMC 49 PCA+LDA Ahmed 50 Ali 50 Andersson Ball 50 Dikovski 50 Kwolek 50 Preis 50 Sinha MMC 50 PCA+LDA50 c) AUC d) MAP Figure 3: Simulations with 31 different C L, C E ) configurations horizontal axes) on four evaluation metrics vertical axes) Michal Balazia FI MU) Gait Recognition from MoCap Data May 2nd, DBI23 / 26,62) 0.35 Ahmed 3,61) Ali 0.30 Andersson 0.25 Ball 0.20 Dikovski 0.15 Kwolek 0.10 Preis 0.05 Sinha 0.00 MMC,62) 3,61) PCA+LDA 4,60) 4,60) 5,59) 5,59) 6,58) 6,58) 7,57) 7,57) 8,56) 8,56) 9,55) 9,55) 0,54) 0,54) 1,53) 1,53) 2,52) 2,52) 3,51) 3,51) 4,50) 4,50) 5,49) 6,48) 7,47) b) SC 5,49) 6,48) 7,47) 8,46) 8,46) 9,45) 9,45) 0,44) 0,44) 1,43) 1,43) 2,42) 2,42) 3,41) 3,41) 4,40) 4,40) 5,39) 5,39) 6,38) 6,38) 7,37) 7,37) 8,36) 8,36) 9,35) 9,35) 30,34) 30,34) 31,33) 31,33) 32,32) 32,32)

24 c) AUC Evaluation: Results % score % score root lhipjoint rhipjoint lfemur rfemur ltibia rtibia lfoot rfoot ltoes rtoes lclavicle rclavicle lhumerus rhumerus lradius rradius lwrist rwrist lhand rhand lfingers rfingers lthumb rthumb lowerback upperback thorax lowerneck upperneck head pelvis torso left arm right arm arms left leg right leg legs left limbs right limbs limbs all but torso all but arms all but legs Figure 4: Four evaluated metrics of the MMC method with incomplete data. In each column a subset of joints is systematically 0.5 excluded from the input: first 31 columns root head) exclude excluded a single joint and last 14 columns pelvis all but legs) exclude multiple joints. 0 Structure of the human body is the following: head, pelvis = {root, lhipjoint, rhipjoint}, left leg = {lfemur, ltibia, lfoot, ltoes}, left arm = {lhumerus, lradius, lwrist, lhand, lfingers, lthumb}, right leg = {rfemur, rtibia, Ahmed rfoot, rtoes}, Andersson right arm = Dikovski {rhumerus, rradius, Preis rwrist, rhand, MMC rfingers, rthumb}, torso = {lowerback, upperback, Ali thorax, Ball lowerneck, upperneck, Kwolek lclavicle, Sinha rclavicle}. Configuration PCA+LDA 9, 55). DCT d) MAP Figure 3: Simulations with 31 different C L, C E ) configurations horizontal axes) on four evaluation metrics vertical axes). ROC root lhipjoint rhipjoint lfemur rfemur ltibia rtibia lfoot rfoot ltoes rtoes lclavicle rclavicle lhumerus rhumerus lradius rradius lwrist rwrist lhand rhand lfingers rfingers lthumb rthumb lowerback upperback thorax lowerneck upperneck head pelvis torso left arm right arm arms left leg right leg legs left limbs right limbs limbs all but torso all but arms all but legs % noise PR % noise a) ROC and PR for an x% noise simulated by multiplying each measured value by a random number in interval x /100, 1 + x /100). excluded ROC 0.80 Ahmed 0.75 Ali 0.70Andersson Ball 0.65 Dikovski 0.60Kwolek Preis 0.55 Sinha 0.50MMC PCA+LDA % noise PR DBI SC 0.30 Ahmed 0.25 Ali 0.20Andersson Ball 0.15 Dikovski 0.10Kwolek Preis 0.05 Sinha 0.00MMC PCA+LDA % noise b) ROC and PR for an x% noise simulated by substituting each measured value with a random value with probability x%. Figure 5: ROC and PR of all methods with noisy data. Configuration 9, 55). Michal Balazia FI MU) Gait Recognition from MoCap Data May 2nd, / 26 ROC DBI PRSC ROC PR Ahm Ali Ande Ball Diko Kwol Preis Sinha MMC PCA+

25 Evaluation Framework and Database Available online at Database extraction drive Implementations of all 20 methods Classifier learning and classification mechanism Evaluation mechanism and 12 performance metrics Michal Balazia FI MU) Gait Recognition from MoCap Data May 2nd, / 26

26 Summary Universal gait features learned on an auxiliary dataset Our approach based on MMC and PCA+LDA Broad evaluation on normal and cross-identity setups MMC learned on 17 identities recognizes 37 identities with 95% CCR MMC learned yet on 7 identities best recognizes 57 identities Evaluation framework and database Thank you for attention. Questions? Michal Balazia FI MU) Gait Recognition from MoCap Data May 2nd, / 26