Action Recognition on Distributed Body Sensor Networks via Sparse Representation

Transcription

1 Action Recognition on Distributed Body Sensor Networks via Sparse Representation Allen Y Yang <yang@eecsberkeleyedu> June 28, 2008 CVPR4HB

2 New Paradigm of Distributed Pattern Recognition Centralized Recognition Distributed Recognition powerful processors (virtually) unlimited memory (virtually) unlimited bandwidth observations in controlled environment simple sensor management mobile processors limited onboard memory band-limited communications high-percentage of occlusion/outliers complex sensor networks

3 New Paradigm of Distributed Pattern Recognition Centralized Recognition Distributed Recognition powerful processors (virtually) unlimited memory (virtually) unlimited bandwidth observations in controlled environment simple sensor management mobile processors limited onboard memory band-limited communications high-percentage of occlusion/outliers complex sensor networks Main constraints for sensor network applications 1 Conserve power consumption 2 Adapt to network configuration changes

4 d-war: Distributed Wearable Action Recognition Architecture Eight sensors on human body Locations are given and fixed Each sensor carries triaxial accelerometer and biaxial gyroscope Sampling frequency: 20Hz Figure: Readings from 8 x-axis accelerometers and x-axis gyroscopes for a stand-kneel-stand sequence

5 Challenges 1 Simultaneous segmentation and classification

6 Challenges 1 Simultaneous segmentation and classification 2 Individual sensors not sufficient to classify full-body motions Single sensors on the upper body can not recognize lower body motions Vice Versa

7 Challenges 1 Simultaneous segmentation and classification 2 Individual sensors not sufficient to classify full-body motions Single sensors on the upper body can not recognize lower body motions Vice Versa 3 Simulate sensor failure and network congestion by different subsets of active sensors

8 Challenges 1 Simultaneous segmentation and classification 2 Individual sensors not sufficient to classify full-body motions Single sensors on the upper body can not recognize lower body motions Vice Versa 3 Simulate sensor failure and network congestion by different subsets of active sensors 4 Identity independence: Figure: Same actions performed by two subjects

9 Outline 1 Overview: Classification framework via l 1 -minimization 2 Individual sensor: Obtains limited classification ability and becomes active only when certain events are locally detected 3 Global classifier: Adapts to the change of active sensors in network and boost multi-sensor recognition

10 Sparse Representation Sparsity A signal is sparse if most of its coefficients are (approximately) zero

11 Sparse Representation Sparsity A signal is sparse if most of its coefficients are (approximately) zero 1 Sparsity in frequency domain 2 Sparsity in spatial domain Figure: 2-D DCT transform Figure: Gene microarray data

12 Sparsity in human visual cortex [Olshausen & Field 1997, Serre & Poggio 2006]

13 Sparsity in human visual cortex [Olshausen & Field 1997, Serre & Poggio 2006] 1 Feed-forward: No iterative feedback loop 2 Redundancy: Average neurons for each feature representation 3 Recognition: Information exchange between stages is not about individual neurons, but rather how many neurons as a group fire together

14 Sparsity and l 1 -Minimization 1 Black gold age [Claerbout & Muir 1973, Taylor, Banks & McCoy 1979] Figure: Deconvolution of spike train

15 Sparsity and l 1 -Minimization 1 Black gold age [Claerbout & Muir 1973, Taylor, Banks & McCoy 1979] Figure: Deconvolution of spike train 2 Basis pursuit [Chen & Donoho 1999]: Given y = Ax and x unknown, x = arg min x 1, subject to y = Ax 3 The Lasso (least absolute shrinkage and selection operator) [Tibshirani 1996] x = arg min x 1, subject to y Ax 2 ɛ

16 Sparse Representation Encodes Membership 1 Notation Training: For K classes, collect training samples {v 1,1,, v 1,n1 },, {v K,1,, v K,nK } R D Test: Present a new y R D, solve for label(y) [1, 2,, K]

17 Sparse Representation Encodes Membership 1 Notation Training: For K classes, collect training samples {v 1,1,, v 1,n1 },, {v K,1,, v K,nK } R D Test: Present a new y R D, solve for label(y) [1, 2,, K] 2 Subspace model: Assume y belongs to Class i: where A i = [v i,1, v i,2,, v i,ni ] y = α i,1 v i,1 + α i,2 v i,2 + + α i,n1 v i,ni, = A i α i,

18 Sparse Representation Encodes Membership 1 Notation Training: For K classes, collect training samples {v 1,1,, v 1,n1 },, {v K,1,, v K,nK } R D Test: Present a new y R D, solve for label(y) [1, 2,, K] 2 Subspace model: Assume y belongs to Class i: where A i = [v i,1, v i,2,, v i,ni ] y = α i,1 v i,1 + α i,2 v i,2 + + α i,n1 v i,ni, = A i α i, 3 Nevertheless, Class i is the unknown variable we need to solve: α 1 α 2 Sparse representation y = [A 1, A 2,, A K ] = Ax α K

19 Sparse Representation Encodes Membership 1 Notation Training: For K classes, collect training samples {v 1,1,, v 1,n1 },, {v K,1,, v K,nK } R D Test: Present a new y R D, solve for label(y) [1, 2,, K] 2 Subspace model: Assume y belongs to Class i: where A i = [v i,1, v i,2,, v i,ni ] y = α i,1 v i,1 + α i,2 v i,2 + + α i,n1 v i,ni, = A i α i, 3 Nevertheless, Class i is the unknown variable we need to solve: α 1 α 2 Sparse representation y = [A 1, A 2,, A K ] = Ax α K Sparse representation encodes membership! x 0 = [ 0 0 α T i 0 0 ] T R n

20 Distributed Action Recognition 1 Training samples: manually segment and normalize to duration h

21 Distributed Action Recognition 1 Training samples: manually segment and normalize to duration h 2 On each sensor node i, stack training actions into vector form v i = [x(1),, x(h), y(1),, y(h), z(1),, z(h), θ(1),, θ(h), ρ(1),, ρ(h)] T R 5h

22 Distributed Action Recognition 1 Training samples: manually segment and normalize to duration h 2 On each sensor node i, stack training actions into vector form v i = [x(1),, x(h), y(1),, y(h), z(1),, z(h), θ(1),, θ(h), ρ(1),, ρ(h)] T R 5h 3 Full body motion Training sample: v = ( v1 v 8 ) Test sample: y = y 1 R 8 5h y 8

23 Distributed Action Recognition 1 Training samples: manually segment and normalize to duration h 2 On each sensor node i, stack training actions into vector form v i = [x(1),, x(h), y(1),, y(h), z(1),, z(h), θ(1),, θ(h), ρ(1),, ρ(h)] T R 5h 3 Full body motion Training sample: v = ( v1 v 8 ) Test sample: y = y 1 R 8 5h y 8 4 Action subspace: If y is from Class i, y = y 1 ( v1 = α i,1 y 8 v8 ) α i,ni ( v1 v8 ) n i = A i α i

24 Sparse Representation 1 Nevertheless, label(y) = i is the unknown membership function to solve: α 1 Sparse Representation: y = [ ] α 2 A 1 A 2 A K = Ax R8 5h α K

25 Sparse Representation 1 Nevertheless, label(y) = i is the unknown membership function to solve: α 1 Sparse Representation: y = [ ] α 2 A 1 A 2 A K = Ax R8 5h α K 2 One solution: x = [0, 0,, α T i, 0,, 0] T Sparse representation encodes membership

26 Sparse Representation 1 Nevertheless, label(y) = i is the unknown membership function to solve: α 1 Sparse Representation: y = [ ] α 2 A 1 A 2 A K = Ax R8 5h α K 2 One solution: x = [0, 0,, α T i, 0,, 0] T Sparse representation encodes membership 3 Two problems: Directly solving the linear system is intractable Seeking the sparsest solution

27 Dimensionality Reduction 1 Construct Fisher/LDA features R i R 10 5h on each node i: ỹ i = R i y i = R i A i x = Ã i x R 10

28 Dimensionality Reduction 1 Construct Fisher/LDA features R i R 10 5h on each node i: ỹ i = R i y i = R i A i x = Ã i x R 10 2 Globally ỹ 1 = ỹ 8 R R 8 y 1 y 8 = RAx = Ãx R8 10

29 Dimensionality Reduction 1 Construct Fisher/LDA features R i R 10 5h on each node i: ỹ i = R i y i = R i A i x = Ã i x R 10 2 Globally ỹ 1 = ỹ 8 R R 8 y 1 y 8 = RAx = Ãx R During the transformation, the data matrix A and x remain unchanged

30 Seeking Sparsest Solution: l 1 -Minimization 1 Ideal solution: l 0 -minimization (P 0 ) x = arg min x 0 st ỹ = Ãx x where 0 simply counts the number of nonzero terms However, such solution is generally NP-hard

31 Seeking Sparsest Solution: l 1 -Minimization 1 Ideal solution: l 0 -minimization (P 0 ) x = arg min x 0 st ỹ = Ãx x where 0 simply counts the number of nonzero terms However, such solution is generally NP-hard 2 Compressed sensing: under mild condition, equivalence relation (P 1 ) x = arg min x 1 st ỹ = Ãx x

32 Seeking Sparsest Solution: l 1 -Minimization 1 Ideal solution: l 0 -minimization (P 0 ) x = arg min x 0 st ỹ = Ãx x where 0 simply counts the number of nonzero terms However, such solution is generally NP-hard 2 Compressed sensing: under mild condition, equivalence relation (P 1 ) x = arg min x 1 st ỹ = Ãx x 3 l 1 -Ball l 1 -Minimization is convex Solution equal to l 0 -minimization Reference: Robust face recognition via sparse representation (in press) PAMI, 2008

33 Distributed Recognition Framework 1 Distributed Sparse Representation ỹ 1 v1 ) ( v1 ) = R (,, x ỹ 8 v 8 1 v 8 n ỹ 1 =R 1(v 1,1,,v 1,n)x ỹ 8 =R 8(v 8,1,,v 8,n)x

34 Distributed Recognition Framework 1 Distributed Sparse Representation ỹ 1 v1 ) ( v1 ) = R (,, x ỹ 8 v 8 1 v 8 n ỹ 1 =R 1(v 1,1,,v 1,n)x ỹ 8 =R 8(v 8,1,,v 8,n)x 2 Local classifier: For each note i, solve ỹ i = R i ( vi,1,, v i,n ) x R 10

35 Distributed Recognition Framework 1 Distributed Sparse Representation ỹ 1 v1 ) ( v1 ) = R (,, x ỹ 8 v 8 1 v 8 n ỹ 1 =R 1(v 1,1,,v 1,n)x ỹ 8 =R 8(v 8,1,,v 8,n)x 2 Local classifier: For each note i, solve ỹ i = R i ( vi,1,, v i,n ) x R 10 3 Adaptive global classifier: Suppose only first L sensors are available (L 8): ỹ 1 R 1 0 v 1 v 1 =,, x R 10L ỹ L 0 R L v L 1 v L n

36 Distributed Recognition Framework 1 Distributed Sparse Representation ỹ 1 v1 ) ( v1 ) = R (,, x ỹ 8 v 8 1 v 8 n ỹ 1 =R 1(v 1,1,,v 1,n)x ỹ 8 =R 8(v 8,1,,v 8,n)x 2 Local classifier: For each note i, solve ỹ i = R i ( vi,1,, v i,n ) x R 10 3 Adaptive global classifier: Suppose only first L sensors are available (L 8): ỹ 1 R 1 0 v 1 v 1 =,, x R 10L ỹ L 0 R L v L 1 v L n 4 The representation x and training matrix A remain invariant

37 Rejection of Invalid Actions (Segmentation) 1 Multiple segmentation hypothesis

38 Rejection of Invalid Actions (Segmentation) 1 Multiple segmentation hypothesis 2 Outlier rejection

39 Rejection of Invalid Actions (Segmentation) 1 Multiple segmentation hypothesis 2 Outlier rejection Outlier Rejection When l 1 -solution is not sparse or concentrated to one subspace, the test sample is invalid Sparsity Concentration Index: SCI(x) = K max i δ i (x) 1 / x 1 1 K 1 [0, 1]

40 Experiment Precision vs Recall: Sensors 2 7 2,7 1,2,7 1-3, 7,8 1-8 Prec [%] Rec [%]

41 Experiment Precision vs Recall: Sensors 2 7 2,7 1,2,7 1-3, 7,8 1-8 Prec [%] Rec [%] Segmentation results using all 8 sensors: (a) Stand-Sit-Stand (b) Sit-Lie-Sit (c) Rotate-Left (d) Go-Downstairs

42 Confusion Tables (a) 1-8 (b) 7

43 Conclusion 1 Mixture subspace model: 10-D action spaces suffice for data communication

44 Conclusion 1 Mixture subspace model: 10-D action spaces suffice for data communication 2 Accurate recognition via sparse representation Full-body network: 99% accuracy with 95% recall Keep one on upper body and one on lower body: 94% accuracy and 82% recall Reduce to single sensors: 90% accuracy

45 Conclusion 1 Mixture subspace model: 10-D action spaces suffice for data communication 2 Accurate recognition via sparse representation Full-body network: 99% accuracy with 95% recall Keep one on upper body and one on lower body: 94% accuracy and 82% recall Reduce to single sensors: 90% accuracy 3 Applications Beyond Wii controllers and iphones

46 Conclusion 1 Mixture subspace model: 10-D action spaces suffice for data communication 2 Accurate recognition via sparse representation Full-body network: 99% accuracy with 95% recall Keep one on upper body and one on lower body: 94% accuracy and 82% recall Reduce to single sensors: 90% accuracy 3 Applications Beyond Wii controllers and iphones Eldertech: falling detection, mobility monitoring

47 Conclusion 1 Mixture subspace model: 10-D action spaces suffice for data communication 2 Accurate recognition via sparse representation Full-body network: 99% accuracy with 95% recall Keep one on upper body and one on lower body: 94% accuracy and 82% recall Reduce to single sensors: 90% accuracy 3 Applications Beyond Wii controllers and iphones Eldertech: falling detection, mobility monitoring Energy Expenditure: lifestyle-related chronic diseases

48

49 Acknowledgments Collaborators Berkeley: Ruzena Bajcsy, Shankar Sastry, Sameer Iyengar Cornell: Philip Kuryloski UT-Dallas: Roozbeh Jafari NSF TRUST Center Funding Support ARO MURI: Heterogeneous Sensor Networks (HSN) Telecom Italia Lab at Berkeley References Robust face recognition via sparse representation (in press) PAMI, 2008 Donoho For most large underdetermined systems of equations, the minimal l 1 -norm near-solution approximates the sparsest near-solution 2004 Candes Compressive sampling 2006 Donoho Neighborly polytopes and sparse solution of underdetermined linear equations 2004