Scheduling Top- Down and Bo4om- Up Processes

Transcription

1 Scheduling Top- Down and Bo4om- Up Processes Song- Chun Zhu, Sinisa Todorovic, and Ales Leonardis At CVPR, Providence, Rhode Island June 16,

2 α, β and γ computing processes in AoG The And-Or graph is a recursive structure. So, consider a node A. 1. Node A at a coarse scale terminates to leaf nodes (ground) 2. Node A is connected to the root. EsJmate the α/β/γ channels when they are applicable an opjmal scheduling problem α β γ

3 Example: Human Faces are Computed in 3 Channels α-channel β-channel γ-channel

4 Human faces in real scenarios α-channel β-channel γ-channel

5 Hierarchical modeling and α, β and γ compujng And-node terminate-node γ: p(face parents) Headshoulder 1. Each node has its own α, β and γ computing processes. β: p(face parts) Face α: p(face compact image data) 2. How much does each channel contribute? Left eye Right eye Nose Mouse Wu & Zhu IJCV11

6 Hierarchical modeling and α, β and γ compujng pg l = arg max pg + log " p( (t(^l)) t(^l)) q( (t(^l))) {z } h X N + p(n) log p( log p(^l _ l )+p(n) NX i=1 log p(x(^l+ i ) X(^l)) {z } parse graph connecjvity = detector (t(^l+ )) t(^l+ i i )) q( (t(^l+ i ))) + X i6=j log p(x(^l+ i i ),X(^l+ j )) i=1 {z } = zoom- in Wu & Zhu IJCV11 + log p( (t(^l )) t(^l )) q( (t(^l )) + log p(x(^l) X(^l {z } = zoom- out ))

7 α processes for the face node α-channels: p(face compact image data) Headshoulder Face Left eye Right eye Nose Mouse

8 β processes for the face node(when its α is off) β-channels: p(face parts), binding Headshoulder Face Left eye Right eye Nose Mouse α-channels of some parts are on

9 β processes for the face node(when its α is off) β-channels: p(face parts), binding Headshoulder Face Left eye Right eye Nose Mouse α-channels of some parts are on

10 β processes for the face node(when its α is off) β-channels: p(face parts), binding Headshoulder Face Left eye Right eye Nose Mouse α-channels of some parts are on

11 β processes for the face node(when its α is off) β-channels: p(face parts), binding Headshoulder Face Left eye Right eye Nose Mouse α-channels of some parts are on

12 γ processes for the face node(when it s α and β is off) γ-channels: p(face parents), predicting α-channels of some parents are on Headshoulder Face Left eye Right eye Nose Mouse

13 γ processes for the face node(when it s α and β is off) γ-channels: p(face parents), predicting α-channels of some parents are on Headshoulder Face Left eye Right eye Nose Mouse

14 γ processes for the face node(when it s α and β is off) γ-channels: p(face parents), predicting α-channels of some parents are on Headshoulder Face Left eye Right eye Nose Mouse

15 γ processes for the face node(when it s α and β is off) γ-channels: p(face parents), predicting α-channels of some parents are on Headshoulder Face Left eye Right eye Nose Mouse

16 In general: recursive α, β and γ channels α β γ

17 α-channel: head-shoulder

18 α-channel: head-shoulder

19 α-channel: head-shoulder

20 α-channel: face

21 α-channel: face

22 α-channel: face

23 α-channel: eye

24 α-channel: eye

25 α-channel: eye

26 α-channel: nose

27 α-channel: nose

28 α-channel: nose

29 α-channel: mouth

30 α-channel: mouth

31 α-channel: mouth

32 All α channels keep things and throw away stuffs by integra(ng α, β and γ channels

33 Integrating α, β and γ channels Thresholds

34 Integrating α, β and γ channels

35 Integrating α, β and γ channels

36 Integrating α, β and γ channels

37 Integrating α, β and γ channels

38 Integrating α, β and γ channels

39 Integrating α, β and γ channels

40 Integrating α, β and γ channels

41 Integrating α, β and γ channels

42 Integrating α, β and γ channels

43 Integrating α, β and γ channels

44 Integrating α, β and γ channels

45 Integrating α, β and γ channels

46 Integrating α, β and γ channels

47 Integrating α, β and γ channels

48 Integrating α, β and γ channels

49 Information contribution α channels

50 Performance improvement red for α, blue for α+β, green for α+γ, cyan for α+β+γ channels

51 Cost- SensiJve and Goal- Guided Inference in AoG Song- Chun Zhu, Sinisa Todorovic, and Ales Leonardis At CVPR, Providence, Rhode Island June 16,

52 Problem Given a high- resolujon and long video, showing SpaJally large scenes, Many people engaged in a wide range of individual acjons group acjvijes

53 Problem Goal: Answer WHAT, WHERE, and WHEN queries about individual acjons group acjvijes

54 MulJscale Event Understanding

55 A Principled Framework Needed for Goal- driven zooming- in or zooming- out, Cost- sensi(ve inference via Scheduling of the bo4om- up and top- down inference Any- Jme inference - - adapt to a given Jme budget

56 And- Or Graph and Parse Graph pg l = arg max pg + log " p( (t(^l)) t(^l)) q( (t(^l))) {z } h X N + p(n) log p( log p(^l _ l )+p(n) NX i=1 log p(x(^l+ i ) X(^l)) {z } parse graph connecjvity = detector (t(^l+ )) t(^l+ i i )) q( (t(^l+ i ))) + X i6=j log p(x(^l+ i i ),X(^l+ j )) i=1 {z } = zoom- in + log p( (t(^l )) t(^l )) q( (t(^l )) + log p(x(^l) X(^l {z } = zoom- out ))

57 Inference - - Alpha of Individual AcJon WaiJng

58 Inference - - Alpha of Individual AcJon WaiJng Walking

59 Inference - - Alpha of Individual AcJon WaiJng Walking

60 Inference - - Bo4om Up (Beta) Group Walking Group Discuss

61 Inference Alpha of Group AcJviJes Group Walking Group Discuss

62 Inference - - Top Down (Gamma) Group Walking Group Discuss

63 Inference - - Top Down (Gamma) WaiJng Walking

64 Inference - - Alpha of Objects Stairs Chairs Food bus

65 Inference - - Bo4om Up (Beta) WaiJng Walking

66 Inference WaiJng Walking

67 Queries about Individual AcJons Only Alpha Zoom- out Zoom- in and Zoom- out

68 Scheduling V ( ) Q (s) =E[G(s, a)] + X T(s, Q (s, ),s 0 ) V Q (s 0 ), =1,...,B s 0 States (s) = Query, All previous acjons, Their rewards unjl this point AcJons (a) = ( 1) Run detector and terminate (Alpha) Bo4om- up (Beta) Top- down (Gamma)

69 Policies, Given a Query (Q) 1. Alpha(Q) 2. Alpha(Q), Bo4om- up, Alpha(Context), Top- down 3. Alpha(Q), Top- down, Alpha(Details), Bo4om- up 4. Alpha(Q), Bo4om- up, Alpha(Context), Top- down, Alpha(Details), Bo4om- up 5. Alpha(Q), Top- down, Alpha(Details), Bo4om- up Alpha(Context), Top- down

70 Group Query

71 Group Query Policy #1 Group Walking Group Discuss

72 Group Query Policy #1 Group Walking Group Discuss

73 Group Query Policy #1 WaiJng Walking

74 Group Query Policy #1 Group Walking Group Discuss

75 Group Query Policy #2 Group Walking Group Discuss

76 Group Query Policy #2 Group Walking Group Discuss

77 Group Query Policy #2

78 Group Query Policy #2

79 Group Query Policy #2 Stairs Chairs Food bus

80 Group Query Policy #2 WaiJng Walking

81 Group Query Policy #2 Group Walking Group Discuss

82 Group Query Policy #2 Group Walking Group Discuss

83 Group Query Alpha Policy #1 Policy #2

84 New Dataset Footage: 106min Frames: 12 fps Res.: 2560x1920 pixels Portable acquisijon system VIRAT Benchmark datasets: single actor single group single acjon

85 Domain Knowledge 5 Group AcJviJes: Walking together, Queuing, Campus tour, Individual AcJons: Walking, Siing, Riding a bike, 16 Objects: Food truck, Vending machine, Bike, Backpack,...