Diverse M-Best Solutions in Markov Random Fields

Transcription

1 Dverse M-Best Solutons n Markov Random Felds Dhruv Batra Vrgna Tech Jont work wth: Students: Payman Yadollahpour (TTIC), Abner Guzman-Rvera (UIUC) Colleagues: Chrs Dyer (CMU), Greg Shakhnarovch (TTIC), Pushmeet Kohl (MSRC), Kevn Gmpel (TTIC)

2 (C) Dhruv Batra 2

3 Ambguty Ambguty Ambguty?? One nstance / Two nstances? (C) Dhruv Batra 3

4 Problems wth MAP Sngle Not Model-Class Predcton Enough Tranng s = Uncertanty Wrong! Data! Msmanagement -- Inherent MAP Approxmaton s NP-Hard Error -- Estmaton Error Ambguty -- Optmzaton Error Make -- Bayes Multple Error Predctons! (C) Dhruv Batra 4

5 Multple Predctons P (X ) x x x x X X MAP Samplng Porway & Zhu, 211! TU & Zhu, 22! Rch Hstory! (C) Dhruv Batra 5

6 Multple Predctons P (X ) X X MAP Samplng Porway & Zhu, 211! TU & Zhu, 22! Rch Hstory! M-Best MAP Flerova et al., 211! Fromer et al., 29! Yanover et al., 23!! Ideally: M-Best Modes (C) Dhruv Batra 6

7 Multple Predctons P (X ) X X MAP Our work: Dverse M-Best n MRFs [ECCV 12]! Samplng M-Best MAP - Don t hope for dversty. Explctly encode t. Ideally: Porway & Zhu, 211! Flerova et al., 211! TU & Zhu, 22! - Not guaranteed Fromer et to al., be 29! modes. M-Best Modes Rch Hstory! Yanover et al., 23! (C) Dhruv Batra 7

8 Example Result CRF Dverse Segmentatons. Re-ranker α ψ(x, y) Re-ranked Lst Top Soluton (C) Dhruv Batra 8

9 Example Result CRF Dverse Segmentatons Stage 1. Dscrmnatve Re-rankng of Dverse Segmentaton Re-ranker Re-ranked Lst α ψ(x, y) [Yadollahpour et al., CVPR13, Wednesday Poster] Stage 2 Top Soluton. (C) Dhruv Batra 9

10 MAP Integer Program max y S(y) = µ θ (s) (s) θ (y ) + θ j (y,y j ) (,j) kx1 (C) Dhruv Batra 1

11 MAP Integer Program max y S(y) = θ (s) + θ j (y,y j ) (,j) 1 kx1 (C) Dhruv Batra 11

12 MAP Integer Program max y S(y) = 1 θ (s) + θ j (y,y j ) (,j) kx1 (C) Dhruv Batra 12

13 MAP Integer Program max y S(y) = θ (s) + θ j (y,y j ) (,j) 1 kx1 (C) Dhruv Batra 13

14 MAP Integer Program max y S(y) = θ (s) + θ j (y,y j ) (,j) 1 kx1 (C) Dhruv Batra 14

15 MAP Integer Program max y S(y) = θ (s) 1 kx1 + (,j) θ j (s, t) µ j (s, t) k 2 x1 (C) Dhruv Batra 15

16 MAP Integer Program max y S(y) = max µ C θ (s) θ µ + θ j µ j (,j) s.t. µ ( ), µ j ( ) {, 1} 1 kx1 + (,j) θ j (s, t) µ j (s, t) k 2 x1 P (X ) X MAP (C) Dhruv Batra 16 X

17 MAP Integer Program max µ C θ µ + θ j µ j (,j) s.t. µ ( ), µ j ( ) {, 1} Graphcuts, BP, Expanson, etc P (X ) X MAP (C) Dhruv Batra 17 X

18 Dverse 2 nd -Best max µ C θ µ + θ j µ j (,j) s.t. µ ( ), µ j ( ) {, 1} (µ, µ (1) ) k Dversty MAP (µ, µ (1) ) k P (X ) X MAP (C) Dhruv Batra 18 X

19 Dverse M-Best max µ C θ µ + θ j µ j (,j) s.t. µ ( ), µ j ( ) {, 1} (µ, µ (1) ) k (µ, µ (2) ) k (µ, µ (M 1) ) k (µ, µ (1) ) k (µ, µ (M 1) ) k (µ, µ (2) ) k P (X ) X MAP (C) Dhruv Batra 19 X

20 Dverse 2 nd -Best max µ C θ µ + θ j µ j (,j) s.t. µ ( ), µ j ( ) {, 1} (µ, µ (1) ) k Q1: How do we solve DvMBest? Q2: What knd of dversty functons are allowed? Q3: How much dversty? (C) Dhruv Batra 2

21 Dverse 2 nd -Best Prmal max µ C θ µ + θ j µ j (,j) s.t. µ ( ), µ j ( ) {, 1} (µ, µ (1) ) k Dversty-Augmented Score +λ (µ, µ (1) ) k Dualze S(y) + Dv(y, y (1) ) P (X ) X X MAP (C) Dhruv Batra 21

22 Dverse 2 nd -Best Lagrangan Relaxaton f(λ) = max µ C θ µ + θ j µ j (,j) s.t. µ ( ), µ j ( ) {, 1} Dversty-Augmented Score +λ (µ, µ (1) ) k Dual f(λ) mn λ f(λ) f(λ) = (µ λ, µ (1) ) k Subgradent Descent f(λ () ) f(λ (1) ) Concave (Non-smooth) Upper-Bound on Dv2Best Score Dv2Best score λ () λ (1) (C) Dhruv Batra 22 λ

23 Dverse 2 nd -Best Lagrangan Relaxaton Many ways to solve: Dversty-Augmented Energy mn θ µ + θ j µ λ 1. f(λ) Subgradent = j µ CAscent. Optmal. Slow. (,j) (µ, µ (1) ) k 2. Bnary Search. s.t. µ ( ), µ j ( ) Optmal {, 1} for M=2. Faster. (µ, µ (1) ) k 3. Grd-search on lambda. Sub-optmal. Fastest. Dualze (C) Dhruv Batra 23

24 Theorem Statement Theorem [Batra et al 12]: Lagrangan Dual corresponds to solvng the Relaxed Prmal: Based on result from [Geoffron 74] Dual mn λ LagranganDual(λ) Relaxed Prmal max µ s.t. θ µ + θ j µ j j µ Co µ ( ), µ j ( ) {, 1} µ C (µ, µ (1) ) k (C) Dhruv Batra 24

25 Effect of Lagrangan Relaxaton µ (5) µ (3) µ (4) µ (2) µ (1) (C) Dhruv Batra 25

26 Effect of Lagrangan Relaxaton µ (5) µ (3) µ (4) µ (2) µ (1) (C) Dhruv Batra 26

27 Effect of Lagrangan Relaxaton [Mezuman et al. UAI13] Parwse Potental Strength Parwse Potental Strength (C) Dhruv Batra 27

28 Dverse 2 nd -Best mn µ C θ µ + θ j µ j (,j) s.t. µ ( ), µ j ( ) {, 1} (µ, µ (1) ) k Q1: How do we solve DvMBest? Q2: What knd of dversty functons are allowed? Q3: How much dversty? (C) Dhruv Batra 28

29 Dversty [Specal Case] -1 Dversty = [Yanover NIPS3; Fromer NIPS9; Flerova Soft11] M-Best MAP [Specal Case] Max Dversty = [Park & Ramanan ICCV11] Hammng Dversty Cardnalty Dversty Any Dversty max µ C S(µ)+λ (µ, µ (1) ) (C) Dhruv Batra 29

30 Hammng Dversty (µ, µ (1) )= V µ µ (1) 1 1 =1 1 1 = (C) Dhruv Batra 3

31 Hammng Dversty (µ, µ (1) )= V µ µ (1) Dversty Augmented Inference: max µ C θ µ + θ j µ j + λ (µ, µ (1) ) (,j) = θ λµ (1) µ + θ j µ j (,j) θ (C) Dhruv Batra 31

32 Hammng Dversty (µ, µ (1) )= V µ µ (1) Dversty Augmented Inference: for = 1,2,...,n θ [x (1) ] -= λ endfor x (2) = Fnd MAP(θ, θ j ) Unchanged. Can stll use graph-cuts! Smply edt node-terms. Reuse MAP machnery! (C) Dhruv Batra 32

33 Dverse 2 nd -Best mn µ C θ µ + θ j µ j (,j) s.t. µ ( ), µ j ( ) {, 1} (µ, µ (1) ) k Q1: How do we solve DvMBest? Q2: What knd of dversty functons are allowed? Q3: How much dversty? (C) Dhruv Batra 33

34 How Much Dversty? P (X ) X Emprcal Soluton: Cross-Val for k More Effcent: Cross-Val for λ (C) Dhruv Batra 34

35 Experments 3 Applcatons Interactve Segmentaton: Hammng, Cardnalty (n paper) Pose Estmaton: Hammng Semantc Segmentaton: Hammng Baselnes: M-Best MAP (No Dversty) Confdence-Based Perturbaton (No Optmzaton) (C) Dhruv Batra 35

36 Interactve Segmentaton Setup Model: Color/Texture + Potts Grd CRF Inference: Graph-cuts Dataset: 5 tran/val/test mages Image + Scrbbles MAP 2 nd Best MAP Dverse 2 nd Best 1-2 Nodes Flpped 1-5 Nodes Flpped (C) Dhruv Batra 36

37 Pose Trackng Setup Model: Mxture of Parts from [Park & Ramanan, ICCV 11] Inference: Dynamc Programmng Dataset: 4 vdeos, 585 frames (C) Dhruv Batra Image Credt: [Yang & Ramanan, ICCV 11] 37

38 CVPR 213 Dversty Tutoral (C) Dhruv Batra 38

39 Pose Trackng Chan CRF wth M states at each tme M Best Solutons (C) Dhruv Batra Image Credt: [Yang & Ramanan, ICCV 11] 39

40 Pose Trackng MAP DvMBest + Vterb (C) Dhruv Batra 4

41 Pose Trackng Better PCP Accuracy 85% 8% 75% 7% 65% 6% 55% DvMBest (Re-ranked) 13% Gan Same Features Same Model [Park & Ramanan, ICCV 11] (Re-ranked) Confdence-based Perturbaton (Re-ranked) 5% 45% #Solutons / Frame (C) Dhruv Batra 41

42 Machne Translaton Input: De Regerung wll de Folter von Hexen unterbnden und gab ene Broschüre heraus MAP Translaton: The government wants the torture of wtch and gave out a booklet (C) Dhruv Batra 42

43 Machne Translaton Input: De Regerung wll de Folter von Hexen unterbnden und gab ene Broschüre heraus 5-Best Translatons: The government wants the torture of wtch and gave out a booklet The government wants the torture of wtch and gave out a booklet The government wants the torture of wtch and gave out a brochure The government wants the torture of wtch and gave out a leaflet The government wants the torture of wtch and gave out a brochure (C) Dhruv Batra 43

44 Machne Translaton Input: De Regerung wll de Folter von Hexen unterbnden und gab ene Broschüre heraus Dverse 5-Best Translatons: The government wants the torture of wtch and gave out a booklet The government wants to stop torture of wtch and ssued a leaflet ssued The government wants to stop the torture of wtches and gave out a brochure The government ntends to the torture of wtchcraft and were ssued a leaflet The government s the torture of wtches stamp out and gave a brochure (C) Dhruv Batra 44

45 Machne Translaton Input: De Regerung wll de Folter von Hexen unterbnden und gab ene Broschüre heraus Dverse 5-Best Translatons: The government wants the torture of wtch and gave out a booklet The government wants to stop torture of wtch and ssued a leaflet ssued The government wants to stop the torture of wtches and gave out a brochure The government ntends to the torture of wtchcraft and were ssued a leaflet The government s the torture of wtches stamp out and gave a brochure Correct Translaton: The government wants to lmt the torture of wtches, a brochure was released (C) Dhruv Batra 45