Efficient Inference in Fully Connected CRFs with Gaussian Edge Potentials

Transcription

1 Efficient Inference in Fully Connected CRFs with Gaussian Edge Potentials Philipp Krähenbühl and Vladlen Koltun Stanford University Presenter: Yuan-Ting Hu 1

2 Conditional Random Field (CRF) E x I = φ u x i I + φ p (x i, x j I) i i j i Unary term Pairwise term X, I : random fields Application: Image segmentation: achieve state-of-the-art performance (in 2011) 2

3 Image Segmentation Example: semantic image segmentation Input Output 3

4 CRF for Image Segmentation E x I = φ u x i I + φ p (x i, x j I) i Unary term i j i Pairwise term X: a random field defined over a set of variables {X 1,, X N } Label of pixels (grass, bench, tree,..) I: a random field defined over a set of variables I 1,, I N Image (observation) 4

5 CRF for Image Segmentation E(x) = ψ u x i + ψ p (x i, x j ) i Unary term i j i Pairwise term Unary term Trained on dataset 5

6 CRF for Image Segmentation E(x) = ψ u x i + ψ p (x i, x j ) i Unary term i j i Pairwise term Pairwise term Impose consistency of the labeling Defined over neighboring pixels ψ p x i, x j = μ x i, x j K m=1 w m k m (f i, f j ) 6

7 CRF for Image Segmentation Pairwise term ψ p x i, x j = μ x i, x j K m=1 w m k m (f i, f j ) k (m) (f i, f j ) is a Gaussian kernel f i, f j is the feature vectors for pixel i and j, e.g., color intensities, is the weight of the m-th kernel μ x i, x j is the label compatibility function w m E(x) = i ψ u x i Unary term + i j i ψ p (x i, x j ) Pairwise term 7

8 CRF for Image Segmentation E x I = φ u x i I + φ p (x i, x j I) i Unary term i j i Pairwise term Neighboring pixels Local connections May not capture the sharp boundaries 8

9 Grid CRF for Image Segmentation Unary term Local connections May not capture the sharp boundaries 9

10 Fully connected CRF for Image Segmentation Dense CRF Fully connected CRF Every node is connected to every other node MCMC inference, 36 hours!! 10

11 Efficient Inference on Fully connected CRF They propose an efficient approximate algorithm for inference on fully connected CRF Inference in 0.2 seconds ~50,000 nodes (apply to pixel level segmentation) Based on a mean field approximation to the CRF distribution 11

12 Mean Field Approximation Mean field update rule for CRF Q i x i = l K = 1 Z i exp{ ψ u x i μ l, l w m k m f i, f j Q j (l )} l L m=1 j i 12

13 Mean Field Approximation Q i x i = l = 1 Z i exp ψ u x i Algorithm l L μ l, l K m=1 Initialize Q : Q i x i = 1 Z i exp{ φ u (x i )} While not converged w m j i Message passing: Q i m (l) = σ j i k m f i, f j Q j l k m f i, f j Q j l 13

14 Mean Field Approximation Q i x i = l = Algorithm 1 Z i exp ψ u x i l L μ l, l Initialize Q : Q i x i = 1 Z i exp{ φ u (x i )} While not converged Message passing: Q i (m) = σj i k m f i, f j Q j l K m=1 w m Qi m (l) Compatibility transform: Q i x i = σ l L μ l, l σk m=1 w m m Qi (l) 14

15 Mean Field Approximation Algorithm Q i x i = l = 1 Z i exp ψ u x i Initialize Q : Q i x i = 1 Z i exp{ φ u (x i )} While not converged Message passing: Q i (m) = σj i k m f i, f j Q j l Q i x i Compatibility transform: Q i x i = σ l L μ l, l σk m=1 w m m Qi (l) Update to calculate Q i x i = l Normalization 15

16 Mean Field Approximation Algorithm O(N 2 ) O(N) O(N) O(N) Q i x i = l = 1 Z i exp ψ u x i Initialize Q : Q i x i = 1 Z i exp{ φ u (x i )} While not converged Message passing: Q i (m) = σj i k m f i, f j Q j l Q i x i Compatibility transform: Q i x i = σ l L μ l, l σk m=1 w m m Qi (l) Update to calculate Q i x i = l Normalization 16

17 Mean Field Approximation Algorithm O(N 2 ) O(N) O(N) O(N) Q i x i = l = 1 Z i exp ψ u x i Initialize Q : Q i x i = 1 Z i exp{ φ u (x i )} While not converged Message passing: Q i (m) = σj i k m f i, f j Q j l Q i x i Compatibility transform: Q i x i = σ l L μ l, l σk m=1 w m m Qi (l) Update to calculate Q i x i = l Normalization 17

18 Efficient Message Passing Message passing Q i (m) = j i k m f i, f j Q j l Gaussian filter k m f i, f j Apply convolution to Q j l 18

19 Efficient Message Passing Message passing (m) Q i = k m j i f i, f j Q j l = G m Q l Q i (l) Gaussian filter k m f i, f j Apply convolution to Q j l Smooth, low-pass filter -> can be reconstructed by a set of samples (by sampling theorem) 19

20 Efficient Message Passing Message passing (m) Q i = k m j i f i, f j Q j l = G m Q l Q i (l) Downsampling Q j l Blur the downsampled signal (apply convolution operator with kernel k (m) ) Upsampling to reconstruct the filtered signal ~ Q i (m) Reduce the time complexity to O(N) 20

21 Mean Field Approximation Algorithm Q i x i = l = 1 Z i exp ψ u x i Initialize Q : Q i x i = 1 Z i exp{ φ u (x i )} Q i x i O(N) O(N) O(N) O(N) While not converged Message passing: Q i (m) = σj i k m f i, f j Q j l Compatibility transform: Q i x i = σ l L μ l, l σk m=1 w m m Qi (l) Update to calculate Q i x i = l Normalization 21

22 Results Image Dense CRF Results 22

23 Results 23

24 Conclusion A fully connected CRF model for pixel level segmentation Efficient inference on the fully connected CRF Linear in number of variables 24

25 Dense CRF as Post-processing Semantic Image Segmentation with Deep Convolutional Nets and Fully Connected CRFs. Chen et al. ICLR 15 25

26 Convergent Inference Parameter Learning and Convergent Inference for Dense Random Fields. Philipp Krähenbühl and Vladlen Koltun. ICML 13. A new efficient inference algorithm in dense CRF that is guaranteed to converge for some specific kernels and label compatibility functions. 26

27 Questions? 27

28 Pairwise Term in the Dense CRF Model Pairwise term They use ψ p x i, x j = μ x i, x j K m=1 w m k m (f i, f j ) p i : position of pixel i I i : color intensity of pixel I θ : hyper parameters 28