Stochastic Variational Inference with Gradient Linearization

Transcription

1 Stochastic Variationa Inference with Gradient Linearization Suppementa Materia Tobias Pötz * Anne S Wannenwetsch Stefan Roth Department of Computer Science, TU Darmstadt Preface In this suppementa materia, we show that SVIGL can be interpreted as a gradient descent approach using a specia preconditioner and provide the gradient inearization for the optica fow and Poisson-Gaussian energies used in the main manuscript Furthermore, we provide a proof for Proposition, show the hyperparameter evauation for SVI with SGD, and give additiona detais of our optica fow experiment in Tabe Finay, we show some exempary resuts of Poisson-Gaussian denoising and provide detais on the experiment on 3D surface reconstruction A SVIGL as Preconditioned Gradient Descent Here, we show that an update step of SVIGL as given in Eq 1 can be interpreted as one iteration of preconditioned gradient descent To simpify notation et A A t and b b t Foowing, eg [9], we have t+1 = A 1 b 0a b t 0b b + A t 0c KL q p 0d Therefore, SVIGL performs gradient descent with preconditioner P = A 1 This interpretation aso aows to introduce a step size parameter α to SVIGL t+1 = t αa 1 KL q p 1a = t αa 1 b + A t 1b = 1 α t + αˆ t+1, 1c with ˆ t+1 = A 1 b denoting the SVIGL estimate as given in Eq 1 In practice, our experiments have shown that the performance of SVIGL is not sensitive to the choice of the step size parameter We thus simpy set α = 1 *Authors contributed equay B Linearized Gradients In the foowing, we show how inearized gradients can be obtained for the presented appications of SVIGL in optica fow estimation and Poisson-Gaussian denoising For other appications, incuding many modes in computer vision, it is possibe to derive parameters A and b in a simiar fashion B1 Optica fow Here, we show the derivation of a inearized gradient for a simpe optica fow energy using the brightness constancy assumption, ie Ex, y =λ D L =1 + λ S T ρ D I t, + x I x 0 L ρ S fj x a =λ D E D x, y + λ S E S x, b with I t, = I + x 0 I1, I = I + x 0, and x 0 denoting the point of approximation of the Tayor inearization The derivations for the EpicFow energy function in Eq 15 are more tedious, but can be done anaogousy Data term In a first step, we derive the inearized gradient for the data energy term Here, it hods that x E D x, y = x ρ D I t, + =ρ D I T x x 0 3a T I t, + x I x 0 I 3b The derivative of the generaized Charbonnier [] used for

2 ρ D can be written as: ρ Dx = x x /c c max1, a + 1 ρ D x x Using Eqs 3b and 4b, we have a/ 1 4a 4b Using the derivative ρ S as given in Eq 4b, we obtain F T j ρ SF j x = = F T j F T j D ρ S Fj x F j x D ρ S Fj x F j x 30a 30b x E D x, y = T = ρ D I t, + x I x 0 I t, I + x, I x, I x I I t, I x, I x 0 I 5 The ast identity Eq 5 aows us to easiy identify a inearized form of the gradient of the data term as with and x E D x, y = A D xxx + b D xx, 6 D ρd A D Ix xx = D ρ D I x I y b D xx = D ρ D I x I y D ρ D Iy 7 D ρd I x I t 1 D ρ A D I y I t 1 x xx 0 8 T Here, x = x 1 1,, x1 L, x 1,, x L denotes the stacked vector of a horizonta and vertica fow components D turns the argument vector into a diagona matrix short for diag{ }, and the product is appied eement-wise Smoothness term For the smoothness term et us first express the convoution f j x as a matrix-vector product F j x, with F j denoting the convoution matrix corresponding to f j and x the vectorized fow as before With that, the gradient of the smoothness term E S can be written as: x E S x = x = L Fj ρ S x 9a F T j ρ S Fj x 9b A S xxx 30c Compete inearized gradient We now summarize the resuts of Eqs 7, 8, and 30c to obtain the inearized gradient as x Ex, y =λ D x E D x, y + λ S x E S x = λ D A D xx + λ S A S xx x + λ D b D x A x xx + b x B Poisson-Gaussian denoising 31a 31b 31c Let us first recap the energy function for Poisson- Gaussian denoising: where Ex, y = λ D L =1 + λ S x y σx L fj ρ S x, 3a =λ D E D x, y + λ S E S x, 3b σx = β 1 x + β 33 We wi derive the inearized gradients for the data term E D and the smoothness term E S separatey Data term x E D x, y The gradient of the data term is given as x y = σx β 1x y σx 4 34a = x σx y σx β 1x σx 4 + β 1xy σx 4 β 1y σx 4 34b 1 =x β 1x σx 4 + β 1y σx 4 σx y σx + β 1y σx 4, 34c

3 where a operations are eement-wise The inearized gradient of the data term can then be obtained as 1 A D xx = D σx β 1x σx 4 + β 1y σx 4 35 y b D xx = σx + β 1y σx 4 36 Smoothness term For the smoothness term we can reuse the inearized gradient derived in Eq 30c Compete inearized gradient We can now put the resuts of Eqs 30c, 35, and 36 together to obtain a inearized gradient of the energy for Poisson-Gaussian denoising, cf Eqs 31a 31c C Proof Proposition In this section, we provide a proof for Proposition of the main paper Proposition An energy function can be inearized with a positive semi-definite matrix A x if it is composed of a sum of energy terms ρ i w i that fufi the foowing conditions: 1 Each penaty function ρ i is symmetric and ρ i w i 0 for a w i 0 Each penaty function ρ i is appied eement-wise on w i, which is of the form w i = K i x + g i y, with fiter matrix K i and g i not depending on x Proof From assuming a symmetric ρ i in, it foows that ρ i is point symmetric Due to ρ i w i 0 for a w i 0 we then find that ρ i w i can be written as ρ iw i ρ i w i w i with a ρ i w i 0 37 For an energy term as described in, the gradient wrt x is given as x ρ i w i = K T i C i K i x + g i y, 38 with C i = D ρ i K i x + g i y 39 A inearization can then be obtained using A i x = K T i C i K i, b i x = K T i C i g i y 40 Since C i is a diagona matrix of non-negative eements Eq 37, A i x is positive semi-definite as x T A i xx = x T K T i C i K i x = v T C i v 0 41 As the sum of positive semi-definite matrices is positive semi-definite, a matrix A x composed of energy terms that fufi and is positive semi-definite D Hyperparameters for SGD In the foowing, we aim to find optima hyperparameters for the SVI baseine based on SGD For a experiments we seect an initia step size α 0, which is cut after each third of iterations by a factor of ten An evauation of the unnormaized KL divergence for optica fow potted against the runtime for different initia step sizes α 0 of SGD is shown in Fig 6a Here, the KL divergence deteriorates severey using SGD with a step size arger than 10 6 For smaer step sizes, SVI with SGD shows a sow convergence such that we set α 0 = 10 6 Foowing the same procedure, we perform severa experiments for Poisson-Gaussian denoising and evauate different settings for the initia step size parameter α 0 of SGD in Fig 7a Again, an initia step size α 0 = 10 6 proves to be most effective Smaer step sizes converge too sowy, whie SGD with bigger step size vaues converges faster but to a worse oca optimum For an initia step size of α 0 = 10 5 optimization diverges immediatey Appying SVI with SGD, we observe in both appications a faster convergence of the KL divergence with a smaer sampe size, but a arger number of iterations, cf Figs 6b and 7b We therefore choose Z = 1 with 4000 iterations of SGD for the experiments in the main paper E Comparison with ProbFowFieds In Tabe of the main paper we evauate the quaity of the posterior variances obtained with SVIGL Here, we foow Wannenwetsch et a [45] and derive an uncertainty measure by computing the margina entropy of the fow at every pixe To have a fair comparison with [45], we use the same EpicFow [3] energy formuation with earned Gaussian scae mixture penaty functions and expicit indicator variabes for their mixture components Since SVIGL is designed for variationa inference in distributions with continuous random variabes, we aternate cosed-form updates of the atent indicator variabes with SVIGL updates for the continuous fow variabes For the discrete update, we approximate the tedious anaytica expectation vaues over the fow variabes with a Monte-Caro estimator cf Eq 7b This effectivey reduces the optimization wrt the indicator variabes to an independent update thus maintaining the ease of use of SVIGL Weighting parameters λ D and λ S are determined on a training set with Bayesian optimization [37] using the F1-score as described in [45] F Resuts of Poisson-Gaussian Denoising Fig 8 shows some exampe resuts of SVIGL appied to Poisson-Gaussian denoising on the BSDS dataset High uncertainties can be observed especiay on object boundaries Due to the high amount of noise, a strong smoothness term

4 a Figure 6 Unnormaized KL divergence vs runtime for optica fow with SVIGL and SVI with SGD with different step sizes a and different numbers of sampes and iterations b Vaues averaged on the vaidation set b a Figure 7 Unnormaized KL divergence vs runtime for denoising with SVIGL and SVI with SGD with different step sizes a and with different numbers of sampes and iterations b Vaues averaged over the BSDS test set b maximizes the PSNR on the training set Therefore, the denoised images tend to be rather smooth in genera G Resuts on Sinte Test As described in Sec 51, we appy SVIGL as we as two MAP baseines on the fu-sized Sinte test images in order to evauate their performance Figure 9 shows a screenshot of the private Sinte benchmark tabe with resuts for both methods SVIGL outperforms the underying FowFieds method [1] as we as the L-BFGS baseine and shows an AEPE resut on par with the corresponding MAP estimate using GL Moreover, SVIGL estimates are competitive with the finetuned version of FowNet [49], ie the state-of-theart baseine for optica fow prediction with convoutiona neura networks H 3D Surface Reconstruction We now give more detais on the appication of SVIGL to 3D surface reconstruction First, we restate the energy of Lipman et a [5], which is given by P E, P, C = x i p j h c i p j i=1 C λ i x i c i h c i c i 4 i=1 i =1 Here, p j P denote the noisy input points, c i C are the current estimates of the smoothed points, and x i the new estimates of the smoothed points Whie the first part of the energy forces the new estimates to be cose to the input points, the second term pushes the reconstructed points apart by penaizing points in that are too cose to points in C The contribution of each term is weighted by the Gaussian kerne h A cosed-form soution to minimizing the above energy is given in [5] This soution is then used in a fixed point scheme as t+1 = arg min E, P, t, 43 where 0 is initiaized as a L projection of the input points In a variationa inference setting, cosed-form updates are no onger possibe due to introducing the additiona vari-

5 Figure 8 Exampes of ground truth eft, noisy images second coumn, estimated cean images third coumn, and uncertainty estimates right from SVIGL on the BSDS test set Figure 9 Screenshot of the private Sinte benchmark tabe fina with resuts for SVIGL, MAP + GL, MAP + L-BFGS, and the origina FowFieds approach [1] status as of March 018 ance variabes σ of the variationa posterior Hence, we empoy SVIGL updates instead To be abe to appy SVIGL, we require a inearization of the energy gradient The specific form of the energy in Eq 19 aows for a diagona inearization: hkci pj k xi E, P, C = xi pj kxi pj k j J i0 I xi ci0 hkci ci0 k kxi ci0 k 44 In tota, we run 10 iterations of Eq 43 In each iteration, we compute a singe SVIGL update with a sampe set size of Z = 5 References [49] E Ig, N Mayer, T Saikia, M Keuper, A Dosovitskiy, and T Brox Fownet 0: Evoution of optica fow estimation with deep networks In CVPR, pages , 017 4