Making Flippy Floppy

Transcription

1 Making Flippy Floppy James V. Burke UW Mathematics Aleksandr Y. Aravkin IBM, T.J.Watson Research Michael P. Friedlander UBC Computer Science Current Topics May 2013 Talking Heads (1983)

2 Imaging Application Migration Velocity Analysis

3 Imaging: Migration Velocity Analysis After collecting seismic data, and having a smooth estimate of the velocity model in the subsurface, high-quality images are obtained by solving an optimization problem for the model update. Smallest 2D images: variable size 1/2 million Target 3D images: variable size billions Depth ( 24 meters) Lateral ( 24 meters) Depth ( 24 meters) Lateral ( 24 meters) / 25

4 Sparse Formulation for Migration BP σ : min x 1 st r J C x 2 σ Problem Specification r m J residual at smooth model estimate smooth velocity estimate Jacobian of forward model Depth ( 24 meters) Lateral ( 24 meters) C Curvelet transform x σ curvelet coefficients of the update error level Depth ( 24 meters) Results Improved recovery compared to LS inversion Lateral ( 24 meters) 3 / 25

5 In the Beginning there was BPDN & LASSO

6 BPDN & LASSO A R m n with m << n Basis Pursuit (Mallet and Zhang (1993), Chen, Donoho, Saunders (1998)) BP: min x 1 st Ax = b Basis Pursuit De-Noising (BPDN) (Chen, Donoho, Saunders (1998)) BP σ : min x 1 st b Ax 2 σ LASSO (Least Absolute Shrinkage and Selection Operator) (Tibshirani (1996)) LS τ : min 1 2 b Ax 2 2 st x 1 τ Lagrangian formulation QP λ : min b Ax + λ x 1 4 / 25

7 BPDN & LASSO A R m n with m << n Basis Pursuit (Mallet and Zhang (1993), Chen, Donoho, Saunders (1998)) BP: min x 1 st Ax = b Basis Pursuit De-Noising (BPDN) (Chen, Donoho, Saunders (1998)) BP σ : min x 1 st b Ax 2 σ LASSO (Least Absolute Shrinkage and Selection Operator) (Tibshirani (1996)) LS τ : min 1 2 b Ax 2 2 st x 1 τ Lagrangian formulation QP λ : min b Ax + λ x 1 Candés, Romberg, and Tao (2006): BP gives least support solutions. 4 / 25

8 BPDN & LASSO A R m n with m << n Basis Pursuit (Mallet and Zhang (1993), Chen, Donoho, Saunders (1998)) BP: min x 1 st Ax = b Basis Pursuit De-Noising (BPDN) (Chen, Donoho, Saunders (1998)) BP σ : min x 1 st b Ax 2 σ Target Problem (SPGl 1 ) LASSO (Least Absolute Shrinkage and Selection Operator) (Tibshirani (1996)) LS τ : min 1 2 b Ax 2 2 st x 1 τ Lagrangian formulation QP λ : min b Ax + λ x 1 Candés, Romberg, and Tao (2006): BP gives least support solutions. 4 / 25

9 BPDN & LASSO A R m n with m << n Basis Pursuit (Mallet and Zhang (1993), Chen, Donoho, Saunders (1998)) BP: min x 1 st Ax = b Basis Pursuit De-Noising (BPDN) (Chen, Donoho, Saunders (1998)) BP σ : min x 1 st b Ax 2 σ Target Problem (SPGl 1 ) LASSO (Least Absolute Shrinkage and Selection Operator) (Tibshirani (1996)) LS τ : min 1 2 b Ax 2 2 st x 1 τ Easiest to solve (SPG) Lagrangian formulation QP λ : min b Ax + λ x 1 Candés, Romberg, and Tao (2006): BP gives least support solutions. 4 / 25

10 BPDN & LASSO A R m n with m << n Basis Pursuit (Mallet and Zhang (1993), Chen, Donoho, Saunders (1998)) BP: min x 1 st Ax = b Optimal Value = τ BP Basis Pursuit De-Noising (BPDN) (Chen, Donoho, Saunders (1998)) BP σ : min x 1 st b Ax 2 σ Target Problem (SPGl 1 ) LASSO (Least Absolute Shrinkage and Selection Operator) (Tibshirani (1996)) LS τ : min 1 2 b Ax 2 2 st x 1 τ Easiest to solve (SPG) Lagrangian formulation QP λ : min b Ax + λ x 1 Candés, Romberg, and Tao (2006): BP gives least support solutions. 4 / 25

11 SPGL1: PROBING THE PARETO FRONTIER FOR BASIS PURSUIT SOLUTIONS van den Berg and Friedlander (2008)

12 Optimal Value Function BP σ : min x 1 st 1 2 Ax b 2 2 σ LS τ : min 1 2 Ax b 2 2 st x 1 τ The key is the value function v(τ) := 1 min x 2 Ax b τ v(τ) = 1 2 Axτ b 2 2 (τ, σ) Algorithm 1 Evaluate v(τ) by solving LS τ inexactly projected gradient 2 Compute v (τ) inexactly duality theory 3 Solve v(τ) = σ Inexact Newton s method τ BP 5 / 25

13 Optimal Value Function: Variational Properties v(τ) := 1 min x 2 Ax b τ Theorem [Berg & F., 2008, 2011] v(τ) 1 v(τ) is convex 2 For all τ (0, τ BP ) v is continuously differentiable v (τ) = λ τ with λ τ = A T r τ r τ = Ax τ b where x τ solves LS τ τ BP 6 / 25

14 Root Finding: v(τ) = σ Approximately solve minimize 1 2 Ax b 2 2 subj to x 1 τ k Newton update τ k+1 τ k (v k σ)/v k Early termination monitor duality gap / 25

15 EXTENSIONS Sparse Optimization with Least-Squares Constraints van den Berg and Friedlander (2011)

16 Gauge Functions U R n non-empty, closed and convex (usually, 0 U ). The gauge functional associated with U is given by γ (x U ) := inf {t x tu, t 0}. Examples: 1 U = B the closed unit ball for the norm γ (x B) = x 2 U = K a convex cone γ (x K ) = δ (x K ) := 3 U = B K γ (x B K ) = x + δ (x K ) { 0, x K + x / K 8 / 25

17 Optimal Value Function v(τ) = 1 2 Axτ b 2 2 BP σ : min γ (x U ) st 1 2 Ax b 2 2 σ LS τ : min 1 2 Ax b 2 2 st γ (x U ) τ The key is the value function v(τ) := 1 min γ(x U ) τ 2 Ax b 2 2 (τ, σ) Algorithm 1 Evaluate v(τ) by solving LS τ inexactly projected gradient 2 Compute v (τ) inexactly duality theory 3 Solve v(τ) = σ Inexact Newton s method τ = γ (xτ U ) 9 / 25

18 Applications for Guage Functionals Sparse optimization with least-squares constraints van der Berg and Friedlander (2011) Non-negative Basis Pursuit Source Localization Mass Spectrometry Nuclear-norm Minimization Matrix Completion Problems 10 / 25

19 HOW DANG FAR DOES THIS FLIPPIN IDEA GO?

20 How far does flipping go? ψ i : X R n R, i = 1, 2, arbitrary functions and X an arbitrary set. epi(ψ) := {(x, µ) ψ(x) µ} v 1 (σ) := inf x X ψ 1(x) + δ ((x, σ) epi(ψ 2 )) v 2 (τ) := inf x X ψ 2(x) + δ ((x, τ) epi(ψ 1 )) P 1,2 (σ) P 2,1 (τ) S 1,2 := { σ R = arg min P 1,2 (σ) {x X ψ 2 (x) = σ } } Then, for every σ S 1,2, (a) v 2 (v 1 (σ)) = σ, and (b) arg min P 1,2 (σ) = arg min P 2,1 (v 1 (σ)) {x X ψ 1 (x) = v 1 (σ)}. Moreover, S 2,1 = {v 1 (σ) σ S 1,2 } and {(σ, v 1 (σ)) σ S 1,2 } = {(v 2 (τ), τ) τ S 2,1 }. 11 / 25

21 Making Flippy Floppy (Easier to solve) v 1 (σ) := inf x X ψ 1(x) + δ ((x, σ) epi(ψ 2 )) v 2 (τ) := inf x X ψ 2(x) + δ ((x, τ) epi(ψ 1 )) P 1,2 (σ) P 2,1 (τ) GOAL: Solve P 1,2 (σ) by solving P 2,1 (τ) for perhaps several values of τ. The van den Berg-Friedlander method: Given σ solve the equation v 2 (τ) = σ for τ = τ σ. Then arg min P 2,1 (τ σ ) = arg min P 1,2 (σ). 12 / 25

22 When is the van den Berg-Friedlander method viable? Key considerations: (A) The problem P 2,1 (τ): v 2 (τ) := inf ψ 2(x) + δ ((x, τ) epi(ψ 1 )) x X must be easily and accurately solvable. (B) We must be able to solve equations of the form v 2 (τ) = σ. (C) v 2 (τ) should have reasonable variational properties (continuity, differentiability, subdifferentiability). 13 / 25

23 When is the van den Berg-Friedlander method viable? Key considerations: (A) The problem P 2,1 (τ): v 2 (τ) := inf ψ 2(x) + δ ((x, τ) epi(ψ 1 )) x X must be easily and accurately solvable. (B) We must be able to solve equations of the form v 2 (τ) = σ. (C) v 2 (τ) should have reasonable variational properties (continuity, differentiability, subdifferentiability). Fact: v 2 is non-increasing in τ > τ min, where τ min := inf {τ P 2,1 (τ) is feasible and finite valued} τ max := sup {τ P 2,1 (τ) is feasible and finite valued} and so is differentiable a.e. (τ min, τ max ). 13 / 25

24 What generalizations should we consider? In the motivating models, we minimize a sparsity inducing regularizing function subject to a linear least-squares misfit measure for the data. Data Misfit Statistical Model Error model Ax b 2 2 b = Ax + ɛ ɛ N (0, I ). Some Alternatives: Statistical Model Misfit Measure Error model Gaussian Laplace Huber Vapnik (ɛ insensitive loss) (a T i x b i ) 2 ɛ i N (0, 1) a T i x b i ɛ i L(0, 1) ρh (ai T x b i ) ɛ i H (0, 1) ρv (ai T x b i ) ɛ i H (0, 1) 14 / 25

25 What generalizations should we consider? In the motivating models, we minimize a sparsity inducing regularizing function subject to a linear least-squares misfit measure for the data. Data Misfit Statistical Model Error model Ax b 2 2 b = Ax + ɛ ɛ N (0, I ). Some Alternatives: Statistical Model Misfit Measure Error model Gaussian Laplace Huber Vapnik (ɛ insensitive loss) Gauss-nik? (a T i x b i ) 2 ɛ i N (0, 1) a T i x b i ɛ i L(0, 1) ρh (ai T x b i ) ɛ i H (0, 1) ρv (ai T x b i ) ɛ i H (0, 1) Hube-nik? 14 / 25

26 Gauss, Laplace, Huber, Vapnik y y x x V (x) = 1 2 x2 Gauss V (x) = x Laplace y y K K x ɛ ɛ x V (x) = Kx 1 2 K2 ; x < K V (x) = 1 2 x2 ; K x K V (x) = Kx 1 2 K2 ; K < x V (x) = x ɛ; x < ɛ V (x) = 0; ɛ x ɛ V (x) = x ɛ; ɛ x Huber Vapnik 15 / 25

27 ROBUSTNESS, SPARSNESS, AND BEYOND! Arbitrary Convex Pairs

28 Assume ρ and φ are closed, proper, and convex P 1 (σ): min φ(x) st ρ(b Ax) σ P 2 (τ): min ρ(b Ax) st φ(x) τ ρ(b Ax) P 1 (σ) is the target problem P 2 (τ) is the easier flipped problem. Problems P 1 (σ) and P 2 (τ) are linked by (τ, σ) v 2 (τ) := min ρ(b Ax) + δ ((x, τ) epi(φ)) φ(x) 16 / 25

29 Assume ρ and φ are closed, proper, and convex P 1 (σ): min φ(x) st ρ(b Ax) σ P 2 (τ): min ρ(b Ax) st φ(x) τ ρ(b Ax) P 1 (σ) is the target problem P 2 (τ) is the easier flipped problem. Problems P 1 (σ) and P 2 (τ) are linked by (τ, σ) v 2 (τ) := min ρ(b Ax) + δ ((x, τ) epi(φ)) φ(x) Broad summary of results: 1 v 2 (τ) is always convex, but may not be differentiable. 2 Solving v 2 (τ) = σ can be solved via an inexact secant method. 3 We have precise knowledge of the variational properties of v 2 (τ) for a large classes of problems P 2 (τ). 16 / 25

30 Convexity of general optimal value functions: Inf-Projection Theorem v 2 (τ) is non-increasing and convex. h(b Ax) Proof: f (x, τ) := ρ(b Ax) + δ ((x, τ) epiφ) is convex in (x, τ). (τ, σ) Therefore, the inf-projection in the variable x is convex in τ: v 2 (τ) = inf f (x, τ). x φ(x) 17 / 25

31 Inexact Secant method for v 2 (τ) = σ Theorem The inexact secant method for finding v 2 (τ) = σ, given by τ k+1 τ k l(τ k) σ m k m k = l(τ k) u(τ k 1 ) (τ k τ k 1 ) 0 < l k v 2 (τ k ) u k h(b Ax) is superlinearly convergent as long as 1 u(τ k ) l(τ k ) shrinks fast enough 2 the left Dini derivative of v 2 (τ) at τ σ is negative. σ τ 2 τ 3 τ 4 τ 5 τ σ φ(x) Tuesday, November 15, / 25

32 Derivatives for Quadratic Support Functions

33 Quadratic Support Functions QS Functions φ(x) := sup [ x, u 1 2 ut Bu] u U U R n is nonempty, closed and convex with 0 U B R n n is symmetric positive semi-definite. Examples: 1 Support functionals: B = 0 2 Gauge functionals: γ ( U ) = δ ( U ) 3 Norms: B = closed unit ball, = γ ( B) 4 Least-squares: U = R n, B = I 5 Huber: U = [ ɛ, ɛ] n, B = I 19 / 25

34 Computing Derivatives for QS Functions φ(x) := sup [ x, u 1 2 ut Bu] u U v(b, τ) := min ρ(b Ax) st φ(x) τ ( ) ū v(b, τ) = µ ( x, ū) satisfy the KKT cond. for P(b, τ) and { µ = max γ ( A T ū U ), ū T ABA T ū/ } 2τ. 20 / 25

35 More specific examples of derivative computations v(b,τ) := min 1 2 b Ax 2 2 st φ(x) τ Optimal Solution: x Optimal Residual: r = A x b 1 Support functionals: φ(x) = δ (x U ), 0 U = v 2(τ) = δ ( A T r U ) = γ ( A T r U ) 2 Gauge functionals: φ(x) = γ (x U ), 0 U = v 2(τ) = γ ( A T r U ) = δ ( A T r U ) 3 Norms: φ(x) = X = v 2(τ) = A T r 4 Huber: φ(x) = sup u [ ɛ,ɛ] n [ x, u 1 2 ut u] = v 2(τ) = max{ɛ A T r, A T r 2 / 2τ} 5 Vapnik: φ(x) = (x ɛ) + + ( x ɛ) + = v 2(τ) = ( A T r + ɛ A T r 1 ) 21 / 25

36 Sparse and Robust Formulation HBP σ : min x 1 st ρ(b Ax) σ Huber 3 2 Signal Recovery Problem Specification x 20-sparse spike train in R 512 b measurements in R 120 A Measurement matrix satisfying RIP ρ Huber function σ error level set at.01 Truth LS LS Truth Residuals 5 outliers Results In the presence of outliers, the robust formulation recovers the spike train, while the standard formulation does not. Huber / 25

37 Sparse and Robust Formulation HBP σ : min 0 x x 1 st ρ(b Ax) σ 4 3 Signal Recovery Problem Specification LS 2 x 20-sparse spike train in R Truth 1 0 b measurements in R 120 Huber 1 A ρ Measurement matrix satisfying RIP Huber function LS Residuals σ error level set at.01 5 outliers Results In the presence of outliers, the robust formulation recovers the spike train, while the standard formulation does not. Truth Huber / 25

38 Comparison with Student s t v N(0, 1) v L(0, 1) T (ν = 1) v 0.5v 2 k 2 vk log(1 + v 2 k) vk 2vk/ vk 2vk/(1 + v 2 k) Gaussian, Laplace, and Student s t Densities, Corresponding Negative Log Likelihoods, and Influence Functions (for scalar v k ). 24 / 25

39 Comparison with Student s t minimize x 0 x 1 subj to ρ(b Ax) σ, Figure: Left, top to bottom: True signal, and reconstructions via least-squares, Huber, and Student s t. Right, top to bottom: true errors, and least-squares, Huber, and Student s t residuals. 25 / 25