Optimal Value Function Methods in Numerical Optimization Level Set Methods

Transcription

1 Optimal Value Function Methods in Numerical Optimization Level Set Methods James V Burke Mathematics, University of Washington, (jvburke@uw.edu) Joint work with Aravkin (UW), Drusvyatskiy (UW), Friedlander (UBC/Davis), Roy (UW) 2016 Joint Mathematics Meeting SIAM Minisymposium on Optimization

2 Motivation Optimization in Large-Scale Inference A range of large-scale data science applications can be modeled using optimization: - Inverse problems (medical and seismic imaging ) - High dimensional inference (compressive sensing, LASSO, quantile regression) - Machine learning (classification, matrix completion, robust PCA, time series) These applications are often solved using side information: - Sparsity or low rank of solution - Constraints (topography, non-negativity) - Regularization (priors, total variation, dirty data) We need efficient large-scale solvers for nonsmooth programs. 2/19

3 The Prototypical Problem Sparse Data Fitting: Find sparse x with Ax b (e.g. model selection, compressed sensing) 3/19

4 The Prototypical Problem Sparse Data Fitting: Find sparse x with Ax b (e.g. model selection, compressed sensing) Convex approaches: BPDN LASSO Lagrangian min x s.t. x Ax b 2 2 σ min x s.t. 1 2 Ax b 2 2 x 1 τ min x 1 2 Ax b λ x 1 3/19

5 The Prototypical Problem Sparse Data Fitting: Find sparse x with Ax b (e.g. model selection, compressed sensing) Convex approaches: BPDN LASSO Lagrangian min x s.t. x Ax b 2 2 σ min x s.t. 1 2 Ax b 2 2 x 1 τ min x 1 2 Ax b λ x 1 BPDN: often most natural and transparent. (e.g. physical considerations guide σ) 3/19

6 The Prototypical Problem Sparse Data Fitting: Find sparse x with Ax b (e.g. model selection, compressed sensing) Convex approaches: BPDN LASSO Lagrangian min x s.t. x Ax b 2 2 σ min x s.t. 1 2 Ax b 2 2 x 1 τ min x 1 2 Ax b λ x 1 BPDN: often most natural and transparent. (e.g. physical considerations guide σ) Lagrangian: ubiquitous in practice. (e.g. no constraints ) 3/19

7 The Prototypical Problem Sparse Data Fitting: Find sparse x with Ax b (e.g. model selection, compressed sensing) Convex approaches: BPDN LASSO Lagrangian min x s.t. x Ax b 2 2 σ min x s.t. 1 2 Ax b 2 2 x 1 τ min x 1 2 Ax b λ x 1 BPDN: often most natural and transparent. (e.g. physical considerations guide σ) Lagrangian: ubiquitous in practice. (e.g. no constraints ) All three are (essentially) equivalent computationally! 3/19

8 The Prototypical Problem Sparse Data Fitting: Find sparse x with Ax b (e.g. model selection, compressed sensing) Convex approaches: BPDN LASSO Lagrangian min x s.t. x Ax b 2 2 σ min x s.t. 1 2 Ax b 2 2 x 1 τ min x 1 2 Ax b λ x 1 BPDN: often most natural and transparent. (e.g. physical considerations guide σ) Lagrangian: ubiquitous in practice. (e.g. no constraints ) All three are (essentially) equivalent computationally! Basis for SPGL1 (van den Berg-Friedlander 08) 3/19

9 Optimal Value or Level Set Framework Problem class: Solve min x X s.t. φ(x) ρ(ax b) σ P(σ) 4/19

10 Optimal Value or Level Set Framework Problem class: Solve min x X s.t. φ(x) ρ(ax b) σ P(σ) Strategy: consider the flipped problem v(τ) := min x X s.t. ρ(ax b) φ(x) τ Q(τ) 4/19

11 Optimal Value or Level Set Framework Problem class: Solve min x X s.t. φ(x) ρ(ax b) σ P(σ) Strategy: consider the flipped problem v(τ) := min x X s.t. ρ(ax b) φ(x) τ Q(τ) Then opt-val(p(σ)) is the minimal root of the equation v(τ) = σ 4/19

12 Optimal Value or Level Set Framework Problem class: Solve min x X s.t. φ(x) ρ(ax b) σ P(σ) Strategy: consider the flipped problem v(τ) := min x X s.t. ρ(ax b) φ(x) τ Q(τ) Then opt-val(p(σ)) is the minimal root of the equation Assume X, ρ, φ are convex v(τ) = σ 4/19

13 Optimal Value or Level Set Framework Problem class: Solve min x X s.t. φ(x) ρ(ax b) σ P(σ) Strategy: consider the flipped problem v(τ) := min x X s.t. ρ(ax b) φ(x) τ Q(τ) Then opt-val(p(σ)) is the minimal root of the equation v(τ) = σ Assume X, ρ, φ are convex = v is convex. 4/19

14 Inexact Root Finding Problem: Find root of the inexactly known convex function v( ) σ. 5/19

15 Inexact Root Finding Problem: Find root of the inexactly known convex function v( ) σ. Bisection is one approach 5/19

16 Inexact Root Finding Problem: Find root of the inexactly known convex function v( ) σ. Bisection is one approach nonmonotone iterates (bad for warmstarts) at best linear convergence (with perfect information) 5/19

17 Inexact Root Finding Problem: Find root of the inexactly known convex function v( ) σ. Bisection is one approach nonmonotone iterates (bad for warmstarts) at best linear convergence (with perfect information) Solution: modified secant approximate Newton methods 5/19

18 Inexact Root Finding: Secant 6/19

19 Inexact Root Finding: Secant 6/19

20 Inexact Root Finding: Secant 6/19

21 Inexact Root Finding: Secant 6/19

22 Inexact Root Finding: Secant 6/19

23 Inexact Root Finding: Secant 6/19

24 Inexact Root Finding: Newton 6/19

25 Inexact Root Finding: Newton 6/19

26 Inexact Root Finding: Newton 6/19

27 Inexact Root Finding: Newton 6/19

28 Inexact Root Finding: Newton 6/19

29 Inexact Root Finding: Newton 6/19

30 Inexact Root Finding: Convergence 7/19

31 Inexact Root Finding: Convergence 7/19

32 Inexact Root Finding: Convergence 7/19

33 Inexact Root Finding: Convergence Key observation: C = C(τ 0 ) is independent of v (τ ). 7/19

34 Minorants from Duality 8/19

35 Minorants from Duality 8/19

36 Minorants from Duality 8/19

37 Robustness: 1 u/l α, where α [1, 2) f f (a) k = 13, α = 1.3 (b) k = 770, α = 1.99 (c) k = 18, α = f f (d) k = 9, α = 1.3 (e) k = 15, α = 1.99 (f) k = 10, α = 1.3 Figure: Inexact secant (top) and Newton (bottom) for f 1 (τ) = (τ 1) 2 10 (first two columns) and f 2 (τ) = τ 2 (last column). Below each panel, α is the oracle accuracy, and k is the number of iterations needed to converge, i.e., to reach f i (τ k ) ɛ = f 2 f 2 9/19

38 Sensor Network Localization (SNL) Given a weighted graph G = (V, E, d) find a realization: p 1,..., p n R 2 with d ij = p i p j 2 for all ij E. 10/19

39 Sensor Network Localization (SNL) SDP relaxation (Weinberger et al. 04, Biswas et al. 06): max tr (X) s.t. P E K(X) d 2 2 σ Xe = 0, X 0 where [K(X)] i,j = X ii + X jj 2X ij. 11/19

40 Sensor Network Localization (SNL) SDP relaxation (Weinberger et al. 04, Biswas et al. 06): max tr (X) s.t. P E K(X) d 2 2 σ Xe = 0, X 0 where [K(X)] i,j = X ii + X jj 2X ij. Intuition: X = P P T and then tr (X) = 1 with p n + 1 i the ith row of P. n p i p j 2 i,j=1 11/19

41 Sensor Network Localization (SNL) SDP relaxation (Weinberger et al. 04, Biswas et al. 06): max tr (X) s.t. P E K(X) d 2 2 σ Xe = 0, X 0 where [K(X)] i,j = X ii + X jj 2X ij. Intuition: X = P P T and then tr (X) = 1 with p n + 1 i the ith row of P. Flipped problem: min P E K(X) d 2 2 s.t. tr X = τ Xe = 0 X 0. n i,j=1 p i p j 2 11/19

42 Sensor Network Localization (SNL) SDP relaxation (Weinberger et al. 04, Biswas et al. 06): max tr (X) s.t. P E K(X) d 2 2 σ Xe = 0, X 0 where [K(X)] i,j = X ii + X jj 2X ij. Intuition: X = P P T and then tr (X) = 1 with p n + 1 i the ith row of P. Flipped problem: min P E K(X) d 2 2 s.t. tr X = τ Xe = 0 X 0. Perfectly adapted for the Frank-Wolfe method. n i,j=1 p i p j 2 11/19

43 Sensor Network Localization (SNL) SDP relaxation (Weinberger et al. 04, Biswas et al. 06): max tr (X) s.t. P E K(X) d 2 2 σ Xe = 0, X 0 where [K(X)] i,j = X ii + X jj 2X ij. Intuition: X = P P T and then tr (X) = 1 with p n + 1 i the ith row of P. Flipped problem: min P E K(X) d 2 2 s.t. tr X = τ Xe = 0 X 0. Perfectly adapted for the Frank-Wolfe method. n i,j=1 Key point: Slater failing (always the case) is irrelevant. p i p j 2 11/19

44 Approximate Newton 1.5 Pareto curve Figure: σ = /19

45 Approximate Newton 1.5 Pareto curve 1.4 Pareto curve Figure: σ = 0.25 Figure: σ = 0 12/19

46 Max-trace 0.6 Newton_max 0.6 Refine positions /19

47 Max-trace 0.6 Newton_max 0.6 Refine positions Newton_min Refine positions /19

48 Conclusion Simple strategy for optimizing over complex domains Rigorous convergence guarantees Insensitivity to ill-conditioning Many applications Sensor Network Localization (D-Krislock-Voronin-Wolkowicz 15) Large scale SDP and LP (cf. Renegar 14) Chromosome reconstruction (Aravkin-Becker-D-Lozano 15) Phase retrieval (Aravkin-Burke-D-Friedlander-Roy 15)... 14/19

49 Conclusion Simple strategy for optimizing over complex domains Rigorous convergence guarantees Insensitivity to ill-conditioning Many applications Sensor Network Localization (D-Krislock-Voronin-Wolkowicz 15) Large scale SDP and LP (cf. Renegar 14) Chromosome reconstruction (Aravkin-Becker-D-Lozano 15) Phase retrieval (Aravkin-Burke-D-Friedlander-Roy 15)... References : Optimal value function methods for numerical optimization, Aravkin-B-Drusvyatskiy-Friedlander-Roy, preprint, Noisy Euclid. distance realization: facial reduction and the Pareto frontier, Cheung-Drusvyatskiy-Krislock-Wolkowicz, submitted, /19

50 More References Probing the pareto frontier for basis pursuit solutions van der Berg - Friedlander SIAM J. Sci. Comput. 31(2008), Sparse optimization with least-squares constraints van der Berg - Friedlander SIOPT 21(2011), Variational Properties of Value Functions. Aravkin - B - Friedlander SIOPT 23(2013), /19

51 Basis Pursuit with Outliers BP σ : min x 1 st ρ(b Ax) σ Standard least-squares: ρ(z) = z 2 or ρ(z) = z /19

52 Basis Pursuit with Outliers BP σ : min x 1 st ρ(b Ax) σ Standard least-squares: ρ(z) = z 2 or ρ(z) = z 2 2. Quantile Huber: τ r κτ 2 2 if r < τκ, 1 ρ κ,τ (r) = 2κ r2 if r [ κτ, (1 τ)κ], (1 τ) r κ(1 τ)2 2, if r > (1 τ)κ. Standard Huber when τ = /19

53 Huber 17/19

54 Sparse and Robust Formulation HBP σ : min x 1 st ρ(b Ax) σ Huber2 3 Signal Recovery Problem Specification Truth x 20-sparse spike train in R 512 b measurements in R 120 LS A Measurement matrix satisfying RIP ρ Huber function σ error level set at.01 5 outliers LS Residuals Results In the presence of outliers, the robust formulation recovers the spike train, while the standard formulation does not. Truth Huber /19

55 Sparse and Robust Formulation HBP σ : min 0 x Problem Specification x 1 st ρ(b Ax) σ LS Signal Recovery x 20-sparse spike train in R b measurements in R 120 A Measurement matrix satisfying RIP ρ Huber function σ error level set at.01 5 outliers Results In the presence of outliers, the robust formulation recovers the spike train, while the standard formulation does not. Truth 1 0 Huber 1 LS Truth Residuals Huber /19