l 1 -magic : Recovery of Sparse Signals via Convex Programming

Transcription

1 l -magic : Recovery of Sparse Signals via Convex Programming Emmanuel Candès and Justin Romberg Caltech October 005 Seven problems A recent series of papers [3 8] develops a theory of signal recovery from highly incomplete information The central results state that a sparse vector x 0 R N can be recovered from a small number of linear measurements b = Ax 0 R K K N or b = Ax 0 + e when there is measurement noise by solving a convex program As a companion to these papers this package includes MATLAB code that implements this recovery procedure in the seven contexts described below The code is not meant to be cuttingedge rather it is a proof-of-concept showing that these recovery procedures are computationally tractable even for large scale problems where the number of data points is in the millions The problems fall into two classes: those which can be recast as linear programs LPs and those which can be recast as second-order cone programs SOCPs The LPs are solved using a generic path-following primal-dual method The SOCPs are solved with a generic log-barrier algorithm The implementations follow Chapter of [] For maximum computational efficiency the solvers for each of the seven problems are implemented separately They all have the same basic structure however with the computational bottleneck being the calculation of the Newton step this is discussed in detail below The code can be used in either small scale mode where the system is constructed explicitly and solved exactly or in large scale mode where an iterative matrix-free algorithm such as conjugate gradients CG is used to approximately solve the system Our seven problems of interest are: Min-l with equality constraints The program P min x subject to Ax = b also known as basis pursuit finds the vector with smallest l norm x := i x i that explains the observations b As the results in [4 6] show if a sufficiently sparse x 0 exists such that Ax 0 = b then P will find it When x A b have real-valued entries P can be recast as an LP this is discussed in detail in [0] Minimum l error approximation Let A be a M N matrix with full rank Given y R M the program P A min x y Ax

2 finds the vector x R N such that the error y Ax has minimum l norm ie we are asking that the difference between Ax and y be sparse This program arises in the context of channel coding [8] Suppose we have a channel code that produces a codeword c = Ax for a message x The message travels over the channel and has an unknown number of its entries corrupted The decoder observes y = c + e where e is the corruption If e is sparse enough then the decoder can use P A to recover x exactly Again P A can be recast as an LP Min-l with quadratic constraints This program finds the vector with minimum l norm that comes close to explaining the observations: P min x subject to Ax b ɛ where ɛ is a user specified parameter As shown in [5] if a sufficiently sparse x 0 exists such that b = Ax 0 + e for some small error term e ɛ then the solution x to P will be close to x 0 That is x x 0 C ɛ where C is a small constant P can be recast as a SOCP Min-l with bounded residual correlation Also referred to as the Dantzig Selector the program P D min x subject to A Ax b γ where γ is a user specified parameter relaxes the equality constraints of P in a different way P D requires that the residual Ax b of a candidate vector x not be too correlated with any of the columns of A the product A Ax b measures each of these correlations If b = Ax 0 +e where e i N 0 σ then the solution x D to P D has near-optimal minimax risk: E x D x 0 Clog N minx 0 i σ see [7] for details For real-valued x A b P D can again be recast as an LP; in the complex case there is an equivalent SOCP It is also true that when x A b are complex the programs P P A P D can be written as SOCPs but we will not pursue this here If the underlying signal is a D image an alternate recovery model is that the gradient is sparse [8] Let x denote the pixel in the ith row and j column of an n n image x and define the operators and D h; x = { x i+j x i < n 0 i = n D x = i D v; x = Dh; x D v; x { x ij+ x j < n 0 j = n The -vector D x can be interpreted as a kind of discrete gradient of the digital image x The total variation of x is simply the sum of the magnitudes of this discrete gradient at every point: TVx := D h; x + D v; x = D x With these definitions we have three programs for image recovery each of which can be recast as a SOCP:

3 Min-TV with equality constraints T V min TVx subject to Ax = b If there exists a piecewise constant x 0 with sufficiently few edges ie D x 0 is nonzero for only a small number of indices then T V will recover x 0 exactly see [4] Min-TV with quadratic constraints T V min TVx subject to Ax b ɛ Examples of recovering images from noisy observations using T V were presented in [5] Note that if A is the identity matrix T V reduces to the standard Rudin-Osher-Fatemi image restoration problem [8] See also [9 3] for SOCP solvers specifically designed for the total-variational functional Dantzig TV T V D min TVx subject to A Ax b γ This program was used in one of the numerical experiments in [7] In the next section we describe how to solve linear and second-order cone programs using modern interior point methods Interior point methods Advances in interior point methods for convex optimization over the past 5 years led by the seminal work [4] have made large-scale solvers for the seven problems above feasible Below we overview the generic LP and SOCP solvers used in the l -magic package to solve these problems A primal-dual algorithm for linear programming In Chapter of [] Boyd and Vandenberghe outline a relatively simple primal-dual algorithm for linear programming which we have followed very closely for the implementation of P P A and P D For the sake of completeness and to set up the notation we briefly review their algorithm here The standard-form linear program is min z c 0 z subject to A 0 z = b f i z 0 where the search vector z R N b R K A 0 is a K N matrix and each of the f i i = m is a linear functional: f i z = c i z + d i for some c i R N d i R At the optimal point z there will exist dual vectors ν R K λ R m λ 0 such that the Karush-Kuhn-Tucker conditions are satisfied: KKT c 0 + A T 0 ν + i λ i c i = 0 λ i f i z = 0 i = m A 0 z = b f i z 0 i = m 3

4 In a nutshell the primal dual algorithm finds the optimal z along with optimal dual vectors ν and λ by solving this system of nonlinear equations The solution procedure is the classical Newton method: at an interior point z k ν k λ k by which we mean f i z k < 0 λ k > 0 the system is linearized and solved However the step to new point z k+ ν k+ λ k+ must be modified so that we remain in the interior In practice we relax the complementary slackness condition λ i f i = 0 to λ k i f i z k = /τ k where we judiciously increase the parameter τ k as we progress through the Newton iterations This biases the solution of the linearized equations towards the interior allowing a smooth welldefined central path from an interior point to the solution on the boundary see [50] for an extended discussion The primal dual and central residuals quantify how close a point z ν λ is to satisfying KKT with in place of the slackness condition: r dual = c 0 + A T 0 ν + i λ i c i r cent = Λf /τ r pri = A 0 z b where Λ is a diagonal matrix with Λ ii = λ i and f = f z f m z T From a point z ν λ we want to find a step z ν λ such that r τ z + z ν + ν λ + λ = 0 3 Linearizing 3 with the Taylor expansion around z ν λ r τ z + z ν + ν λ + λ r τ z ν λ + J rτ z νλ z ν λ where J rτ z νλ is the Jacobian of r τ we have the system 0 AT 0 C T z ΛC 0 F v = c 0 + A T 0 ν + i λ ic i Λf /τ A λ A 0 z b where m N matrix C has the c T i as rows and F is diagonal with F ii = f i z We can eliminate λ using: λ = ΛF C z λ /τf 4 leaving us with the core system C T F ΛC A T 0 A 0 0 z = ν c0 + /τc T f A T 0 ν b A 0 z 5 With the z ν λ we have a step direction To choose the step length 0 < s we ask that it satisfy two criteria: z + s z and λ + s λ are in the interior ie f i z + s z < 0 λ i > 0 for all i The norm of the residuals has decreased sufficiently: r τ z + s z ν + s ν λ + s λ αs r τ z ν λ where α is a user-sprecified parameter in all of our implementations we have set α = 00 4

5 Since the f i are linear functionals item is easily addressed We choose the maximum step size that just keeps us in the interior Let and set I + f = {i : c i z > 0} I λ {i : λ < 0} s max = 099 min{ { f i z/ c i z i I + f } { λ i/ λ i i I λ }} Then starting with s = s max we check if item above is satisfied; if not we set s = β s and try again We have taken β = / in all of our implementations When r dual and r pri are small the surrogate duality gap η = f T λ is an approximation to how close a certain z ν λ is to being opitmal ie c 0 z c 0 z η The primal-dual algorithm repeats the Newton iterations described above until η has decreased below a given tolerance Almost all of the computational burden falls on solving 5 When the matrix C T F ΛC is easily invertible as it is for P or there are no equality constraints as in P A P D 5 can be reduced to a symmetric positive definite set of equations When N and K are large forming the matrix and then solving the linear system of equations in 5 is infeasible However if fast algorithms exist for applying C C T and A 0 A T 0 we can use a matrix free solver such as Conjugate Gradients CG is iterative requiring a few hundred applications of the constraint matrices roughly speaking to get an accurate solution A CG solver based on the very nice exposition in [9] is included with the l -magic package The implementations of P P A P D are nearly identical save for the calculation of the Newton step In the Appendix we derive the Newton step for each of these problems using notation mirroring that used in the actual MATLAB code A log-barrier algorithm for SOCPs Although primal-dual techniques exist for solving second-order cone programs see [ 3] their implementation is not quite as straightforward as in the LP case Instead we have implemented each of the SOCP recovery problems using a log-barrier method The log-barrier method for which we will again closely follow the generic but effective algorithm described in [ Chap ] is conceptually more straightforward than the primal-dual method described above but at its core is again solving for a series of Newton steps Each of P T V T V T V D can be written in the form min z c 0 z subject to A 0 z = b where each f i describes either a constraint which is linear f i = c i z + d i f i z 0 i = m 6 or second-order conic f i z = Ai z c i z + d i the A i are matrices the c i are vectors and the d i are scalars The standard log-barrier method transforms 6 into a series of linearly constrained programs: min c 0 z + z τ k log f i z subject to A 0 z = b 7 i 5

6 where τ k > τ k The inequality constraints have been incorporated into the functional via a penalty function which is infinite when the constraint is violated or even met exactly and smooth elsewhere As τ k gets large the solution z k to 7 approaches the solution z to 6: it can be shown that c 0 z k c 0 z < m/τ k ie we are within m/τ k of the optimal value after iteration k m/τ k is called the duality gap The idea here is that each of the smooth subproblems can be solved to fairly high accuracy with just a few iterations of Newton s method especially since we can use the solution z k as a starting point for subproblem k + At log-barrier iteration k Newton s method which is again iterative proceeds by forming a series of quadratic approximations to 7 and minimizing each by solving a system of equations again we might need to modify the step length to stay in the interior The quadratic approximation of the functional f 0 z = c 0 z + log f i z τ in 7 around a point z is given by where g z is the gradient f 0 z + z z + g z z + H z z z := qz + z i g z = c 0 + τ and H z is the Hessian matrix H z = τ f i z f iz f i z T i i f i z f iz + τ i f i z f i z Given that z is feasible that A 0 z = b in particular the z that minimizes qz + z subject to A 0 z = b is the solution to the set of linear equations Hz A τ T 0 z = τg A 0 0 v z 8 The vector v which can be interpreted as the Lagrange multipliers for the quality constraints in the quadratic minimization problem is not directly used In all of the recovery problems below with the exception of T V there are no equality constraints A 0 = 0 In these cases the system 8 is symmetric positive definite and thus can be solved using CG when the problem is large scale For the T V problem we use the SYMMLQ algorithm which is similar to CG and works on symmetric but indefinite systems see [6] With z in hand we have the Newton step direction The step length s is chosen so that f i z + s z < 0 for all i = m The functional has decreased suffiently: f 0 z + s z < f 0 z + αs z g z z where α is a user-specified parameter each of the implementations below uses α = 00 This requirement basically states that the decrease must be within a certain percentage of that predicted by the linear model at z As before we start with s = and decrease by multiples of β until both these conditions are satisfied all implementations use β = / The complete log-barrier implementation for each problem follows the outline: The choice of log x for the barrier function is not arbitrary it has a property termed self-concordance that is very important for quick convergence of 7 to 6 both in theory and in practice see the very nice exposition in [7] 6

7 Inputs: a feasible starting point z 0 a tolerance η and parameters µ and an initial τ Set k = Solve 7 via Newton s method followed by the backtracking line search using z k as an initial point Call the solution z k 3 If m/τ k < η terminate and return z k 4 Else set τ k+ = µτ k k = k + and go to step In fact we can calculate in advance how many iterations the log-barrier algorithm will need: log m log η log τ barrier iterations = log µ The final issue is the selection of τ Our implementation chooses τ conservatively; it is set so that the duality gap m/τ after the first iteration is equal to c 0 z 0 In Appendix we explicitly derive the equations for the Newton step for each of P T V T V T V D again using notation that mirrors the variable names in the code 3 Examples To illustrate how to use the code the l -magic package includes m-files for solving specific instances of each of the above problems these end in examplem in the main directory 3 l with equality constraints We will begin by going through leq examplem in detail This m-file creates a sparse signal takes a limited number of measurements of that signal and recovers the signal exactly by solving P The first part of the procedure is for the most part self-explainatory: % put key subdirectories in path if not already there pathpath /Optimization ; pathpath /Data ; % load random states for repeatable experiments load RandomStates rand state rand_state; randn state randn_state; % signal length N = 5; % number of spikes in the signal T = 0; % number of observations to make K = 0; % random +/- signal x = zerosn; q = randpermn; xq:t = signrandnt; 7

8 We add the Optimization directory where the interior point solvers reside and the Data directories to the path The file RandomStatesm contains two variables: rand state and randn state which we use to set the states of the random number generators on the next two lines we want this to be a repeatable experiment The next few lines set up the problem: a length 5 signal that contains 0 spikes is created by choosing 0 locations at random and then putting ± at these locations The original signal is shown in Figure a The next few lines: % measurement matrix disp Creating measurment matrix ; A = randnkn; A = ortha ; disp Done ; % observations y = A*x; % initial guess = min energy x0 = A *y; create a measurement ensemble by first creating a K N matrix with iid Gaussian entries and then orthogonalizing the rows The measurements y are taken and the minimum energy solution x0 is calculated x0 which is shown in Figure is the vector in {x : Ax = y} that is closest to the origin Finally we recover the signal with: % solve the LP tic xp = leq_pdx0 A [] y e-3; toc The function leq pdm found in the Optimization subdirectory implements the primal-dual algorithm presented in Section ; we are sending it our initial guess x0 for the solution the measurement matrix the third argument which is used to specify the transpose of the measurement matrix is unnecessary here and hence left empty since we are providing A explicitly the measurements and the precision to which we want the problem solved leq pd will terminate when the surrogate duality gap is below 0 3 Running the example file at the MATLAB prompt we have the following output: >> leq_example Creating measurment matrix Done Iteration = tau = 6433e+0 Primal = 650e+0 PDGap = 59e+0 Dual res = 693e+00 Primal res = 568e-5 Hp condition number = 486e-0 Iteration = tau = 96e+0 Primal = 9358e+0 PDGap = 7903e+0 Dual res = 706e-0 Primal res = 497e-5 Hp condition number = 5e-0 Iteration = 3 tau = 79e+0 Primal = 5567e+0 PDGap = 375e+0 Dual res = 950e-0 Primal res = 6003e-4 Hp condition number = 9606e-05 Iteration = 4 tau = 4689e+0 Primal = 439e+0 PDGap = 84e+0 Dual res = 575e-0 Primal res = 3307e-4 Hp condition number = 9e-04 Iteration = 5 tau = 66e+0 Primal = 3646e+0 PDGap = 66e+0 Dual res = 56e-0 Primal res = 44e-4 Hp condition number = 84e-04 Iteration = 6 tau = 57e+03 Primal = 879e+0 PDGap = 8853e+00 Dual res = 5549e-0 Primal res = 473e-4 Hp condition number = 7396e-05 Iteration = 7 tau = 047e+04 Primal = 097e+0 PDGap = 9780e-0 Dual res = 5549e-04 Primal res = 4e-3 Hp condition number = 05e-06 Iteration = 8 tau = 9605e+04 Primal = 00e+0 PDGap = 066e-0 Dual res = 5549e-06 Primal res = 376e-3 Hp condition number = 90e-07 Iteration = 9 tau = 88e+05 Primal = 00e+0 PDGap = 6e-0 Dual res = 5549e-08 Primal res = 830e- Hp condition number = 6793e-09 Iteration = 0 tau = 8084e+06 Primal = 000e+0 PDGap = 67e-03 Dual res = 5549e-0 Primal res = 33e- Hp condition number = 889e- Iteration = tau = 747e+07 Primal = 000e+0 PDGap = 38e-04 Dual res = 5550e- Primal res = 48e-0 Hp condition number = 076e- Elapsed time is seconds The recovered signal xp is shown in Figure c The signal is recovered to fairly high accuracy: >> normxp-x 8

9 a Original b Minimum energy reconstruction c Recovered Figure : D recovery experiment for l minimization with equality constraints a Original length 5 signal x consisting of 0 spikes b Minimum energy linear reconstruction x0 c Minimum l reconstruction xp ans = 4746e-05 3 Phantom reconstruction A large scale example is given in tveq phantom examplem This files recreates the phantom reconstruction experiment first published in [4] The Shepp-Logan phantom shown in Figure a is measured at K = 548 locations in the D Fourier plane; the sampling pattern is shown in Figure b The image is then reconstructed exactly using T V The star-shaped Fourier-domain sampling pattern is created with % number of radial lines in the Fourier domain L = ; % Fourier samples we are given [MMhmhmhi] = LineMaskLn; OMEGA = mhi; The auxiliary function LineMaskm found in the Measurements subdirectory creates the starshaped pattern consisting of lines through the origin The vector OMEGA contains the locations of the frequencies used in the sampling pattern This example differs from the previous one in that the code operates in large-scale mode The measurement matrix in this example is making the system 8 far too large to solve or even store explicitly In fact the measurment matrix itself would require almost 3 gigabytes of memory if stored in double precision Instead of creating the measurement matrix explicitly we provide function handles that take a vector x and return Ax As discussed above the Newton steps are solved for using an implicit algorithm To create the implicit matrix we use the function handles A A_fhpz OMEGA; At At_fhpz OMEGA n; The function A fhpm takes a length N vector we treat n n images as N := n vectors and returns samples on the K frequencies Actually since the underlying image is real A fhpm 9

10 a Phantom b Sampling pattern c Min energy d min-tv reconstruction Figure : Phantom recovery experiment return the real and imaginary parts of the D FFT on the upper half-plane of the domain shown in Figure b To solve T V we call xp = tveq_logbarrierxbp A At y e- e-8 600; The variable xbp is the initial guess which is again the minimal energy reconstruction shown in Figure c y are the measurements ande- is the desired precision The sixth input is the value of µ the amount by which to increase τ k at each iteration; see Section The last two inputs are parameters for the large-scale solver used to find the Newton step The solvers are iterative with each iteration requiring one application of A and one application of At The seventh and eighth arguments above state that we want the solver to iterate until the solution has precision 0 8 that is it finds a z such that Hz g / g 0 8 or it has reached 600 iterations The recovered phantom is shown in Figure d We have X T V X / X Optimization routines We include a brief description of each of the main optimization routines type help <function> in MATLAB for details Each of these m-files is found in the Optimization subdirectory 0

11 cgsolve Solves Ax = b where A is symmetric positive definite using the Conjugate Gradient method ldantzig pd ldecode pd leq pd lqc logbarrier lqc newton Solves P D the Dantzig selector using a primal-dual algorithm Solves the norm approximation problem P A for decoding via linear programming using a primal-dual algorithm Solves the standard Basis Pursuit problem P using a primal-dual algorithm Barrier outer iterations for solving quadratically constrained l minimization P Newton inner iterations for solving quadratically constrained l minimization P tvdantzig logbarrier Barrier iterations for solving the TV Dantzig selector T V D tvdantzig newton Newton iterations for T V D tveq logbarrier Barrier iterations for equality constrained TV minimizaiton T V tveq newton Newton iterations for T V tvqc logbarrier Barrier iterations for quadratically constrained TV minimization T V tvqc newton Newton iterations for T V

12 Appendix A l minimization with equality constraints When x A and b are real then P can be recast as the linear program min u i subject to x i u i 0 xu i x i u i 0 Ax = b which can be solved using the standard primal-dual algorithm outlined in Section again see [ Chap] for a full discussion Set f u;i := x i u i f u;i := x i u i with λ u;i λ u;i the corresponding dual variables and let f u be the vector f u; f u;n T and likewise for f u λ u λ u Note that δi f u;i = δ i δi f u;i = δ i f u;i = 0 f u;i = 0 where δ i is the standard basis vector for component i Thus at a point x u; v λ u λ u the central and dual residuals are Λu f r cent = u /τ Λ u f u λu λ r dual = u + A T v λ u λ u and the Newton step 5 is given by: Σ Σ A T Σ Σ 0 x u = w /τ fu + fu A T v w := /τ fu + fu A 0 0 v b Ax with Σ = Λ u F u w 3 Λ u Fu Σ = Λ u Fu + Λ u Fu The F for example are diagonal matrices with F ii = f ;i and f ;i we can eliminate and solve Σ x = Σ Σ Σ x = Σ x w Σ Σ w A T v u = Σ w Σ x AΣ AT v = w 3 AΣ x w Σ x Σ Σ w = /f ;i Setting This is a K K positive definite system of equations and can be solved using conjugate gradients Given x u v we calculate the change in the inequality dual variables as in 4: λ u = Λ u Fu x + u λ u /τfu λ u = Λ u Fu x + u λ u /τfu

13 B l norm approximation The l norm approximation problem P A can also be recast as a linear program: min xu M m= u m subject to Ax u y 0 Ax u + y 0 recall that unlike the other 6 problems here the M N matrix A has more rows than columns For the primal-dual algorithm we define Given a vector of weights σ R M σ m f u;m = m m f u = Ax u y f u = Ax u + y A T σ σ σ m f u;m fu T A ;m = T ΣA A T Σ ΣA Σ σ m f u;m = At a point x u; λ u λ u the dual residual is A r dual = T λ u λ u λ u λ u and the Newton step is the solution to A T Σ A A T Σ x = Σ A Σ u where m m Σ = Λ u F u Σ = Λ u F u A T σ σ σ m f u;m fu T A ;m = T ΣA A T Σ ΣA Σ /τ A T f u /τ f u Λ u F u Λ u F u Setting Σ x = Σ Σ Σ we can eliminate u = Σ w Σ A x and solve A T Σ x A x = w A T Σ Σ w + fu + f := for x Again A T Σ x A is a N N symmetric positive definite matrix it is straightforward to verify that each element on the diagonal of Σ x will be strictly positive and so the Conjugate Gradients algorithm can be used for large-scale problems Given x u the step directions for the inequality dual variables are given by u λ u = Λ u Fu A x u λ u /τfu λ u = Λ u Fu A x + u λ u /τfu w w C l Dantzig selection An equivalent linear program to P D in the real case is given by: min u i subject to x u 0 xu i x u 0 A T r ɛ 0 A T r ɛ 0 3

14 where r = Ax b Taking f u = x u f u = x u f ɛ = A T r ɛ f ɛ = A T r ɛ the residuals at a point x u; λ u λ u λ ɛ λ ɛ the dual residual is λu λ r dual = u + A T Aλ ɛ λ ɛ λ u λ u and the Newton step is the solution to A T AΣ a A T A + Σ Σ x = Σ Σ u where Again setting we can eliminate and solve /τ A T A f ɛ Σ = Λ u F u Σ = Λ u F u Σ a = Λ ɛ F ɛ + f ɛ /τ f u Λ u F u Λ u F u Λ ɛ F ɛ Σ x = Σ Σ Σ u = Σ w Σ x A T AΣ a A T A + Σ x x = w Σ Σ w fu + f u + f u := for x As before the system is symmetric positive definite and the CG algorithm can be used to solve it Given x u the step directions for the inequality dual variables are given by λ u = Λ u Fu x u λ u /τfu λ u = Λ u Fu x u λ u /τfu λ ɛ = Λ ɛ Fɛ A T A x λ ɛ /τfɛ λ ɛ = Λ ɛ Fɛ A T A x λ ɛ /τfɛ w w D l minimization with quadratic constraints The quadractically constrained l minimization problem P can be recast as the second-order cone program min u i subject to x u 0 xu i x u 0 Ax b ɛ 0 Taking f u = x u f u = x u f ɛ = Ax b ɛ we can write the Newton step as in 8 at a point x u for a given τ as Σ fɛ A T A + fɛ A T rr T A Σ x f u = fu + fɛ A T r Σ Σ u τ fu fu := 4 w w

15 where r = Ax b and As before we set and eliminate u leaving us with the reduced system Σ x f ɛ Σ = F u Σ = F u + F u + F u Σ x = Σ Σ Σ u = Σ w Σ x A T A + fɛ A T rr T A x = w Σ Σ w which is symmetric positive definite and can be solved using CG E Total variation minimization with equality constraints The equality constrained TV minimization problem min x TVx subject to Ax = b can be rewritten as the SOCP min tx t st D x t Ax = b Defining the inequality functions f t = D t i j = n 9 we have D T f t = D x t δ f t f T t = D T D xx T D T D t D T D xδ T t δ x T D T D t δ δ T D f t = D 0 0 δ δ T where δ is the Kronecker vector that is in entry and zero elsewhere For future reference: D T σ f t = h ΣD h x + Dv T ΣD v x σt σ f t ft T = σ f t = BΣB T BT Σ ΣT B T ΣT D T h ΣD h + D T v ΣD v 0 0 Σ where Σ = diag{σ } T = diagt D h has the D h; as rows and likewise for D v and B is a matrix that depends on x: B = D T h Σ h + D T v Σ v with Σ h = diagd h x Σ v = diagd v x 5

16 The Newton system 8 for the log-barrier algorithm is then H BΣ A T Σ B T Σ 0 x t = DT h F t D h x + Dv T F t D v x τ Ft t := A 0 0 v 0 w w 0 where Eliminating t H = D T h F t D h + D T v F t D v + BF t B T t = Σ w Σ B T x = Σ w Σ Σ h D h x Σ Σ v D v x the reduced N + K N + K system is H A T x w = A 0 v 0 0 with H = H BΣ Σ BT = D T h Σ b Σ h F t D h + D T v Σ b Σ v F t D v + D T h Σ b Σ h Σ v D v w = w BΣ Σ w + D T v Σ b Σ h Σ v D h = w D T h Σ h + D T v Σ v Σ Σ w Σ b = F t Σ Σ The system of equations 0 is symmetric but not positive definite Note that D h and D v are very sparse matrices and hence can be stored and applied very efficiently This allows us to again solve the system above using an iterative method such as SYMMLQ [6] F Total variation minimization with quadratic constraints We can rewrite T V as the SOCP t subject to D x t i j = n min xt Ax b ɛ where D is as in Taking f t as in 9 and with f ɛ = where r = Ax b f ɛ = A T r f 0 ɛ fɛ T = Ax b ɛ A T rr T A 0 f 0 0 ɛ = A A Also D f t = D 0 0 δ δ T f ɛ = A A

17 The Newton system is similar to that in equality constraints case: H BΣ x D T Σ B T = h Ft D h x + Dv T F t D v x + fɛ A T r Σ t τ tft := where tft = t /f t and Again eliminating t H = Dh T Ft D h + Dv T Ft D v + BFt B T fɛ A T A + fɛ A T rr T A Σ = T Ft Σ = Ft + Ft T t = Σ w Σ Σ h D h x Σ Σ v D v x w w the key system is where H x = w D T h Σ h + D T v Σ v Σ Σ w H = H BΣ Σ BT = D T h Σ b Σ h F t D h + D T v Σ b Σ v F t D v + Σ b = F t D T h Σ b Σ h Σ v D v + D T v Σ b Σ h Σ v D h fɛ A T A + fɛ A T rr T A Σ Σ The system above is symmetric positive definite and can be solved with CG G Total variation minimization with bounded residual correlation The T V Dantzig problem has an equivalent SOCP as well: t subject to D x t i j = n min xt A T Ax b ɛ 0 A T Ax b ɛ 0 The inequality constraint functions are f t = D x t i j = n f ɛ = A T Ax b ɛ f ɛ = A T Ax b ɛ with and σ f ɛ; = A T Aσ 0 σ f ɛ; = σ f ɛ; fɛ T ; = σ f ɛ; fɛ T ; = A T Aσ 0 A T AΣA T A

18 Thus the log barrier Newton system is nearly the same as in the quadratically constrained case: H BΣ x D T Σ B T = h Ft D h x + Dv T F t D v x + A T Afɛ fɛ w Σ t τ tft := w where H = Dh T Ft D h + Dv T Ft D v + BFt B T + A T AΣ a A T A Σ = T Ft Σ = Ft Σ a = Fɛ Eliminating t as before + F t T + F ɛ t = Σ w Σ Σ h D h x Σ Σ v D v x the key system is where H x = w D T h Σ h + D T v Σ v Σ Σ w H = D T h Σ b Σ h F t D h + D T v Σ b Σ v F t D v + Σ b = F t D T h Σ b Σ h Σ v D v + D T v Σ b Σ h Σ v D h + A T AΣ a A T A Σ Σ References [] F Alizadeh and D Goldfarb Second-order cone programming Math Program Ser B 95: [] S Boyd and L Vandenberghe Convex Optimization Cambridge University Press 004 [3] E Candès and J Romberg Quantitative robust uncertainty principles and optimally sparse decompositions To appear in Foundations of Comput Math 005 [4] E Candès J Romberg and T Tao Robust uncertainty principles: Exact signal reconstruction from highly incomplete frequency information Submitted to IEEE Trans Inform Theory June 004 Available on thearxiv preprint server: mathgm/ [5] E Candès J Romberg and T Tao Stable signal recovery from incomplete and inaccurate measurements Submitted to Communications on Pure and Applied Mathematics March 005 [6] E Candès and T Tao Near-optimal signal recovery from random projections and universal encoding strategies submitted to IEEE Trans Inform Theory November 004 Available on the ArXiV preprint server: mathca/04054 [7] E Candès and T Tao The Dantzig selector: statistical estimation when p is much smaller than n Manuscript May 005 [8] E J Candès and T Tao Decoding by linear programming To appear in IEEE Trans Inform Theory December 005 8

19 [9] T Chan G Golub and P Mulet A nonlinear primal-dual method for total variation-based image restoration SIAM J Sci Comput 0: [0] S S Chen D L Donoho and M A Saunders Atomic decomposition by basis pursuit SIAM J Sci Comput 0: [] D Goldfarb and W Yin Second-order cone programming methods for total variation-based image restoration Technical report Columbia University 004 [] H Hintermüller and G Stadler An infeasible primal-dual algorithm for TV-based infconvolution-type image restoration To appear in SIAM J Sci Comput 005 [3] M Lobo L Vanderberghe S Boyd and H Lebret Applications of second-order cone programming Linear Algebra and its Applications 84: [4] Y E Nesterov and A S Nemirovski Interior Point Polynomial Methods in Convex Programming SIAM Publications Philadelphia 994 [5] J Nocedal and S J Wright Numerical Optimization Springer New York 999 [6] C C Paige and M Saunders Solution of sparse indefinite systems of linear equations SIAM J Numer Anal 4 September 975 [7] J Renegar A mathematical view of interior-point methods in convex optimization MPS- SIAM Series on Optimization SIAM 00 [8] L I Rudin S Osher and E Fatemi Nonlinear total variation noise removal algorithm Physica D 60: [9] J R Shewchuk An introduction to the conjugate gradient method without the agonizing pain Manuscript August 994 [0] S J Wright Primal-Dual Interior-Point Methods SIAM Publications 997 9