Efficient quadratic penalization through the partial minimization technique

Transcription

1 This article has been accepted for pblication in a ftre isse of this jornal, bt has not been flly edited Content may change prior to final pblication Citation information: DOI 9/TAC , IEEE Transactions on Atomatic Control Efficient qadratic penalization throgh the partial imization techniqe Aleksandr Y Aravkin Dmitriy Drsvyatskiy Tristan van Leewen Abstract Common comptational problems, sch as parameter estimation in dynamic models and PDE constrained optimization, reqire data fitting over a set of axiliary parameters sbject to physical constraints over an nderlying state Naive qadratically penalized formlations, commonly sed in practice, sffer from inherent ill-conditioning We show that srprisingly the partial imization techniqe reglarizes the problem, making it well-conditioned This viewpoint sheds new light on variable projection techniqes, as well as the penalty method for PDE constrained optimization, and motivates robst extensions In addition, we otline an inexact analysis, showing that the partial imization sbproblem can be solved very loosely in each iteration We illstrate the theory and algorithms on bondary control, optimal transport, and parameter estimation for robst dynamic inference I INTRODUCTION In this work, we consider a strctred class of optimization problems having the form y, fy + g sbject to Ay q Here, f : R n R is convex and smooth, q R n is a fixed vector, and A is a smoothly varying invertible matrix For now, we make no assmptions on g : R d R, thogh in practice, it is typically either a smooth or a simple nonsmooth fnction Optimization problems of this form often appear in PDE constrained optimization [6], [2], [23], Kalman filtering [2], [4], [5], bondary control problems [], [9], and optimal transport [], [3] Typically, encodes axiliary variables while y encodes the state of the system; the constraint Ay q corresponds to a discretized PDE describing the physics Since the discretization Ay q is already inexact, it is appealing to relax it in the formlation A seegly naive relaxation approach is based on the qadratic penalty: y, F y, : fy + g + Ay q 2 2 Here > is a relaxation parameter for the eqality constraints in, corresponding to relaxed physics The classical qadratic penalty method in nonlinear programg proceeds by applying an iterative optimization algorithm to the nconstrained problem 2 ntil some teration criterion is satisfied, then increasing, and repeating the procedre with the previos iterate sed as a warm start For a detailed discssion, see eg [7, Section Department of Applied Mathematics, University of Washington, Seattle, WA saravkin@wed Department of Mathematics, University of Washington, Seattle, WA ddrsv@wed Department of Mathematics, Utrecht University, Utrecht, Nethelands TvanLeewen@nl logf-f * logf-f * iteration iteration Fig : Top panel shows logf k F of L-BFGS with partial imization applied to 2 for the bondary control problem in Section III Each partial imization solves a least sqares problem in y Bottom panel shows logf k F of L-BFGS withot partial imization applied to 2, for the same vales of Both methods are initialized at random, and y that imizes F, Performance of L-BFGS withot partial imization degrades as increases, while performance of L-BFGS with partial imization is insensitive to 7] The athors of [22] observe that this strategy helps to avoid extraneos local ima, in contrast to the original formlation From this consideration alone, the formlation 2 appears to be sefl Conventional wisdom teaches s that the qadratic penalty techniqe is rarely appropriate The difficlty is that one mst allow to tend to infinity in order to force near-feasibility in the original problem ; the residal error Ay q at an optimal pair, y for the problem 2 is at best on the c 27 IEEE Personal se is permitted, bt repblication/redistribtion reqires IEEE permission See for more information

2 This article has been accepted for pblication in a ftre isse of this jornal, bt has not been flly edited Content may change prior to final pblication Citation information: DOI 9/TAC , IEEE Transactions on Atomatic Control 2 order of O/ Conseqently, the maximal eigenvale of the Hessian of the penalty term scales linearly with and the problems 2 become comptationally difficlt Indeed, the maximal eigenvale of the Hessian deteres the behavior of nmerical methods gradient descent, qasi-newton far away from the soltion a regime in which most hge scale problems are solved Figre is a simple nmerical illstration of this inherent difficlty on a bondary control problem; see Section III for more details on the problem formlation The bottom panel in the figre tracks progress of the objective fnction in 2 when an L-BFGS method is applied jointly in the variables, y After iterations of L-BFGS, the objective vale significantly increases with increasing, while the actal imal vale of the objective fnction converges to that of, and so hardly changes In other words, performance of the method scales poorly with, illstrating the ill-conditioning In this paper, we show that by sing a simple partial imization step, this complexity blow-p can be avoided entirely The reslting algorithm is perfectly sited for many large-scale problems, where satisfying the constraint Ay q to high accracy is not reqired or even possible The strategy is straightforward: we rewrite 2 as ϕ + g, 3 where the fnction ϕ is defined implicitly by { ϕ fy + Ay q 2} 4 y We will call ϕ the redced fnction Thogh this approach of imizing ot the variable y, sometimes called variable projection, is widely sed eg [7], [], [22], little theoretical jstification for its speriority is known In this work, we show that not only does partial imization perform well nmerically for the problem class 2, bt is also theoretically gronded We prove that srprisingly the Lipschitz constant of ϕ is bonded by a constant independent of Therefore, iterative methods can be applied directly to the formlation 3 The performance of the new method is illstrated sing a toy example top panel of Figre We se L-BFGS to attack the oter problem; solving it within 35 iterations The inner solver for the toy example simply solves the least sqares problem in y The inner problem 4 can be solved efficiently since its condition nmber is nearly independent of When f is a convex qadratic and A is sparse, one can apply sparse direct solvers or iterative methods sch as LSQR [8] More generally, when f is an arbitrary smooth convex fnction, one can apply first-order methods, which converge globally linearly with the rate governed by the condition nmber of the strongly convex objective in 4 Qasi-newton methods or variants of Newton s method are also available; even if g is nonsmooth, BFGS methods can still be applied The otline of the paper is as follows In Section II, we present complexity garantees of the partial imization techniqe In Section III, we nmerically illstrate the overall approach on bondary control and optimal transport problems, and on tning an oscillator from very noisy measrements II THEORY In this section, we show that the proposed framework is insensitive to the parameter Throghot we assme that f and A are C 2 -smooth, f is convex, and A is invertible for every R n To shorten the formlas, in this section, we will se the symbol A instead of A throghot Setting the stage, define the fnction A qick comptation shows ϕ, y fy + 2 A y q 2 y ϕ, y fy + A T A y q, ϕ, y G, y T A y q, where G, y is the Jacobian with respect to of the map A y Clearly the Lipschitz constant Lip ϕ scales with This can be detrimental to nmerical methods For example, basic gradient descent will find a point k, y k satisfying ϕ k, y k 2 < ɛ after at most O Lip ϕϕ,x ϕ ɛ iterations [6, Section 23] As discssed in the introdction, imizing ϕ amonts to the imization problem ϕ for the redced fnction ϕ defined in 4 Note since f is convex and A is invertible, the fnction ϕ, admits a niqe imizer, which we denote by y Appealing to the classical implicit fnction theorem eg [2, Theorem 58] we dedce that ϕ is differentiable with ϕ ϕ, y G, y T A y q We aim to pper bond the Lipschitz constant of ϕ by a qantity independent of We start by estimating the residal A y q Throghot the paper, we se the following simple identity Given an invertible map F : R n R n and invertible matrix C, for any points x R n and nonzero R, we have F x F x, and 2 C F C T C T F C We often apply this observation to the invertible map F x fx + Bx, where B is a positive definite matrix Lemma Residal bond For any point, the ineqality holds: f A q A y q Proof Note that the imizers y of ϕ, are characterized by first order optimality conditions 5 fy + A T A y q 6 Applying the implicit fnction theorem, we dedce that y depends C 2 -smoothly on with y given by 2 fy + A T A A T A y q On the other hand, from the eqality 6 we have y f + A T A A T f q + AT A A T q c 27 IEEE Personal se is permitted, bt repblication/redistribtion reqires IEEE permission See for more information

3 This article has been accepted for pblication in a ftre isse of this jornal, bt has not been flly edited Content may change prior to final pblication Citation information: DOI 9/TAC , IEEE Transactions on Atomatic Control 3 Therefore, we dedce A y q A f + AT A A T q q A T f A + I I q Define now the operator F : A T and the point z : F q Note that F x x f A + I f A x Letting L be a Lipschitz constant of F, we obtain A y q F q F z L q z L q F q L A T fa q Now the inverse fnction theorem yields for any point y the ineqality F y F F y 2 f A F y + I, where the last ineqality follows from the fact that by convexity of f A all eigenvales of 2 f A are nonnegative Ths we may set L, completing the proof For ease of reference, we record the following direct corollary Corollary For any point, we have y A q A A y q A f A q Next we will compte the Hessian of ϕ, y, and se it to show that the norm of the Hessian of ϕ is bonded by a constant independent of Defining ] ] R, y, v [G, y T v and K, v [A T v, we can partition the Hessian as follows: [ ] 2 ϕ ϕ ϕ y where ϕ, y ϕ y ϕ yy G, y T G, y + R, y, A y q, ϕ yy, y 2 fy + A T A, ϕ y, y K, A y q + A T G, y See [22, Section 4] for more details Moreover, it is known that the Hessian of the redced fnction ϕ admits the expression [22, Eqation 22] 2 ϕ ϕ, y ϕ y, y ϕ yy, y ϕ y, y, 7 which is simply the Schr complement of ϕ yy, y in 2 ϕ, y We define the operator norms G, y T : sp [G, y T v], v A T : sp [A T v] v Using this notation, we can prove the following key bonds Corollary 2 For any points and y, the ineqalities hold: 2 ϕ yy, y A, f A q R, y, A y q G, y, f A q A T K, A y q Proof The first bond follows by the ineqality ϕ yy, y A A T 2 fya + I A T A 2, and the remaining bonds are immediate from Lemma Next, we need the following elementary linear algebraic fact Lemma 2 For any positive semidefinite matrix B and a real >, we have I I + B 2 B Proof Define the matrix F I I + B and consider an arbitrary point z Observing the ineqality I + B and defining the point p : I + Bz, we obtain F z z I + B z I + B p I + B z I + B p z Bz B z Since this holds for all z, the reslt follows Ptting all the pieces together, we can now prove the main theorem of this section Theorem Norm of the redced Hessian The operator norm of 2 ϕ is bonded by a qantity CA, f, q, independent of Proof To simplify the proof, define G : G, y, R : R, y, A y q, K : K, A y q, and ϕ yy, y When, the operator norm of 2 φ has a trivial bond c 27 IEEE Personal se is permitted, bt repblication/redistribtion reqires IEEE permission See for more information

4 This article has been accepted for pblication in a ftre isse of this jornal, bt has not been flly edited Content may change prior to final pblication Citation information: DOI 9/TAC , IEEE Transactions on Atomatic Control 4 directly from eqation 7 For large, after rearranging 7, we can write 2 ϕ as follows: R 2 K T K + K T A T G + G T A K + G T G 2 G T A A T G 8 Corollary 2 implies that the operator norm of the first row of 8 is bonded above by the qantity L G + L2 A T 2 A 2 + 2L A T A 2 A T G, where we set L : f A q Notice that the expression in 9 is independent of We rewrite the second row of 8 sing the explicit expression for : G T G 2 G T A A T G 9 G T G G T A T 2 fy A + I G G T I Applying Lemma 2 with B A T G T G 2 G T A A T G Setting A T 2 fy A + I G 2 fy A, we have G 2 A T 2 fy A CA, f, q, : L G + 2L A T A 2 A T G + G 2 A 2 2 fy + L2 A T 2 A 2 For >, the last term is always trivially bonded by L 2 A T 2 A 2 and the reslt follows A Inexact analysis of the projection sbproblem In practice, one can rarely evalate ϕ exactly It is therefore important to nderstand how inexact soltions of the inner problems 4 impact iteration complexity of the oter problem The reslts presented in the previos section form the fondation for sch an analysis For simplicity, we assme that g is smooth, thogh the reslts can be generalized, as we comment on shortly In this section, we compte the overall complexity of the partial imization techniqe when the oter nonconvex imization problem 3 is solved by an inexact gradient descent algorithm When g is nonsmooth, a completely analogos analysis applies to the prox-gradient method We only focs here on gradient descent, as opposed to more sophisticated methods, since the analysis is straightforward We expect qasi- Newton methods and limited memory variants to exhibit exactly the same behavior eg Figre We do not perform a similar analysis here for inexact qasi-newton methods, as the global efficiency estimates even for exact qasi-newton methods for nonconvex problems are poorly nderstood Define the fnction H : g + ϕ Let β > be the Lipschitz constant of the gradient H g + ϕ Fix a constant c >, and sppose that in each iteration k, we compte a vector v k with v k H k c k Consider then the inexact gradient descent method k+ k β v k Then we dedce H k+ H k H k, β v k + β 2 β v k 2 v 2β k H k 2 H k 2 Hence we obtain the convergence garantee: i,,k H i 2 k k H i 2 i 2β H H k 2β H H k + k k v k H k 2 i + c2 π 2 6k β2 2 + c 2 π 2 /6 k where is sed to go from line to line 2 Now, if we compte g exactly, the qestion is how many inner iterations are needed to garantee v k H k c k For fixed k, the inner objective is ϕ k, y fy + 2 A ky q 2 The condition nmber ratio of Lipschitz constant of the gradient over the strong convexity constant of ϕ k, y in y is κ k : Lipf A k 2 + A k 2 A k 2 Notice that κ k converges to the sqared condition nmber of A k as Gradient descent on the fnction ϕ k, garantees y i y 2 ɛ after κ k log y y 2 ɛ iterations Then we have ϕ k, y i ϕ k, y G, y y i y Since we want the left hand side to be bonded by c k, we simply need to ensre y i y 2 c 2 k 2 2 G k, y 2 Therefore the total nmber of inner iterations is no larger than y κ k log y 2 G k, y 2 c 2 k 2 2, which grows very slowly with k and with In particlar, the nmber of iterations to solve the inner problem scales as logk to achieve a global k rate in H 2 If instead we se a fast-gradient method [6, Section 22] for imizing ϕ k,, we can replace κ k with the mch better qantity κ k throghot c 27 IEEE Personal se is permitted, bt repblication/redistribtion reqires IEEE permission See for more information

5 This article has been accepted for pblication in a ftre isse of this jornal, bt has not been flly edited Content may change prior to final pblication Citation information: DOI 9/TAC , IEEE Transactions on Atomatic Control 5 III NUMERICAL ILLUSTRATIONS In this section, we present two representative examples of PDE constrained optimization bondary control and optimal transport and a problem of robst dynamic inference In each case, we show that practical experience spports theoretical reslts from the previos section In particlar, in each nmerical experiment, we stdy the convergence behavior of the proposed method as increases A Bondary control In bondary control, the goal is to steer a system towards a desired state by controlling its bondary conditions Perhaps the simplest example of sch a problem is the following Given a sorce qx, defined on a domain Ω, we seek bondary conditions sch that the soltion to the Poisson problem y q y Ω for x Ω is close to a desired state y d Discretizing the PDE yields the system Ay + B q where A is a discretization of the Laplace operator on the interior of the domain and B coples the interior gridpoints to the bondary The corresponding PDE-constrained optimization problem is given by,y 2 y y d 2 2 sbject to Ay + B q, whereas the penalty formlation reads,y 2 y y d Ay + B q 2 2 Since both terms are qadratic in y, we can qickly solve for y explicitly Nmerical experiments: In this example, we consider an L-shaped domain with a sorce q shaped like a Gassian bell, as depicted in figre 2 Or goal is to get a constant distribtion y d in the entire domain The soltion for is shown in figre 3 a To solve the optimization problem we se a steepest-descent method with a fixed stepsize, detered from the Lipschitz constant of the gradient The reslt of the constrained formlation is shown in figre 3 b We see that by adapting the bondary conditions we get a more even distribtion The convergence behavior for varios vales of is shown in figre 4 a We see that as, the behavior tends towards that of the constrained formlation, as expected The Lipschitz constant of the gradient evalated at the initial point, as a fnction of is shown in figre 4 b; the crve levels off as the theory predicts B Optimal transport The second class of PDE-constrained problems we consider comes from optimal transport, where the goal is to detere a mapping, or flow, that optimally transforms one mass density fnction into another Say we have two density fnctions y x and y T x, with x Ω, we can formlate the problem as Fig 2: L-shaped domain with the sorce fnction q a b Fig 3: Bondary vales,, and soltion in the interior for the initial and optimized bondary vales are depicted in a and b respectively gradient - -2 inf iteration a Lipschitz constant Fig 4: The convergence plots for varios vales of are depicted in a, while b shows the dependence of the nmerically compted Lipschitz constant on t, x finding a flowfield, t, x, sch that y 2 t, x T x yt, x and y x y, x, where yt, x solves y t + y Discretizing sing an implicit Lax-Friedrichs scheme [2], the PDE reads Ay q, b c 27 IEEE Personal se is permitted, bt repblication/redistribtion reqires IEEE permission See for more information

6 This article has been accepted for pblication in a ftre isse of this jornal, bt has not been flly edited Content may change prior to final pblication Citation information: DOI 9/TAC , IEEE Transactions on Atomatic Control 6 y data Fig 8: Left: Densities, Gassian black dash, Hber red solid, and Stdent s t ble dot Right: Negative Log Likelihoods gradient Fig 5: The initial left and desired right mass density are shown yt Fig 6: The mass density obtained after optimization left and the corresponding time-averaged flow field right are shown where q contains the initial condition and we have I + tb M I + tb 2 A M M I + tb N with M a for-point averaging matrix and B containing the discretization of the derivative terms Adding reglarization to promote smoothness of and y in time [2], we obtain the problem ,y iteration a 2 P y y T α2 2 yt Ldiag inf sbject to Ay q Lipschitz constant Fig 7: Convergence plots for varios vales of are shown in a, while b shows the nmerically compted Lipschitz constant as a fnction of b, Here, P restricts the soltion y to t T, α is a reglarization parameter and L is a block matrix with I on the main and pper diagonal The penalized formlation is,y 2 P y y T α2 2 yt Ldiag + 2 Ay q 2 2 Again the partial imization in y amont to imizing a qadratic fnction Nmerical experiments: For the nmerical example we consider the domain Ω [, ] 2, discretized with x /6 and T /32 with a stepsize of t /8 The initial and desired state are depicted in figre 5 The reslting state obtained at time T and the corresponding time-averaged flowfield are depicted in figre 6 The initial flow was generated by iid samples from a standard Gassian random variable To imize 2, we sed a steepest-descent method with constant step size, sing the largest eigenvale of the Gass-Newton Hessian at the initial as an estimate of the Lipschitz constant The convergence behavior for varios vales of as well as the corresponding estimates of the Lipschitz constant at the final soltion are shown in figre 7 C Robst dynamic inference with the penalty method In many settings, data is natrally very noisy, and a lot of effort mst be spent in pre-processing and cleaning before applying standard inversion techniqes To narrow the scope, consider dynamic inference, where we wish to infer both hidden states and nknown parameters driven by an nderlying ODE Recent efforts have focsed on developing inference formlations that are robst to otliers in the data [3], [4], [9], sing convex penalties sch as l, Hber [4] and non-convex penalties sch as the Stdent s t log likelihood in place of the least sqares penalty The goal is to develop formlations and estimators that achieve adeqate performance when faced with otliers; these may arise either as gross measrement errors, or real-world events that are not modeled by the dynamics Figre 8 shows the probability density fnctions and penalties corresponding to Gassian, Hber, and Stdent s t densities Qadratic tail growth corresponds to extreme decay of the Gassian density for large inpts, and linear growth of the inflence of any measrement on the fit In contrast, Hber and Stdent s t have linear and sblinear tail growth, respectively, which ensres every observation has bonded inflence We focs on the Hber fnction [4], since it is both C - smooth and convex In particlar, the fnction fy in and 4 is chosen to be a composition of the Hber with an observation model Note that Hber is not C 2, so this case is c 27 IEEE Personal se is permitted, bt repblication/redistribtion reqires IEEE permission See for more information

7 This article has been accepted for pblication in a ftre isse of this jornal, bt has not been flly edited Content may change prior to final pblication Citation information: DOI 9/TAC , IEEE Transactions on Atomatic Control 7 not immediately captred by the theory we propose However, Hber can be closely approximated by a C 2 fnction [8], and then the theory flly applies For or nmerical examples, we apply the algorithm developed in this paper directly to the Hber formlation We illstrate robst modeling sing a simple representative example Consider a 2-dimensional oscillator, governed by the following eqations: [ ] [ y ] [ ] [ ] y sinωt + 3 y 2 y 2 where we can interpret as the freqency, and 2 is the damping Discretizing in time, we have [ ] k+ [ y 2 t 2 t 2 ] [ ] k [ ] y sinωtk + t y 2 t y 2 We now consider direct observations of the second component, z k [ ] [ ] k y + w y k, w k N, R k 2 We can formlate the joint inference problem on states and measrements as follows: φ, y :,y 2 ρ R /2 Hy z st Gy v, 4 with v k sinωt k, v the initial condition for y, and ρ is either the least sqares or Hber penalty, and we se the following definitions: R diag{r k } H diag{h k } y vec{y k } z vec{z,, z N } I G G 2 I G N I Note in particlar that there are only two nknown parameters, ie R 2, while the state y lies in R 2N, with N the nmber of modeled time points The redced optimization problem for is given by f : 2 ρr /2 HG v z To compte the derivative of the ODE-constrained problem, we can se the adjoint state method Defining the Lagrangian Ly, x, R 2 ρ /2 Hy z + x, Gy v, we write down the optimality conditions L and obtain y G v x G T H T R /2 ρr /2 HG v z f x, Gy The inexact penalized problem is given by ϕ, y,y 2 ρ R /2 Hy z + 2 Gy v 2 5 TABLE I: Reslts for Kalman experiment Penalty method for both least sqares and hber achieves the same reslts for moderate vales of as does the projected formlation While Hber reslts converge to nearly the tre parameters, least sqares reslts converge to an incorrect parameter estimate ρ Iter Opt 3 l , 98 5 l , 43 7 l , 9 l , 8 l , 8 3 h h , 7 h , 9 h , h , We then immediately find y arg y 2 ρ R /2 Hy z + Gy v 2 2 φ Gy Gy v When ρ is the least sqares penalty, y is available in closed form However, when ρ is the Hber, y reqires an iterative algorithm Rather than solving for y sing a first-order method, we se IPsolve, an interior point method well sited for Hber [5] Even thogh each iteration reqires inversions of systems of size ON, these systems are very sparse, and the complexity of each iteration to compte y is ON for any piecewise linear qadratic fnction [5] Once again, we see that the comptational cost does not scale with Nmerical Experiments: We simlate a data contaation scenario by solving the ODE 3 for the particlar parameter vale 2, The second component of the reslting state y is observed, and the observations are contaated In particlar, in addition to Gassian noise with standard deviation σ, for % of the measrements niformly distribted errors in [, 2] are added The state y R 24 is finely sampled over 4 periods For the least sqares and the Hber penalty with κ, we solved both the ODE constrained problem 4 and the penalized version 5 for { 3, 5, 7, 9 } The reslts are presented in Table I The Hber formlation behaves analogosly to the formlation sing least sqares; in particlar the oter projected fnction in is no more difficlt to imize And, as expected, the robst Hber penalty finds the correct vales for the parameters A state estimate generated from -estimates corresponding to large is shown in Figre 9 The hberized approach is able to ignore the otliers, and recover both better estimates of the nderlying dynamics parameters, and the tre observed and hidden components of the state y IV CONCLUSIONS In this paper, we showed that, contrary to conventional wisdom, the qadratic penalty techniqe can be sed effectively c 27 IEEE Personal se is permitted, bt repblication/redistribtion reqires IEEE permission See for more information

8 This article has been accepted for pblication in a ftre isse of this jornal, bt has not been flly edited Content may change prior to final pblication Citation information: DOI 9/TAC , IEEE Transactions on Atomatic Control Tre L2 recovery Hber recovery measrements penalty composed with a linear model as the fnction fy Nmerical reslts illstrated the overall approach, inclding convergence behavior of the penalized projected scheme, as well as modeling advantages of robst penalized formlations Acknowledgment Research of A Aravkin was partially spported by the Washington Research Fondation Data Science Professorship Research of D Drsvyatskiy was partially spported by the AFOSR YIP award FA The research of T van Leewen was in part financially spported by the Netherlands Organisation of Scientific Research NWO as part of research programme Tre L2 recovery Hber recovery Fig 9: Top panel: first samples of state y 2 solid black, least sqares recovery red dash-dot and hber recovery ble dash Noisy measrements z appear as ble diamonds on the plot, with otliers shown at the top of the panel Bottom panel: first samples of state y solid black, least sqares recovery red dash-dot and Hber recovery ble dash for control and PDE constrained optimization problems, if done correctly In particlar, when combined with the partial imization techniqe, we showed that the penalized projected scheme g + {fy + 2 } Ay b 2 y has the following advantages: The Lipschitz constant of the gradient of the oter fnction in is bonded as, and hence we can effectively analyze the global convergence of first-order methods 2 Convergence behavior of the data-reglarized convex inner problem is controlled by parametric matrix A, a fndamental qantity that does not depend on 3 The inner problem can be solved inexactly, and in this case, the nmber of inner iterations of a first-order algorithm needs to grow only logarithmically with and the oter iterations conter, to preserve the natral rate of gradient descent As an immediate application, we extended the penalty method in [22] to convex robst formlations, sing the Hber REFERENCES [] L Ambrosio and N Gigli A User s Gide to Optimal Transport In Modelling and Optimisation of Flows on Networks, pages 55 Springer, 23 [2] B D Anderson and J B Moore Optimal filtering 979, 979 [3] A Aravkin, B Bell, J Brke, and G Pillonetto An l -Laplace robst Kalman smoother IEEE Transactions on Atomatic Control, 2 [4] A Aravkin, J Brke, and G Pillonetto Robst and trend-following Stdent s t Kalman smoothers SIAM Jornal on Control and Optimization, 525: , 24 [5] A Aravkin, J V Brke, and G Pillonetto Sparse/robst estimation and Kalman smoothing with nonsmooth log-concave densities: Modeling, comptation, and theory Jornal of Machine Learning Research, 4: , 23 [6] A Aravkin, M Friedlander, F Herrmann, and T van Leewen Robst inversion, dimensionality redction, and randomized sampling Mathematical Programg, 34: 25, 22 [7] A Y Aravkin and T van Leewen Estimating nisance parameters in inverse problems Inverse Problems, 28:56, 22 [8] K P Bbe and R T Langan Hybrid l/l2 imization with applications to tomography Geophysics, 624:83 95, 997 [9] S Farahmand, G B Giannakis, and D Angelosante Dobly Robst Smoothing of Dynamical Processes via Otlier Sparsity Constraints IEEE Transactions on Signal Processing, 59: , 2 [] G Golb and V Pereyra Separable nonlinear least sqares: the variable projection method and its applications Inverse Problems, 92:R R26, Apr 23 [] M Ggat Control problems with the wave eqation SIAM Jornal on Control and Optimization, 485:326 35, 29 [2] E Haber and L Hanson Model Problems in PDE-Constrained Optimization Technical report, 27 [3] S Haker, L Zh, A Tannenbam, and S Angenent Optimal Mass Transport for Registration and Warping International Jornal of Compter Vision, 63:225 24, dec 24 [4] P J Hber Robst Statistics John Wiley and Sons, 24 [5] R E Kalman A New Approach to Linear Filtering and Prediction Problems Transactions of the AMSE - Jornal of Basic Engineering, 82D:35 45, 96 [6] Y Nesterov Introdctory lectres on convex optimization, volme 87 of Applied Optimization Klwer Academic Pblishers, Boston, MA, 24 A basic corse [7] J Nocedal and S Wright Nmerical optimization Springer Series in Operations Research and Financial Engineering Springer, New York, second edition, 26 [8] C C Paige and M A Sanders LSQR: An algorithm for sparse linear eqations and sparse least sqares ACM Transactions on Mathematical Software TOMS, 8:43 7, 982 [9] P Philip Optimal Control and Partial Differential Eqations Technical report, LMU Berlin, 29 [2] R Rockafellar and R Wets Variational Analysis, volme 37 Springer, 998 [2] A Tarantola Inverse Problem Theory SIAM, 25 [22] T van Leewen and F J Herrmann A penalty method for pdeconstrained optimization in inverse problems Inverse Problems, 32:57, 25 [23] J Viriex and S Operto An overview of fll-waveform inversion in exploration geophysics Geophysics, 746:WCC WCC26, c 27 IEEE Personal se is permitted, bt repblication/redistribtion reqires IEEE permission See for more information