Rank-Deficient Nonlinear Least Squares Problems and Subset Selection

Transcription

1 Rank-Deficient Nonlinear Least Squares Problems and Subset Selection Ilse Ipsen North Carolina State University Raleigh, NC, USA Joint work with: Tim Kelley and Scott Pope

2 Motivating Application Overview Modeling cardiovascular systems Extract biomarkers: Nonlinear parameter estimation Nonlinear dependencies among parameters Computation Solution of nonlinear least squares problem by Levenberg-Marquardt trust region algorithm Rank deficient Jacobians Errors in Jacobian evaluation How to regularize the Jacobian? Truncated SVD: NO Column subset selection: YES

3 Modeling Cardiovascular Systems Goal: Identify parameters that regulate blood flow Cardiovascular system = lumped 5-compartment model Blood flow, volume, pressure, resistance, compliance Vvc Veins Cer Veins qacp Arteries Cer Art Vac Cvc pvc Racp pac Cac Rvc qvc Vvs Sys Veins pvs Cvs qmv Rmv Left Ventricle Vlv plv Clv(t) qav Rav qac Rac Sys Art Vas pas Cas qasp Rasp [Pope, Olufsen, Ellwein, Novak, Kelley]

4 Computation System of 5 ODEs with N = 16 parameters Parameter vector p R N y = F(t,y;p) y(0) = y 0 Observations d j at M time points t j M N Nonlinear residual y(t 1,p) d 1 R(p) =. y(t M,p) d M Identify parameters p that minimize difference between measured and computed quantities min p R(p) T R(p)/2

5 Nonlinear Least Squares Problem min p R(p) T R(p)/2 Jacobian J n R (p n ) at current iterate p n Levenberg-Marquardt trust region algorithm For n = 0,1,2... ( ) 1 p n+1 = p n ν n I +Jn T J n J T n R(p n ) ν n = 0 and J n full column rank: Gauss Newton Here: ν n 0 and J n rank deficient

6 Levenberg-Marquardt Algorithm Inside a Levenberg-Marquardt Step: While iterate has not changed Trial step s = ( ν n I +J T n J n ) 1J T n R(p n ) Trial iterate p t = p n +s if p t good enough then p n+1 = p t, ν n+1 keep or decrease ν n else ν n+1 increase ν n Ideally: ν n 0, or at least ν n bounded p n converge to minimizer, or at least stationary point But here: Poor convergence (Levenberg-Marquardt stagnates) Gradient at solution not small Accuracy of solution???

7 Convergence Analysis Near solution manifold Assuming exact arithmetic Nonlinear iterations with rank deficient Jacobians: Ben-Israel 1966, Boggs 1976, Deuflhard & Heindl 1979, Schaback 1985

8 Behavior of Levenberg-Marquardt Iterates Assumptions Initial iterate p 0 close enough to a solution p J(p) Lipschitz continuous R(p ) small but not necessarily zero Then we can show Levenberg-Marquardt parameters ν n remain bounded Iterates p n approach solution manifold If p n converge then they converge to some solution (Cauchy sequence) Still need to show that p n converge

9 Convergence of Levenberg-Marquardt Iterates Model nonlinear dependence among parameters: R(p) = R (B(p)) B : R M R K If K = N then Jacobian J has full column rank Assumptions: Sufficiently close to a solution p R and B uniformly Lipschitz continuously differentiable All K singular values of B uniformly bounded away from 0 All K singular values of R uniformly bounded away from 0 R(b) T R(b)/2 has unique minimizer Then: p n converge to a solution r-linearly

10 Summary: Convergence Analysis Assumptions: Near solution manifold Initial iterate p 0 sufficiently close to a solution p Nonlinear residual R(p ) small but not necessarily zero Dependence among parameters: R(p) = R (B(p)) B : R M R K Jacobians B and R have rank K All Jacobians sufficiently smooth Then: Iterates p n converge to some solution But this assumes exact arithmetic! What happens in finite precision?

11 Finite Precision Issues Computation of trial step Effect of errors in computed Jacobian Regularization of Jacobian: Truncated SVD subset selection Singular vector perturbations: Stewart 1973

12 Computation of Trial Step p n+1 = p n +s Trial step ( J s = νi +J J) T T R Works for ν = 0 and rank deficient J Computed as minimum norm solution to linear least squares problem min x Ax b A = ( ) J νi b = ( ) R 0 Illconditioned or illposed if ν small, J rank deficient

13 Full Rank Jacobian J has full column rank: s = ( νi +J T J ) 1 J T R Condition number κ ν (J) := ( νi +J T J ) 1 J T J J = J +E has full column rank, E ǫ J ( T J) 1 JT s = νi + J R Relative error in trial step s s s κ ν (J) ( 1+ R ) ǫ J s Error in J amplified by conditioning of J and nonlinear residual R

14 Rank Deficient Jacobian: Regularization Exact trial step s = ( νi +J T J ) J T R Truncate SVD of J: Truncated Jacobian J t has singular values σ 1 σ K > σ K+1 = = σ N = 0 Truncated trial step is ( J s t = νi +Jt t) T J T t R Computed as minimum norm solution of min x A t x b A t = ( ) Jt νi b = ( ) R 0 Linear least squares problem still ill-conditioned or ill-posed

15 Truncated SVD: Errors in Jacobian SVD of exact Jacobian: J = UΣV T Nonzero singular values σ 1 σ K > 0 κ ν (J) := σ 1 max σ K σ σ 1 rank(j) = K σ ν +σ 2 SVD of perturbed Jacobian: J+E = Ũ ΣṼT, E F ǫ J Ũ, Ṽ rotations of U, V by angles θ J t truncated SVD of J +E rank( J t ) = K Trial step of truncated perturbed Jacobian ( s t = νi + J t T J ) t J t T R

16 Relative Error for Truncated SVD For ǫ sufficiently small s t s s t κ ν (J) [ ] R 1+(1+2 J tanθ) ǫ+o(ǫ 2 ) J s t tanθ: Accuracy of singular vectors of truncated Jacobian J t Error in J t amplified by Conditioning of J Inaccuracy of singular vectors Nonlinear residual R Trial step from truncated SVD not accurate if J close to matrix of rank K 1: κ ν (J) 1 Singular vectors have low accuracy: tanθ 0 Nonlinear residual R large

17 Alternative Regularization: Subset Selection Choose K very linearly independent columns J 1 from J J = ( J 1 J 2 ) Trial step ( ) 1J ŝ = νi +J1 T T J 1 1 R Computed as solution of min x A 1 x b A 1 = ( ) J1 νi b = ( ) R 0 Linear least squares problem now well-conditioned

18 Subset Selection: Errors in Jacobian J 1 selected by strong RRQR [Gu & Eisenstat 1996] σ j 1+K(N K) σ j (J 1 ) σ j 1 j K Singular values of J 1 close to largest singular values of J Perturbed Jacobian J = J +E rank(j +E) K, E ǫ J Select same columns for J 1 and J 1 : ( J1 J2 ) ( s = νi + J 1 T J ) 1 1 JT 1 R

19 Subset Selection: Relative Error Condition number κ ν (J 1 ) = σ 1 max σ K σ σ 1 σ ν +σ 2 σ K = σ K / 1+K(N K) Relative error in subset selection trial step s ŝ s κ ν (J 1 ) ( 1+ R ) J s ǫ Error in J amplified by conditioning of J 1 and nonlinear residual R Same as full rank bound applied to J 1

20 Numerical Experiments Goal: Design simplest possible setting to reproduce failures from truncated SVD observed in cardiovascular model

21 Numerical Experiments Driven harmonic oscillator ( δ)y +(c 1 +c 2 )y +k y = 2sin(5t) y(0) = y 0, y (0) = y 0 4 parameters p = ( δ c 1 c 2 k ) T Numerical solution ỹ(t j ) from Matlab ode15s Nonlinear residual ỹ(t 1 ) d 1 R(p) =. ỹ(t M ) d M Estimate p by solving nonlinear least squares problem min p R(p) T R(p)/2

22 Numerical Experiments: Assumptions ( δ)y +(c 1 +c 2 )y +k y = 2sin(5t) Highly accurate Jacobians: Compute columns of J from sensitivities y/ p Zero residual: Data d from exact parameters p = ( ) T Initial guess p 0 = ( ) T Singular values of initial Jacobian: One zero singular value by design: Need to recover c 1 +c 2 = 1 R c 1 = R c 2

23 Numerical Experiments: Zero Residual Assumptions for subset selection: Rank K = 3 of Jacobian is known Subset selection applied only to initial Jacobian All Levenberg-Marquardt iterations work with same K columns Parameters corresponding to N K = 1 non-selected columns set to nominal values Exact parameters p = ( ) T Truncated SVD: p = ( ) T Subset selection: p = ( ) T A little more accurate

24 Convergence History: Zero Residual Truncated SVD Gradient Norm Least Squares Error Subset selection Iterations Iterations Subset selection converges faster and slightly more accurate than truncated SVD

25 Numerical Experiments: Non-Zero Residual ( δ)y +(c 1 +c 2 )y +k 0 y = 2sin(5t) Non-zero residual Componentwise relative perturbation of data d by 10 4 Singular values of Jacobian have not changed: Exact parameters p = ( ) T Truncated SVD: p = ( ) T δ completely wrong Subset selection: p = ( ) T Much more accurate

26 Truncated SVD Subset Selection What is really going on?

27 General Least Squares Problems min x Ax b A is M N, M N Singular values σ 1 σ K σ K+1 σ N > 0 Least squares problem with illconditioned matrix Truncated SVD Singular values σ 1 σ K σ K+1 = = σ N = 0 Least squares problem now ill-posed Subset Selection: K columns of A selected by strong RRQR Singular values σ 1 σ K / 1+K(N k) 0 Least squares problem with wellconditioned matrix

28 The Problem with Truncated SVD min x Ax b A has singular values σ 1 σ k σ r > 0 s = A b is minimal norm solution Truncated SVD: min x A t x b A t has singular values σ 1 σ k s t = A tb is minimal norm solution, residual r t = b As t Relative error s t s s t σ 1 σ r r t A s t Small residual does not imply that s t accurate Bound independent of how many singular values truncated

29 Summary Parameter estimation with nonlinear dependences Expressed as nonlinear least squares problem Solved by Levenberg-Marquardt trust region algorithm Rank deficient Jacobians Errors in Jacobian evaluation, non-zero residuals How to regularize Jacobian: Truncated SVD: NO Subset selection: Yes