LSMR: An iterative algorithm for least-squares problems

Transcription

1 LSMR: An iterative algorithm for least-squares problems David Fong Michael Saunders Institute for Computational and Mathematical Engineering (icme) Stanford University Copper Mountain Conference on Iterative Methods Copper Mountain, Colorado Apr 5 9, 2010 David Fong, Michael Saunders LSMR Algorithm 1/31

2 LSMR in one slide solve Ax = b min Ax b 2 David Fong, Michael Saunders LSMR Algorithm 2/31

3 LSMR in one slide solve Ax = b min Ax b 2 ) ( ) min A b ( x λi 0 2 David Fong, Michael Saunders LSMR Algorithm 2/31

4 LSMR in one slide solve Ax = b min Ax b 2 ) ( ) min A b ( x λi 0 2 LSQR CG on the normal equation LSMR MINRES on the normal equation David Fong, Michael Saunders LSMR Algorithm 2/31

5 LSMR in one slide solve Ax = b min Ax b 2 ) ( ) min A b ( x λi 0 2 LSQR CG on the normal equation LSMR MINRES on the normal equation Almost same complexity as LSQR Better convergence properties for inexact solves David Fong, Michael Saunders LSMR Algorithm 2/31

6 LSQR Iterative algorithm for ( ) ( min A b x λi 0) 2 David Fong, Michael Saunders LSMR Algorithm 3/31

7 LSQR Iterative algorithm for Properties ( ) ( min A b x λi 0) 2 A is rectangular (m n) and often sparse A can be an operator CG on the normal equation (A T A + λ 2 I)x = A T b Av, A T u plus O(m + n) operations per iteration David Fong, Michael Saunders LSMR Algorithm 3/31

8 Monotone convergence of residual Measure of Convergence r = b Ax r ˆr, A T r 0 David Fong, Michael Saunders LSMR Algorithm 4/31

9 Monotone convergence of residual Measure of Convergence r = b Ax r ˆr, A T r 0 LSQR r LSQR log A T r Name:lp fit1p, Dim:1677x627, nnz:9868, id=80 2 Name:lp fit1p, Dim:1677x627, nnz:9868, id= r log A T r iteration count iteration count David Fong, Michael Saunders LSMR Algorithm 4/31

10 Monotone convergence of residual Measure of Convergence r = b Ax r ˆr, A T r 0 LSQR LSMR r log A T r Name:lp fit1p, Dim:1677x627, nnz:9868, id=80 lsqr lsmr 2 1 Name:lp fit1p, Dim:1677x627, nnz:9868, id=80 lsqr lsmr r log A T r iteration count iteration count David Fong, Michael Saunders LSMR Algorithm 4/31

11 LSMR Algorithm David Fong, Michael Saunders LSMR Algorithm 5/31

12 Golub-Kahan bidiagonalization Given A (m n) and b (m 1) Direct bidiagonalization U T AV = B David Fong, Michael Saunders LSMR Algorithm 6/31

13 Golub-Kahan bidiagonalization Given A (m n) and b (m 1) Direct bidiagonalization U T AV = B Iterative bidiagonalization 1 β 1 u 1 = b, α 1 v 1 = A T u 1 2 for = 1, 2,..., set β +1 u +1 = Av α u α +1 v +1 = A T u +1 β +1 v David Fong, Michael Saunders LSMR Algorithm 6/31

14 Golub-Kahan bidiagonalization (2) The process can be summarized by AV = U +1 B A T U +1 = V +1 L T +1 where α 1 β 2 α 2 L = β α α 1, B β 2 α 2 = β α β +1 = ( L β +1 e T ) David Fong, Michael Saunders LSMR Algorithm 7/31

15 Golub-Kahan bidiagonalization (3) V spans the Krylov subspace: span{v 1,..., v } = span{a T b, (A T A)A T b,..., (A T A) 1 A T b} Define x = V y Subproblem to solve min r = min β 1 e 1 B y (LSQR) y y ( ) min A T r = min y y β B T 1 e 1 B β +1 e T y (LSMR) where r = b Ax, β = α β David Fong, Michael Saunders LSMR Algorithm 8/31

16 Golub-Kahan bidiagonalization (3) V spans the Krylov subspace: span{v 1,..., v } = span{a T b, (A T A)A T b,..., (A T A) 1 A T b} Define x = V y Subproblem to solve min r = min β 1 e 1 B y (LSQR) y y ( ) min A T r = min y y β B T 1 e 1 B β +1 e T y (LSMR) where r = b Ax, β = α β David Fong, Michael Saunders LSMR Algorithm 8/31

17 Least squares subproblem min y ( ) B β T B 1e 1 y β +1 e T David Fong, Michael Saunders LSMR Algorithm 9/31

18 Least squares subproblem min y = min y ( ) B β T B 1e 1 y β +1 e T ( ) R β T R 1e 1 y q T Q +1 B = R ( ) R, R T q 0 = β +1 e David Fong, Michael Saunders LSMR Algorithm 9/31

19 Least squares subproblem min y = min y = min t ( ) B β T B 1e 1 y β +1 e T ( ) R β T R ( ) R 1e 1 y q T Q +1 B =, R T q R 0 = β +1 e ( ) R β T 1e 1 t ϕ e T t = R y, q = ( β +1 /(R ), )e = ϕ e David Fong, Michael Saunders LSMR Algorithm 9/31

20 Least squares subproblem min y = min y = min t = min t ( ) B β T B 1e 1 y β +1 e T ( ) R β T R ( ) R 1e 1 y q T Q +1 B =, R T q R 0 = β +1 e ( ) R β T 1e 1 t ϕ e T t = R y, q = ( β +1 /(R ), )e = ϕ e ( ) ( ) ( ) ( ) z R R T β1e 1 R z t ζ +1 0 Q+1 = ϕ e T 0 0 ζ+1 David Fong, Michael Saunders LSMR Algorithm 9/31

21 Least squares subproblem min y = min y = min t = min t ( ) B β T B 1e 1 y β +1 e T ( ) R β T R ( ) R 1e 1 y q T Q +1 B =, R T q R 0 = β +1 e ( ) R β T 1e 1 t ϕ e T t = R y, q = ( β +1 /(R ), )e = ϕ e ( ) ( ) ( ) ( ) z R R T β1e 1 R z t ζ +1 0 Q+1 = ϕ e T 0 0 ζ+1 Things to note x = V y, t = R y, z = R t, two cheap QRs David Fong, Michael Saunders LSMR Algorithm 9/31

22 Least squares subproblem (2) Remember x = V y, t = R y, z = R t David Fong, Michael Saunders LSMR Algorithm 10/31

23 Least squares subproblem (2) Remember x = V y, t = R y, z = R t Key steps to compute x x = V y David Fong, Michael Saunders LSMR Algorithm 10/31

24 Least squares subproblem (2) Remember x = V y, t = R y, z = R t Key steps to compute x x = V y = W t R T W T = V T David Fong, Michael Saunders LSMR Algorithm 10/31

25 Least squares subproblem (2) Remember x = V y, t = R y, z = R t Key steps to compute x x = V y = W t R T W T = V T = W z RT W T = W T David Fong, Michael Saunders LSMR Algorithm 10/31

26 Least squares subproblem (2) Remember x = V y, t = R y, z = R t Key steps to compute x x = V y = W t R T W T = V T = W z RT W T = W T = x 1 + ζ w where z = ( ζ 1 ζ 2 ζ ) T David Fong, Michael Saunders LSMR Algorithm 10/31

27 Flow chart of LSMR A,b Golub-Kahan bidiagonalization QR decompositions Update x Estimate bacward error large small Solution x David Fong, Michael Saunders LSMR Algorithm 11/31

28 Flow chart of LSMR Computational cost A,b Av, A T v, 3m + 3n Golub-Kahan bidiagonalization O(1) QR decompositions 3n Update x O(1) Estimate bacward error small Solution x large David Fong, Michael Saunders LSMR Algorithm 11/31

29 Computational and storage requirement Storage Wor m n m n MINRES on A T Ax = A T b Av 1 x, v 1, v 2, w 1, w 2 8 LSQR Av, u x, v, w 3 5 LSMR Av, u x, v, h, h 3 6 where h, h are scalar multiples of w, w David Fong, Michael Saunders LSMR Algorithm 12/31

30 Numerical experiments Test Data University of Florida Sparse Matrix Collection LPnetlib: Linear Programming Problems A = (Problem.A) b = Problem.c (127 problems) David Fong, Michael Saunders LSMR Algorithm 13/31

31 Numerical experiments Test Data University of Florida Sparse Matrix Collection LPnetlib: Linear Programming Problems A = (Problem.A) b = Problem.c (127 problems) Solve min Ax b 2 with LSQR and LSMR Bacward error tests: nnz(a) Reorthogonalization: nnz(a) David Fong, Michael Saunders LSMR Algorithm 13/31

32 Bacward errors - optimal Optimal bacward error µ(x) min E E s.t. (A + E)T (A + E)x = (A + E) T b David Fong, Michael Saunders LSMR Algorithm 14/31

33 Bacward errors - optimal Optimal bacward error µ(x) min E E s.t. (A + E)T (A + E)x = (A + E) T b Higham 1995: µ(x) = σ min (C), C [ A ( )] r x I rrt. r 2 Cheaper estimate (Grcar, Saunders, & Su 2007): ( ) ( ) A r K = r v = y = argmin Ky v x 0 y µ(x) = Ky x David Fong, Michael Saunders LSMR Algorithm 14/31

34 Bacward error estimates - E 1, E 2 Again, (A + E i ) T (A + E i )x = (A + E i ) T b Two estimates given by Stewart (1975 and 1977): E 1 = ext x 2, E 1 = e x, e = ˆr r E 2 = rrt A r 2, E 2 = AT r r where ˆr is the residual for the exact solution David Fong, Michael Saunders LSMR Algorithm 15/31

35 r for LSQR and LSMR Name:lp greenbeb, Dim:5598x2392, nnz:31070, id=100 lsqr lsmr r iteration count David Fong, Michael Saunders LSMR Algorithm 16/31

36 log 10 ( E 2 ) for LSQR and LSMR 0 Name:lp pilot ja, Dim:2267x940, nnz:14977, id=88 E2 LSQR E2 LSMR log(e2) iteration count David Fong, Michael Saunders LSMR Algorithm 17/31

37 Bacward errors for LSQR 1 0 Name:lp cre a, Dim:7248x3516, nnz:18168, id=93 E1 LSQR E2 LSQR Optimal LSQR 1 2 log(bacward Error) iteration count David Fong, Michael Saunders LSMR Algorithm 18/31

38 Bacward errors for LSMR 1 0 Name:lp cre a, Dim:7248x3516, nnz:18168, id=93 E1 LSMR E2 LSMR Optimal LSMR 1 2 log(bacward Error) iteration count David Fong, Michael Saunders LSMR Algorithm 19/31

39 Optimal bacward errors for LSQR and LSMR 0 Name:lp pilot, Dim:4860x1441, nnz:44375, id=107 Optimal LSQR Optimal LSMR 1 2 log( A T r / r ) iteration count David Fong, Michael Saunders LSMR Algorithm 20/31

40 Space-time trade-offs LSMR is well-suited for limited memory computations. What if we have more memory Av expensive Can we speed things up? David Fong, Michael Saunders LSMR Algorithm 21/31

41 Space-time trade-offs LSMR is well-suited for limited memory computations. What if we have more memory Av expensive Can we speed things up? Some ideas: Reorthogonalization Restarting Local reorthogonalization David Fong, Michael Saunders LSMR Algorithm 21/31

42 Reorthogonalization Golub-Kahan process Infinite precision Finite precision U, V orthonormal Lose orthogonality At most min(m, n) iterations Could tae 10n or more David Fong, Michael Saunders LSMR Algorithm 22/31

43 Reorthogonalization Golub-Kahan process Infinite precision Finite precision U, V orthonormal Lose orthogonality At most min(m, n) iterations Could tae 10n or more Apply modified Gram-Schmidt to u +1 and/or v +1 : u u (u T j u)u j j =, 1, 2,... (similarly for v) David Fong, Michael Saunders LSMR Algorithm 22/31

44 Effects of reorthogonalization on various problems 1 0 Name:lp ship12l, Dim:5533x1151, nnz:16276, id=91 NoOrtho OrthoU OrthoV OrthoUV 1 0 Name:lp scfxm2, Dim:1200x660, nnz:5469, id=62 NoOrtho OrthoU OrthoV OrthoUV log(e2) 3 4 log(e2) iteration count iteration count 1 0 Name:lp agg2, Dim:758x516, nnz:4740, id=58 NoOrtho OrthoU OrthoV OrthoUV 0 1 Name:lpi gran, Dim:2525x2658, nnz:20111, id=94 NoOrtho OrthoU OrthoV OrthoUV log(e2) log(e2) iteration count iteration count David Fong, Michael Saunders LSMR Algorithm 23/31

45 Orthogonality of U Original MGS on U MGS on V MGS on U,V David Fong, Michael Saunders LSMR Algorithm 24/31

46 Orthogonality of V Original MGS on U MGS on V MGS on U,V David Fong, Michael Saunders LSMR Algorithm 25/31

47 What we learnt so far Reorthogonalizing V (only) is sufficient Reorthogonalizing U (only) is nearly as good x converges the same for all options What can be improved May still use too much memory Need more flexibility for space-time trade-off David Fong, Michael Saunders LSMR Algorithm 26/31

48 Reorthogonalization with Restarting Restarting LSMR r = b Ax min A x r David Fong, Michael Saunders LSMR Algorithm 27/31

49 Reorthogonalization with Restarting Restarting LSMR r = b Ax min A x r Restarting leads to stagnation 0 1 Name:lp maros, Dim:1966x846, nnz:10137, id=81 NoOrtho Restart5 Restart10 Restart50 NoRestart 1 0 Name:lp cre c, Dim:6411x3068, nnz:15977, id=90 NoOrtho Restart5 Restart10 Restart50 NoRestart Bacward Error 3 4 Bacward Error iteration count iteration count David Fong, Michael Saunders LSMR Algorithm 27/31

50 Local reorthogonalization Reorthogonalize wrto only the last l vectors Partial speed-up Less memory Depends on efficiency of Av and A T u David Fong, Michael Saunders LSMR Algorithm 28/31

51 Local reorthogonalization Reorthogonalize wrto only the last l vectors Partial speed-up Less memory Depends on efficiency of Av and A T u Speed up with local reorthogonalization Name:lp fit1p, Dim:1677x627, nnz:9868, id=80 NoOrtho Local5 Local10 Local50 FullOrtho 2 1 Name:lp bnl2, Dim:4486x2324, nnz:14996, id=89 NoOrtho Local5 Local10 Local50 FullOrtho Bacward Error Bacward Error iteration count iteration count David Fong, Michael Saunders LSMR Algorithm 28/31

52 Conclusions Advantages of LSMR All the good properties of LSQR Monotone convergence of A T r r still monotonic Cheap near-optimal bacward error estimate reliable stopping rule Bacward error almost surely monotonic David Fong, Michael Saunders LSMR Algorithm 29/31

53 Acnowledgement Michael Saunders Chris Paige Stanford Graduate Fellowship David Fong, Michael Saunders LSMR Algorithm 30/31

54 Questions LSMR in MATLAB and slides for this tal are downloadable at David Fong, Michael Saunders LSMR Algorithm 31/31