Golub-Kahan iterative bidiagonalization and determining the noise level in the data

Transcription

1 Golub-Kahan iterative bidiagonalization and determining the noise level in the data Iveta Hnětynková,, Martin Plešinger,, Zdeněk Strakoš, * Charles University, Prague ** Academy of Sciences of the Czech republic, Prague *** Technical University, Liberec with thanks to P. C. Hansen, M. Kilmer and many others SIAM ALA, Monterey, October

2 Six tons large scale real world ill-posed problem: 2

3 Solving large scale discrete ill-posed problems is frequently based upon orthogonal projections-based model reduction using Krylov subspaces, see, e.g., hybrid methods. This can be viewed as approximation of a Riemann-Stieltjes distribution function via matching moments. 3

4 Consider the Riemann-Stieltjes distribution function ω(λ) with the n points of increase associated with the HPD matrix B and the normalized inital vector s, (or with the transfer function given by the Laplace transform of a linear dynamical system determined by B, s ). Then s (λi B) 1 s = n j=1 ω j λ λ j F n (λ), where λ j, j = 1,..., n denote the eigenvalues of B and ω j the squared size of the component of s in the corresponding invariant subspace respectively. 4

5 The continued fraction on the right hand side is given by F n (λ) R n(λ) P n (λ) λ γ 1 λ γ 2 1 δ 2 2 λ γ 3... δ λ γ n 1 δ2 n λ γ n and the entries γ 1,..., γ n and δ 2,..., δ n form the Jacobi matrix 5

6 T n γ 1 δ 2 δ 2 γ δ n δ n γ n, δ l > 0, l = 2,..., n. Consider the kth Gauss-Christoffel quadrature approximation ω (k) (λ) of the Riemann-Stieltjes distribution function ω(λ). Its algebraic degree is 2k 1, i.e., it matches the first 2k moments ξ l 1 = λ l 1 dω(λ) = k j=1 ω (k) j {λ (k) j } l 1, l = 1,...,2k. 6

7 The nodes and weights of ω (k) (λ) are given by the eigenvalues and the corresponding squared first elements of the normalized eigenvectors of T k. Expansion of the continued fraction F n (λ) in terms of the decreasing powers of λ and the approximation by its kth convergent F k (λ) gives F n (λ) = 2k l=1 ξ l 1 λ l + O ( 1 ) λ 2k+1 = F k (λ) + O ( 1 ) λ 2k+1. Here F k (λ) approximates F n (λ) with the error proportional to λ (2k+1), which represents the minimal partial realization matching the first 2k moments, cf. [Stieltjes , Chebyshev ]. 7

8 Discrete ill-posed problem, the smallest node and weight in approximation of ω(λ): 10 0 ω(λ): nodes σ j 2, weights (b/β1,u j ) 2 nodes (θ 1 (k) ) 2, weights (p1 (k),e1 ) weight δ 2 noise the leftmost node 8

9 Outline 1. Problem formulation 2. Golub-Kahan iterative bidiagonalization, Lanczos tridiagonalization, and approximation of the Riemann-Stieltjes distribution function 3. Propagation of the noise in the Golub-Kahan bidiagonalization 4. Determination of the noise level 5. Numerical illustration 6. Summary and future work 9

10 Consider an ill-posed square nonsingular linear algebraic system A x b, A R n n, b R n, with the right-hand side corrupted by a white noise b = b exact + b noise 0 R n, b exact b noise, and the goal to approximate x exact A 1 b exact. The noise level δ noise bnoise b exact 1. 10

11 Properties (assumptions): matrices A, A T, AA T have a smoothing property; left singular vectors u j of A represent increasing frequencies as j increases; the system A x = b exact satisfies the discrete Picard condition. Discrete Picard condition (DPC): On average, the components (b exact, u j ) of the true right-hand side b exact in the left singular subspaces of A decay faster than the singular values σ j of A, j = 1,..., n. White noise: The components (b noise, u j ), j = 1,..., n do not exhibit any trend. 11

12 Problem Shaw: Noise level, Singular values, and DPC: [Hansen Regularization Tools] σ j (b, u j ) for δ noise = (b, u j ) for δ noise = 10 8 (b, u j ) for δ noise = σ j, double precision arithmetic σ j, high precision arithmetic (b exact, u j ), high precision arithmetic singular value number singular value number 12

13 Regularization is used to suppress the effect of errors in the data and extract the essential information about the solution. In hybrid methods, see [O Leary, Simmons 81], [Hansen 98], or [Fiero, Golub Hansen, O Leary 97], [Kilmer, O Leary 01], [Kilmer, Hansen, Español 06], [O Leary, Simmons 81], the outer bidiagonalization is combined with an inner regularization e.g., truncated SVD (TSVD), or Tikhonov regularization of the projected small problem (i.e. of the reduced model). The bidiagonalization is stopped when the regularized solution of the reduced model matches some selected stopping criteria. 13

14 Stopping criteria are typically based, amongst others, see [Björk 88], [Björk, Grimme, Van Dooren 94], on estimation of the L-curve [Calvetti, Golub, Reichel 99], [Calvetti, Morigi, Reichel, Sgallari 00], [Calvetti, Reichel 04]; estimation of the distance between the exact and regularized solution [O Leary 01]; the discrepancy principle [Morozov 66], [Morozov 84]; cross validation methods [Chung, Nagy, O Leary 04], [Golub, Von Matt 97], [Nguyen, Milanfar, Golub 01]. For an extensive study and comparison see [Hansen 98], [Kilmer, O Leary 01]. 14

15 Stopping criteria based on information from residual vectors: A vector ˆx is a good approximaton to x exact = A 1 b exact if the corresponding residual approximates the (white) noise in the data ˆr b Aˆx b noise. Behavior of ˆr can be expressed in the frequency domain using discrete Fourier transform, see [Rust 98], [Rust 00], [Rust, O Leary 08], or discrete cosine transform, see [Hansen, Kilmer, Kjeldsen 06], and then analyzed using statistical tools cumulative periodograms. 15

16 This talk: Under the given (quite natural) assumptions, the Golub-Kahan iterative bidiagonalization reveals the noise level δ noise. 16

17 Outline 1. Problem formulation 2. Golub-Kahan iterative bidiagonalization, Lanczos tridiagonalization, and approximation of the Riemann-Stieltjes distribution function 3. Propagation of the noise in the Golub-Kahan bidiagonalization 4. Determination of the noise level 5. Numerical illustration 6. Summary and future work 17

18 Golub-Kahan iterative bidiagonalization (GK) of A : given w 0 = 0, s 1 = b / β 1, where β 1 = b, for j = 1,2,... α j w j = A T s j β j w j 1, w j = 1, β j+1 s j+1 = A w j α j s j, s j+1 = 1. Let S k = [ s 1,..., s k ], W k = [ w 1,..., w k ] be the associated matrices with orthonormal columns. 18

19 Denote L k = α 1 β 2 α β k α k, L k+ = α 1 β 2 α β k α k β k+1 = [ L k e T k β k+1 ], the bidiagonal matrices containing the normalization coefficients. Then GK can be written in the matrix form as A T S k = W k L T k, A W k = [ S k, s k+1 ] Lk+ = S k+1 L k+. 19

20 GK is closely related to the Lanczos tridiagonalization of the symmetric matrix A A T with the starting vector s 1 = b / β 1, A A T S k = S k T k + α k β k+1 s k+1 e T k, T k = L k L T k = α 2 1 α 1 β 1 α 1 β 1 α β α k 1 β k, α k 1 β k α 2 k + β2 k i.e. the matrix L k from GK represents a Cholesky factor of the symmetric tridiagonal matrix T k from the Lanczos process. 20

21 Approximation of the distribution function: The Lanczos tridiagonalization of the given (SPD) matrix B R n n generates at each step k a non-decreasing piecewise constant distribution function ω (k), with the nodes being the (distinct) eigenvalues of the Lanczos matrix T k and the weights ω (k) j being the squared first entries of the corresponding normalized eigenvectors [Hestenes, Stiefel 52]. The distribution functions ω (k) (λ), k = 1, 2,... represent Gauss- Christoffel quadrature (i.e. minimal partial realization) approximations of the distribution function ω(λ), [Hestenes, Stiefel 52], [Fischer 96], [Meurant, Strakoš 06]. 21

22 Consider the SVD L k = P k Θ k Q k T, P k = [p (k) 1,..., p(k) k ], Q k = [q (k) 1,..., q(k) k ], Θ k = diag(θ (k) 1,..., θ(k) n ), with the singular values ordered in the increasing order, 0 < θ (k) 1 <... < θ (k) k. Then T k = L k L T k = P k Θ 2 k P T k is the spectral decomposition of T k, (θ (k) l ) 2 are its eigenvalues (the Ritz values of AA T ) and p (k) l its eigenvectors (which determine the Ritz vectors of AA T ), l = 1,..., k. 22

23 Summarizing: The GK bidiagonalization generates at each step k the distribution function ω (k) (λ) with nodes (θ (k) l ) 2 and weights ω (k) l = (p (k) l, e 1 ) 2 that approximates the distribution function ω(λ) with nodes σj 2 and weights ω j = (b/β 1, u j ) 2, where σj 2, u j are the eigenpairs of A A T, for j = n,..., 1. Note that unlike the Ritz values (θ (k) l ) 2, the squared singular values are enumerated in descending order. σ 2 j 23

24 Outline 1. Problem formulation 2. Golub-Kahan iterative bidiagonalization, Lanczos tridiagonalization, and approximation of the Riemann-Stieltjes distribution function 3. Propagation of the noise in the Golub-Kahan bidiagonalization 4. Determination of the noise level 5. Numerical illustration 6. Summary and future work 24

25 GK starts with the normalized noisy right-hand side s 1 = b / b. Consequently, vectors s j contain information about the noise. Can this information be used to determine the level of the noise in the observation vector b? Consider the problem Shaw from [Hansen Regularization Tools] (computed via [A,b exact,x] = shaw(400)) with a noisy righthand side (the noise was artificially added using the MATLAB function randn). As an example we set δ noise bnoise b exact =

26 Components of several bidiagonalization vectors s j computed via GK with double reorthogonalization: s 1 s 6 s 11 s 16 s s 18 s 19 s 20 s 21 s

27 The first 80 spectral coefficients of the vectors s j in the basis of the left singular vectors u j of A: U T s 1 U T s 6 U T s 11 U T s 16 U T s U T s 18 U T s 19 U T s 20 U T s 21 U T s

28 Signal space noise space diagrams: s 1 > s 2 s 5 > s 6 s 6 > s 7 s 10 > s 11 s 13 > s 14 s 15 > s 16 s 16 > s 17 s 17 > s 18 s 18 > s 19 s 19 > s 20 s 20 > s 21 s 21 > s 22 s k (triangle) and s k+1 (circle) in the signal space span{u 1,..., u k+1 } (horizontal axis) and the noise space span{u k+2,..., u n } (vertical axis). 28

29 The noise is amplified with the ratio α k /β k+1 : GK for the spectral components: α 1 (V T w 1 ) = Σ(U T s 1 ), β 2 (U T s 2 ) = Σ(V T w 1 ) α 1 (U T s 1 ), and for k = 2,3,... α k (V T w k ) = Σ(U T s k ) β k (V T w k 1 ), β k+1 (U T s k+1 ) = Σ(V T w k ) α k (U T s k ). 29

30 Since dominance in Σ(U T s k ) and (V T w k 1 ) is shifted by one component, in α k (V T w k ) = Σ(U T s k ) β k (V T w k 1 ), one can not expect a significant cancelation, and therefore α k β k. Whereas Σ(V T w k ) and (U T s k ) do exhibit dominance in the direction of the same components. If this dominance is strong enough, then the required orthogonality of s k+1 and s k in β k+1 (U T s k+1 ) = Σ(V T w k ) α k (U T s k ) can not be achieved without a significant cancelation, and one can expect β k+1 α k.. 30

31 Absolute values of the first 25 components of Σ(V T w k ), α k (U T s k ), and β k+1 (U T s k+1 ) for k = 7, β 8 /α 7 = (left) and for k = 12, β 13 /α 12 = (right), Shaw with the noise level δ noise = : 10 0 Σ V T w 7 α 7 U T s 7 β 8 U T s Σ V T w 12 α 12 U T s 12 β 13 U T s

32 Outline 1. Problem formulation 2. Golub-Kahan iterative bidiagonalization, Lanczos tridiagonalization, and approximation of the Riemann-Stieltjes distribution function 3. Propagation of the noise in the Golub-Kahan bidiagonalization 4. Determination of the noise level 5. Numerical illustration 6. Summary and future work 32

33 Depending on the noise level, the smaller nodes of ω(λ) are completely dominated by noise, i.e., there exists an index J noise such that for j J noise (b/β1, uj) 2 (b noise /β1, uj) 2 and the weight of the set of the associated nodes is given by δ 2 n j=j noise (b noise /β 1, u j ) 2. Recall that the large nodes σ1 2, σ2 2,... are well-separated (relatively to the small ones) and their weights on average decrease faster than σ1 2, σ2 2, see (DPC). Therefore the large nodes essentially control the behavior of the early stages of the Lanczos process. 33

34 At any iteration step, the weight corresponding to the smallest node (θ (k) 1 )2 must be larger than the sum of weights of all σ 2 j smaller than this (θ (k) 1 )2, see [Fischer, Freund 94]. As k increases, some (θ (k) 1 )2 eventually approaches (or becomes smaller than) the node σ 2 J noise, and its weight becomes (p (k) 1, e 1) 2 δ 2 δ 2 noise. The weight (p (k) 1, e 1) 2 corresponding to the smallest Ritz value (θ (k) 1 )2 is strictly decreasing. At some iteration step it sharply starts to (almost) stagnate on the level close to the squared noise level δnoise 2. 34

35 Square roots of the weights (p (k) 1, e 1) 2, k = 1, 2,..., Shaw with the noise level δ noise = : δ noise k noise = 18 1 = iteration number k 35

36 Square roots of the weights (p (k) 1, e 1) 2, k = 1, 2,..., Shaw with the noise level δ noise = 10 4 : δ noise k noise = 8 1 = iteration number k 36

37 The smallest node and weight in approximation of ω(λ): 10 0 ω(λ): nodes σ j 2, weights (b/β1,u j ) 2 nodes (θ 1 (k) ) 2, weights (p1 (k),e1 ) weight δ 2 noise the leftmost node 37

38 The smallest node and weight in approximation of ω(λ): 10 0 ω(λ): nodes σ j 2, weights (b/β1,u j ) 2 nodes (θ 1 (k) ) 2, weights (p1 (k),e1 ) weight δ 2 noise the leftmost node 38

39 Outline 1. Problem formulation 2. Golub-Kahan iterative bidiagonalization, Lanczos tridiagonalization, and approximation of the Riemann-Stieltjes distribution function 3. Propagation of the noise in the Golub-Kahan bidiagonalization 4. Determination of the noise level 5. Numerical illustration 6. Summary and future work 39

40 Image deblurring problem, image size pixels, problem dimension n = , the exact solution (left) and the noisy right-hand side (right), δ noise = x exact b exact + b noise 40

41 Square roots of the weights (p (k) 1, e 1) 2, k = 1, 2,... (top) and error history of LSQR solutions (bottom): ( p (k) 1, e 1 ), b noise 2 / b exact 2 = ( p 1 (k), e 1 ) δ noise Error history x k LSQR x exact the minimal error, k =

42 The best LSQR reconstruction (left), x LSQR 41, and the corresponding componentwise error (right). GK without any reorthogonalization! LSQR reconstruction with minimal error, x LSQR 41 Error of the best LSQR reconstruction, x exact x LSQR

43 Outline 1. Problem formulation 2. Golub-Kahan iterative bidiagonalization, Lanczos tridiagonalization, and the approximation of the Riemann-Stieltjes distribution function 3. Propagation of the noise in the Golub-Kahan bidiagonalization 4. Determination of the noise level 5. Numerical illustration 6. Summary and future work 43

44 Message: Using GK, information about the noise can be obtained in a straightforward way. Future work: Large scale problems; Behavior in finite precision arithmetic (GK without reorthogonalization); Regularization; Denoising; Colored noise. 44

45 References Golub, Kahan: Calculating the singular values and pseudoinverse of a matrix, SIAM J. B2, Hansen: Rank-deficient and discrete ill-posed problems, SIAM Monographs Math. Modeling Comp., Hansen, Kilmer, Kjeldsen: Exploiting residual information in the parameter choice for discrete ill-posed problems, BIT, Hnětynková, Strakoš: Lanczos tridiag. and core problem, LAA, Meurant, Strakoš: The Lanczos and CG algorithms in finite precision arithmetic, Acta Numerica, Paige, Strakoš: Core problem in linear algebraic systems, SIMAX, Rust: Truncating the SVD for ill-posed problems, Technical Report, Rust, O Leary: Residual periodograms for choosing regularization parameters for ill-posed problems, Inverse Problems, Hnětynková, Plešinger, Strakoš: The regularizing effect of the Golub-Kahan iterative bidiagonalization and revealing the noise level, BIT,

46 Main message: Whenever you see a blurred elephant which is a bit too noisy, the best thing is to apply the GK iterative bidiagonalization. Full version of the talk can be found at 46

47 Thank you for your kind attention! 47