COMPUTATIONAL ISSUES RELATING TO INVERSION OF PRACTICAL DATA: WHERE IS THE UNCERTAINTY? CAN WE SOLVE Ax = b?

Transcription

1 COMPUTATIONAL ISSUES RELATING TO INVERSION OF PRACTICAL DATA: WHERE IS THE UNCERTAINTY? CAN WE SOLVE Ax = b? Rosemary Renaut rosie BRIDGING THE GAP? OCT 2, 2012

2 Discussion Yuen: Solve large scale problems efficiently - find solutions and features - edge resolution Reusable kernels -effective libraries - better utilize HPC/GPU environments How can you differentiate your data Solve for solution and features together Does this all mesh together successfully? How does analysis of computation fit?

3 Discussion Yuen: Solve large scale problems efficiently - find solutions and features - edge resolution Reusable kernels -effective libraries - better utilize HPC/GPU environments How can you differentiate your data Solve for solution and features together Does this all mesh together successfully? How does analysis of computation fit?

4 Discussion Yuen: Solve large scale problems efficiently - find solutions and features - edge resolution Reusable kernels -effective libraries - better utilize HPC/GPU environments How can you differentiate your data Solve for solution and features together Does this all mesh together successfully? How does analysis of computation fit?

5 Discussion Yuen: Solve large scale problems efficiently - find solutions and features - edge resolution Reusable kernels -effective libraries - better utilize HPC/GPU environments How can you differentiate your data Solve for solution and features together Does this all mesh together successfully? How does analysis of computation fit?

6 Discussion Yuen: Solve large scale problems efficiently - find solutions and features - edge resolution Reusable kernels -effective libraries - better utilize HPC/GPU environments How can you differentiate your data Solve for solution and features together Does this all mesh together successfully? How does analysis of computation fit?

7 Discussion Yuen: Solve large scale problems efficiently - find solutions and features - edge resolution Reusable kernels -effective libraries - better utilize HPC/GPU environments How can you differentiate your data Solve for solution and features together Does this all mesh together successfully? How does analysis of computation fit?

8 Goals Numerical instability? Impact on solving ill-posed problems? What problem did we actually solve? Example - linear case least squares? Little mathematics - many images Nonlinear LS use multiple LLS cases

9 Illustration:Blurred Signal Restoration Figure: Standard blurred signal, desire to find signal and its features

10 Inverse Problem given model A, Condition e + 05data b find x, noise.0001 x = A 1 b (inverse crime) - need an alternative feasible solution

11 Tikhonov Regularized Solutions x(λ) and derivative Lx for changing λ Figure: We cannot capture Lx (red) from the solution (green): Notice that Lx decreases as λ increases

12 Example TV Solution: 1D: No Updates for the parameters

13 Example TV Solution: 1D Updates for the parameters

14 TV Solutions - Solve for both Lx and x concurrently: Split Bregman Formulation (Goldstein and Osher, 2009) Introduce d Lx and let R(x) = λ2 2 d Lx µ d 1 (x, d)(λ, µ) = arg min x,d {1 2 Ax b λ2 2 d Lx µ d 1 } Alternating minimization separates steps for d from x Various versions of the iteration can be defined. Fundamentally: S1 : x (k+1) = arg min{ 1 x 2 Ax b λ2 2 Lx (d(k+1) g (k) ) 2 2} S2 : d (k+1) = arg min d { λ2 2 d (Lx(k+1) + g (k) ) µ d 1 } S3 : g (k+1) = g (k) + Lx (k+1) d (k+1).

15 So how does this go? 1. Inverse problem we need regularization 2. For feature extraction we need more than Tikhonov Regularization - e.g. TV 3. The TV iterates over many Tikhonov solutions 4. Both techniques are parameter dependent 5. Moreover the parameters are needed 6. We need to fully understand the Tikhonov and ill-posed problems 7. Can we do blackbox solvers? 8. Be careful

16 Decompositions: SVD A = UΣV T, GSVD A = UGZ T, L = V MZ T 1. A (full column rank): u i, v i left and right singular vectors, σ i spectral values of A x = n i=1 u T i b v i σ i A weighted linear combination of the basis vectors v i 2. Tikhonov Regularization I is a spectral filtering x filt = n i=1 γ i ( ut i b )v i σ i 3. Generalized Tikhonov: Generalized SVD expansion ( ) p x (k+1) ν i u T i = b νi 2 + λ2 µ 2 + λ2 µ i vi T n h(k) i νi 2 + λ2 µ 2 z i + (u T i b)z i i i=1 i=p+1

17 An example: n = 32 Left Singular Vectors and Basis Depend on A Figure: The first few left singular vectors u i and basis vectors v i. Can we use these basis vectors

18 Second example: n = 84 Left Singular Vectors and Basis Depend on A Figure: The first few left singular vectors u i and basis vectors v i. Can we use these basis vectors

19 Not Always so Bad a case with n = 64 Figure: Left singular vectors u i and basis vectors v i. Can we use these basis vectors

20 Solutions expressed with respect to a basis Given A is the basis a good representation for true basis of A? Mathematical backward stability - for SVD the basis is not too far from appropriate orthogonal manifold But although the basis is orthogonal - it eventually contributes noise When we look at residual for r = Ax b is r small sufficient? What is small Need to start looking at the noise entering the residual. Need to extend statistical techniques to examining the stability in a new context?

21 Cumulative Periodogram for the left / right singular vectors Figure: On left the left singular vectors and on the right the basis vectors v. Low frequency vectors lie above the diagonal and high frequency below the diagonal. White noise follows the diagonal

22 Second Example Cumulative Periodogram for the left / right singular vectors Figure: On left the left singular vectors and on the right the basis vectors v. Low frequency vectors lie above the diagonal and high frequency below the diagonal. White noise follows the diagonal

23 Cumulative Periodogram for the left / right singular vectors Figure: On left the left singular vectors and on the right the basis vectors v. Low frequency vectors lie above the diagonal and high frequency below the diagonal. White noise follows the diagonal

24 Measure Deviation from Straight Line: testing for white noise Figure: Calculate the cumulative periodogram, measure deviation from white noise line, assess proportion of vector outside Kolmogorov Smirnov test at a 5% confidence level. Cannot expect to use more than 9 vectors in the expansion for x. Additional terms are contaminated by noise - independent of noise in b

25 Measure Deviation from Straight Line: testing for white noise Figure: Calculate the cumulative periodogram, measure deviation from white noise line, assess proportion of vector outside Kolmogorov Smirnov test at a 5% confidence level. Cannot expect to use more than 12 vectors in the expansion for x. Additional terms are contaminated by noise - independent of noise in b

26 Measure Deviation from Straight Line: testing for white noise Figure: Calculate the cumulative periodogram, measure deviation from white noise line, assess proportion of vector outside Kolmogorov Smirnov test at a 5% confidence level. Use all but last 2 vectors in the expansion for x. Additional terms are contaminated by noise - independent of noise in b

27 Testing for the GSVD Basis Figure: First Example: It pays to truncate the SVD before finding the GSVD to give the basis for x

28 Testing for the GSVD Basis Figure: Second Example: It pays to truncate the SVD before finding the GSVD to give the basis for x

29 Observations Libraries need to include additional methods for assessing reliability of the residual Even when committing the inverse crime we will not achieve the solution if we cannot approximate the basis correctly. We need all basis vectors which contain the high frequency terms in order to approximate a solution with high frequency components - e.g. edges. Reminder - this is independent of the data. But is an indication of an ill-posed problem. In this case the data that is modified exhibits in the matrix A decomposition.

30 Summary/Conclusions/Computational Issues Basis vectors are subject to noise and contaminate the solution independent the data. Nonlinear least squares use repeated solutions of least squares problems - SVD/GSVD analysis is relevant Solutions are obtained in a contaminated basis if we do not recognize the noise from the basis- how can we estimate uncertainty due to noise in data Analysis here in terms of SVD/GSVD - but equivalent results apply when using Krylov methods, need to examine the basis We have to truncate the basis but implies that we will not see high frequency in the solutions This is work in progress - lots to discuss

31 Power Spectrum for detecting white noise : a time series analysis technique Suppose for a given vector y that it is a time series indexed by position, i.e. index i. Diagnostic 1 Does the histogram of entries of y generate histogram consistent with y N(0, 1)? (i.e. independent normally distributed with mean 0 and variance 1) Not practical to automatically look at a histogram and make an assessment Diagnostic 2 Test the expectation that y i are selected from a white noise time series. Take the Fourier transform of y and form cumulative periodogram z from power spectrum c j c j = (dft(y) j 2 i=1, z j = c j q i=1 c, j = 1,..., q, i Automatic: Test is the line (z j, j/q) close to a straight line with slope 1 and length 5/2?

32 Advantages of the SB formulation Update for g: updates the Lagrange multiplier g S3 : g (k+1) = g (k) + Lx (k+1) d (k+1). This is just - a vector update Update for d: S2 : d = arg min{µ d 1 + λ2 d 2 d c 2 2}, c = Lx + g = arg min d { d 1 + γ 2 d c 2 2}, γ = λ2 µ. This is achieved using soft thresholding. - in place operation

33 The Tikhonov Step of the Algorithm S1 : x (k+1) = arg min{ 1 x 2 Ax b λ2 2 Lx (d(k+1) g (k) ) 2 2} Standard least squares update using a Tikhonov regularizer. x = arg min{ 1 x 2 Ax b λ2 2 Lx h 2 2}, h = d g Disadvantages of the formulation update for x: A Tikhonov LS update each step Right hand side deepndent on k Regularization parameter λ - dependent on k? Threshold parameter µ - dependent on k?