ECE295, Data Assimila0on and Inverse Problems, Spring 2015

Transcription

1 ECE295, Data Assimila0on and Inverse Problems, Spring April, Intro; Linear discrete Inverse problems (Aster Ch 1 and 2) Slides 8 April, SVD (Aster ch 2 and 3) Slides 15 April, RegularizaFon (ch 4) 22 April, Sparse methods (ch ) 29 April, more on Sparse 6 May, Bayesian methods and Monte Carlo methods (ch 11) 13 May, IntroducFon to sequenfal Bayesian methods, Kalman Filter (KF) 20 May, Ensemple Kalman Filer (EnKF) 27 May, EnKF, ParFcle Filter, 3 June, Markov Chain Monte Carlo Homework: Just the code to me (I dont need anything else). Call the files LastName_ExXX. Homework is due 8am on Wednesday. 8 April: Hw 1: Download the matlab codes for the book (cd_5.3) from this website 15 April: SVD analysis: SVD homework. You can also try replacing the matrix in the Shaw problem with the beamforming sensing matrix. The sensing matrix is available here. 22 April Late April: Beamforming May: Ice- flow from GPS

2 Recap: Linear discrete inverse problems The Least squares solution minimizes the prediction error. Goodness of fit criteria tells us whether the least squares model adequately fits the data, given the level of noise. Chi-square with N-M degrees of freedom The covariance matrix describes how noise propagates from the data to the estimated model Chi-square with M degrees of freedom Gives confidence intervals The resolution matrix describes how the estimated model relates to the true model 37

3 Recap: Goodness of fit and model covariance Once a best fit solution has been obtained we test goodness of fit with a chi-square test (assuming Gaussian statistics) If the model passes the goodness of fit test we may proceed to evaluating model covariance (if not then your data errors are probably too small) Evaluate model covariance matrix Plot model or projections of it onto chosen subsets of parameters Calculate confidence intervals using projected equation Where 2 follows a 2 1 distribution 28

4 Recap: Linear discrete inverse problems The Least squares solution minimizes the prediction error. Goodness of fit criteria tells us whether the least squares model adequately fits the data, given the level of noise. Chi-square with N-M degrees of freedom The covariance matrix describes how noise propagates from the data to the estimated model Chi-square with M degrees of freedom Gives confidence intervals The resolution matrix describes how the estimated model relates to the true model 37

5 SVD

6 Data and model null spaces 112

7 Moore-Penrose inverse and data null space d = Gm G = U p S p V T p The Moore-Penrose pseudo or generalized inverse of G is written It is the unique matrix that satisfies four special properties G = V p Sp 1 Up T GG G = G (GG ) T = GG G GG = G (G G) T = G G Even when G has zero eigenvalues the Moore-Penrose inverse always exists and has desirable properties m = G d obs = V p Sp 1 Up T d obs 113

8 Properties of the Moore-Penrose inverse U_p and V_p always exist and there are FOUR possible situations to consider Case 1: No model or data null space U o and V o do not exist Case 2: Only a model null space exists U o does not exist, V o exists Case 3: Only a data null space exists U o exists, V o does not exist Case 4: Both data and model null space exists U o and V o exists 114

9 Covariance and Resolution of the pseudo inverse How does data noise propagate into the model? What is the model covariance matrix for the generalized inverse? For the case C d = 2 I C M = G C d (G ) T G = V p Sp 1 Up T C M = 2 G (G ) T = 2 V p Sp 2 Vp T Recall that S p is a diagonal matrix of singular ordered values S p = diag[s 1,s 2,...,s p ] C M = 2 p X i=1 v i v T i s 2 i As the number of singular values, p, increases the variance of the model parameters increases! 120

10 Covariance and Resolution of the pseudo inverse How is the estimated model related to the true model? Model resolution matrix R = G G = V p Sp 1 Up T U p S p Vp T = V p V T p As p increases the model null space decreases m = Rm true G = V p Sp 1 Up T p M : V T p V 1 p, R I As the number of singular values, p, increases the resolution of the model parameters increases! We see the trade-off between variance and resolution 121

11 Worked example: tomography Using rays 1-4 G = d = Gm G T G = This has eigenvalues 0, 2, 4, 6. V p = V o = Gv o = 0 s 1 2 = 6 s 2 2 = 4 s 3 2 = 2 s 42 = 0 122

12 Worked example: tomography Using rays 1-4 G = d = Gm G T G = This has eigenvalues 0, 2, 4, 6. V p = V o = Gv o = 0 s 1 2 = 6 s 2 2 = 4 s 3 2 = 2 s 42 = 0 122

13 Worked example: Eigenvectors S 12 =6 S 22 = V p = S 3 2 = V o =

14 Worked example: tomography Using eigenvalues all non zero eigenvalues s 1, s 2 and s 3 the resolution matrix becomes m = Rm true = V p V T p m true R = V p = Input model Recovered model 124

15 Worked example: tomography Using eigenvalues s 1, s 2 and s 3 the model covariance becomes C M = 2 p X i=1 v i v T i s 2 i C M = s 1 2 = 2 s 22 = 4 s 3 2 = C M =

16 Worked example: tomography Repeat using only one singular value s 3 =6 Model resolution matrix V p = R = V p V T p = Model covariance matrix Input Output C M = 2 p X i=1 v i v T i s 2 i =

17 Recap: Singular value decomposition There may exist a model null space -> models that can not be constrained by the data. There may exist a data null space -> data that can not be fit by any model. The general linear discrete inverse problem may be simultaneously under and over determined (mix-determined). Singular value decomposition is a framework for dealing with ill-posed problems. The Pseudo inverse is constructed using SVD and provides a unique model with desirable properties. Fits the data in a least squares sense Gives a minimum length model (no component in the null space) Model Resolution and Covariance can be traded off by choosing the number of eigenvalues to use in reconstruction. 127

18 Regulariza0on

19 Regularization The idea behind SVD is to limit the degree of freedom in the model and fit the data to an acceptable level. Retain only those features necessary to fit the data. A general framework for solving non-unique inverse problems is to introduce regularization. Regularization makes a non-unique problem become a unique problem. How does it do this? REGULARIZATION We want to minimize a combination of data misfit and some property of the model that measures extravagant behaviour, e.g. (m) =(dg(m)) T C 1 d (dg(m))+(mm o) T Cm 1 (mm o ) Inverse problem nonlinear optimization problem 142

20 Solutions to inverse problems Optimal data fitting model Extremal model Data acceptable models The general framework is optimization (m) =(dg(m) T Cd 1 (dg(m)+(mm o) T CM 1 (mm o) 143

21 Damped least squares We seek the model that minimizes (d, m) =(dgm) T Cd 1 (dgm)+(mm o) T CM 1 (mm o) After some algebra we get m DLS =(G T C 1 D G+C1 M )1 (G T C 1 D d+c1 M m o) This is the damped least squares solution. A special case is to minimise a weighted sum of the data misfit and the model norm. min (d Gm) m 2 2 Normal equations become (G T G + I)m = G T d this system of linear equations will have a unique solution which is called the Tikhonov solution 144

22 L-curve solution Choose that is nearest the elbow of the curve With this value of we can construct a particular Tikhonov solution (G T G + I)m = G T d Perform repeat solutions of the normal equations for different and select the one which lies near the elbow of the trade-off curve Elbow is estimated visually -> potentially subjective if elbow is not clear See example 5.1 of Aster et al. (2005) 145

23 Tikhonov: Discrepancy principle Discrepancy principle gives us a value of and consequently a value for If the N data have Gaussian errors with known variance N(0, 2 ) then we would expect that on average that each residual (d i G I,j m j ) would be approximately. Hence we have d Gm 2 = N 2 c.f.2 N (50%) = N We get an expected value of the norm of the prediction error from the number of data and the standard deviation of the data (G T G + I)m = G T d Perform repeated solutions until d Gm = 147

24 Tikhonov regularization: Shaw problem (G T G + I)m = G T d L-curve for the Shaw problem Fortunately a nice clear elbow is visible in this log-log plot, Trade off parameter is easily picked. N[0, (10 6 ) 2 ] L = = = = Tikhonov solution for L-curve solution L m =

25 SVD and Tikhonov regularization Tkihonov solution G = U p S p V T p m =(G T G + I) 1 G T d It can be shown... Page of Aster et al (2005) Look! m = that this can be written as MX s 2 Ã! i ui d i=1 s 2 i + v i s i Now we get very different behaviour m = px Ã! ui d i=1 s i SVD v i Let f i = s2 i s 2 i + Filter factors As singular values s i 0 the solution is not highly sensitive to noise Because f i 0 rather than. If s i >> then f i 1 If s i << then f i 0 SVD s i >s p f i =1 s i <s p f i =0 filters out (damps) the unstable influence of the small eigenvalues 154