Advanced Numerical Linear Algebra: Inverse Problems

Transcription

1 Advanced Numerical Linear Algebra: Inverse Problems Rosemary Renaut Spring 23

2 Some Background on Inverse Problems Constructing PSF Matrices The DFT Rosemary Renaut February 4, 23

3 References Deblurring Images Matrices Spectra and Filtering, Hansen, Nagy, and O Leary, SIAM 26 asuedu/doi/book/37/ pch/hno/ Computational Methods for Inverse Problem, Vogel, SIAM 22 vogel/book/ book/37/ Rank Deficient and Discrete Ill-Posed Inverse Problems, Hansen, SIAM pch/regutools/ book/37/ Discrete Inverse Problems: Insights and Algorithms book/37/ Parameter Estimation and Inverse Problems, Aster, Borchers and Thurber

4 Illustrative Example: Fredholm first kind integral equation g(x) = h(x, x )f (x )dx, < x, x < This is a one dimensional example of signal blur - similar to the two dimensional image blur problem f is the light source intensity g is the measured image intensity h is the kernel characterizing blurring effects When h(x, x ) = h(x x ) The kernel is spatially invariant Typical choice of h(x) in D Gaussian h(x) = σ 2π exp( x 2 2σ 2 ), σ > Out of focus h(x) = { C l x u otherwise

5 Forward Problem: Given f, h calculate g Suppose that the integral is approximated using numerical quadrature yielding a matrix A Let dx = /n, x i = idx, i n Then A ij = w j h ij = w j h(x i, x j ) or w j h((i j)dx), where w j is the weight for the quadrature, eg w j = dx f is sampled at f (x j ), g(x) is sampled at x i Matrix equation g = Af describes the linear relationship Requirement b a h(x)dx (for PSF no energy is introduced, the signal is spread) leads to the normalization j w jh(x j ) =

6 Example of two Blurs, D Figure: Notice that without restoration of the data it will be difficult to extract image features

7 Matrices D Figure: These are the matrices for N = and 2 -notice the structure

8 Inverse Problem solve g Af when the problem ill-posed The solution depends on the conditioning of A, method and on the noise in measurements g 25 no noise inverse crime with backslash 25 no noise take inverse 4 with noise

9 Inverse Problem solve g Af when the problem ill-posed: Larger N = 2 The solution depends on the conditioning of A, method and on the noise in measurements g 5 5 no noise inverse crime with backslash no noise take inverse x with noise

10 Inverse Problem solve g Af problem that is spatially variant model N = x 3 data solutions

11 Inverse Problem solve g Af problem that is spatially variant model N = x 3 data x 2 solutions and with noise

12 Inverse Problem solve g Af problem that is spatially variant model N = x data solutions

13 Inverse Problem solve g Af problem that is spatially variant model N = x data solutions

14 Two Dimensional Problem The model is the same just extended to two dimensions: G(x, y) exact = H(x, x, y, y )X(x, y )dx dy H(x, x, y, y ) is often smooth, Impulse Response or Point Spread Function of imaging system X(x, y) is the original exact image, or light source When H is invariant in space isotropic H(x, x, y, y ) = H(x x, y y ) H might represent a projection as for example with PET images Typically a noisy image is obtained G(x, y) = H(x, x, y, y )X(x, y )dx dy + E(x, y) Often the noise distribution E(x, y) is known Distribution of E should be used in finding a solution

15 Example of restoration of 2D image with noise Reconstruction with noise N(, 5 )

16 Actual PET Images are Obtained from Projected Data Figure: Sinogram Data

17 Noisy Sinogram x

18 Typical PET Frames - noisy dependent on time window of data collection

19 D Formulation for the Convolution Blurred Signal g(x) = h(x x )f (x )dx Limits Take h(x) = for x > r then integration is limited g(x) = x+r x r h(x x )f (x )dx Description Flip the PSF function to obtain h( x ) 2 Shift by x to give h(x x ) 3 Then we see that g is a weighted local average of f with weights from h

20 Flip and Translate f (t) = < t < 2π h(t) = exp( t) < t < 2π Convolution g(x) = h(x t)f (t)dt < x < 2π g(x) = = x h(x t)f (t)dt = exp( (x t))dt 2π min {x,2π} h(x t)dt = max {,x 2π} < x < 2π = exp( x)(exp(x) ) = exp( x) exp( (x t))dt h(x) h( x) h(π x) g(x) g(x) f(x)

21 True Convolution: h(t) = 5 < t < f (t) = exp( t) < t < 2π x+ g(x) = f (t)dt = 2 max {,x } 2 exp( t) x+ max {,x } = { exp( x)(2 sinh()) x > 2 ( exp( (x + ))) x Convolution h( x) f(x) g(x)

22 Discrete Convolution of two sequences Discrete Convolution assuming normalization to ignore constant weighting in PSF h g i = j h i j f j Assume f R n, h R m and n > m g i = (f h) i = j Ω(i) f j h i j+ Ω(i) is the set of integers for the sum and may depend on i, depending on how the extent of h outside the defined range of the signal is implemented Equivalently we suppose that f is given only for a limited range, a Full convolution will require elements of f beyond the defined range Elements of f and h are only defined for positive index

23 The equations g = h f g 2 = h 2 f + h f 2 = g m = h m f + h m f 2 + h f m = g n = h m f n m+ + h m f n m+2 + h f n = g m+n = Notice that Ω(i) = m, i = n : m Ω(i) = {max(, i + m),, min(i, n)} h m f n For other i not all values of h because relevant indices of f are not defined

24 Matrix Formulation g g 2 g 3 g m g m+ g n g m+n = h h 2 h h 3 h 2 h h m h m h h m h m h h m h m h h m f f 2 f 3 f m f m+ f n Notice that the middle block contains the terms which can be calculated using all values of h In this formulation the lack of use of h amounts to padding f by zeros outside the valid domain

25 The complete Toeplitz System g g ṃ g n g m+n = h m h m h h m h m h h m h m h w f f n y w l, y l correspond to padding of f i Total number of padded entries is m f R m+n Pad h to length m + n similarly (Note that for only n entries for g, use the middle n entries) w l = y l = corresponds to using zero boundary conditions on f

26 Other Boundary Conditions : Periodic on f Suppose that the original values of f are assumed to be periodic Then we fill in the entries in the Toeplitz matrix: g m 2 g m g m g m+ g n g n+ g n+2 = h m 2 h m 3 h m h m h m h m 2 h h m h m h m h h m h m h h m h m h h h m h m h 2 h 2 h h m h 3 f f 2 f 3 f m f m+ f n This is called a CIRCULANT matrix Each row (column) is a periodic shift of previous row (column)

27 Reflexive Boundary Conditions Assume that the function f is reflected at the boundary For example look at the equations for g m and g n+ yielding g m = h m f + h m f + h m 2 f 2 + h f m g n+ = h m f n m+2 + h m f n m+3 + h 2 f n + h f n g m = (h m + h m )f + h m 2 f h f m g n+ = h m f n m+2 + h m f n m (h 2 + h )f n A matrix which has constant entries on each anti diagonal is a Hankel matrix Hence the resulting matrix is Toeplitz+Hankel

28 Example of Forward Operation Different Boundary Conditions Global Domain FFT Trunc and global Trunc Domain FFT Trunc : Periodic BC Trunc : Zero BC Trunc : RBC data gauss motion Global Domain FFT Trunc and global Trunc Domain FFT Trunc : Periodic BC Trunc : Zero BC 5 5 Trunc : RBC data gauss motion Figure: Convolution With Different Boundary Conditions

29 Example of Inversion Different Boundary Conditions Deblurred Gauss Deblurred Gauss 2 data Zero BC 2 data Zero BC Periodic Periodic Reflexive Reflexive Deblurred Step Deblurred Step 2 data Zero BC 2 data Zero BC Periodic Periodic Reflexive Reflexive Figure: Convolution With Different Boundary Conditions

30 Forward and Inversion Different Boundary Conditions: Small N Global Domain FFT Trunc and global Trunc Domain FFT Deblurred Gauss data Zero BC Periodic Reflexive Trunc : Periodic BC Trunc : Zero BC Trunc : RBC Deblurred Step data gauss motion 2 5 data Zero BC Periodic Reflexive Figure: Convolution With Different Boundary Conditions

31 d example code rosie/classes/spring23_ files/onedexample_pubhtml rosie/classes/spring23_ files/htmlfeb4/oned_basis_example_pub_v2html

32 Extension to 2D : In Matrix Notation Let x = vec(x), b = vec(g), be the matrices X, G with entries stacked as a vector ie we take the entries of X R m n column wise and set N = mn, vec(x) = x x 2 x n RN Then the convolution operator is rewritten as Ax = b where A represents the PSF H, and entries of A depend on how this is implemented Condition of the problem can still be investigated naïvely for the linear system This requires the generation of the matrix A for a two dimensional problem

33 Descriptive Version of 2D Convolution Regard the PSF as a mask with elements in an array 2 Rotate the PSF array by 8 3 To compute convolution for a certain pixel we have to line up the central point of the rotated mask on that pixel 4 Multiply the corresponding elements within the mask 5 Sum the elements within the mask 6 Take account of boundary conditions as for the D Case

34 Simple Example X = x x 2 x 3 x 2 x 22 x 23 x 3 x 32 x 33 H = h h 2 h 3 h 2 h 22 h 23 h 3 h 32 h 33 G = g g 2 g 3 g 2 g 22 g 23 g 3 g 32 g 33 Rotated H, align with x 22 and multiply Hflip = h 33 h 32 h 3 h 23 h 22 h 2 X Hflip = x h 33 x 2 h 32 x 3 h 3 x 2 h 23 x 22 h 22 x 23 h 2 h 3 h 2 h x 3 h 3 x 32 h 2 x 33 h where X H denotes element wise multiplication We then sum all the entries In the Vec notation we have g 22 = vec(hflip) T vec(x) as an inner product All entries are obtained as inner products

35 Calculating all entries for the case of zero boundary conditions g g 2 g 3 g 2 g 22 g 32 g 3 g 23 g 33 = h 22 h 2 h 2 h h 32 h 22 h 2 h 3 h 2 h h 32 h 22 h 3 h 2 h 23 h 3 h 22 h 2 h 2 h h 33 h 23 h 3 h 32 h 22 h 2 h 3 h 2 h h 33 h 23 h 32 h 22 h 3 h 2 h 23 h 3 h 22 h 2 h 33 h 23 h 3 h 32 h 22 h 2 h 33 h 23 h 32 h 22 Note that each block is of Toeplitz form and that as a block matrix the matrix is also Toeplitz Matrix is block Toeplitz with Toeplitz Blocks BTTB x x 2 x 3 x 2 x 22 x 32 x 3 x 23 x 33

36 Constructing matrix A: Boundary Conditions PSF is usually of finite extent, but for a finite section of an image, convolution with the PSF uses values outside the specific image If this is not handled correctly, boundary effects will contaminate the image, leading to artifacts This is true whether for the forward or inverse model Suggestions Zero Padding: Embed the image in a larger image, with sufficient zeros to handle the extent of the PSF O O O O X O O O O Introduces edges, discontinuities, leads to ringing and artificial black borders Matrix is block Toeplitz with Toeplitz Blocks BTTB

37 Constructing matrix A Periodicity: Embed the image periodically in a larger image X X X X X X X X X Introduces edges, discontinuities, leads to ringing and artificial black borders Matrix is block Circulant with Circulant Blocks BCCB Matrix is a sum of BTTB, BTHB,BHTB, BHHB H for a Hankel block

38 Constructing matrix A Periodicity: Embed the image periodically in a larger image X X X X X X X X X Introduces edges, discontinuities, leads to ringing and artificial black borders Matrix is block Circulant with Circulant Blocks BCCB Reflexive: Reflect and embed X lrud X ud X lrud X lr X X lr X lrud X ud X lrud Obvious reflections about the central block: no edges Matrix is a sum of BTTB, BTHB,BHTB, BHHB H for a Hankel block

39 Separable PSF: A = cr T c coefficients of blur across columns of the image r coefficients of blur across rows of the image c = (c, c 2,, c m ) T r = (r, r 2,, r n ) T The PSF matrix A is a rank one matrix A kj = c k r j A = c r c r 2 c r 3 c 2 r c 2 r 2 c 2 r 3 c 3 r c 3 r 2 c 3 r 3 Then let A r = r 2 r r 3 r 2 r r 3 r 2 A c = c 2 c c 3 c 2 c c 3 c 2 A r and A c inherit the structure of the D matrices dependent on how boundary conditions are applied We obtain the overall blur matrix A = A r A c

40 Kronecker Product To obtain the two dimensional PSF matrix A which acts on the image in vec form we map A using its special structure A r A c = = a a 2 a n a 2 a 22 a 2n a m a m2 a mn A c a A c a 2 A c a n A c a 2 A c a 22 A c a 2n A c a m A c a m2 A c a mn A c

41 Evaluating using the Kronecker Product Assume A c R m m and A r R n n, for an image of size m n Then, for the exact blurred image, G = A c XA T r Left multiplication by A c applies PSF to each column of X, is vertical blur Left multiplication of X T by A r applies PSF to each column of X T, which are rows of X Hence the horizontal blur is applied as (A r X T ) T = XA T r Let A = A r A c be the Kronecker product of size mn mn, the ij block of A is (A r ) ij A c Then G = A c XA T r in vector form is b = Ax

42 Summary: Constructing matrix A Typically we only see a finite region of an image that extends in all directions We have to be careful in constructing the matrix A Remark The blurring matrix A depends on two ingredients The PSF defines the blur But the boundary conditions imposed on the image, and hence on A, specify assumptions on the scene outside the image, which then impact the reconstruction of the image Remark Practically, we look for algorithms that do not calculate A

43 Practical Implementation: Fourier Transform Discrete convolution of two equal length sequences is accomplished by point wise multiplication in the Fourier domain Suppose f is a sequence of length n with components f j The unitary DFT of f is ˆf with component k ˆfk = n The inverse DFT is given by f j = n n j= n k= f j e 2πi(j )(k ) n ˆfk e 2πi(j )(k ) n Notice that the sequences are periodic with period n: f j+n = f j Extending to length m + n can be done by periodicity, or reflection at the boundaries

44 Fourier transform as a Matrix Operator It may be useful to note that the Fourier transform can be implemented (thought of) as a matrix operation: ˆf = Ff F kj = e 2πi(k )(j ) n n f = F ˆf = F f Fkj = e 2πi(k )(j ) n n using the fact that the matrix F is unitary, F F = I n

45 Summary Implementation of the D Convolution by FFT g i = (f h) i = j Ω(i) f j h i j+, i = : m + n Pad sequences with zeros to length m + n Further pad to length power of two 2 Transform both sequences to frequency domain by DFT 3 Multiply DFTS pointwise ĝ i = ˆf i ĥ i 4 Inverse transform ĝ to yield g sampled at x i, i = : m + n 5 For real data ignore imaginary part of FFT which occurs due to floating point accuracy 6 Extract needed components of g

46 Constructing Ax for structured A Spectral decompositions Solutions in terms of spectral decompositions Rosemary Renaut January, 23

47 Circulant Matrix Definition Toeplitz A is circulant if rows / columns are circular shifts h h h (n 3) h (n 2) h 2 h h (n 4) h (n 3) A = (a ij ) = (h i j+ ) = h n h 2 h h h n h n h 2 h Write A = toeplitz(h ext ) where h ext is the extension of h of length 2n h ext = (h (n 2), h (n 3),, h, h, h 2,, h n ) For circulant it is periodic with period n and A = circulant(h, h 2, h n ) Important : h is the first column of A

48 Relating circulant matrices and the DFT Definition (Circulant Right Shift Matrix) R = or R kj = δ([j k + ] n) Notation δ(j) = only for j = Note right shift property R(h, h 2, h 3,, h n) T = (h n, h,, h n ) T Induction: R j (h, h 2, h 3,, h n) T = (h n j+, h n j+2,, h n j ) T R j = Row j

49 Expressing the Circulant Matrix A = h h n h 3 h 2 h 2 h h 4 h 3 = h n h 2 h h n h n h n h 2 h n h j+ R j (periodicity h n+ = h ) j= Let ω = exp( 2πi/n), then nf kj = ω (k )(j ), nf kj = ω (k )(j ) and use n n ω (j )(k ) = δ([k ] n), F F = FF = I n j= Then it is immediate to show (i) R = F diag(, ω, ω 2,, ω n )F = F diag(ω l )F (ii) RF l = ω l F l, R j = F diag(ω l ) j F Here F l is the l th column of F l = : n

50 The Spectral Decomposition We apply the spectral decomposition results for R to show A = n h j+ R j j= = F diag( n h j+ ω (l )j )F j= = nf diag(ĥ)f = F ΛF, Λ = n diag(ĥ, ĥ2,, ĥn) ( nĥl, F l ) are the eigenvalue - eigenvector pairs of A To find eigenvalues of A observe FA = ΛF and F = (/ n)(,,, ) T hence ( nfa ) l = λ l

51 Computing Ax If the system is nonsingular A = F Λ F Note also that A is normal AA = F ΛFF ΛF = F (Λ) 2 F = A A Ax is calculated via Ax = F diag(nĥ)fx, transform Fx = ˆx = nf (ĥ ˆx) convolution ŷ = ĥ ˆx = n y inverse y = F ŷ Practically if Λ is known we write Ax = y where y = F (ˆλ ˆx) To calculate the product Ax when A is Toeplitz, embed in a larger circulant matrix (see Vogel chap 5) A similar spectral decomposition is obtained

52 The Two Dimensional BCCB Case : Simple Recall the earlier case with 3 3 PSF h 22 h 2 h 32 h 2 h h 3 h 23 h 3 h 33 h 32 h 22 h 2 h 3 h 2 h h 33 h 23 h 3 h 2 h 32 h 22 h h 3 h 2 h 3 h 33 h 23 h 23 h 3 h 33 h 22 h 2 h 32 h 2 h h 3 h 33 h 23 h 3 h 32 h 22 h 2 h 3 h 2 h = h 3 h 33 h 23 h 2 h 32 h 22 h h 3 h 2 h 2 h h 3 h 23 h 3 h 33 h 22 h 2 h 32 h 3 h 2 h h 33 h 23 h 3 h 32 h 22 h 2 h h 3 h 2 h 3 h 33 h 23 h 2 h 32 h 22 H 2 H H 3 H 3 H 2 H H H 3 H 2 H = circulant(h, h 2, h 3 ) = circulant(h, ) H 2 = circulant(h 2, h 22, h 32 ) = circulant(h,2 ) H 3 = circulant(h 3, h 23, h 33 ) = circulant(h,3 ) and the entire PSF can be obtained from the first column of A

53 The general case for BCCB A(h) = H 2 H H 4 H 3 H 3 H 2 H H 4 H ny H3 H 2 H H H ny H 3 H 2 Each block is circulant, H j = circulant(h,j ), h of size n x n y A is of size n x n y n x n y, n = n x n y The first column defines h Again A is normal and we have a spectral decomposition A = F diag( n vec(ĥ))f = F ΛF Here F = F r F c and ĥ is the two dimensional FFT of h of size n x n y ĥ = F(h) = (F r F c )h Calculation scheme: A vec(x) = vec(f (ˆλ vec(fx)))

54 Calculating the Spectral Decomposition Λ for 2D BCCB As for the D case to find eigenvalues of A observe FA = ΛF and F = (/ n)(,,, ) T, (look at the Kronecker product) Hence ( nfa ) l = λ l For the BCCB first column of A gives on h Suppose center point of h in spatial domain is at index center = (r, c ) Matlab can be used to find this column A and F acting on A is accomplished as the 2D transform fft2(circshift(h, center)) ie circshift performs necessary circular shift of the PSF matrix Note that fft2 in Matlab implements nf but also ifft2 implements / nf

55 Spectral Decomposition Λ for Reflexive : Hansen (2) Assume underlying Toeplitz matrix is symmetric, ie the PSF is doubly symmetric about the central point: fliplr(h) = flipud(h) = h The flip operators in Matlab flip columns(rows) of array As D circulant use expansion: A = n j= h js j for basis matrices S j (S j ) lk = l k = j l + k = j l + k = 2n + 2 j To find the decomposition use the discrete cosine transform (DCT) dct vectors w l, l = : n are eigenvectors of each of basis matrices S j w l = c l (cos((l )θ ), cos((l )θ 2 ),, cos((l )θ n)) T, l = : n (2k )π 2 θ k =, k = : n, c l = l >, c = 2n n n Matrix W n defined with columns w l is orthogonal Using the eigenvector decomposition for each basis matrix, the spectrum for A can be obtained A = Ae = W nλw T n e W T n A = ΛW T n e Eigenvalues are λ l = [dct(a )] l /[dct(e )] l because dct(x) = W T n x

56 DCT analysis extends for the 2D case with reflexive BCs A = W T ΛW where W is the orthogonal two-dimensional DCT matrix 2D DCT is calculated for array X by computing DCT of each column, followed by DCT of each row Inverse is calculated equivalently A = W T Λ W Use Matlab functions dct2 and idct2 in the image processing toolbox Alternatively, toolbox for the HNO book! The spectrum is obtained from dct2(dctshift(h, center))/dct2(e ) DCT Implementation uses real arithmetic rather than complex for the FFT Note cost equivalent to FFT cost

57 Summary: Spectral Decomposition for (Most) PSF Matrices :BTTB, BCCB, Double Symmetric Reflexive Given h the PSF array, center = (r, c ) the central index of the PSF Eigenvalues of A are obtained by a transform applied to shifted h A spectral decomposition exists A = U ΛU which can be used for forward and inverse operations All forward and inverse operations use transforms and convolution We do not know anything about the ordering of the eigenvalues in Λ in terms of size

58 The Singular Value Decomposition (SVD): General A Suppose, here, A R N N : A = UΣV T, U and V are orthogonal U T U = I N = V T V, σ i are the singular values of A, Σ = diag(σ i ) σ σ 2 σ N u i (columns of U) are the left singular vectors of A v i (columns of V ) are the right singular vectors of A Assume σ N >, A is nonsingular, then A = V Σ U T = N i= v i u T i σ i, A = N σ i u i v T i i= Solution expressed with the SVD components is a weighted linear combination of basis vectors v i x = N i= ( ut i b )v i σ i

59 The Two Dimensional Result We note that x = vec(x) Likewise we can write V i = array(v i ), where array is the reverse of vec converting vector to array Then X = = N i= N i= ( ut i b σ i )array(v i ) ( ut i b σ i )V i This is a decomposition in the basis images V i

60 Kronecker Product Relations 2 Transpose is distributive (A B)vec(X) = vec(bxa T ) (A B) T = (A T B T ) 3 When both matrices are invertible (A B) = A B 4 Mixed product property for matrices of appropriate dimensions (A B)(C D) = (AB CD) 5 The Kronecker product is associative, but not commutative

61 Spectral Expansion for DFT : BCCB A = F ΛF Use F = F r F c, the one dimensional FT matrices x = F Λ Fb = (F r F c ) Λ vec(f c GF T r ) Notice vec(f c GFr T ) is the two dimensional FT of the blurred image G Let vec( G) = Λ vec(f c GFr T ) X = Fc G(F r ) T = conj(f c ) G conj(fr T ) X = G ij conj(f c,i f T r,j ) i j f c,i and f r,j are the i th and j th columns of F c and F r This is a decomposition in the basis images f c,i f T r,j

62 Spectral Expansion for the Reflexive Case A = W T ΛW Use W = W r W c, the one dimensional DCT matrices x = W T Λ W b = (W r W c ) T Λ vec(w c GW T r ) vec(w c GW T r ) is the two dimensional DCT of the blurred image G Let vec( G) = Λ vec(w c GW T r ) X = (W c ) T G (Wr ) X = G ij (w c,i w T r,j ) i j w c,i and w r,j are the i th and j th columns of W c and W r This is a decomposition in the basis images w c,i w T r,j

63 The Separable PSF: A = cr T Matrix A is obtained as Kronecker product A = A r A c Ax = A c XA T r SVD of each matrix A r = U r Σ r V T r SVD of Kronecker product A c = U c Σ c V T c (U r Σ r V T r ) (U c Σ c V T c ) = (U r U c )(Σ r Σ c )(V r V c ) T Extend the one dimensional SVD solution for b = Ax X = V c Σ c U T c GU r Σ r Vr T vec(u c GUr T ) = (U r U c ) T vec(g) represents the spectral transform of the blurred image G Let G be the matrix Σ c Uc T GU r Σ r then X = V c GV T r = (V r V c ) G = Gij (v c,i v T r,j ) v c,i and v r,j are the i th and j th columns of V c and V r This is a decomposition in the basis images v c,i v T r,j

64 Summary: Spectral Decomposition of the Solution For all cases we have a decomposition of the solution as a sum of basis images For the Fourier case the order of the spectral components is not defined as it is for the SVD We can examine the solution by examining the spectral components in each case G ij are equivalent to the weights u T i b/σ i of the SVD We note that the DFT and DCT provide feasible approaches for finding the defining spectrum, the large scale SVD may be too expensive for practical problems

65 The Noise Practically the measured data is contaminated by noise, we denote by e or array(e) = E The spectral decomposition acts on the noise term in the same way it acts on the exact right hand side b eg x = r i= ( ut i (b exact + e) σ i )v i = x exact + r i= ( ut i e σ i r is the rank of A If e is uniform, we expect u T i e to be similar magnitude i Note cond(a) = σ /σ r Typically, expect σ i to decay to zero Hence cond(a) may be large σ i small represents high frequency component in the sense that u i, v i have more sign changes as i increases ) is the coefficient of v i in the error image, the weight of v i in the error image, where v i is associated with a spatial frequency When /σ i is large the contribution of the high frequency error is magnified due to ( ut i e σ ) ( ut i e σ i )v i

66 Example of singular values dependent on Width of Gaussian PSF and two noise levels Example taken from Hansen, Nagy and O Leary, Figure 55

67 Example of basis images for Middle PSF in 55, using SVD Example taken from Hansen, Nagy and O Leary, Figure 57

68 Example of basis images : Periodic Boundary Conditions Example taken from Hansen, Nagy and O Leary, Figure 58 Notice the periodicity effect on the basis images

69 Example of basis images : Reflexive Boundary Conditions Example taken from Hansen, Nagy and O Leary, Figure 59 Notice the reflexive effect on the basis images

70 Observations σ i depend on matrix A, which depends on width of the PSF σi decay faster for wider PSF than narrow PSF implemented boundary conditions In particular the basis images satisfy the boundary conditions Moreover u T i b depends on the image, and on the noise level in the image Thus (u T i b)/σ i depends on A, e and b When σ i is small relative u T i e solution may be contaminated

71 Example of basis images: Periodic BCCB using DFT Example taken from Hansen, Nagy and O Leary, Figure 5 Notice that DFT basis images are quite different

72 Example of basis images: Doubly Symmetric PSF using DCT Example taken from Hansen, Nagy and O Leary, Figure 5 Notice that DCT basis images are again also quite different

73 Comparing magnitudes of Spectral Components in DFT, DCT and SVD Bases: separable Gaussian blur Example taken from Hansen, Nagy and O Leary, Figure 52 Ordered: we see FFT and DCT have fewer large SVs

74 Summary: The Spectral Decomposition or SVD SVD assists with understanding the impact of noise on the solution process Spectral decomposition shows how the solution is composed of series of basis images Practically SVD is not useful for large scale problems Practically when A arises from image deblurring the FFT and DCT can be used to find the spectral decomposition and to efficiently implement forward operations Ay, A T y and A y for any y Spectral decomposition suggests truncation to remove the components which lead to contamination of the solution

75 Truncating the SVD expansion The SVD expansion shows the impact of the noise on the calculation of x = r i= ( ut i b σ i )v i Therefore it seems reasonable to consider the truncated solution x k = Here A k is a rank k matrix k i= ( ut i b σ i )v i = A k b The use of the truncated expansion is feasible only if we can first calculate the SVD of A efficiently The limit on the sum k is regarded as a regularization parameter We can change k and obtain different images Choice of k controls the degree of low pass filtering which is applied ie controls the attenuation of the high frequency components Look at the example: the image is over or under smoothed dependent on k

76 Example of Truncated SVD Example from Hansen, Nagy and O Leary, Fig 5 Notice under/over smoothing dependent on choice of k in the TSVD