Regularization Parameter Estimation for Least Squares: A Newton method using the χ 2 -distribution

Transcription

1 Regularization Parameter Estimation for Least Squares: A Newton method using the χ 2 -distribution Rosemary Renaut, Jodi Mead Arizona State and Boise State September 2007 Renaut and Mead (ASU/Boise) Scalar Newton method September / 31

2 Outline 1 Introduction- Ill-posed least squares Some Standard Methods 2 A Statistically based method: Chi squared Method Background Algorithm Single Variable Newton Method Extend for General D: Generalized Tikhonov Observations 3 Results 4 Conclusions 5 References Renaut and Mead (ASU/Boise) Scalar Newton method September / 31

3 Regularized Least Squares for Ax = b Assume: A R m n, b R m, x R n, and the system is ill-posed. Generalized Tikhonov regularization, operator D acts on x. ˆx = argmin J(x) = argmin{ Ax b W 2 b + D(x x 0 ) 2 W x }. (1) Assume N (A) N (D) = Statistically W b is inverse covariance matrix for data b. Standard: W x = λ 2 I, λ unknown penalty parameter: (1) is ˆx(λ) = argmin J(x) = argmin{ Ax b 2 W b + λ 2 D(x x 0 ) 2 }. (2) Renaut and Mead (ASU/Boise) Scalar Newton method September / 31

4 Regularized Least Squares for Ax = b Assume: A R m n, b R m, x R n, and the system is ill-posed. Generalized Tikhonov regularization, operator D acts on x. ˆx = argmin J(x) = argmin{ Ax b W 2 b + D(x x 0 ) 2 W x }. (1) Assume N (A) N (D) = Statistically W b is inverse covariance matrix for data b. Standard: W x = λ 2 I, λ unknown penalty parameter: (1) is ˆx(λ) = argmin J(x) = argmin{ Ax b 2 W b + λ 2 D(x x 0 ) 2 }. (2) Renaut and Mead (ASU/Boise) Scalar Newton method September / 31

5 Regularized Least Squares for Ax = b Assume: A R m n, b R m, x R n, and the system is ill-posed. Generalized Tikhonov regularization, operator D acts on x. ˆx = argmin J(x) = argmin{ Ax b W 2 b + D(x x 0 ) 2 W x }. (1) Assume N (A) N (D) = Statistically W b is inverse covariance matrix for data b. Standard: W x = λ 2 I, λ unknown penalty parameter: (1) is ˆx(λ) = argmin J(x) = argmin{ Ax b 2 W b + λ 2 D(x x 0 ) 2 }. (2) Renaut and Mead (ASU/Boise) Scalar Newton method September / 31

6 Regularized Least Squares for Ax = b Assume: A R m n, b R m, x R n, and the system is ill-posed. Generalized Tikhonov regularization, operator D acts on x. ˆx = argmin J(x) = argmin{ Ax b W 2 b + D(x x 0 ) 2 W x }. (1) Assume N (A) N (D) = Statistically W b is inverse covariance matrix for data b. Standard: W x = λ 2 I, λ unknown penalty parameter: (1) is ˆx(λ) = argmin J(x) = argmin{ Ax b 2 W b + λ 2 D(x x 0 ) 2 }. (2) Question: What is the correct λ? Renaut and Mead (ASU/Boise) Scalar Newton method September / 31

7 Some standard approaches I: L-curve - Find the corner Let r(λ) = (A(λ) A)b: Influence Matrix A(λ) = A(A T W b A + λ 2 D T D) 1 A T Plot log( Dx ), log( r(λ) ) Trade off contributions. Expensive - requires range of λ. GSVD makes calculations efficient. Not statistically based Renaut and Mead (ASU/Boise) Scalar Newton method September / 31

8 Some standard approaches I: L-curve - Find the corner Let r(λ) = (A(λ) A)b: Influence Matrix A(λ) = A(A T W b A + λ 2 D T D) 1 A T Plot log( Dx ), log( r(λ) ) Trade off contributions. Expensive - requires range of λ. GSVD makes calculations efficient. Not statistically based Renaut and Mead (ASU/Boise) Scalar Newton method September / 31

9 Some standard approaches I: L-curve - Find the corner Let r(λ) = (A(λ) A)b: Influence Matrix A(λ) = A(A T W b A + λ 2 D T D) 1 A T Plot log( Dx ), log( r(λ) ) Trade off contributions. Expensive - requires range of λ. GSVD makes calculations efficient. Not statistically based Find corner Renaut and Mead (ASU/Boise) Scalar Newton method September / 31

10 Some standard approaches I: L-curve - Find the corner Let r(λ) = (A(λ) A)b: Influence Matrix A(λ) = A(A T W b A + λ 2 D T D) 1 A T Plot log( Dx ), log( r(λ) ) Trade off contributions. Expensive - requires range of λ. GSVD makes calculations efficient. Not statistically based Find corner No corner Renaut and Mead (ASU/Boise) Scalar Newton method September / 31

11 Some standard approaches I: L-curve - Find the corner Let r(λ) = (A(λ) A)b: Influence Matrix A(λ) = A(A T W b A + λ 2 D T D) 1 A T Plot log( Dx ), log( r(λ) ) Trade off contributions. Expensive - requires range of λ. GSVD makes calculations efficient. Not statistically based Find corner No corner Renaut and Mead (ASU/Boise) Scalar Newton method September / 31

12 Some standard approaches I: L-curve - Find the corner Let r(λ) = (A(λ) A)b: Influence Matrix A(λ) = A(A T W b A + λ 2 D T D) 1 A T Plot log( Dx ), log( r(λ) ) Trade off contributions. Expensive - requires range of λ. GSVD makes calculations efficient. Not statistically based Find corner No corner Renaut and Mead (ASU/Boise) Scalar Newton method September / 31

13 Some standard approaches I: L-curve - Find the corner Let r(λ) = (A(λ) A)b: Influence Matrix A(λ) = A(A T W b A + λ 2 D T D) 1 A T Plot log( Dx ), log( r(λ) ) Trade off contributions. Expensive - requires range of λ. GSVD makes calculations efficient. Not statistically based Find corner No corner Renaut and Mead (ASU/Boise) Scalar Newton method September / 31

14 Some standard approaches II: Generalized Cross-Validation (GCV) Minimizes GCV function b Ax(λ) 2 W b [trace(i m A(λ))] 2, which estimates predictive risk. Expensive - requires range of λ. GSVD makes calculations efficient. Statistically based Requires minimum Renaut and Mead (ASU/Boise) Scalar Newton method September / 31

15 Some standard approaches II: Generalized Cross-Validation (GCV) Minimizes GCV function b Ax(λ) 2 W b [trace(i m A(λ))] 2, which estimates predictive risk. Expensive - requires range of λ. GSVD makes calculations efficient. Statistically based Requires minimum Multiple minima Renaut and Mead (ASU/Boise) Scalar Newton method September / 31

16 Some standard approaches II: Generalized Cross-Validation (GCV) Minimizes GCV function b Ax(λ) 2 W b [trace(i m A(λ))] 2, which estimates predictive risk. Expensive - requires range of λ. GSVD makes calculations efficient. Statistically based Requires minimum Multiple minima Sometimes flat Renaut and Mead (ASU/Boise) Scalar Newton method September / 31

17 Some standard approaches II: Generalized Cross-Validation (GCV) Minimizes GCV function b Ax(λ) 2 W b [trace(i m A(λ))] 2, which estimates predictive risk. Expensive - requires range of λ. GSVD makes calculations efficient. Statistically based Requires minimum Multiple minima Sometimes flat Renaut and Mead (ASU/Boise) Scalar Newton method September / 31

18 Some standard approaches II: Generalized Cross-Validation (GCV) Minimizes GCV function b Ax(λ) 2 W b [trace(i m A(λ))] 2, which estimates predictive risk. Expensive - requires range of λ. GSVD makes calculations efficient. Statistically based Requires minimum Multiple minima Sometimes flat Renaut and Mead (ASU/Boise) Scalar Newton method September / 31

19 Some standard approaches II: Generalized Cross-Validation (GCV) Minimizes GCV function b Ax(λ) 2 W b [trace(i m A(λ))] 2, which estimates predictive risk. Expensive - requires range of λ. GSVD makes calculations efficient. Statistically based Requires minimum Multiple minima Sometimes flat Renaut and Mead (ASU/Boise) Scalar Newton method September / 31

20 Some standard approaches II: Generalized Cross-Validation (GCV) Minimizes GCV function b Ax(λ) 2 W b [trace(i m A(λ))] 2, which estimates predictive risk. Expensive - requires range of λ. GSVD makes calculations efficient. Statistically based Requires minimum Multiple minima Sometimes flat Renaut and Mead (ASU/Boise) Scalar Newton method September / 31

21 Some standard approaches III: Unbiased Predictive Risk Estimation (UPRE) Minimize expected value of predictive risk: Minimize UPRE function b Ax(λ) 2 W b +2 trace(a(λ)) m Expensive - requires range of λ. GSVD makes calculations efficient. Statistically based Minimum needed Renaut and Mead (ASU/Boise) Scalar Newton method September / 31

22 Some standard approaches III: Unbiased Predictive Risk Estimation (UPRE) Minimize expected value of predictive risk: Minimize UPRE function b Ax(λ) 2 W b +2 trace(a(λ)) m Expensive - requires range of λ. GSVD makes calculations efficient. Statistically based Minimum needed Renaut and Mead (ASU/Boise) Scalar Newton method September / 31

23 Some standard approaches III: Unbiased Predictive Risk Estimation (UPRE) Minimize expected value of predictive risk: Minimize UPRE function b Ax(λ) 2 W b +2 trace(a(λ)) m Expensive - requires range of λ. GSVD makes calculations efficient. Statistically based Minimum needed Renaut and Mead (ASU/Boise) Scalar Newton method September / 31

24 Some standard approaches III: Unbiased Predictive Risk Estimation (UPRE) Minimize expected value of predictive risk: Minimize UPRE function b Ax(λ) 2 W b +2 trace(a(λ)) m Expensive - requires range of λ. GSVD makes calculations efficient. Statistically based Minimum needed Renaut and Mead (ASU/Boise) Scalar Newton method September / 31

25 Some standard approaches III: Unbiased Predictive Risk Estimation (UPRE) Minimize expected value of predictive risk: Minimize UPRE function b Ax(λ) 2 W b +2 trace(a(λ)) m Expensive - requires range of λ. GSVD makes calculations efficient. Statistically based Minimum needed Renaut and Mead (ASU/Boise) Scalar Newton method September / 31

26 Some standard approaches III: Unbiased Predictive Risk Estimation (UPRE) Minimize expected value of predictive risk: Minimize UPRE function b Ax(λ) 2 W b +2 trace(a(λ)) m Expensive - requires range of λ. GSVD makes calculations efficient. Statistically based Minimum needed Renaut and Mead (ASU/Boise) Scalar Newton method September / 31

27 Development A New Statistically based method: The Chi squared Method Its Background A Newton algorithm Some Examples Future Work Renaut and Mead (ASU/Boise) Scalar Newton method September / 31

28 Development A New Statistically based method: The Chi squared Method Its Background A Newton algorithm Some Examples Future Work Renaut and Mead (ASU/Boise) Scalar Newton method September / 31

29 Development A New Statistically based method: The Chi squared Method Its Background A Newton algorithm Some Examples Future Work Renaut and Mead (ASU/Boise) Scalar Newton method September / 31

30 Development A New Statistically based method: The Chi squared Method Its Background A Newton algorithm Some Examples Future Work Renaut and Mead (ASU/Boise) Scalar Newton method September / 31

31 Development A New Statistically based method: The Chi squared Method Its Background A Newton algorithm Some Examples Future Work Renaut and Mead (ASU/Boise) Scalar Newton method September / 31

32 General Result: Tikhonov (D = I) Cost functional at min is χ 2 r.v. Theorem (Rao:73, Tarantola, Mead (2007)) J(x) = (b Ax) T C b 1 (b Ax) + (x x 0 ) T C x 1 (x x 0 ), x and b are stochastic (need not be normal) r = b Ax 0 are iid. (Assume no components are zero) Matrices C b = W b 1 and C x = W x 1 are SPD - Then for large m, minimium value of J is a random variable it follows a χ 2 distribution with m degrees of freedom. Renaut and Mead (ASU/Boise) Scalar Newton method September / 31

33 General Result: Tikhonov (D = I) Cost functional at min is χ 2 r.v. Theorem (Rao:73, Tarantola, Mead (2007)) J(x) = (b Ax) T C b 1 (b Ax) + (x x 0 ) T C x 1 (x x 0 ), x and b are stochastic (need not be normal) r = b Ax 0 are iid. (Assume no components are zero) Matrices C b = W b 1 and C x = W x 1 are SPD - Then for large m, minimium value of J is a random variable it follows a χ 2 distribution with m degrees of freedom. Renaut and Mead (ASU/Boise) Scalar Newton method September / 31

34 General Result: Tikhonov (D = I) Cost functional at min is χ 2 r.v. Theorem (Rao:73, Tarantola, Mead (2007)) J(x) = (b Ax) T C b 1 (b Ax) + (x x 0 ) T C x 1 (x x 0 ), x and b are stochastic (need not be normal) r = b Ax 0 are iid. (Assume no components are zero) Matrices C b = W b 1 and C x = W x 1 are SPD - Then for large m, minimium value of J is a random variable it follows a χ 2 distribution with m degrees of freedom. Renaut and Mead (ASU/Boise) Scalar Newton method September / 31

35 General Result: Tikhonov (D = I) Cost functional at min is χ 2 r.v. Theorem (Rao:73, Tarantola, Mead (2007)) J(x) = (b Ax) T C b 1 (b Ax) + (x x 0 ) T C x 1 (x x 0 ), x and b are stochastic (need not be normal) r = b Ax 0 are iid. (Assume no components are zero) Matrices C b = W b 1 and C x = W x 1 are SPD - Then for large m, minimium value of J is a random variable it follows a χ 2 distribution with m degrees of freedom. Renaut and Mead (ASU/Boise) Scalar Newton method September / 31

36 General Result: Tikhonov (D = I) Cost functional at min is χ 2 r.v. Theorem (Rao:73, Tarantola, Mead (2007)) J(x) = (b Ax) T C b 1 (b Ax) + (x x 0 ) T C x 1 (x x 0 ), x and b are stochastic (need not be normal) r = b Ax 0 are iid. (Assume no components are zero) Matrices C b = W b 1 and C x = W x 1 are SPD - Then for large m, minimium value of J is a random variable it follows a χ 2 distribution with m degrees of freedom. Renaut and Mead (ASU/Boise) Scalar Newton method September / 31

37 Implications: Theorem implies m 2z α/2 < J(ˆx) < m + 2z α/2 for confidence interval (1 α), ˆx the solution. Equivalently, when D = I, m 2z α/2 < r T (AC x A T + C b ) 1 r < m + 2z α/2. Renaut and Mead (ASU/Boise) Scalar Newton method September / 31

38 Implications: Theorem implies m 2z α/2 < J(ˆx) < m + 2z α/2 for confidence interval (1 α), ˆx the solution. Equivalently, when D = I, m 2z α/2 < r T (AC x A T + C b ) 1 r < m + 2z α/2. Renaut and Mead (ASU/Boise) Scalar Newton method September / 31

39 Implications: Theorem implies m 2z α/2 < J(ˆx) < m + 2z α/2 for confidence interval (1 α), ˆx the solution. Equivalently, when D = I, m 2z α/2 < r T (AC x A T + C b ) 1 r < m + 2z α/2. Renaut and Mead (ASU/Boise) Scalar Newton method September / 31

40 Implications: Theorem implies m 2z α/2 < J(ˆx) < m + 2z α/2 for confidence interval (1 α), ˆx the solution. Equivalently, when D = I, m 2z α/2 < r T (AC x A T + C b ) 1 r < m + 2z α/2. Note no assumptions on W x : it is completely general Renaut and Mead (ASU/Boise) Scalar Newton method September / 31

41 Implications: Theorem implies m 2z α/2 < J(ˆx) < m + 2z α/2 for confidence interval (1 α), ˆx the solution. Equivalently, when D = I, m 2z α/2 < r T (AC x A T + C b ) 1 r < m + 2z α/2. Note no assumptions on W x : it is completely general Can we use the result to obtain an efficient algorithm? Renaut and Mead (ASU/Boise) Scalar Newton method September / 31

42 First attempt: a New Algorithm for Estimating Model Covariance Algorithm (Mead 07) Given confidence interval parameter α, initial residual r = b Ax 0 and estimate of the data covariance C b, find L x which solves the nonlinear optimization. Renaut and Mead (ASU/Boise) Scalar Newton method September / 31

43 First attempt: a New Algorithm for Estimating Model Covariance Algorithm (Mead 07) Given confidence interval parameter α, initial residual r = b Ax 0 and estimate of the data covariance C b, find L x which solves the nonlinear optimization. Minimize L x L T x 2 F Subject to m 2z α/2 < r T (AL x L T x A T + C b ) 1 r < m + 2z α/2 AL x L T x A T + C b well-conditioned. Renaut and Mead (ASU/Boise) Scalar Newton method September / 31

44 First attempt: a New Algorithm for Estimating Model Covariance Algorithm (Mead 07) Given confidence interval parameter α, initial residual r = b Ax 0 and estimate of the data covariance C b, find L x which solves the nonlinear optimization. Minimize L x L T x 2 F Subject to m 2z α/2 < r T (AL x L T x A T + C b ) 1 r < m + 2z α/2 AL x L T x A T + C b well-conditioned. Expensive Renaut and Mead (ASU/Boise) Scalar Newton method September / 31

45 Single Variable Approach: Seek efficient, practical algorithm Let W x = σ 2 x I, where regularization parameter λ = 1/σ x. Use SVD to implement U b Σ b Vb T and define s = U b W 1/2 b r: Find σ x such that = W b 1/2 A, svs σ 1 σ 2... σ p m 2z α/2 < s T 1 diag( σi 2 σx )s < m + 2z α/2. Equivalently, find σx 2 such that F (σ x ) = s T 1 diag( 1 + σxσ 2 i 2 )s m = 0. Renaut and Mead (ASU/Boise) Scalar Newton method September / 31

46 Single Variable Approach: Seek efficient, practical algorithm Let W x = σ 2 x I, where regularization parameter λ = 1/σ x. Use SVD to implement U b Σ b Vb T and define s = U b W 1/2 b r: Find σ x such that = W b 1/2 A, svs σ 1 σ 2... σ p m 2z α/2 < s T 1 diag( σi 2 σx )s < m + 2z α/2. Equivalently, find σx 2 such that F (σ x ) = s T 1 diag( 1 + σxσ 2 i 2 )s m = 0. Renaut and Mead (ASU/Boise) Scalar Newton method September / 31

47 Single Variable Approach: Seek efficient, practical algorithm Let W x = σ 2 x I, where regularization parameter λ = 1/σ x. Use SVD to implement U b Σ b Vb T and define s = U b W 1/2 b r: Find σ x such that = W b 1/2 A, svs σ 1 σ 2... σ p m 2z α/2 < s T 1 diag( σi 2 σx )s < m + 2z α/2. Equivalently, find σx 2 such that F (σ x ) = s T 1 diag( 1 + σxσ 2 i 2 )s m = 0. Renaut and Mead (ASU/Boise) Scalar Newton method September / 31

48 Single Variable Approach: Seek efficient, practical algorithm Let W x = σ 2 x I, where regularization parameter λ = 1/σ x. Use SVD to implement U b Σ b Vb T and define s = U b W 1/2 b r: Find σ x such that = W b 1/2 A, svs σ 1 σ 2... σ p m 2z α/2 < s T 1 diag( σi 2 σx )s < m + 2z α/2. Equivalently, find σx 2 such that F (σ x ) = s T 1 diag( 1 + σxσ 2 i 2 )s m = 0. Renaut and Mead (ASU/Boise) Scalar Newton method September / 31

49 Single Variable Approach: Seek efficient, practical algorithm Let W x = σ 2 x I, where regularization parameter λ = 1/σ x. Use SVD to implement U b Σ b Vb T and define s = U b W 1/2 b r: Find σ x such that = W b 1/2 A, svs σ 1 σ 2... σ p m 2z α/2 < s T 1 diag( σi 2 σx )s < m + 2z α/2. Equivalently, find σx 2 such that F (σ x ) = s T 1 diag( 1 + σxσ 2 i 2 )s m = 0. Scalar Root Finding: Newton s Method Renaut and Mead (ASU/Boise) Scalar Newton method September / 31

50 Extension to Generalized Tikhonov Define ˆx GTik = argminj D (x) = argmin{ Ax b 2 W b + D(x x 0 ) 2 W x }, (3) Renaut and Mead (ASU/Boise) Scalar Newton method September / 31

51 Extension to Generalized Tikhonov Define ˆx GTik = argminj D (x) = argmin{ Ax b 2 W b + D(x x 0 ) 2 W x }, (3) Theorem For large m, the minimium value of J D is a random variable which follows a χ 2 distribution with m n + p degrees of freedom. (Assuming that no components of r are zero) Renaut and Mead (ASU/Boise) Scalar Newton method September / 31

52 Extension to Generalized Tikhonov Define ˆx GTik = argminj D (x) = argmin{ Ax b 2 W b + D(x x 0 ) 2 W x }, (3) Theorem For large m, the minimium value of J D is a random variable which follows a χ 2 distribution with m n + p degrees of freedom. (Assuming that no components of r are zero) Proof. Use the Generalized Singular Value Decomposition for [W b 1/2 A, W x 1/2 D] Renaut and Mead (ASU/Boise) Scalar Newton method September / 31

53 Extension to Generalized Tikhonov Define ˆx GTik = argminj D (x) = argmin{ Ax b 2 W b + D(x x 0 ) 2 W x }, (3) Theorem For large m, the minimium value of J D is a random variable which follows a χ 2 distribution with m n + p degrees of freedom. (Assuming that no components of r are zero) Proof. Use the Generalized Singular Value Decomposition for [W b 1/2 A, W x 1/2 D] Find W x such that J D is χ 2 with m n + p d.o.f. Renaut and Mead (ASU/Boise) Scalar Newton method September / 31

54 Newton Root Finding W x = σx 2 I p Let GSVD of [W 1/2 b A, D] [ Υ A = U 0 m n n ] X T D = V [M, 0 p n p ]X T, γ i are the generalized singular values m = m n + p p i=1 s2 i δ γ i 0 m i=n+1 s2 i, s i = s i /(γi 2 σx 2 + 1), i = 1,..., p t i = s i γ i. Renaut and Mead (ASU/Boise) Scalar Newton method September / 31

55 Newton Root Finding W x = σx 2 I p Let GSVD of [W 1/2 b A, D] [ Υ A = U 0 m n n ] X T D = V [M, 0 p n p ]X T, γ i are the generalized singular values m = m n + p p i=1 s2 i δ γ i 0 m i=n+1 s2 i, s i = s i /(γi 2 σx 2 + 1), i = 1,..., p t i = s i γ i. Find root of p i=1 ( 1 γi 2 σ 2 +1 )s2 i + m i=n+1 s2 i Solve F = 0, where = m F(σ x ) = s T s m and F (σ x ) = 2σ x t 2 2. Renaut and Mead (ASU/Boise) Scalar Newton method September / 31

56 An Illustrative Example: phillips Fredholm integral equation (Hansen) Add noise to b Standard deviation σ bi =.01 b i +.1b max Covariance matrix C b = σb 2I m = W 1 b Example Error 10% σ 2 b average of σ2 b i is the original b and noisy data. Renaut and Mead (ASU/Boise) Scalar Newton method September / 31

57 An Illustrative Example: phillips Fredholm integral equation (Hansen) Compare Solutions: + is reference x 0. is exact. L-Curve o Three other solutions: UPRE, GCV and χ 2 method (blue, magenta, black) Each method gives different solution - but UPRE, GCV and χ 2 are comparable. Comparison with new method Renaut and Mead (ASU/Boise) Scalar Newton method September / 31

58 Observations: Example F Initialization GCV, UPRE, L-curve, χ 2 use GSVD (or SVD). Algorithm is cheap as compared to GCV, UPRE, L-curve. F is monotonic decreasing, even Renaut and Mead (ASU/Boise) Scalar Newton method September / 31

59 Observations: Example F Initialization GCV, UPRE, L-curve, χ 2 use GSVD (or SVD). Algorithm is cheap as compared to GCV, UPRE, L-curve. F is monotonic decreasing, even Solution either exists and is unique for positive σ Renaut and Mead (ASU/Boise) Scalar Newton method September / 31

60 Observations: Example F Initialization GCV, UPRE, L-curve, χ 2 use GSVD (or SVD). Algorithm is cheap as compared to GCV, UPRE, L-curve. F is monotonic decreasing, even Solution either exists and is unique for positive σ Or no solution exists F(0) < 0. Renaut and Mead (ASU/Boise) Scalar Newton method September / 31

61 Observations: Example F Initialization GCV, UPRE, L-curve, χ 2 use GSVD (or SVD). Algorithm is cheap as compared to GCV, UPRE, L-curve. F is monotonic decreasing, even Solution either exists and is unique for positive σ Or no solution exists F(0) < 0. Theoretically, lim σ F > 0 possible. Equivalent to λ = 0. No regularization needed. Renaut and Mead (ASU/Boise) Scalar Newton method September / 31

62 Observations: Example F Initialization GCV, UPRE, L-curve, χ 2 use GSVD (or SVD). Algorithm is cheap as compared to GCV, UPRE, L-curve. F is monotonic decreasing, even Solution either exists and is unique for positive σ Or no solution exists F(0) < 0. Theoretically, lim σ F > 0 possible. Equivalent to λ = 0. No regularization needed. Renaut and Mead (ASU/Boise) Scalar Newton method September / 31

63 Remark on F (0) < 0 Notice, when F (0) < 0, m is too big relative to J. Equivalently, there are insufficient degrees of freedom. Notice J(ˆx) = P 1/2 s 2 2, P = diag(1/((γ iσ) 2 + 1), 0 n p, I m n ) In particular J(ˆx(0)) = P 1/2 (0)s 2 2 = y, for some y. If y < m, set m = floor(y) Theorem is revised to: m = min{floor(j(0)), m n + p}. In example J(0) 39, F (0) 461. On right m = 38. Renaut and Mead (ASU/Boise) Scalar Newton method September / 31

64 Remark on F (0) < 0 Notice, when F (0) < 0, m is too big relative to J. Equivalently, there are insufficient degrees of freedom. Notice J(ˆx) = P 1/2 s 2 2, P = diag(1/((γ iσ) 2 + 1), 0 n p, I m n ) In particular J(ˆx(0)) = P 1/2 (0)s 2 2 = y, for some y. If y < m, set m = floor(y) Theorem is revised to: m = min{floor(j(0)), m n + p}. In example J(0) 39, F (0) 461. On right m = 38. Renaut and Mead (ASU/Boise) Scalar Newton method September / 31

65 Remark on F (0) < 0 Notice, when F (0) < 0, m is too big relative to J. Equivalently, there are insufficient degrees of freedom. Notice J(ˆx) = P 1/2 s 2 2, P = diag(1/((γ iσ) 2 + 1), 0 n p, I m n ) In particular J(ˆx(0)) = P 1/2 (0)s 2 2 = y, for some y. If y < m, set m = floor(y) Theorem is revised to: m = min{floor(j(0)), m n + p}. In example J(0) 39, F (0) 461. On right m = 38. Renaut and Mead (ASU/Boise) Scalar Newton method September / 31

66 Remark on F (0) < 0 Notice, when F (0) < 0, m is too big relative to J. Equivalently, there are insufficient degrees of freedom. Notice J(ˆx) = P 1/2 s 2 2, P = diag(1/((γ iσ) 2 + 1), 0 n p, I m n ) In particular J(ˆx(0)) = P 1/2 (0)s 2 2 = y, for some y. If y < m, set m = floor(y) Theorem is revised to: m = min{floor(j(0)), m n + p}. In example J(0) 39, F (0) 461. On right m = 38. Renaut and Mead (ASU/Boise) Scalar Newton method September / 31

67 Remark on F (0) < 0 Notice, when F (0) < 0, m is too big relative to J. Equivalently, there are insufficient degrees of freedom. Notice J(ˆx) = P 1/2 s 2 2, P = diag(1/((γ iσ) 2 + 1), 0 n p, I m n ) In particular J(ˆx(0)) = P 1/2 (0)s 2 2 = y, for some y. If y < m, set m = floor(y) Theorem is revised to: m = min{floor(j(0)), m n + p}. In example J(0) 39, F (0) 461. On right m = 38. Renaut and Mead (ASU/Boise) Scalar Newton method September / 31

68 Example: Seismic Signal Restoration Real data set of 48 signals of length 500. The point spread function is derived from the signals Solve Pf = g, where P is psf matrix, g is signal and restore f. Calculate the signal variance pointwise over all 48 signals. Compare restoration of S-wave with derivative orders 0, 1, 2 Weighting matrices are I, σg 2 I, and diag(σg 2 i ), cases 1, 2, and 3. Renaut and Mead (ASU/Boise) Scalar Newton method September / 31

69 Tikhonov Regularization Observations Reduced Degrees of Freedom Relevant! Degrees of Freedom found automatically Case 2 and 1 have different solutions Case 3 leaves the features of the signal Renaut and Mead (ASU/Boise) Scalar Newton method September / 31

70 First and Second Order Derivative Restoration Observations Here derivative smoothing is not desirable Case 3 preserves signal characterist Given value is λ, th regularization parameter. λ increases with derivative order - m smoothing. Renaut and Mead (ASU/Boise) Scalar Newton method September / 31

71 Comparison with L-curve and UPRE Solutions Observations L-curve underestimates λ severely. UPRE and χ 2 are similar when DOF limited on χ 2. UPRE underestima for case 2 and 3 weighting. Renaut and Mead (ASU/Boise) Scalar Newton method September / 31

72 Newton s Method converges in 5 10 Iterations l cb Iterations k mean std e e e e e e e e e e e e e e e e e e + 00 Table: Convergence characteristics for problem phillips with n = 40 over 500 runs Renaut and Mead (ASU/Boise) Scalar Newton method September / 31

73 Newton s Method converges in 5 10 Iterations l cb Iterations k mean std e e e e e e e e e e e e e e e e e e 01 Table: Convergence characteristics for problem blur with n = 36 over 500 runs Renaut and Mead (ASU/Boise) Scalar Newton method September / 31

74 Estimating The Error and Predictive Risk Error l cb χ 2 L GCV UPRE mean mean mean mean e e e e e e e e e e e e e e e e e e e e e e e e 03 Table: Error characteristics for problem phillips with n = 60 over 500 runs with error contaminated x 0. Relative errors larger than.009 removed. Results are comparable Renaut and Mead (ASU/Boise) Scalar Newton method September / 31

75 Estimating The Error and Predictive Risk Risk l cb χ 2 L GCV UPRE mean mean mean mean e e e e e e e e e e e e e e e e e e e e e e e e 02 Table: Error characteristics for problem phillips with n = 60 over 500 runs χ 2 method does not give best estimate of risk Renaut and Mead (ASU/Boise) Scalar Newton method September / 31

76 Estimating The Error and Predictive Risk Error Histogram Normal noise on rhs, first order derivative, C b = σ 2 I Renaut and Mead (ASU/Boise) Scalar Newton method September / 31

77 Estimating The Error and Predictive Risk Error Histogram Exponential noise on rhs, first order derivative, C b = σ 2 I Renaut and Mead (ASU/Boise) Scalar Newton method September / 31

78 Conclusions χ 2 Newton algorithm is cost effective It performs as well ( or better) than GCV and UPRE when statistical information is available. Should be method of choice when statistical information is provided Method can be adapted to find W b if W x is provided. Renaut and Mead (ASU/Boise) Scalar Newton method September / 31

79 Future Work Analyse for truncated expansions (TSVD and TGSVD) -reduce the degrees of freedom. Further theoretical analysis and simulations with other noise distributions. Comparison new work of Rust & O Leary Can it be extended for nonlinear regularization terms? (TV?) Development of the nonlinear least squares for general diagonal W x. Efficient calculation of uncertainty information, covariance matrix. Nonlinear problems? Renaut and Mead (ASU/Boise) Scalar Newton method September / 31

80 Some Solutions: with no prior information x 0 Illustrated are solutions and error bars No Statistical Information Solution is Smoothed With statistical information C b = diag(σ 2 b i ) Renaut and Mead (ASU/Boise) Scalar Newton method September / 31

81 Some Generalized Tikhonov Solutions: First Order Derivative No Statistical Information C b = diag(σ 2 b i ) Renaut and Mead (ASU/Boise) Scalar Newton method September / 31

82 Some Generalized Tikhonov Solutions: Prior x 0 : Solution not smoothed No Statistical Information C b = diag(σ 2 b i ) Renaut and Mead (ASU/Boise) Scalar Newton method September / 31

83 Some Generalized Tikhonov Solutions: x 0 = 0: Exponential noise No Statistical Information C b = diag(σ 2 b i ) Renaut and Mead (ASU/Boise) Scalar Newton method September / 31

84 Relationship to Discrepancy Principle The discrepancy principle can be implemented by a Newton method. Finds σ x such that the regularized residual satisfies Consistent with our notation σ 2 b = 1 m b Ax(σ) 2 2. (4) p 1 ( γi 2 σ )2 si 2 + i=1 m i=n+1 s 2 i = m, (5) Weight in the first sum is squared here, otherwise functional is the same. But discrepancy principle often oversmooths. What happens here? Renaut and Mead (ASU/Boise) Scalar Newton method September / 31

85 Major References Bennett A, 2005 Inverse Modeling of the Ocean and Atmosphere (Cambridge University Press) Hansen, P. C., 1994, Regularization Tools: A Matlab Package for Analysis and Solution of Discrete Ill-posed Problems, Numerical Algorithms Mead J., 2007, A priori weighting for parameter estimation, J. Inv. Ill-posed Problems, to appear. Rao, C. R., 1973, Linear Statistical Inference and its applications, Wiley, New York. Tarantola A 2005 Inverse Problem Theory and Methods for Model Parameter Estimation (SIAM). Vogel, C. R., Computational Methods for Inverse Problems, (SIAM), Frontiers in Applied Mathematics. Renaut and Mead (ASU/Boise) Scalar Newton method September / 31

86 blur Atmospheric (Gaussian PSF) (Hansen): Again with noise Solution on Left and Degraded on the Right Renaut and Mead (ASU/Boise) Scalar Newton method September / 31

87 blur Atmospheric (Gaussian PSF) (Hansen): Again with noise Solutions using x 0 = 0, Generalized Tikhonov Second Derivative 5% noise Renaut and Mead (ASU/Boise) Scalar Newton method September / 31

88 blur Atmospheric (Gaussian PSF) (Hansen): Again with noise olutions using x 0 = 0, Generalized Tikhonov Second Derivative 10% noise Renaut and Mead (ASU/Boise) Scalar Newton method September / 31