Solving Regularized Total Least Squares Problems

Transcription

1 Solving Regularized Total Least Squares Problems Heinrich Voss Hamburg University of Technology Institute of Numerical Simulation Joint work with Jörg Lampe TUHH Heinrich Voss Total Least Squares Harrachov, August / 42

2 Regularized total least squres problems Total Least Squares Problem The ordinary Least Squares (LS) method assumes that the system matrix A of a linear model is error free, and all errors are confined to the right hand side b. TUHH Heinrich Voss Total Least Squares Harrachov, August / 42

3 Regularized total least squres problems Total Least Squares Problem The ordinary Least Squares (LS) method assumes that the system matrix A of a linear model is error free, and all errors are confined to the right hand side b. Given A R m n, b R m, m n Find x R n and b R m such that b = min! subject to Ax = b + b. TUHH Heinrich Voss Total Least Squares Harrachov, August / 42

4 Regularized total least squres problems Total Least Squares Problem The ordinary Least Squares (LS) method assumes that the system matrix A of a linear model is error free, and all errors are confined to the right hand side b. Given A R m n, b R m, m n Find x R n and b R m such that b = min! subject to Ax = b + b. Obviously equivalent to: Find x R n such that Ax b = min! TUHH Heinrich Voss Total Least Squares Harrachov, August / 42

5 Regularized total least squres problems Total Least Squares In practical applications it may happen that all data are contaminated by noise, for instance because the matrix A is also obtained from measurements. TUHH Heinrich Voss Total Least Squares Harrachov, August / 42

6 Regularized total least squres problems Total Least Squares In practical applications it may happen that all data are contaminated by noise, for instance because the matrix A is also obtained from measurements. If the true values of the observed variables satisfy linear relations, and if the errors in the observations are independent random variables with zero mean and equal variance, then the total least squares (TLS) approach often gives better estimates than LS. TUHH Heinrich Voss Total Least Squares Harrachov, August / 42

7 Regularized total least squres problems Total Least Squares In practical applications it may happen that all data are contaminated by noise, for instance because the matrix A is also obtained from measurements. If the true values of the observed variables satisfy linear relations, and if the errors in the observations are independent random variables with zero mean and equal variance, then the total least squares (TLS) approach often gives better estimates than LS. Given A R m n, b R m, m n Find A R m n, b R m and x R n such that [ A, b] 2 F = min! subject to (A + A)x = b + b, (1) where F denotes the Frobenius norm. TUHH Heinrich Voss Total Least Squares Harrachov, August / 42

8 Regularized total least squres problems Total Least Squares In practical applications it may happen that all data are contaminated by noise, for instance because the matrix A is also obtained from measurements. If the true values of the observed variables satisfy linear relations, and if the errors in the observations are independent random variables with zero mean and equal variance, then the total least squares (TLS) approach often gives better estimates than LS. Given A R m n, b R m, m n Find A R m n, b R m and x R n such that [ A, b] 2 F = min! subject to (A + A)x = b + b, (1) where F denotes the Frobenius norm. In statistics this approach is called errors-in-variables problem or orthogonal regression, in image deblurring blind deconvolution TUHH Heinrich Voss Total Least Squares Harrachov, August / 42

9 Regularized total least squres problems Regularized Total Least Squares Problem If A and [A, b] are ill-conditioned, regularization is necessary. TUHH Heinrich Voss Total Least Squares Harrachov, August / 42

10 Regularized total least squres problems Regularized Total Least Squares Problem If A and [A, b] are ill-conditioned, regularization is necessary. Let L R k n, k n and δ > 0. Then the quadratically constrained formulation of the Regularized Total Least Squares (RTLS) problem reads: Find A R m n, b R m and x R n such that [ A, b] 2 F = min! subject to (A + A)x = b + b, Lx 2 δ 2. TUHH Heinrich Voss Total Least Squares Harrachov, August / 42

11 Regularized total least squres problems Regularized Total Least Squares Problem If A and [A, b] are ill-conditioned, regularization is necessary. Let L R k n, k n and δ > 0. Then the quadratically constrained formulation of the Regularized Total Least Squares (RTLS) problem reads: Find A R m n, b R m and x R n such that [ A, b] 2 F = min! subject to (A + A)x = b + b, Lx 2 = δ 2. TUHH Heinrich Voss Total Least Squares Harrachov, August / 42

12 Regularized total least squres problems Regularized Total Least Squares Problem If A and [A, b] are ill-conditioned, regularization is necessary. Let L R k n, k n and δ > 0. Then the quadratically constrained formulation of the Regularized Total Least Squares (RTLS) problem reads: Find A R m n, b R m and x R n such that [ A, b] 2 F = min! subject to (A + A)x = b + b, Lx 2 = δ 2. Using the orthogonal distance this problems can be rewritten as (cf. Golub, Van Loan 1980) Find x R n such that f (x) := Ax b x 2 = min! subject to Lx 2 = δ 2. TUHH Heinrich Voss Total Least Squares Harrachov, August / 42

13 Regularized total least squres problems Regularized Total Least Squares Problem ct. Theorem 1: Let N (L) be the null space of L. If then f = inf{f (x) : Lx 2 = δ 2 } < f (x) := admits a global minimum. Ax 2 min x N (L), x 0 x 2 (1) Ax b x 2 = min! subject to Lx 2 = δ 2. (2) TUHH Heinrich Voss Total Least Squares Harrachov, August / 42

14 Regularized total least squres problems Regularized Total Least Squares Problem ct. Theorem 1: Let N (L) be the null space of L. If then f = inf{f (x) : Lx 2 = δ 2 } < f (x) := admits a global minimum. Ax 2 min x N (L), x 0 x 2 (1) Ax b x 2 = min! subject to Lx 2 = δ 2. (2) Conversely, if problem (2) admits a global minimum, then f Ax 2 min x N (L), x 0 x 2 TUHH Heinrich Voss Total Least Squares Harrachov, August / 42

15 Regularized total least squres problems Regularized Total Least Squares Problem ct. Under the condition (1) problem (2) is equivalent to the quadratic optimization problem Ax b 2 f (1 + x 2 ) = min! subject to Lx 2 = δ 2, (3) i.e. x is a global minimizer of problem (2) if and only if it is a global minimizer of (3). TUHH Heinrich Voss Total Least Squares Harrachov, August / 42

16 Regularized total least squres problems Regularized Total Least Squares Problem ct. Under the condition (1) problem (2) is equivalent to the quadratic optimization problem Ax b 2 f (1 + x 2 ) = min! subject to Lx 2 = δ 2, (3) i.e. x is a global minimizer of problem (2) if and only if it is a global minimizer of (3). For fixed y R n find x R n such that g(x; y) := Ax b 2 f (y)(1 + x 2 ) = min! subject to Lx 2 = δ 2. (P y ) TUHH Heinrich Voss Total Least Squares Harrachov, August / 42

17 Regularized total least squres problems Regularized Total Least Squares Problem ct. Under the condition (1) problem (2) is equivalent to the quadratic optimization problem Ax b 2 f (1 + x 2 ) = min! subject to Lx 2 = δ 2, (3) i.e. x is a global minimizer of problem (2) if and only if it is a global minimizer of (3). For fixed y R n find x R n such that g(x; y) := Ax b 2 f (y)(1 + x 2 ) = min! subject to Lx 2 = δ 2. (P y ) Lemma 1 (Sima, Van Huffel, Golub 2004) Problem (P y ) admits a global minimizer if and only if f (y) min x N (L),x 0 x T A T Ax x T. x TUHH Heinrich Voss Total Least Squares Harrachov, August / 42

18 Regularized total least squres problems RTLSQEP Method (Sima, Van Huffel, Golub 2004) Lemma 2 Assume that y satisfies conditions of Lemma 1 and Ly = δ, and let z be a global minimizer of problem (P y ). Then it holds that f (z) f (y). TUHH Heinrich Voss Total Least Squares Harrachov, August / 42

19 Regularized total least squres problems RTLSQEP Method (Sima, Van Huffel, Golub 2004) Lemma 2 Assume that y satisfies conditions of Lemma 1 and Ly = δ, and let z be a global minimizer of problem (P y ). Then it holds that f (z) f (y). Proof (1 + z ) 2 (f (z) f (y)) = g(z; y) g(y; y) = (1 + y 2 )(f (y) f (y)) = 0. TUHH Heinrich Voss Total Least Squares Harrachov, August / 42

20 Regularized total least squres problems RTLSQEP Method (Sima, Van Huffel, Golub 2004) Lemma 2 Assume that y satisfies conditions of Lemma 1 and Ly = δ, and let z be a global minimizer of problem (P y ). Then it holds that f (z) f (y). Proof (1 + z ) 2 (f (z) f (y)) = g(z; y) g(y; y) = (1 + y 2 )(f (y) f (y)) = 0. Require: x 0 satisfying conditions of Lemma 1 and Lx 0 = δ. for m = 0, 1, 2,... until convergence do Determine global minimizer x m+1 of end for g(x; x m ) = min! subject to Lx 2 = δ 2. TUHH Heinrich Voss Total Least Squares Harrachov, August / 42

21 Regularized total least squres problems RTLS Method Obviously, 0 f (x m+1 ) f (x m ) TUHH Heinrich Voss Total Least Squares Harrachov, August / 42

22 Regularized total least squres problems RTLS Method Obviously, 0 f (x m+1 ) f (x m ) The (P x m) can be solved via the first order necessary optimality conditions (A T A f (x m )I)x + λl T Lx = A T b, Lx 2 = δ 2. TUHH Heinrich Voss Total Least Squares Harrachov, August / 42

23 Regularized total least squres problems RTLS Method Obviously, 0 f (x m+1 ) f (x m ) The (P x m) can be solved via the first order necessary optimality conditions (A T A f (x m )I)x + λl T Lx = A T b, Lx 2 = δ 2. Although g( ; x m ) in general is not convex these conditions are even sufficient if the Lagrange parameter is chosen maximal. TUHH Heinrich Voss Total Least Squares Harrachov, August / 42

24 Regularized total least squres problems RTLS Method Theorem 2 Assume that (ˆλ, ˆx) solves the first order conditions. (A T A f (y)i)x + λl T Lx = A T b, Lx 2 = δ 2. ( ) If Ly = δ and ˆλ is the maximal Lagrange multiplier then ˆx is a global minimizer of problem (P y ). TUHH Heinrich Voss Total Least Squares Harrachov, August / 42

25 Regularized total least squres problems RTLS Method Theorem 2 Assume that (ˆλ, ˆx) solves the first order conditions. (A T A f (y)i)x + λl T Lx = A T b, Lx 2 = δ 2. ( ) If Ly = δ and ˆλ is the maximal Lagrange multiplier then ˆx is a global minimizer of problem (P y ). Proof The statement follows immediately from the following equation which can be shown similarly as in W. Gander (1981): If (λ j, z j ), j = 1, 2, are solutions of ( ) then it holds that g(z 2 ; y) g(z 1 ; y) = 1 2 (λ 1 λ 2 ) L(z 1 z 2 ) 2. TUHH Heinrich Voss Total Least Squares Harrachov, August / 42

26 A quadratic eigenproblem A quadratic eigenproblem (A T A f (y)i)x + λl T Lx = A T b, Lx 2 = δ 2 ( ) TUHH Heinrich Voss Total Least Squares Harrachov, August / 42

27 A quadratic eigenproblem A quadratic eigenproblem (A T A f (y)i)x + λl T Lx = A T b, Lx 2 = δ 2 ( ) If L is square and nonsingular: Let z := Lx. Then Wz + λz := L T (A T A f (y)i)l 1 z + λz = L T A T b =: h, z T z = δ 2. TUHH Heinrich Voss Total Least Squares Harrachov, August / 42

28 A quadratic eigenproblem A quadratic eigenproblem (A T A f (y)i)x + λl T Lx = A T b, Lx 2 = δ 2 ( ) If L is square and nonsingular: Let z := Lx. Then Wz + λz := L T (A T A f (y)i)l 1 z + λz = L T A T b =: h, z T z = δ 2. u := (W + λi) 2 h h T u = z T z = δ 2 h = δ 2 hh T u TUHH Heinrich Voss Total Least Squares Harrachov, August / 42

29 A quadratic eigenproblem A quadratic eigenproblem (A T A f (y)i)x + λl T Lx = A T b, Lx 2 = δ 2 ( ) If L is square and nonsingular: Let z := Lx. Then Wz + λz := L T (A T A f (y)i)l 1 z + λz = L T A T b =: h, z T z = δ 2. u := (W + λi) 2 h h T u = z T z = δ 2 h = δ 2 hh T u (W + λi) 2 u δ 2 hh T u = 0. TUHH Heinrich Voss Total Least Squares Harrachov, August / 42

30 A quadratic eigenproblem A quadratic eigenproblem (A T A f (y)i)x + λl T Lx = A T b, Lx 2 = δ 2 ( ) If L is square and nonsingular: Let z := Lx. Then Wz + λz := L T (A T A f (y)i)l 1 z + λz = L T A T b =: h, z T z = δ 2. u := (W + λi) 2 h h T u = z T z = δ 2 h = δ 2 hh T u (W + λi) 2 u δ 2 hh T u = 0. If ˆλ is the right-most real eigenvalue, and the corresponding eigenvector is scaled such that h T u = δ 2 then the solution of problem ( ) is recovered as x = L 1 (W + ˆλI)u. TUHH Heinrich Voss Total Least Squares Harrachov, August / 42

31 A quadratic eigenproblem A quadratic eigenproblem ct. If L R k n with linearly independent rows and k < n, the first order conditions can be reduced to a quadratic eigenproblem (W + λi) 2 u δ 2 hh T u = 0. where W m = (C f (x m )D S(T f (x m )I n k ) 1 S T ) h m = g D(T f (x m )I n k ) 1 c with C, D R k k, S R k n k, T R n k n k, g R k, c R n k, and C, D, T are symmetric matrices. TUHH Heinrich Voss Total Least Squares Harrachov, August / 42

32 Nonlinear maxmin characterization Nonlinear maxmin characterization Let T (λ) C n n, T (λ) = T (λ) H, λ J R an open interval (maybe unbounded). TUHH Heinrich Voss Total Least Squares Harrachov, August / 42

33 Nonlinear maxmin characterization Nonlinear maxmin characterization Let T (λ) C n n, T (λ) = T (λ) H, λ J R an open interval (maybe unbounded). For every fixed x C n, x 0 assume that the real function f ( ; x) : J R, f (λ; x) := x H T (λ)x is continuous, and that the real equation f (λ, x) = 0 has at most one solution λ =: p(x) in J. TUHH Heinrich Voss Total Least Squares Harrachov, August / 42

34 Nonlinear maxmin characterization Nonlinear maxmin characterization Let T (λ) C n n, T (λ) = T (λ) H, λ J R an open interval (maybe unbounded). For every fixed x C n, x 0 assume that the real function f ( ; x) : J R, f (λ; x) := x H T (λ)x is continuous, and that the real equation f (λ, x) = 0 has at most one solution λ =: p(x) in J. Then equation f (λ, x) = 0 implicitly defines a functional p on some subset D of C n which we call the Rayleigh functional. TUHH Heinrich Voss Total Least Squares Harrachov, August / 42

35 Nonlinear maxmin characterization Nonlinear maxmin characterization Let T (λ) C n n, T (λ) = T (λ) H, λ J R an open interval (maybe unbounded). For every fixed x C n, x 0 assume that the real function f ( ; x) : J R, f (λ; x) := x H T (λ)x is continuous, and that the real equation f (λ, x) = 0 has at most one solution λ =: p(x) in J. Then equation f (λ, x) = 0 implicitly defines a functional p on some subset D of C n which we call the Rayleigh functional. Assume that (λ p(x))f (λ, x) > 0 for every x D, λ p(x). TUHH Heinrich Voss Total Least Squares Harrachov, August / 42

36 Nonlinear maxmin characterization maxmin characterization (V., Werner 1982) Let sup v D p(v) J and assume that there exists a subspace V C n of dimension l such that V D and inf p(v) J. v V D TUHH Heinrich Voss Total Least Squares Harrachov, August / 42

37 Nonlinear maxmin characterization maxmin characterization (V., Werner 1982) Let sup v D p(v) J and assume that there exists a subspace V C n of dimension l such that V D and inf p(v) J. v V D Then T (λ)x = 0 has at least l eigenvalues in J, and for j = 1,..., l the j-largest eigenvalue λ j can be characterized by λ j = max dim V =j, V D inf v V D p(v). (1) TUHH Heinrich Voss Total Least Squares Harrachov, August / 42

38 Nonlinear maxmin characterization maxmin characterization (V., Werner 1982) Let sup v D p(v) J and assume that there exists a subspace V C n of dimension l such that V D and inf p(v) J. v V D Then T (λ)x = 0 has at least l eigenvalues in J, and for j = 1,..., l the j-largest eigenvalue λ j can be characterized by λ j = max dim V =j, V D inf v V D p(v). (1) For j = 1,..., l every j dimensional subspace Ṽ Cn with Ṽ D and λ j = inf v Ṽ D p(v) is contained in D {0}, and the maxmin characterization of λ j can be replaced by λ j = min p(v). v V \{0} max dim V =j, V \{0} D TUHH Heinrich Voss Total Least Squares Harrachov, August / 42

39 Back to Nonlinear maxmin characterization T (λ)x := ( (W + λi) 2 δ 2 hh T ) x = 0 (QEP) TUHH Heinrich Voss Total Least Squares Harrachov, August / 42

40 Back to Nonlinear maxmin characterization T (λ)x := ( (W + λi) 2 δ 2 hh T ) x = 0 (QEP) f (λ, x) = x H T (λ)x = λ 2 x 2 + 2λx H Wx + Wx 2 x H h 2 /δ 2, x 0 is a parabola which attains its minimum at λ = x H Wx x H x. TUHH Heinrich Voss Total Least Squares Harrachov, August / 42

41 Back to Nonlinear maxmin characterization T (λ)x := ( (W + λi) 2 δ 2 hh T ) x = 0 (QEP) f (λ, x) = x H T (λ)x = λ 2 x 2 + 2λx H Wx + Wx 2 x H h 2 /δ 2, x 0 is a parabola which attains its minimum at λ = x H Wx x H x. Let J = ( λ min, ) where λ min is the minimum eigenvalue of W. Then f (λ, x) = 0 has at most one solution p(x) J for every x 0. Hence, the Rayleigh functional p of (QEP) corresponding to J is defined, and the general conditions are satisfied. TUHH Heinrich Voss Total Least Squares Harrachov, August / 42

42 Nonlinear maxmin characterization Characterization of maximal real eigenvalue Let V min be the eigenspace of W corresponding to λ min. Then for every x min V min f ( λ min, x min ) = x H min(w λ min ) 2 x min x H minh 2 /δ 2 = x H minh 2 /δ 2 0 TUHH Heinrich Voss Total Least Squares Harrachov, August / 42

43 Nonlinear maxmin characterization Characterization of maximal real eigenvalue Let V min be the eigenspace of W corresponding to λ min. Then for every x min V min f ( λ min, x min ) = x H min(w λ min ) 2 x min x H minh 2 /δ 2 = x H minh 2 /δ 2 0 Hence, if x H min h 0 for some x min V min, then x min D. TUHH Heinrich Voss Total Least Squares Harrachov, August / 42

44 Nonlinear maxmin characterization Characterization of maximal real eigenvalue Let V min be the eigenspace of W corresponding to λ min. Then for every x min V min f ( λ min, x min ) = x H min(w λ min ) 2 x min x H minh 2 /δ 2 = x H minh 2 /δ 2 0 Hence, if x H min h 0 for some x min V min, then x min D. If h V min, and if the minimum eigenvalue µ min of T ( λ min ) is negative, then for the corresponding eigenvector y min it holds and y min D. f ( λ min, y min ) = y H mint ( λ min )y min = µ min y min 2 < 0, TUHH Heinrich Voss Total Least Squares Harrachov, August / 42

45 Nonlinear maxmin characterization Characterization of maximal real eigenvalue Let V min be the eigenspace of W corresponding to λ min. Then for every x min V min f ( λ min, x min ) = x H min(w λ min ) 2 x min x H minh 2 /δ 2 = x H minh 2 /δ 2 0 Hence, if x H min h 0 for some x min V min, then x min D. If h V min, and if the minimum eigenvalue µ min of T ( λ min ) is negative, then for the corresponding eigenvector y min it holds and y min D. f ( λ min, y min ) = y H mint ( λ min )y min = µ min y min 2 < 0, If h V min, and T ( λ min ) is positive semi-definite, then f ( λ min, x) = x H T ( λ min )x 0 for every x 0, and D =. TUHH Heinrich Voss Total Least Squares Harrachov, August / 42

46 Nonlinear maxmin characterization Characterization of maximal real eigenvalue ct. Assume that D. For x H h = 0 it holds that i.e. x D. f (λ, x) = (W + λi)x 2 > 0 for every λ J, TUHH Heinrich Voss Total Least Squares Harrachov, August / 42

47 Nonlinear maxmin characterization Characterization of maximal real eigenvalue ct. Assume that D. For x H h = 0 it holds that i.e. x D. f (λ, x) = (W + λi)x 2 > 0 for every λ J, Hence, D does not contain a two-dimensional subspace of R n, and therefore J contains at most one eigenvalue of (QEP). TUHH Heinrich Voss Total Least Squares Harrachov, August / 42

48 Nonlinear maxmin characterization Characterization of maximal real eigenvalue ct. Assume that D. For x H h = 0 it holds that i.e. x D. f (λ, x) = (W + λi)x 2 > 0 for every λ J, Hence, D does not contain a two-dimensional subspace of R n, and therefore J contains at most one eigenvalue of (QEP). If λ C is a non-real eigenvalue of (QEP) and x a corresponding eigenvector, then x H T (λ)x = λ 2 x 2 + 2λx H Wx + Wx 2 x H h 2 /δ 2 = 0. Hence, the real part of λ satisfies real(λ) = x H Wx x H x λ min. TUHH Heinrich Voss Total Least Squares Harrachov, August / 42

49 Theorem 3 Nonlinear maxmin characterization Let λ min be the minimal eigenvalue of W, and V min be the corresponding eigenspace. TUHH Heinrich Voss Total Least Squares Harrachov, August / 42

50 Theorem 3 Nonlinear maxmin characterization Let λ min be the minimal eigenvalue of W, and V min be the corresponding eigenspace. If h V min and T ( λ min ) is positive semi-definite, then ˆλ := λ min is the maximal real eigenvalue of (QEP). TUHH Heinrich Voss Total Least Squares Harrachov, August / 42

51 Theorem 3 Nonlinear maxmin characterization Let λ min be the minimal eigenvalue of W, and V min be the corresponding eigenspace. If h V min and T ( λ min ) is positive semi-definite, then ˆλ := λ min is the maximal real eigenvalue of (QEP). Otherwise, the maximal real eigenvalue is the unique eigenvalue ˆλ of (QEP) in J = ( λ min, ), and it holds ˆλ = max x D p(x). TUHH Heinrich Voss Total Least Squares Harrachov, August / 42

52 Theorem 3 Nonlinear maxmin characterization Let λ min be the minimal eigenvalue of W, and V min be the corresponding eigenspace. If h V min and T ( λ min ) is positive semi-definite, then ˆλ := λ min is the maximal real eigenvalue of (QEP). Otherwise, the maximal real eigenvalue is the unique eigenvalue ˆλ of (QEP) in J = ( λ min, ), and it holds ˆλ = max x D p(x). ˆλ is the right most eigenvalue of (QEP), i.e. real(λ) λ min ˆλ for every eigenvalue λ ˆλ of (QEP). TUHH Heinrich Voss Total Least Squares Harrachov, August / 42

53 Example Nonlinear maxmin characterization TUHH Heinrich Voss Total Least Squares Harrachov, August / 42

54 Nonlinear maxmin characterization Example: close up TUHH Heinrich Voss Total Least Squares Harrachov, August / 42

55 Nonlinear maxmin characterization Positivity of ˆλ Sima et al. claimed that the right-most eigenvalue problem is always positive. TUHH Heinrich Voss Total Least Squares Harrachov, August / 42

56 Nonlinear maxmin characterization Positivity of ˆλ Sima et al. claimed that the right-most eigenvalue problem is always positive. Simplest counter example: If W is positive definite with eigenvalue λ j > 0, then λ j are the only eigenvalues of the quadratic eigenproblem (W + λi) 2 x = 0, and if the term δ 2 hh T is small enough, then the quadratic problem will have no positive eigenvalue, but the right most eigenvalue will be negative. TUHH Heinrich Voss Total Least Squares Harrachov, August / 42

57 Nonlinear maxmin characterization Positivity of ˆλ Sima et al. claimed that the right-most eigenvalue problem is always positive. Simplest counter example: If W is positive definite with eigenvalue λ j > 0, then λ j are the only eigenvalues of the quadratic eigenproblem (W + λi) 2 x = 0, and if the term δ 2 hh T is small enough, then the quadratic problem will have no positive eigenvalue, but the right most eigenvalue will be negative. However, in quadratic eigenproblems occurring in regularized total least squares problems δ and h are not arbitrary, but regularization only makes sense if δ Lx TLS where x TLS denotes the solution of the total least squares problem without regularization. The following theorem characterizes the case that the right most eigenvalue is negative. TUHH Heinrich Voss Total Least Squares Harrachov, August / 42

58 Nonlinear maxmin characterization Positivity of ˆλ ct. Theorem 4 The maximal real eigenvalue ˆλ of the quadratic problem (W + λi) 2 x δ 2 hh T x = 0 is negative if and only if W is positive definite and W 1 h < δ. TUHH Heinrich Voss Total Least Squares Harrachov, August / 42

59 Nonlinear maxmin characterization Positivity of ˆλ ct. Theorem 4 The maximal real eigenvalue ˆλ of the quadratic problem (W + λi) 2 x δ 2 hh T x = 0 is negative if and only if W is positive definite and W 1 h < δ. For the quadratic eigenproblem occuring in regularized total least squares it holds that W 1 h = L(A T A f (x)i) 1 A T b. TUHH Heinrich Voss Total Least Squares Harrachov, August / 42

60 Nonlinear maxmin characterization Positivity of ˆλ ct. Theorem 4 The maximal real eigenvalue ˆλ of the quadratic problem (W + λi) 2 x δ 2 hh T x = 0 is negative if and only if W is positive definite and W 1 h < δ. For the quadratic eigenproblem occuring in regularized total least squares it holds that W 1 h = L(A T A f (x)i) 1 A T b. For the standard case L = I the right-most eigenvalue ˆλ is always nonnegative if δ < x TLS. TUHH Heinrich Voss Total Least Squares Harrachov, August / 42

61 Nonlinear maxmin characterization Convergence Theorem 5 Any limit point x of the sequence {x m } constructed by RTLSQEP is a global minimizer of Ax b 2 f (x) = 1 + x 2 subject to Lx 2 = δ 2. TUHH Heinrich Voss Total Least Squares Harrachov, August / 42

62 Nonlinear maxmin characterization Convergence Theorem 5 Any limit point x of the sequence {x m } constructed by RTLSQEP is a global minimizer of Ax b 2 f (x) = 1 + x 2 subject to Lx 2 = δ 2. Proof: Let x be a limit point of {x m }, and let {x m j } be a subsequence converging to x. Then x m j solves the first order conditions (A T A f (x m j 1 )I)x m j + λ mj L T Lx m j = A T b. TUHH Heinrich Voss Total Least Squares Harrachov, August / 42

63 Nonlinear maxmin characterization Convergence Theorem 5 Any limit point x of the sequence {x m } constructed by RTLSQEP is a global minimizer of Ax b 2 f (x) = 1 + x 2 subject to Lx 2 = δ 2. Proof: Let x be a limit point of {x m }, and let {x m j } be a subsequence converging to x. Then x m j solves the first order conditions (A T A f (x m j 1 )I)x m j + λ mj L T Lx m j = A T b. From the monotonicity of f (x m ) it follows that lim j f (x m j 1 ) = f (x ) TUHH Heinrich Voss Total Least Squares Harrachov, August / 42

64 Proof ct. Nonlinear maxmin characterization Since W (y) and h(y) depend continuously on y the sequence of right-most eigenvalues {λ mj } converges to some λ, and x satisfies (A T A f (x )I)x λ L T Lx = A T b, Lx 2 = δ 2, where λ is the maximal Lagrange multiplier. TUHH Heinrich Voss Total Least Squares Harrachov, August / 42

65 Proof ct. Nonlinear maxmin characterization Since W (y) and h(y) depend continuously on y the sequence of right-most eigenvalues {λ mj } converges to some λ, and x satisfies (A T A f (x )I)x λ L T Lx = A T b, Lx 2 = δ 2, where λ is the maximal Lagrange multiplier. Hence x is a global minimizer of g(x; x ) = min! subject to Lx 2 = δ 2, TUHH Heinrich Voss Total Least Squares Harrachov, August / 42

66 Proof ct. Nonlinear maxmin characterization Since W (y) and h(y) depend continuously on y the sequence of right-most eigenvalues {λ mj } converges to some λ, and x satisfies (A T A f (x )I)x λ L T Lx = A T b, Lx 2 = δ 2, where λ is the maximal Lagrange multiplier. Hence x is a global minimizer of g(x; x ) = min! subject to Lx 2 = δ 2, and for y R n with Ly 2 = δ 2 it follows that 0 = g(x ; x ) g(y; x ) = Ay b 2 f (x )(1 + y 2 ) = (f (y) f (x ))(1 + y 2 ), i.e. f (y) f (x ). TUHH Heinrich Voss Total Least Squares Harrachov, August / 42

67 Numerical considerations Quadratic eigenproblem The quadratic eigenproblems T m (λ)z = (W m + λi) 2 z 1 δ 2 h mhmz T = 0 can be solved by linearization Krylov subspace method for QEP (Li & Ye 2003) SOAR (Bai & Su 2005) nonlinear Arnoldi method (Meerbergen 2001, V. 2004) TUHH Heinrich Voss Total Least Squares Harrachov, August / 42

68 Numerical considerations Krylov subspace method: Li & Ye 2003 For A, B R n n such that some linear combination of A and B is a matrix of rank q with an Arnoldi-type process a matrix Q R n m+q+1 with orthonormal columns and two matrices H a R m+q+1 m and H b R m+q+1 m with lower bandwidth q + 1 are determined such that AQ(:, 1 : m) = Q(:, 1 : m + q + 1)H a and BQ(:, 1 : m) = Q(:, 1 : m + q + 1)H b. TUHH Heinrich Voss Total Least Squares Harrachov, August / 42

69 Numerical considerations Krylov subspace method: Li & Ye 2003 For A, B R n n such that some linear combination of A and B is a matrix of rank q with an Arnoldi-type process a matrix Q R n m+q+1 with orthonormal columns and two matrices H a R m+q+1 m and H b R m+q+1 m with lower bandwidth q + 1 are determined such that AQ(:, 1 : m) = Q(:, 1 : m + q + 1)H a and BQ(:, 1 : m) = Q(:, 1 : m + q + 1)H b. Then approximations to eigenpairs of the quadratic eigenproblem (λ 2 I λa B)y = 0 are obtained from its projection onto spanq(:, 1 : m) which reads (λ 2 I m λh a (1 : m, 1 : m) H b (1 : m, 1 : m))z = 0. TUHH Heinrich Voss Total Least Squares Harrachov, August / 42

70 Numerical considerations Krylov subspace method: Li & Ye 2003 For A, B R n n such that some linear combination of A and B is a matrix of rank q with an Arnoldi-type process a matrix Q R n m+q+1 with orthonormal columns and two matrices H a R m+q+1 m and H b R m+q+1 m with lower bandwidth q + 1 are determined such that AQ(:, 1 : m) = Q(:, 1 : m + q + 1)H a and BQ(:, 1 : m) = Q(:, 1 : m + q + 1)H b. Then approximations to eigenpairs of the quadratic eigenproblem (λ 2 I λa B)y = 0 are obtained from its projection onto spanq(:, 1 : m) which reads (λ 2 I m λh a (1 : m, 1 : m) H b (1 : m, 1 : m))z = 0. For the quadratic eigenproblem ((W + λi) 2 δ 2 hh T )x = 0 the algorithm of Li & Ye is applied with A = W and B = hh T from which the projected problem (θ 2 I 2θH a(1 : m, 1 : m) H a(1 : m+2, 1 : m) T H a(1 : m+2, m) δ 2 H b (1 : m, 1 : m))z = 0 is easily obtained. TUHH Heinrich Voss Total Least Squares Harrachov, August / 42

71 Numerical considerations SOAR: Bai & Su 2005 The Second Order Arnoldi Reduction method is based on the observation that the Krylov space of the linearization ( ( ) ( ) ( ) A B y I O y = λ I O) z O I z ( ) of (λ 2 r0 I λa B)y = 0 with initial vector has the form 0 where K k = {( ( ) r0 r1,, 0) r 0 ( r2 r 1 ),..., ( rk 1 r 1 = Ar 0 r j = Ar j 1 + Br j 2, for j 2. r k 2 )} TUHH Heinrich Voss Total Least Squares Harrachov, August / 42

72 Numerical considerations SOAR: Bai & Su 2005 The Second Order Arnoldi Reduction method is based on the observation that the Krylov space of the linearization ( ( ) ( ) ( ) A B y I O y = λ I O) z O I z ( ) of (λ 2 r0 I λa B)y = 0 with initial vector has the form 0 where K k = {( ( ) r0 r1,, 0) r 0 ( r2 r 1 ),..., ( rk 1 r 1 = Ar 0 r j = Ar j 1 + Br j 2, for j 2. r k 2 )} The entire information on K k is therefore contained in the Second Order Krylov Space G k (A, B) = span{r 0, r 1,..., r k 1 }. SOAR determines an orthonormal basis of G k (A, B). TUHH Heinrich Voss Total Least Squares Harrachov, August / 42

73 Numerical considerations Nonlinear Arnoldi Method 1: start with initial basis V, V T V = I 2: determine preconditioner M T (σ) 1, σ close to wanted eigenvalue 3: find largest eigenvalue µ of V T T (µ)vy = 0 and corresponding eigenvector y 4: set u = Vy, r = T (µ)u 5: while r / u > ɛ do 6: v = Mr 7: v = v VV T v 8: ṽ = v/ v, V = [V, ṽ] 9: find largest eigenvalue µ of V T T (µ)vy = 0 and corresponding eigenvector y 10: set u = Vy, r = T (µ)u 11: end while TUHH Heinrich Voss Total Least Squares Harrachov, August / 42

74 Numerical considerations Reuse of information The convergence of W m and h m suggests to reuse information from the previous iterations when solving T m (λ)z = 0. TUHH Heinrich Voss Total Least Squares Harrachov, August / 42

75 Numerical considerations Reuse of information The convergence of W m and h m suggests to reuse information from the previous iterations when solving T m (λ)z = 0. Krylov subspace methods for T m (λ)z = 0 can be started with the solution z m 1 of T m 1 (λ)z = 0. TUHH Heinrich Voss Total Least Squares Harrachov, August / 42

76 Numerical considerations Reuse of information The convergence of W m and h m suggests to reuse information from the previous iterations when solving T m (λ)z = 0. Krylov subspace methods for T m (λ)z = 0 can be started with the solution z m 1 of T m 1 (λ)z = 0. The nonlinear Arnoldi method can use thick starts, i.e. the projection method for T m (λ)z = 0 can be initialized by V m 1 where z m 1 = V m 1 u j 1, and u j 1 is an eigenvector of V T m 1 T m 1(λ)V m 1 u = 0. TUHH Heinrich Voss Total Least Squares Harrachov, August / 42

77 Numerical considerations Thick and early updates W j = C f (x j )D S(T f (x j )I n k ) 1 S T h j = g D(T f (x j )I n k ) 1 c with C, D R k k, S R k n k, T R n k n k, g R k, c R n k. TUHH Heinrich Voss Total Least Squares Harrachov, August / 42

78 Numerical considerations Thick and early updates W j = C f (x j )D S(T f (x j )I n k ) 1 S T h j = g D(T f (x j )I n k ) 1 c with C, D R k k, S R k n k, T R n k n k, g R k, c R n k. Hence, in order to update the projected problem V T (W m+1 + λi) 2 Vu 1 δ 2 V T h m hmvu T = 0 one has to keep only CV, DV, S T V, and g T V. TUHH Heinrich Voss Total Least Squares Harrachov, August / 42

79 Numerical considerations Thick and early updates W j = C f (x j )D S(T f (x j )I n k ) 1 S T h j = g D(T f (x j )I n k ) 1 c with C, D R k k, S R k n k, T R n k n k, g R k, c R n k. Hence, in order to update the projected problem V T (W m+1 + λi) 2 Vu 1 δ 2 V T h m h T mvu = 0 one has to keep only CV, DV, S T V, and g T V. Since it is inexpensive to obtain updates of W m and h m we decided to terminate the inner iteration long before convergence, namely if the residual of the quadratic eigenvalue was reduced by at least This reduced the computing time further. TUHH Heinrich Voss Total Least Squares Harrachov, August / 42

80 Numerical considerations Example: shaw(2000); Li & Ye 10 4 shaw(2000) convergence history of Li & Ye method residual norm inner iterations TUHH Heinrich Voss Total Least Squares Harrachov, August / 42

81 Numerical considerations shaw(2000); Early updates 10 2 shaw(2000) convergence history NL Arnoldi early termination residual norm inner iterations TUHH Heinrich Voss Total Least Squares Harrachov, August / 42

82 Numerical considerations Necessary Condition: Golub, Hansen, O Leary 1999 The RTLS solution x satisfies (A T A + λ I I + λ L L T L)x = A T b, Lx 2 = δ 2 where λ I = f (x) λ L = 1 δ 2 (bt (Ax b) λ I ) TUHH Heinrich Voss Total Least Squares Harrachov, August / 42

83 Numerical considerations Necessary Condition: Golub, Hansen, O Leary 1999 The RTLS solution x satisfies (A T A + λ I I + λ L L T L)x = A T b, Lx 2 = δ 2 where λ I = f (x) λ L = 1 δ 2 (bt (Ax b) λ I ) Eliminating λ I : (A T A f (x)i + λ L L T L)x = A T b, Lx 2 = δ 2, iterating on f (x m ), and solving via quadratic eigenproblems yields the method of Sima, Van Huffel and Golub. TUHH Heinrich Voss Total Least Squares Harrachov, August / 42

84 Numerical considerations Necessary Condition: Renaut, Guo 2002,2005 The RTLS solution x satisfies the eigenproblem ( ( ) x x B(x) = λ 1) I 1 where ( ) B(x) = [A, b] T L [A, b] + λ L (x) T L 0 0 δ 2 λ I = f (x) λ L = 1 δ 2 (bt (Ax b) λ I ) TUHH Heinrich Voss Total Least Squares Harrachov, August / 42

85 Numerical considerations Necessary Condition: Renaut, Guo 2002,2005 The RTLS solution x satisfies the eigenproblem ( ( ) x x B(x) = λ 1) I 1 where ( ) B(x) = [A, b] T L [A, b] + λ L (x) T L 0 0 δ 2 λ I = f (x) λ L = 1 δ 2 (bt (Ax b) λ I ) ( ( ) ) Conversely, if ˆλ, ˆx is an eigenpair of B(ˆx), and 1 λ L (ˆx) = 1 (b T (Aˆx b) + f (ˆx)), then ˆx satisfies the necessary conditions of δ 2 the last slide, and the eigenvalue is given by ˆλ = f (ˆx). TUHH Heinrich Voss Total Least Squares Harrachov, August / 42

86 Numerical considerations Method of Renaut, Guo 2005 For θ R + let (xθ T, 1)T be the eigenvector corresponding to the smallest eigenvalue of ( ) B(θ) := [A, b] T L [A, b] + θ T L 0 0 δ 2 ( ) TUHH Heinrich Voss Total Least Squares Harrachov, August / 42

87 Numerical considerations Method of Renaut, Guo 2005 For θ R + let (xθ T, 1)T be the eigenvector corresponding to the smallest eigenvalue of ( ) B(θ) := [A, b] T L [A, b] + θ T L 0 0 δ 2 ( ) Problem Determine θ such that g(θ) = 0 where g(θ) := Lx θ 2 δ x θ 2. TUHH Heinrich Voss Total Least Squares Harrachov, August / 42

88 Numerical considerations Method of Renaut, Guo 2005 For θ R + let (xθ T, 1)T be the eigenvector corresponding to the smallest eigenvalue of ( ) B(θ) := [A, b] T L [A, b] + θ T L 0 0 δ 2 ( ) Problem Determine θ such that g(θ) = 0 where g(θ) := Lx θ 2 δ x θ 2. Given θ k, the eigenvalue problem (*) is solved by the Rayleigh quotient iteration, and θ k is updated according to θ k+1 = θ k + θ k δ 2 g(θ k). TUHH Heinrich Voss Total Least Squares Harrachov, August / 42

89 Typical g Numerical considerations 4 Plot of g(λ L ) g(λ L ) λ L TUHH Heinrich Voss Total Least Squares Harrachov, August / 42

90 Back tracking Numerical considerations 16 x g(λ L ) λ L TUHH Heinrich Voss Total Least Squares Harrachov, August / 42

91 Close up Numerical considerations x g(λ L ) λ L TUHH Heinrich Voss Total Least Squares Harrachov, August / 42

92 Typical g 1 Numerical considerations 0.08 Plot of the inverse of g(λ L ) λ L g(λ ) L TUHH Heinrich Voss Total Least Squares Harrachov, August / 42

93 Numerical considerations Rational interpolation Assumption A: Let θ 1 < θ 2 < θ 3 such that g(θ 3 ) < 0 < g(θ 1 ). TUHH Heinrich Voss Total Least Squares Harrachov, August / 42

94 Numerical considerations Rational interpolation Assumption A: Let θ 1 < θ 2 < θ 3 such that g(θ 3 ) < 0 < g(θ 1 ). Let p 1 < p 2 be the poles of g 1, let h := α + βx + γx 2 (p 2 x)(x p 1 ) be the interpolation of g 1 at (g(θ j ), θ j ), j = 1, 2, 3, and set θ 4 = h(0). TUHH Heinrich Voss Total Least Squares Harrachov, August / 42

95 Numerical considerations Rational interpolation Assumption A: Let θ 1 < θ 2 < θ 3 such that g(θ 3 ) < 0 < g(θ 1 ). Let p 1 < p 2 be the poles of g 1, let h := α + βx + γx 2 (p 2 x)(x p 1 ) be the interpolation of g 1 at (g(θ j ), θ j ), j = 1, 2, 3, and set θ 4 = h(0). Drop θ 1 or θ 3 such that the remaining θ values satisfy assumption A, and repeat until convergence. TUHH Heinrich Voss Total Least Squares Harrachov, August / 42

96 Numerical considerations Rational interpolation Assumption A: Let θ 1 < θ 2 < θ 3 such that g(θ 3 ) < 0 < g(θ 1 ). Let p 1 < p 2 be the poles of g 1, let h := α + βx + γx 2 (p 2 x)(x p 1 ) be the interpolation of g 1 at (g(θ j ), θ j ), j = 1, 2, 3, and set θ 4 = h(0). Drop θ 1 or θ 3 such that the remaining θ values satisfy assumption A, and repeat until convergence. To evaluate g(θ) one has to solve the eigenvalue problem B(θ)x = µx which can be done efficiently by the nonlinear Arnoldi method starting with the entire search space of the previous step. TUHH Heinrich Voss Total Least Squares Harrachov, August / 42

97 Numerical example Numerical example We added white noise to the data of phillips(n) and deriv2(n) with noise level 1%, and chose L to be an approximate first derivative. The following tabel contains the average CPU time for 100 test problems of dimensions n = 1000, n = 2000, and n = 4000 (CPU: Pentium D, 3.4 GHz). TUHH Heinrich Voss Total Least Squares Harrachov, August / 42

98 Numerical example Numerical example We added white noise to the data of phillips(n) and deriv2(n) with noise level 1%, and chose L to be an approximate first derivative. The following tabel contains the average CPU time for 100 test problems of dimensions n = 1000, n = 2000, and n = 4000 (CPU: Pentium D, 3.4 GHz). problem n SOAR Li & Ye NL Arn. R & G phillips deriv TUHH Heinrich Voss Total Least Squares Harrachov, August / 42

99 Numerical example Numerical example ct. We added white noise to the data of phillips(n) and deriv2(n) with noise level 10%, and chose L to be an approximate first derivative. The following tabel contains the average CPU time for 100 test problems of dimensions n = 1000, n = 2000, and n = 4000 (CPU: Pentium D, 3.4 GHz). problem n SOAR Li & Ye NL Arn. R & G phillips deriv TUHH Heinrich Voss Total Least Squares Harrachov, August / 42

100 Conclusions Conclusions Using minmax theory for nonlinear eigenproblems we proved a localization and characterization result for the maximum real eigenvalue of a quadratic eigenproblem occuring in TLS problems. TUHH Heinrich Voss Total Least Squares Harrachov, August / 42

101 Conclusions Conclusions Using minmax theory for nonlinear eigenproblems we proved a localization and characterization result for the maximum real eigenvalue of a quadratic eigenproblem occuring in TLS problems. The right most eigenvalue is not necesarily nonnegative TUHH Heinrich Voss Total Least Squares Harrachov, August / 42

102 Conclusions Conclusions Using minmax theory for nonlinear eigenproblems we proved a localization and characterization result for the maximum real eigenvalue of a quadratic eigenproblem occuring in TLS problems. The right most eigenvalue is not necesarily nonnegative The maximum eigenvalue can be determined efficiently by the nonlinear Arnoldi method, and taking advantage of thick starts and early updates it can be accelerated substantially. TUHH Heinrich Voss Total Least Squares Harrachov, August / 42

103 Conclusions Conclusions Using minmax theory for nonlinear eigenproblems we proved a localization and characterization result for the maximum real eigenvalue of a quadratic eigenproblem occuring in TLS problems. The right most eigenvalue is not necesarily nonnegative The maximum eigenvalue can be determined efficiently by the nonlinear Arnoldi method, and taking advantage of thick starts and early updates it can be accelerated substantially. Likewise, The approach of Renaut & Guo can be significantly improved by rational interpolation and the nonlinear Arnoldi method with thick starts. TUHH Heinrich Voss Total Least Squares Harrachov, August / 42

104 Conclusions Conclusions Using minmax theory for nonlinear eigenproblems we proved a localization and characterization result for the maximum real eigenvalue of a quadratic eigenproblem occuring in TLS problems. The right most eigenvalue is not necesarily nonnegative The maximum eigenvalue can be determined efficiently by the nonlinear Arnoldi method, and taking advantage of thick starts and early updates it can be accelerated substantially. Likewise, The approach of Renaut & Guo can be significantly improved by rational interpolation and the nonlinear Arnoldi method with thick starts. THANK YOU FOR YOUR ATTENTION TUHH Heinrich Voss Total Least Squares Harrachov, August / 42