Total Least Squares and Chebyshev Norm

Transcription

1 Total Least Squares and Chebyshev Norm Milan Hladík 1 & Michal Černý 2 1 Department of Applied Mathematics Faculty of Mathematics & Physics Charles University in Prague, Czech Republic 2 Department of Econometrics & DYME Research Center Faculty of Computes Science & Statistics University of Economics in Prague, Czech Republic ICCS 2015, Reykjavik M. Hladík & M. Černý (Prague, CZ) Total Least Squares and Chebyshev Norm 1 / 23

2 To recall: linear regression, OLS and TLS Classical model: b = Ax b M. Hladík & M. Černý (Prague, CZ) Total Least Squares and Chebyshev Norm 2 / 23

3 To recall: linear regression, OLS and TLS Classical model: b = Ax b known matrix of regressors M. Hladík & M. Černý (Prague, CZ) Total Least Squares and Chebyshev Norm 3 / 23

4 To recall: linear regression, OLS and TLS Classical model: b = Ax b known observations of the dependent variable known matrix of regressors M. Hladík & M. Černý (Prague, CZ) Total Least Squares and Chebyshev Norm 4 / 23

5 To recall: linear regression, OLS and TLS Classical model: b = Ax b known known observations matrix of the of regressors regression parameters dependent to estimate! variable M. Hladík & M. Černý (Prague, CZ) Total Least Squares and Chebyshev Norm 5 / 23

6 To recall: linear regression, OLS and TLS Classical model: b = Ax b known known observations matrix of the of regressors regression parameters dependent to estimate! variable M. Hladík & M. Černý (Prague, CZ) Total Least Squares and Chebyshev Norm 6 / 23

7 To recall: linear regression, OLS and TLS Classical model: b = Ax b errors known known observations matrix of the of regressors regression parameters dependent to estimate! variable M. Hladík & M. Černý (Prague, CZ) Total Least Squares and Chebyshev Norm 7 / 23

8 To recall: linear regression, OLS and TLS Classical model: b = Ax b errors known known observations matrix of the of regressors regression parameters dependent to estimate! variable Good estimator: Ordinary Least Squares (OLS) Find b,x s.t.: Ax = b+ b is solvable and b 2 is minimal x = (A T A) 1 A T b M. Hladík & M. Černý (Prague, CZ) Total Least Squares and Chebyshev Norm 8 / 23

9 To recall: linear regression, OLS and TLS Classical model: b = Ax b errors known known Errors-in-variables (EIV) model: b = Ax b M. Hladík & M. Černý (Prague, CZ) Total Least Squares and Chebyshev Norm 9 / 23

10 To recall: linear regression, OLS and TLS Classical model: b = Ax b errors known known Errors-in-variables (EIV) model: b = Ax b errors known M. Hladík & M. Černý (Prague, CZ) Total Least Squares and Chebyshev Norm 10 / 23

11 To recall: linear regression, OLS and TLS Classical model: b = Ax b known known Errors-in-variables (EIV) model: b = Ax b errors errors known : known is A = A A errors M. Hladík & M. Černý (Prague, CZ) Total Least Squares and Chebyshev Norm 11 / 23

12 To recall: linear regression, OLS and TLS Classical model: b = Ax b errors known known Errors-in-variables (EIV) model: b = Ax b A = A A Good estimator: Total Least Squares (TLS) Find A, b,x s.t.: (A + A)x = b+ b is solvable and ( A, b) F is minimal M. Hladík & M. Černý (Prague, CZ) Total Least Squares and Chebyshev Norm 12 / 23

13 A reformulation TLS finds: A, b s.t. (A + A)x = b + b is solvable and ( A, b) F is minimal, where Q F = Qij trace(q 2 = T Q) is the Frobenius norm. i,j Our problem (Chebyshev Norm Problem, CNP): find A, b s.t. (A + A)x = b + b is solvable and ( A, b) max is minimal, where Q max = max Q ij is the Chebyshev norm. i,j M. Hladík & M. Černý (Prague, CZ) Total Least Squares and Chebyshev Norm 13 / 23

14 Motivation Why to replace F by another norm? Robustness arguments a usage of different norms is a usual method in robust statistics ( F is sensitive to outliers and often ill-conditioned); Estimation theory arguments under certain probabilistic assumptions on the errors A, b, the solution obtained from the Chebyshev Norm Problem gives a consistent estimator for the EIV model. M. Hladík & M. Černý (Prague, CZ) Total Least Squares and Chebyshev Norm 14 / 23

15 Intermezzo: Interval computation Definition. Interval (m n)-matrix is a system of matrices A = [A,A] = {A R m n : A A A}, where A A R m n are given and is understood componentwise. Definition. Solution set of a system of interval-valued linear equations Ax = b is defined as S(A,b) = {x R n : ( A A)( b b) Ax = b}. Interval-theoretic reformulation of the Chebyshev Norm Problem (CNP): Find the minimum δ such that S([A δe,a +δe],[b δe,b +δe]), where E is the all-one matrix and e is the all-one vector. M. Hladík & M. Černý (Prague, CZ) Total Least Squares and Chebyshev Norm 15 / 23

16 Oettli-Prager Theorem Lemma (Oettli-Prager). S(A,b) = {x R n : A C x b C A x +b }, where A C = 1 2 (A+A) is the center matrix and A = 1 2 (A A) is the radius matrix. Corollary characterization of the CNP system: S([A δe,a +δe],[b δe,b +δe]) where D s = diag(s). = {x R n : A x b δe x +δe} = (A δed s )x b +δe, x R : ( A δed s )x b+δe, s {±1} n D s x 0, M. Hladík & M. Černý (Prague, CZ) Total Least Squares and Chebyshev Norm 16 / 23

17 Example of the CNP system s {±1} n { (A δed s )x b +δe, ( A δed s )x b +δe, D s x 0 } 0.6 δ = 0.8 δ = A = b = δ = 0.5 δ = M. Hladík & M. Černý (Prague, CZ) Total Least Squares and Chebyshev Norm 17 / 23

18 Another example of the CNP system { (A δed s )x b +δe, ( A δed s )x b +δe, D s x 0 } s {±1} n β n, (δ n ) = A = b = M. Hladík & M. Černý (Prague, CZ) Total Least Squares and Chebyshev Norm 18 / 23

19 Reduction to 2 n GLFPs To recall: To solve CNP, we are to find the minimum δ such that the CNP system { (A δed s )x b +δe, ( A δed s )x b +δe, D s x 0 } s {±1} n is nonempty. The main observation: In a given orthant s {±1} n, it suffices to solve the following generalized linear-fractional programming (GLFP) problem: min x R n max i {1,...,m} j {0,1} ( 1) 1 j A i x +( 1)j b i e T D s x +1 s.t. D s x 0, where A i is the i-th row of A. An important (well-known) fact. GLFP can be solved in polynomial time via interior point methods. M. Hladík & M. Černý (Prague, CZ) Total Least Squares and Chebyshev Norm 19 / 23

20 Remarks We have shown: Bad news. The algorithm is exponential in n, the number of regression parameters. Good news. The algorithm is not exponential in m, the number of observations. Since usually n m, we can say: Corollary. As long as n = O(1) (i.e., n is a constant independent of m), the method runs in polynomial time. Comment. In practive we work with regression models with up to n = 20 (say) regression parameters. And 2 20 is large, but still tractable. The main question. Can we achieve a better algorithm? Theorem. The answer is NO. (CNP is NP-hard.) M. Hladík & M. Černý (Prague, CZ) Total Least Squares and Chebyshev Norm 20 / 23

21 Example: a simulation study A probabilistic setup: two regression parameters, their true values are zero the observations of the regressors are contaminated by independent errors sampled from Unif( γ,γ), where γ > 0 is a parameter the observations of the dependent variable are contaminated by independent errors sampled from Unif( γ, γ) M. Hladík & M. Černý (Prague, CZ) Total Least Squares and Chebyshev Norm 21 / 23

22 Example: a simulation study estimate of γ δ opt γ = 0.6 γ = 0.4 γ = No of observations k x opt γ = 0.6 γ = 0.2 γ = No of observations estimate of x (true val = 0) k M. Hladík & M. Černý (Prague, CZ) Total Least Squares and Chebyshev Norm 22 / 23

23 Remarks and conclusions Further results. Since the CNP problem is NP-hard, we are interested in designing heuristics. We have also designed some methods for poly-time computable lower bounds, poly-time computable upper bounds. Current work. Now we are investigating under which probabilistic assumptions on the errors A, b the CNP problem gives a consistent estimator of the regression parameters and what is the speed of convergence. Other norms. The TLS problem is interesting not only with the Chebyshev norm. Other matrix norms are of interest as well. Thank you for your attention. M. Hladík & M. Černý (Prague, CZ) Total Least Squares and Chebyshev Norm 23 / 23