Lecture 3 - Least Squares

Size: px

Start display at page:

Download "Lecture 3 - Least Squares"

Abraham Merritt
5 years ago
Views:

Lecture 3 - Least Squares In January 1, 1801, an Italian monk pages 1,2 are from Giuseppe Piazzi, discovered a faint, http://www.keplersdiscovery. nomadic object through his telescope in com/asteroid.

1 Lecture 3 - Least Squares In January 1, 1801, an Italian monk pages 1,2 are from Giuseppe Piazzi, discovered a faint, nomadic object through his telescope in com/asteroid.html Palermo, correctly believing it to reside in the orbital region between Mars and Jupiter. Piazzi watched the object for 41 days but then fell ill, and shortly thereafter the wandering star strayed into the halo of the Sun and was lost to observation. The newly-discovered planet had been lost, and astronomers had a mere 41 days of observation covering a tiny arc of the night from which to attempt to compute an orbit and find the planet again. Amir Beck Introduction to Nonlinear Optimization Lecture Slides - Least Squares 1 / 18

2 Carl Friedrich Gauss The dean of the French astrophysical establishment, Pierre-Simon Laplace ( ), declared that it simply could not be done. In Germany, the 24 years old German mathematician Car Friedrich Gauss had considered that this type of problem to determine a planet s orbit from a limited handful of observations - commended itself to mathematicians by its difficulty and elegance. Gauss discovered a method for computing the planet s orbit using only three of the original observations and successfully predicted where Ceres might be found (now considered to be a dworf planet). The prediction catapulted him to worldwide acclaim. Amir Beck Introduction to Nonlinear Optimization Lecture Slides - Least Squares 2 / 18

3 Formulation Consider the linear system Ax b, (A R m n, b R m ) Assumption: A has a full column rank, that is, rank(a) = n. When m > n, the system is usually inconsistent and a common approach for finding an approximate solution is to pick the solution of the problem (LS) min Ax b 2. Amir Beck Introduction to Nonlinear Optimization Lecture Slides - Least Squares 3 / 18

4 The Least Squares Solution The LS problem is the same as { min f (x) x T A T Ax 2b T Ax + b 2}. x R n Amir Beck Introduction to Nonlinear Optimization Lecture Slides - Least Squares 4 / 18

5 The Least Squares Solution The LS problem is the same as { min f (x) x T A T Ax 2b T Ax + b 2}. x R n 2 f (x) = 2A T A 0 Amir Beck Introduction to Nonlinear Optimization Lecture Slides - Least Squares 4 / 18

6 The Least Squares Solution The LS problem is the same as { min f (x) x T A T Ax 2b T Ax + b 2}. x R n 2 f (x) = 2A T A 0 Therefore, the unique optimal solution x LS is the solution f (x) = 0, namely, (A T A)x LS = A T b normal equations Amir Beck Introduction to Nonlinear Optimization Lecture Slides - Least Squares 4 / 18

7 The Least Squares Solution The LS problem is the same as { min f (x) x T A T Ax 2b T Ax + b 2}. x R n 2 f (x) = 2A T A 0 Therefore, the unique optimal solution x LS is the solution f (x) = 0, namely, (A T A)x LS = A T b normal equations x LS = (A T A) 1 A T b. Amir Beck Introduction to Nonlinear Optimization Lecture Slides - Least Squares 4 / 18

8 A Numerical Example Consider the inconsistent linear system x 1 + 2x 2 = 0 2x 1 + x 2 = 1 3x 1 + 2x 2 = 1 Amir Beck Introduction to Nonlinear Optimization Lecture Slides - Least Squares 5 / 18

9 A Numerical Example Consider the inconsistent linear system x 1 + 2x 2 = 0 2x 1 + x 2 = 1 3x 1 + 2x 2 = 1 To find the least squares solution, we will solve the normal equations: T T ( ) x1 = 2 1 1, x which is the same as ( ) ( ) x1 = 10 9 x 2 ( ) 5 x 3 LS = ( ) 15/26. 8/26 Amir Beck Introduction to Nonlinear Optimization Lecture Slides - Least Squares 5 / 18

10 A Numerical Example Consider the inconsistent linear system x 1 + 2x 2 = 0 2x 1 + x 2 = 1 3x 1 + 2x 2 = 1 To find the least squares solution, we will solve the normal equations: T T ( ) x1 = 2 1 1, x which is the same as ( ) ( ) x1 = 10 9 x 2 ( ) 5 x 3 LS = ( ) 15/26. 8/26 Note that Ax LS = ( 0.038; 0.846; 1.115), so that the errors are b Ax LS = sq. err.= ( 0.115) 2 = Amir Beck Introduction to Nonlinear Optimization Lecture Slides - Least Squares 5 / 18

11 Data Fitting Linear Fitting: Data: (s i, t i ), i = 1, 2,..., m, where s i R n and t i R. Assume that an approximate linear relation holds: t i s T i x, i = 1, 2,..., m The corresponding least squares problem is: m (s T i x t i ) 2. equivalent formulation: where min x R n i=1 min x R n Sx t 2, s T 1 t 1 s T 2 S =., t = t 2.. s T m Amir Beck Introduction to Nonlinear Optimization Lecture Slides - Least Squares 6 / 18 t m

12 Illustration Amir Beck Introduction to Nonlinear Optimization Lecture Slides - Least Squares 7 / 18

13 Example of Polynomial Fitting Given a set of points in R 2 : (u i, y i ), i = 1, 2,..., m for which the following approximate relation holds for some a 0,..., a d : d a j u j i y i, i = 1,..., m. j=0 Amir Beck Introduction to Nonlinear Optimization Lecture Slides - Least Squares 8 / 18

14 Example of Polynomial Fitting Given a set of points in R 2 : (u i, y i ), i = 1, 2,..., m for which the following approximate relation holds for some a 0,..., a d : d a j u j i y i, i = 1,..., m. j=0 The system is 1 u 1 u1 2 u1 d a 0 y 0 1 u 2 u2 2 u2 d a = y 1.. } 1 u m um 2 {{ um d } a d y m U Amir Beck Introduction to Nonlinear Optimization Lecture Slides - Least Squares 8 / 18

15 Example of Polynomial Fitting Given a set of points in R 2 : (u i, y i ), i = 1, 2,..., m for which the following approximate relation holds for some a 0,..., a d : d a j u j i y i, i = 1,..., m. j=0 The system is 1 u 1 u1 2 u1 d a 0 y 0 1 u 2 u2 2 u2 d a = y 1.. } 1 u m um 2 {{ um d } a d y m U The least squares solution is of course well defined if the m (d + 1) matrix is of full column rank. This is true when all the u i s are different from each other (why?) Amir Beck Introduction to Nonlinear Optimization Lecture Slides - Least Squares 8 / 18

16 Regularized Least Squares There are several situations in which the least squares solution does not give rise to a good estimate of the true vector x. In these cases, a regularized problem (called regularized least squares (RLS)) is often solved: (RLS) min Ax b 2 + λr(x). x Here λ is the regularization parameter and R( ) is the regularization function (also called a penalty function). quadratic regularization is a specific choice of regularization function: Amir Beck Introduction to Nonlinear Optimization Lecture Slides - Least Squares 9 / 18

17 Regularized Least Squares There are several situations in which the least squares solution does not give rise to a good estimate of the true vector x. In these cases, a regularized problem (called regularized least squares (RLS)) is often solved: (RLS) min Ax b 2 + λr(x). x Here λ is the regularization parameter and R( ) is the regularization function (also called a penalty function). quadratic regularization is a specific choice of regularization function: min Ax b 2 + λ Dx 2. The optimal solution of the above problem is x RLS = (A T A + λd T D) 1 A T b. what kind of assumptions are needed to assure that A T A + λd T D is invertible? Amir Beck Introduction to Nonlinear Optimization Lecture Slides - Least Squares 9 / 18

18 Regularized Least Squares There are several situations in which the least squares solution does not give rise to a good estimate of the true vector x. In these cases, a regularized problem (called regularized least squares (RLS)) is often solved: (RLS) min Ax b 2 + λr(x). x Here λ is the regularization parameter and R( ) is the regularization function (also called a penalty function). quadratic regularization is a specific choice of regularization function: min Ax b 2 + λ Dx 2. The optimal solution of the above problem is x RLS = (A T A + λd T D) 1 A T b. what kind of assumptions are needed to assure that A T A + λd T D is invertible? (answer: Null(A) Null(D) = {0}) Amir Beck Introduction to Nonlinear Optimization Lecture Slides - Least Squares 9 / 18

19 Application - Denoising Suppose that a noisy measurement of a signal x R n is given: b = x + w. x is the unknown signal, w is the unknown noise and b is the (known) measures vector. Amir Beck Introduction to Nonlinear Optimization Lecture Slides - Least Squares 10 / 18

20 Application - Denoising Suppose that a noisy measurement of a signal x R n is given: b = x + w. x is the unknown signal, w is the unknown noise and b is the (known) measures vector. The least squares problem: MEANINGLESS. min x b 2. Amir Beck Introduction to Nonlinear Optimization Lecture Slides - Least Squares 10 / 18

21 Application - Denoising Suppose that a noisy measurement of a signal x R n is given: b = x + w. x is the unknown signal, w is the unknown noise and b is the (known) measures vector. The least squares problem: MEANINGLESS. min x b 2. Regularization is performed by exploiting some a priori information. For example, if the signal is smooth in some sense, then R( ) can be chosen as n 1 R(x) = (x i x i+1 ) 2. i=1 Amir Beck Introduction to Nonlinear Optimization Lecture Slides - Least Squares 10 / 18

22 Denoising contd. R( ) can also be written as R(x) = Lx 2 where L R (n 1) n is given by L = Amir Beck Introduction to Nonlinear Optimization Lecture Slides - Least Squares 11 / 18

23 Denoising contd. R( ) can also be written as R(x) = Lx 2 where L R (n 1) n is given by L = The resulting regularized least squares problem is min x x b 2 + λ Lx 2 Hence, x RLS (λ) = (I + λl T L) 1 b. Amir Beck Introduction to Nonlinear Optimization Lecture Slides - Least Squares 11 / 18

24 Example - true and noisy signals Amir Beck Introduction to Nonlinear Optimization Lecture Slides - Least Squares 12 / 18

25 RLS reconstructions Amir Beck Introduction to Nonlinear Optimization Lecture Slides - Least Squares 13 / 18

26 Nonlinear Least Squares The least squares problem min Ax b 2 is often called linear least squares. In some applications we are given a set of nonlinear equations: f i (x) b i, i = 1, 2,..., m. Amir Beck Introduction to Nonlinear Optimization Lecture Slides - Least Squares 14 / 18

27 Nonlinear Least Squares The least squares problem min Ax b 2 is often called linear least squares. In some applications we are given a set of nonlinear equations: f i (x) b i, i = 1, 2,..., m. The nonlinear least squares (NLS) problem is the one of finding an x solving the problem m min (f i (x) b i ) 2. i=1 As opposed to linear least squares, there is no easy way to to solve NLS problems. However, there are some dedicated algorithms for this problem, which we will explore later on. Amir Beck Introduction to Nonlinear Optimization Lecture Slides - Least Squares 14 / 18

28 Circle Fitting Linear Least Squares in Disguise Given m points a 1, a 2,..., a m R n, the circle fitting problem seeks to find a circle C(x, r) = {y R n : y x = r} that best fits the m points Amir Beck Introduction to Nonlinear Optimization Lecture Slides - Least Squares 15 / 18

29 Mathematical Formulation of the CF Problem Approximate equations: x a i r, i = 1, 2,..., m. Amir Beck Introduction to Nonlinear Optimization Lecture Slides - Least Squares 16 / 18

30 Mathematical Formulation of the CF Problem Approximate equations: x a i r, i = 1, 2,..., m. To avoid nondifferentiability, consider the squared version: x a i 2 r 2, i = 1, 2,..., m. Amir Beck Introduction to Nonlinear Optimization Lecture Slides - Least Squares 16 / 18

31 Mathematical Formulation of the CF Problem Approximate equations: x a i r, i = 1, 2,..., m. To avoid nondifferentiability, consider the squared version: x a i 2 r 2, i = 1, 2,..., m. Nonlinear least squares formulation: min x R n,r R + m ( x a i 2 r 2 ) 2. i=1 Amir Beck Introduction to Nonlinear Optimization Lecture Slides - Least Squares 16 / 18

32 Reduction to a Least Squares Problem min x,r { m } ( 2a T i x + x 2 r 2 + a i 2 ) 2 : x R n, r R. i=1 Amir Beck Introduction to Nonlinear Optimization Lecture Slides - Least Squares 17 / 18

33 Reduction to a Least Squares Problem min x,r { m } ( 2a T i x + x 2 r 2 + a i 2 ) 2 : x R n, r R. i=1 Making the change of variables R = x 2 r 2, the above problem reduces to { } m f (x, R) ( 2a T i x + R + a i 2 ) 2 : x 2 R. min x R n,r R i=1 Amir Beck Introduction to Nonlinear Optimization Lecture Slides - Least Squares 17 / 18

34 Reduction to a Least Squares Problem min x,r { m } ( 2a T i x + x 2 r 2 + a i 2 ) 2 : x R n, r R. i=1 Making the change of variables R = x 2 r 2, the above problem reduces to { } m f (x, R) ( 2a T i x + R + a i 2 ) 2 : x 2 R. min x R n,r R i=1 The constraint x 2 R can be dropped (will be shown soon), and therefore the problem is equivalent to the LS problem { m } (CF-LS) min ( 2a T i x + R + a i 2 ) 2 : x R n, R R. x,r i=1 Amir Beck Introduction to Nonlinear Optimization Lecture Slides - Least Squares 17 / 18

35 Redundancy of the Constraint x 2 R We will show that any optimal solution (ˆx, ˆR) of (CF-LS) automatically satisfies ˆx 2 ˆR. Amir Beck Introduction to Nonlinear Optimization Lecture Slides - Least Squares 18 / 18

36 Redundancy of the Constraint x 2 R We will show that any optimal solution (ˆx, ˆR) of (CF-LS) automatically satisfies ˆx 2 ˆR. Otherwise, if ˆx 2 < ˆR, then 2a T i ˆx + ˆR + a i 2 > 2a T i ˆx + ˆx 2 + a i 2 = ˆx a i 2 0, i = 1,..., m. Amir Beck Introduction to Nonlinear Optimization Lecture Slides - Least Squares 18 / 18

37 Redundancy of the Constraint x 2 R We will show that any optimal solution (ˆx, ˆR) of (CF-LS) automatically satisfies ˆx 2 ˆR. Otherwise, if ˆx 2 < ˆR, then 2a T i ˆx + ˆR + a i 2 > 2a T i ˆx + ˆx 2 + a i 2 = ˆx a i 2 0, i = 1,..., m. Thus, f (ˆx, ˆR) = m ( 2a T i ˆx + ˆR + a i 2) 2 m ( > 2a T i ˆx + ˆx 2 + a i 2) 2 = f (ˆx, ˆx 2 ), i=1 i=1 Contradiction to the optimality of (ˆx, ˆR). Amir Beck Introduction to Nonlinear Optimization Lecture Slides - Least Squares 18 / 18

Lecture 3 - Least Squares

Lecture 3 - Least Squares In January 1, 1801, an Italian monk pages 1,2 are from Giuseppe Piazzi, discovered a faint, http://www.keplersdiscovery. nomadic object through his telescope in com/asteroid.html