Least Squares and QP Optimization

Transcription

1 Least Squares and QP Optimization The following was implemented in Maple by Marcus Davidsson (2012) Stephen Boyd

2

3

4

5

6

7

8

9 Assume that we have a system of equations: y=x.w where y is vector, X is a square matrix and w is a weight vector. For example:

10 The objective is to find the vector w. This is done by introducing an error vector r = X.w - y. Note that the above system of equations is deterministic hence we dont need an error vector. An error vector is only included to illustrate the basic logic. Our system now looks like: In order for w to be an acceptable solution w has to minimize the total error (r[1] + r[2]). Since we are only interested in the absolute error (minus or plus does not matter) our objective becomes to minimize the sum of squares (r[1]^2 + r[2]^2). Our objective function can therefore be written as:

11 It turnes out that the Least Square (LS) solution to the above problem is given by: We can show this as follows: y = X.w # multiply both sides by X' X'.y = X'.X.w # solve for w (Normal equations) (X'.X)^-1.X'.y = w # caution Note that it is important to check that the matrix X is invertable. Not all matricies are invertable. A invertable matrix has the property X^-1.X = X.X^-1 = I where I is the Identity Matrix. In our case the matrix indeed is invertable hence the above equation holds. We can plot the objective function as follows:

12

13 20 w w 1 Now there also exist an graphical interpretation to the above problem as well:

14 A = 8 90 o B = 6 C =? Pythagoras theorem A^2 + B^2 = C^2 C = sqrt( 8^2 + 6^2) = 10 C= hypotenuse defined as the longest side of a right-angled triangle 3? [4,3] Vector Example 4^2 + 3^2 =?^2? = sqrt(4^2 + 3^2) = 5 [0,0] 4? = w = Euclidean Norm (L2) = lenght of the hypotenuse due to Pythagoras theorem w = sqrt w'.w = sqrt w 1 w 2 $ In our case we get: w 1 w 2 = sqrt( w[1]^2 + w[2]^2) w = sqrt w'.w = sqrt 4 3 $ 4 3 = sqrt( 4^2 + 3^2) = 5 In our previous example the error vector r = X.w - y was given by:

15 We can plot such system of equations as follows: w w1 w1+2*w2-10

16 We know that the optimal point is [-2,6] which is located where these two lines cross each other. The dashed black line represent the error vector. This is the vector that we want to minimize. Now in order for us to use Pythagoras theorem we have to make sure that the above triangle is a right-angled triangle. For two linear functions: a1*w[1] + b1*w[2] + c1 = 0 a2*w[1] + b2*w[2] + c2 = 0 The two lines will be perpendicular (90 degree angle) if and only if a1*a2 + b1*b2 = 0. i) Two lines or curves are orthogonal if they are perpendicular at their point of intersection. In our case we have: 0 (1) ii) Two vectors are orthogonal if their dot product (inner product) is

17 equal to zero. 0 0 (2) iii) A orthogonal matrix is necessarily square A[1..n,1..n] and invertible A^-1.A = A.A^-1 = I. This is an important property! Since the two lines were orthogonal our problem can be written as: We can show this as follows:

18 Warning, there are zero degrees of freedom (3) In the previous example we assumed that X was a square matrix. When X is a non-square matrix X[1..m,1..n] then we have two scenarios: Column-Dominant Matrix = more columns than rows (there are less linear equations than unknown variables ie underdetermined (df<0) Row-Dominant Matrix = more rows than columns (there are more linear equations than unknown variables ie overdetermined (df>0)

19 ) *Note df is defined at the # of rows - # of columns in matrix X In both these cases it turnes out that the normal equation solution (NES) given by error. will introduce significant estimation We can now note that there exist orthogonalization algorithms such as QR & SD decompositions. The Maple Command LinearAlgebra[LeastSquares] "For a floating-point column-dominant matrix a solution is computed by using a singular values decomposition (SVD) by default."

20 "For a floating-point row-dominant matrix a solution is computed by using a QR decomposition (QRD) by default." We can compare the three different approached as follows:

21

22 Column-Dominant Matrix Estimated w df=k23 80 Column-Dominant Matrix Estimated y df=k NES SVD NES SVD y

23

24 Row-Dominant Matrix Estimated w df=165 Row-Dominant Matrix Estimated y df= NES QRD NES QRD y

25

26 Square Matrix Estimated w df=0 Square Matrix Estimated y df= NES SVD/QRD NES SVD/QRD y

27 We can now look at a more pratical example ie portfolio optimization. We first note that the two methods below (minimize portfolio variance for a given portfolio expected return) will produce the same allocations if the matrix is square ie the number of columns=number of rows. where R is the return matrix and ERR is the vector of ER Note-1. The vector ERR is specificed by the user and not calculated on the data! Note-2. This equation is simply found from the normal equations where y=expected return and X= return matrix We can show this as follows:

28

29

30 Square Return Matrix QP NES We can now look what happens when the return matrix is not square ie column-dominate.

31 Warning, problem appears to be unbounded

32

33 25000 Column-Dominate Return Matrix QP NES We can now show that the Least Square (LS) solution and the Quadratic Programming (QP) solution for the objective of minimizing portfolio variance for a given portfolio expected return will be the same when the return matrix is either square or columndominant. This is a very convenient feature because a lot of financial datasets used for portfolio optimization often have more columns than rows ie the global universe might have thousands of stocks.

34 The Maple procedures that we use are: We can show this as follows:

35

36 Warning, problem appears to be unbounded Warning, necessary conditions met but sufficient conditions not satisfied

37

38 Optimization.QPSolve.M Optimization.QPSolve.A Optimization.LSSolve.M Optimization.LSSolve.A LinearAlgebra.LeastSquares When the return matrix is row-dominant the LS and QP solution will be different.

39

40

41

42 Optimization.LSSolve.M LinearAlgebra.LeastSquares Optimization.LSSolve.A Optimization.QPSolve.A Optimization.QPSolve.M We can see in the previous example that the Maple command was by far the fastest one. It also also produces better allocations ie lower risk. In Maple help pages for we can read: When the residuals in the objective function and the constraints are all linear (as in our case), then an active-set method is used ie a good QP-

43 Solver We can now show how we can use for a column-dominated matrix to included a simple budget constraint. The analysis is now done directly on the Stock Price and we assume that our Portfolio Capital (PC) must grow 5% each period ie where t=0..nr In the first period t=0 which means that PC=PC hence we can not buy more stocks than we can afford ie we are constrained by our budget. The analysis is done the same way as the previous example PC represent the y-variable and the x-variables is the matrix of stock prices.

44

45

46 Optimization.LSSolve.M Optimization.LSSolve.A We can do some forward testing as follows:

47

48 Portfolio Value Tracking Error