Linear Programing and Inversions

Transcription

1 Linear Programing and Inversions Linear programming (LP, also called linear optimization) is a method to achieve the best outcome (such as maximum profit or lowest cost) in a mathematical model whose requirements are represented by linear relationships. Linear programming is a special case of optimization (WIKI) y= a x + b The solution tries to set the left side = right side), or achieve the minimum differences between the two sides of the equal sign. This is really an optimization problem, or minimization problem with the solution (sometimes we refer to as models ) produce the least amount of errors (so called the best fits/matches between the two sides of the equation. Example of a model: the line y that fits two or more points (x 1, y 1 ), (x 2, y 2 ). Then a and b are model coefficients If we have multiple equations, or a system of linear equations, we have a linear algebra problem involving matrices and vectors. 1

2 What we have done here borders on the art of inversions, i.e., given AX = B, solve for X by finding the inverse of something (e.g., A). Where it all began: Radon transform: (Johan Radon, 1917): integral of function over a straight line segment. Medical Tomography Brain Scanning: Could the best valentine s gift to your love ones a freshly taken brainogram? The spots of red shows your love, not your words! 2 2

3 Review: Matrices and Determinants Symmetric Diagonal Upper Triangular Lower Triangular Zero Identity Suppose Addition and Subtraction Multiplication A B C 3

4 Further review: Transpose of a Matrix switching rows and columns Determinant of a Matrix (1) Determinant of a 2 x 2 The determinant of this matrix becomes: Determinant of 3 x 3 4

5 Matrices and Vector Operations (Python) import numpy as np A = np.array([[1, 2, 3], [1, 0, 2]]) B = np.array([[2, 1, 2], [1, 0, 3]]) print('a + B = ') print(a+b) print('\n A - B = ') print(a-b) print() A = np.array([[1, 2,], [3, 4]]) B = np.array([[5, 6 ], [7, 8]]) print(a) print(b) print('\n A * B =') print(a*b) # this is element-wise multiplication! print('\n A dot B =') print(np.dot(a, B)) v = np.array([[1], [2], [3]]) # column vector print(v) print('transpose of column vector v') print(np.transpose(v)) 5 print(v.t) # transpose method from array object

6 Output: A + B = [[3 3 5] [2 0 5]] A - B = [[-1 1 1] [ 0 0-1]] [[1 2] [3 4]] [[5 6] [7 8]] A * B = [[ 5 12] [21 32]] A dot B = [[19 22] [43 50]] [[1] [2] [3]] transpose of column vector v [[1 2 3]] Continued here rank of matrix A (2, 2) [[1 2 3]] matrix A [[1 2] [3 4]] determinant of A = -2 # Code below shows that a nonsquare matrix has determinant of 0 M_ill = np.array([[1, 2, 3], [0, 0, 0], [0, 0, 0]]) print('\n non-square matrix M_ill') print(m_ill) print('determinant = %g\n' % np.linalg.det(m_ill)) Output: non-square matrix M_ill [[1 2 3] [0 0 0] [0 0 0]] determinant = 0 6

7 Python script import numpy as np a = np.zeros([2, 2]) a b = np.ones([2, 2]) b c = np.full([2, 2], 5) c d = np.eye(3) d e = np.random.random((2, 2)) e Output: array([[0., 0.], [0., 0.]]) array([[1., 1.], [1., 1.]]) array([[1., 0., 0.], [0., 1., 0.], [0., 0., 1.]]) array([[ , ], [ , ]]) 7

8 Linearization We have encountered problems in the past where a system of equations needs to be solved for the coefficients of interpolation function (e.g., line or cubic polynomials). Example: for a given 4 data points, we would like to find a function to interpolate the value in between or beyond. The key is to set up the system of a equations to solve for the same number of unknowns (coefficients) b(x) (x 2, b 2 ) Cubic polynomial (x 4, b 4 ) 8 (x 1, b 1 ) (x 3, b 3 ) x This could be done using a Cubic polynomial, where we need to solve a 4 x 4 system of equations for 4 data points: a 1 + a 2 x i + a 3 x i 2 + a 4 x i 3 = b i In other words: a 1 + a 2 x 1 + a 3 x1 2 + a 4 x3 1 = b 1 a 1 + a 2 x 2 + a 3 x2 2 + a 4x2 3 = b 2 a 1 + a 2 x 3 + a 3 x3 2 + a 4 x 3 3 = b 3 The unknowns we want to find are a 0, a 1, a 2 and a 3.

9 Numerical solution: Write the system of equations as the sum of unknown coefficients multiply by the independent variable. Cubic Polynomial Interpolation (governing equation) y " $ $ $ $ # x 1 0 x 2 0 x 1 1 x 2 1 x 1 2 x 2 2 x 1 3 x x n 0 x n 1 x n 2 x n 3 " b % " a1% 1 % $ ' ' $ ' ' a2 $ b 2 ' $ ' = $...' ' $ a3' $ ' ' $ '... & # a4& $ ' # $ &' b n A X = B How to solve? Let s start with a simpler inverse problem: find intercept and slope of a linear trend. a + bx = y (governing equation) x " 1 x 1 % " y 1 % $ ' $ ' $ 1 x 2 ' " $ 1 x 3 ' a % $ y 2 ' $ ' = $ y $ ' # b 3 ' & $ ' $ ' $... ' # $ 1 x n &' # $ y n &' 9

10 Redundancy y Perfect line fit If perfectly aligned, any two points can be used to solve for slope+intercept x Determinedness: If there are more linearly independent equations than model coefficients, then this problem is called Over-determined. In this case, if we could show the independence of the 5 data points, the system is over-determined due to only 2 unknowns (a and b). over-determined under-determined y y No single line can go through all points x Infinite solutions x 10

11 Solving Matrix equations --- Gaussian Elimination Can use determinant methods such as Cramer s rule, but the most useful approach is Gaussian Elim. Most matrix decomposition methods is based upon this method. First, let use a simple example: Step 1: Forward Elim. Get rid of x1 in second Eq. by subtracting first Eq x 2 from second Upper triangular matrix form Start at the bottom Step 2: Back substitution Note: determinant does not change (and nonzero!). 0th step 1st step 11

12 Utopian Case: a direct (perfect, no noise) solution exists, one can use linear algebra to solve C X = D C -1 C X = C -1 D X = C -1 D For a very simple 2 x 2 problem, Remarks: (1) If the determinant of the matrix is 0 (singular, imagine c=-2a and d=-2b), in computing the inverse, we have 1/0 which is infinite ---> no matrix inverse. (2) Unless C is a diagonal matrix, the inverse of the matrix is a matrix containing the reciprocal of each element. (3) Special case Comment: If any of the diagonal =0, the matrix is singular again because 1. that row doesn t count, and 2. 1/0 is infinite. 12

13 Python implementation Write in matrix form: 2 1 C = [ 4 1] D = [ 4 2] X = [ x 1 x 2 ] direct_inversion_2by2.py # This code solves a direct solution for a 2x2 import numpy as np C = np.array([[2, 1], [4, -1]]) D = np.array([4, 2]) print('c is:\n', C) print('d is:\n', D) Cinv = np.array([[-1, -1], [-4, 2]])/np.linalg.det(C) print('\nc inverse is:\n', Cinv) print('\ncinv * C = \n, \ np.dot(cinv, C)) X = np.dot(cinv, D) Identity print('\nfinally, X = \n', X) # built-in function print('built-in function solution =\n', np.linalg.solve(c, D)) Output C is: [[ 2 1] [ 4-1]] D is: [4 2] C inverse is: [[ ] [ ] Cinv * C = [[1. 0.] [0. 1.]] Finally, X = [1. 2.] Built-in function solution = [1. 2.] 13

14 Matrix properties vs. Solution Types: (1) Ranks: The maximum number of linearly independent rows of a matrix is called the Row Rank of a matrix. The number of columns is called the Column Rank of a matrix. Sometimes, people use the word Rank to represent Row Rank. (2) Linear independence: Set of vectors x i are considered linearly independent if and only if solution to a i * x i =0 is a i =0 (in other words, no vector can be expressed as a linear combination of other vectors.). (3) Singularity: a matrix is singular if Rank< # rows. (4) When a set of linear equations is described by a singular matrix, two things may happen: 1. No solution 2. Solution is there but not unique (need some more constraints like adding a damping term to solve) Problem 1: Ill Conditioned Matrix Suppose we have the following set x 1 - x 2 = 1-2x 1 + 2x 2 = -2 This looks like we could solve for the two unknowns from 2 equations, issue: The second equation is just the first multiplied by a linear factor 2. So in reality, the rank (row rank) is 1 and not 2 (maximum number of rows). Solution is x 1 = 1 + x 2 (infinite solutions). Example 2: x 1 - x 2 = 1-2x 1 + 2x 2 = 5 14 NO SOLUTION

15 Problem 2: Small elements on the diagonal Eventually turn the matrix into an upper triangular form so that the last row has only 1 non-zero element (diagonal). Solve that equation for the last element of the unknown vector, substitute into second last row, solve for second last variable, and so on Issue: Method will fail for small or large diagonal element. Example: Forward Back Sub Examine the accuracy (single precision) of (2-1/e) Suppose =0.1 x = 0.2 x 10 = x x x 10 9 = x 10 9 = x

16 Due to the same type of round-off error, 16 So equation 2 becomes In fact, we can do better by hand, (1, 1) does satisfy the original equations, becomes In other words, my hand calculation is better than computers.

17 Fix the computer problem: F. El. Permute elements. B. Sub Why does it work? In this example, by switching the two rows, we have effectively moved the near-zero diagonal element to the side so diagonal is no longer too large or too small. This operation (switching row or column) is called pivoting. Scaled Partial Pivoting Method: (1) Start with row vector index i (2) Find scale factor = largest absolute element of each row S i = max a ij (1 <= j <= n, 1 <=i <= n) (3) Use equation for which is the largest, where p is the pivoting row number 17

18 Lets use the 2 x 2 equation to demonstrate So start with row 1, the scaling factor (absolute max) is S 1 =1 For Row 2, the absolute maximum S 2 = 1 a 11 = e/1 = e a 21 = 1/1 = 1 Since a 11 is very small and a 21 > a 11, switch. Now go to row 2 (new pivoting row), is the diagonal too small, if so switch with rows below (don t exist in this 2x2 problem). Implementation of scaled partial pivoting loop over i, find S i = max a ij (0 <= j <= n-1, 0 <=i <=n-1) end loop loop over i, if fabs(a ii ) < threshold loop over remaining index j > i compute Find row k=j that has the max value end loop Replace current pivot row by row k endif end loop