1/60 Chia-Ping Chen Professor Department of Computer Science and Engineering National Sun Yat-sen University Linear Algebra
Gaussian Elimination 2/60
Alpha Go Linear algebra begins with a system of linear equations. 3/60
Example { 1x + 2y = 3 4x + 5y = 6 elimination determinants x = y = 2, x = 1 3 2 1 3 6 5 4 6 1 2 = 1, y = 1 2 4 5 4 5 = 2 4/60
Perspectives row picture 1x + 2y = 3 4x + 5y = 6 solution: point of intersection column picture 1x + 2y = 3 4x + 5y = 6 solution: combination coefficients 5/60
Idea Eliminate unknowns in equations until it is easy to solve. recursion: reduce n unknowns in n equations to n 1 unknowns in n 1 equations base case: solve 1 unknown in 1 equation back substitution 6/60
Picture n equations in n unknowns (eli.) (sub.) (n 1) equations in (n 1) unknowns (eli.) (sub.). (eli.) (sub.) 1 equation in 1 unknown 7/60
Example 2u + v + w = 5 4u 6v = 2 2u + 7v + 2w = 9 2u + v + w = 5 8v 2w = 12 8v + 3w = 14 2u + v + w = 5 8v 2w = 12 w = 2 w = 2 v = 1 u = 1 8/60
Pivots and Multipliers unknown to eli. pivots multipliers u 2 2, -1 v -8-1 w 1-9/60
Complexity: Elimination Consider the elimination of the kth unknown using the kth equation. For each equation below 1 multiplication to determine the multiplier (n k + 2) multiplications to multiply the kth equation There are (n k) equations to eliminate the kth unknown from, so the total number of multiplications is (1 + n k + 2)(n k) There are (n 1) unknowns to eliminate, so the total number of multiplications is n 1 (1 + n k + 2)(n k) = O(n 3 ) k=1 10/60
Complexity: Back Substitution Consider solving the k th unknown counting from the last k 1 multiplications to substitute the values for the already-solved k 1 unknowns 1 multiplication to solve the k th unknown The total number of multiplications is k 1 + 1 = k There are n unknowns to solve, so the total number of multiplications is n k = O(n 2 ) k =1 11/60
Matrix Multiplication 12/60
Definition Let a 11... a 1l b 11... b 1n A m l =....., B l n =..... a m1... a ml b l1... b ln The multiplication is a matrix with elements C = AB l c ij = a ik b kj k=1 of order m n. 13/60
Case: Dot Product For m = n = 1, we have AB = [ b ] 11 a 11... a 1l. b l1 l = a 1k b k1 k=1 This is called dot product. 14/60
Case: Cross Product For l = 1, we have AB = a 11. a m1 This is called cross product. [ b11... b 1n ] a 11 b 11... a 11 b 1n =..... a m1 b 11... a m1 b 1n 15/60
Case: Linear Combination of Columns For n = 1 B = b 11. b l1 AB is a dot product if A is seen as a row of column vectors b 11 AB = a 1... a l. = a 1b 11 + + a l b l1 b l1 l = a k b k1 k=1 It is a linear combination of the column vectors of A. 16/60
Case: Linear Combination of Rows For m = 1 A = [ a 11... a 1l ] AB is a dot product if B is seen as a column of row vectors AB = [ ] a 11... a 1l l = a 1k b k: k=1 b 1:. b l: = a 11b 1: + + a 1l b l: It is a linear combination of the row vectors of B. 17/60
Element by Element AB is a cross product if we treat A as a column of row vectors and B as a row of column vectors. AB = a 1:. a m: b 1... b n a 1: b 1... a 1: b n =..... a m: b 1... a m: b n AB is constructed one element at a time. 18/60
Matrix by Matrix AB is a dot product if we treat A as a row of column vectors and B as a column of row vectors. AB = a 1... a l = a 1 b 1: + + a l b l: l = a k b k: k=1 AB is constructed one matrix at a time. b 1:. b l: 19/60
Column by Column AB is a scalar product if we treat A as a whole and B as a row of column vectors. AB = A b 1... b n = Ab 1... Ab n AB is constructed one column vector at a time. The jth column vector Ab j = a k b kj k is a linear combination of the column vectors of A. 20/60
Row by Row AB is a scalar product if we treat A as a column of row vectors and B as a whole. AB = a 1:. a m: B = a 1: B. a m: B AB is constructed one row vector at a time. The ith row vector a i: B = a ik b k: k is a linear combination of the row vectors of B. 21/60
Example AB = ele. = mat. = col. = row = [ ] 1 1 1 2 3 1 3 = 4 2 0 2 0 [ 7 ] 7 2 10 [ ] 1 1 + 2 1 + 3 2 1 1 + 2 3 + 3 0 4 1 + 2 1 + 0 2 4 1 + 2 3 + 0 0 [ ] [ ] [ ] 1 [ 1 ] 2 [1 ] 3 [2 ] 1 + 3 + 0 4 2 0 [ [ ] [ ] [ ] [ ] [ ] 1 2 3 1 2 1 + 1 + 2 1 + 3 + 0 4 2 0 4 2 1 [ 1 1 ] + 2 [ 1 3 ] + 3 [ 2 0 ] 4 [ 1 1 ] + 2 [ 1 3 ] + 0 [ 2 0 ] [ ]] 3 0 22/60
Properties associative (AB)C = A(BC) distributive non-commutative A(B + C) = AB + AC (A + B)C = AC + BC AB BA 23/60
Example 1 0 0 1 0 0 1 0 0 A = 2 1 0, B = 0 1 0, C = 0 1 0 0 0 1 1 0 1 0 1 1 AB = BA? AC = CA? 24/60
Elimination and Matrix 25/60
A Linear Equation A linear equation a 1 x 1 + + a n x n = b can be represented by a T x = b where a T = [ x ] 1 a 1... a n, x =. x n 26/60
A System of Linear Equations A system of linear equations can be represented by Ax = b where 2u + v + w = 5 4u 6v = 2 2u + 7v + 2w = 9 A = 2 1 1 4 6 0, x = 2 7 2 Ax = b u 5 v, b = 2 w 9 27/60
Row Operation Subtract from a row by a multiple of another row The other rows are unchanged. a i: a i: ka j: 28/60
Elimination and Row Operation Elimination is equivalent to row operation. { 8v 2w = 12 8v + 3w = 14 { 8v 2w = 12 w = 2 is equivalent to [ 8 2 ] 12 8 3 14 [ 8 2 ] 12 0 1 2 29/60
Gaussian Elimination Gaussian elimination is a sequence of row operations. Ax b { }} {{ }} { 2 1 1 u 5 4 6 0 v = 2 2 7 2 w 9 Ux c { }} { {}} { 2 1 1 u 5 0 8 2 v = 12 0 0 1 w 2 is equivalent to [A b] = 2 1 1 5 4 6 0 2 2 7 2 9 2 1 1 5 0 8 2 12 = [U c] 0 0 1 2 30/60
Elementary Matrix An elementary matrix is almost the same as an identity matrix (remember?) except that there is one non-zero non-diagonal element. 1 0 0 I 3 = 0 1 0 0 0 1 1 0 0 E 3 (2, 1; 2) = 2 1 0 0 0 1 31/60
Row Operation and Elementary Matrix Row operation is equivalent to multiplication of elementary matrix. a 1: a 2: a 3: 1 0 0 2 1 0 0 0 1 row operation a 1: a 2: a 3: = a 1: a 2: 2a 1: a 3: a 1: a 2: 2a 1: a 3: row operation = 1 0 0 2 1 0 0 0 1 32/60
Elimination and Elementary Matrix Gaussian elimination is equivalent to a sequence of elementary matrices. [A b] = is equivalent to 2 1 1 5 4 6 0 2 2 7 2 9 GFE [A b] = [U c] 2 1 1 5 0 8 2 12 = [U c] 0 0 1 2 where 1 0 0 1 0 0 1 0 0 E = 2 1 0, F = 0 1 0, G = 0 1 0 0 0 1 1 0 1 0 1 1 33/60
Inverse of Row Operation The inverse of a row operation is a row operation. Consider a row operation that subtracts from the second row by twice the first row. To reverse this operation, one can adds to the second row by twice the first row. 1 0 0 1 0 0 E = 2 1 0 E 1 = 2 1 0 0 0 1 0 0 1 34/60
Reversal where GFEA = U A = E 1 F 1 G 1 U 1 1 1 0 0 1 0 0 1 0 0 E 1 F 1 G 1 = 2 1 0 0 1 0 0 1 0 0 0 1 1 0 1 0 1 1 1 0 0 1 0 0 1 0 0 = 2 1 0 0 1 0 0 1 0 0 0 1 1 0 1 0 1 1 1 0 0 = 2 1 0 1 1 1 1 35/60
LU Decomposition Barring row exchanges, a non-singular matrix A (meaning Ax = b has exactly one solution) can be factorized by A = LU where L is lower-triangular and U is upper-triangular. 2 1 1 1 0 0 2 1 1 4 6 0 = 2 1 0 0 8 2 2 7 2 1 1 1 0 0 1 L and U are related to the multipliers and the pivots. 36/60
Permutation Matrix A permutation matrix is a re-arrangement of the rows or columns of an identity matrix. 0 1 0 P = 1 0 0 0 0 1 37/60
Multiplication by a Permutation Matrix From left P A {}} {{ }} { 1 0 0 a 1: 0 0 1 a 2: 0 1 0 a 3: = a 1: a 3: a 2: From right A P { }} {{ }} { 1 0 0 a 1 a 2 a 3 0 0 1 = a 1 a 3 a 2 0 1 0 38/60
LU Decomposition with Permutation General LU decomposition of a non-singular matrix PA = LU where P is a permutation matrix. 39/60
Inverse 40/60
Definition Given A, there may be a matrix B such that AB = I = BA B is an inverse of A. 41/60
Uniqueness If an inverse of a matrix exists, then it is unique. The inverse of A is denoted by A 1 Let B 1 and B 2 be inverses of A. B 1 AB 2 = B 1 AB 2 (B 1 A)B 2 = B 1 (AB 2 ) B 2 = B 1 42/60
Relation to Singularity An invertible matrix is non-singular. Suppose A 1 exists. Ax = b A 1 Ax = A 1 b x = A 1 b 43/60
Inverse of Product (AB) 1 = B 1 A 1 (AB)(B 1 A 1 ) = ABB 1 A 1 = AA 1 = I 44/60
Computing Inverse Suppose A 1 exists, with A 1 = u v... z AA 1 = I A u v... z = i 1 i 2... i n Au = i 1, Av = i 2,..., Az = i n So A 1 can be found by Gaussian elimination. 45/60
Gauss-Jordan Method The columns of A 1 can be found simultaneously. [ A I ] row operations [ U L 1] row operations [ I U 1 L 1] = [ I A 1] 46/60
Example 2 1 1 A = 4 6 0 2 7 2 [ ] 2 1 1 1 0 0 A I = 4 6 0 0 1 0 2 7 2 0 0 1 2 1 1 1 0 0 0 8 2 2 1 0 = [ U L 1] 0 0 1 1 1 1 3 5 3 1 0 0 4 16 8 0 1 0 1 3 1 2 8 4 = [ I U 1 L 1] = [ I A 1] 0 0 1 1 1 1 47/60
Gauss-Jordan with Row Exchange Let A be non-singular with PA = LU where P is a permutation matrix. Then [ A I ] row exchanges [ PA PI ] row operations [ I U 1 L 1 PI ] = [ I A 1 P 1 PI ] = [ I A 1] 48/60
Transpose 49/60
Definition The mirror image of a matrix across the diagonal line. [ ] T 2 1 1 = 4 6 0 2 4 1 6 1 0 50/60
Transpose of a Product (AB) T = B T A T ( (AB) T ) ij = (AB) ji K = a jk b ki = k=1 K k=1 (B T ) ik (A T ) kj = ( B T A T ) ij 51/60
Transpose of an Inverse (A 1 ) T = (A T ) 1 AA 1 = I (AA 1 ) T = I (A 1 ) T (A T ) = I (A T ) 1 = (A 1 ) T 52/60
Symmetric Matrix A matrix is symmetric if A T = A R T R is symmetric (R T R) T = R T (R T ) T = (R T R) 53/60
Differential Equations and Linear Equations 54/60
Differential Equations in Physics classical mechanics electrodynamics (Maxwell equations) quantum mechanics (Schrödinger equation) 55/60
An Ordinary Differential Equation Consider an ordinary differential equation with boundary condition d 2 u(x) dx 2 = f (x), 0 x 1 u(0) = 0, u(1) = 0 56/60
A Computational Approach u(x) can be approximately and numerically solved. Consider discrete points x i = ih, i = 1, 2,..., n where Let h = ( 1 ) n + 1 u i = u(x i ) f i = f (x i ) 57/60
Approximation to the Derivatives du dx x=xi u(x i + x) u(x i ) = lim x 0 x u(x i+1) u(x i ) x i+1 x i = u i+1 u i h d 2 u dx 2 x=xi = d ( ) du dx dx x=xi du dx (x i) du (x dx i 1) h u i+1 2u i + u i 1 h 2 58/60
Substitution Through approximation is converted to d 2 u(x) dx 2 = f (x), 0 x 1 u i+1 + 2u i u i 1 = h 2 f i, i = 1,..., n 59/60
Example Consider n = 5, f (x) = 6x The system of linear equations for an approximate solution of the differential equation is 2 1 0 0 0 u 1 1 1 2 1 0 0 u 2 0 1 2 1 0 u 3 = 1 2 3 36 0 0 1 2 1 u 4 4 0 0 0 1 2 u 5 5 60/60