Linear Systems and LU CS 205A: Mathematical Methods for Robotics, Vision, and Graphics Doug James (and Justin Solomon) CS 205A: Mathematical Methods Linear Systems and LU 1 / 48
Homework 1. Homework 0: Posted on website. Due next Thurs (use Gradescope) 2. Late Policy: The deadline to upload your work to Gradescope will be Thursday 11:59 pm. You have a total of 3 late periods. Using a late period means you can submit an assignment by Sunday 11:59 pm. If you exhaust your late periods, late assignments will be penalized at 50%. No work will be accepted after the late deadline. CS 205A: Mathematical Methods Linear Systems and LU 2 / 48
Office Hours Leon Yao Tue 12:30-2:30pm (Huang Basement) Rafael Musa Wed 2-4pm (Huang Basement) Alex Jin Thu 10am-noon (Lathrop Tech Lounge) Prof. Doug James Thu 5-7pm (Gates 362) CS 205A: Mathematical Methods Linear Systems and LU 3 / 48
Other Announcements New Section Room: Time: Friday, 11:30-12:20pm Place: Gates B03 (new!) Julia programming refs on webpage CS 205A: Mathematical Methods Linear Systems and LU 4 / 48
Linear Systems A x = b A R m n x R n b R m CS 205A: Mathematical Methods Linear Systems and LU 5 / 48
( 1 0 0 1 Case 1: Solvable ) ( x y ) = ( 1 1 ) Completely Determined CS 205A: Mathematical Methods Linear Systems and LU 6 / 48
Case 2: No Solution ( 1 0 1 0 ) ( x y ) = ( 1 1 Overdetermined ) CS 205A: Mathematical Methods Linear Systems and LU 7 / 48
Case 3: Infinitely Many Solutions ( 1 0 1 0 ) ( x y ) = ( 1 1 Underdetermined ) CS 205A: Mathematical Methods Linear Systems and LU 8 / 48
No Other Cases Proposition If A x = b has two distinct solutions x 0 and x 1, it has infinitely many solutions. CS 205A: Mathematical Methods Linear Systems and LU 9 / 48
Common Misconception Solvability can depend on b! ( ) ( ) ( ) 1 0 x 1 = 1 0 y 0 CS 205A: Mathematical Methods Linear Systems and LU 10 / 48
Dependence on Shape Proposition Tall matrices admit unsolvable right hand sides. Proposition Wide matrices admit right hand sides with infinite numbers of solutions. CS 205A: Mathematical Methods Linear Systems and LU 11 / 48
For Now All matrices will be: Square Invertible CS 205A: Mathematical Methods Linear Systems and LU 12 / 48
Inverting Matrices Do not compute A 1 if you do not need it. Not the same as solving A x = b Can be slow and poorly conditioned CS 205A: Mathematical Methods Linear Systems and LU 13 / 48
Example y z = 1 3x y + z = 4 x + y 2z = 3 0 1 1 1 3 1 1 4 1 1 2 3 Permute rows Row scaling Forward/back substitution CS 205A: Mathematical Methods Linear Systems and LU 14 / 48
Row Operations: Permutation σ : {1,..., m} {1,..., m} P σ e σ(1) e σ(2) e σ(m) CS 205A: Mathematical Methods Linear Systems and LU 15 / 48
Row Operations: Row Scaling S a a 1 0 0 0 a 2 0...... 0 0 a m CS 205A: Mathematical Methods Linear Systems and LU 16 / 48
Row Operations: Elimination Scale row k by constant c and add result to row l. E I + c e l e k CS 205A: Mathematical Methods Linear Systems and LU 17 / 48
Solving via Elimination Matrices 0 1 1 1 3 1 1 4 1 1 2 3 Reverse order! CS 205A: Mathematical Methods Linear Systems and LU 18 / 48
Introducing Gaussian Elimination Big idea: General strategy to solve linear systems via row operations. CS 205A: Mathematical Methods Linear Systems and LU 19 / 48
Elimination Matrix Interpretation A x = b E 1 A x = E 1 b E 2 E 1 A x = E 2 E 1 b. E k E 2 E 1 A x = E k E 2 E 1 b I n n A 1 CS 205A: Mathematical Methods Linear Systems and LU 20 / 48
( ) A b = Shape of Systems CS 205A: Mathematical Methods Linear Systems and LU 21 / 48
Pivot CS 205A: Mathematical Methods Linear Systems and LU 22 / 48
Row Scaling 1 CS 205A: Mathematical Methods Linear Systems and LU 23 / 48
Forward Substitution 1 0 0 0 CS 205A: Mathematical Methods Linear Systems and LU 24 / 48
Forward Substitution 1 0 1 0 0 0 0 CS 205A: Mathematical Methods Linear Systems and LU 25 / 48
Upper Triangular Form 1 0 1 0 0 1 0 0 0 1 CS 205A: Mathematical Methods Linear Systems and LU 26 / 48
Back Substitution 1 0 0 1 0 0 0 1 0 0 0 0 1 CS 205A: Mathematical Methods Linear Systems and LU 27 / 48
Back Substitution 1 0 0 0 1 0 0 0 0 1 0 0 0 0 1 CS 205A: Mathematical Methods Linear Systems and LU 28 / 48
Back Substitution 1 0 0 0 0 1 0 0 0 0 1 0 0 0 0 1 CS 205A: Mathematical Methods Linear Systems and LU 29 / 48
Steps of Gaussian Elimination 1. Forward substitution: For each row i = 1, 2,..., m Scale row to get pivot 1 For each j > i, subtract multiple of row i from row j to zero out pivot column 2. Backward substitution: For each row i = m, m 1,..., 1 For each j < i, zero out rest of column CS 205A: Mathematical Methods Linear Systems and LU 30 / 48
Total Running Time O(n 3 ) CS 205A: Mathematical Methods Linear Systems and LU 31 / 48
Problem ( 0 1 ) A = 1 0 CS 205A: Mathematical Methods Linear Systems and LU 32 / 48
Even Worse ( ε 1 ) A = 1 0 CS 205A: Mathematical Methods Linear Systems and LU 33 / 48
Pivoting Pivoting Permuting rows and/or columns to avoid dividing by small numbers or zero. Partial pivoting Full pivoting 1 10 10 0 0.1 9 0 4 6.2 CS 205A: Mathematical Methods Linear Systems and LU 34 / 48
Reasonable Use Case A x 1 = b 1 A x 2 = b 2. Can we restructure A to make this more efficient? Does each solve take O(n 3 ) time? CS 205A: Mathematical Methods Linear Systems and LU 35 / 48
Observation Steps of Gaussian elimination depend only on structure of A. Avoid repeating identical arithmetic on A? CS 205A: Mathematical Methods Linear Systems and LU 36 / 48
Another Clue: Upper Triangular Systems 1 0 1 0 0 1 0 0 0 1 CS 205A: Mathematical Methods Linear Systems and LU 37 / 48
After Back Substitution 1 0 0 1 0 0 0 1 0 0 0 0 1 No need to subtract the 0 s explicitly! O(n) time CS 205A: Mathematical Methods Linear Systems and LU 38 / 48
Next Pivot: Same Observation 1 0 0 0 1 0 0 0 0 1 0 0 0 0 1 Observation Triangular systems can be solved in O(n 2 ) time. CS 205A: Mathematical Methods Linear Systems and LU 39 / 48
Upper Triangular Part of A A x = b. M k M 1 A x = M k M 1 b Define: U M k M 1 A CS 205A: Mathematical Methods Linear Systems and LU 40 / 48
Lower Triangular Part U = M k M 1 A A = (M1 1 M 1 k )U LU CS 205A: Mathematical Methods Linear Systems and LU 41 / 48
Why Is L Triangular? S diag(a 1, a 2,...) E I + c e l e k Proposition The product of triangular matrices is triangular. CS 205A: Mathematical Methods Linear Systems and LU 42 / 48
Solving Systems Using LU A x = b LU x = b 1. Solve L y = b using forward substitution. 2. Solve U x = y using backward substitution. O(n 2 ) (given LU factorization) CS 205A: Mathematical Methods Linear Systems and LU 43 / 48
LU: Compact Storage U U U U L U U U L L U U L L L U Assumption: Diagonal elements of L are 1. Warning: Do not multiply this matrix! CS 205A: Mathematical Methods Linear Systems and LU 44 / 48
Computing LU Factorization Small modification of forward substitution step to keep track of L. 1 1 See textbook for pseudocode. CS 205A: Mathematical Methods Linear Systems and LU 45 / 48
Question Does every A admit a factorization A = LU? CS 205A: Mathematical Methods Linear Systems and LU 46 / 48
Recall: Pivoting Pivoting Permuting rows and/or columns to avoid dividing by small numbers or zero. Partial pivoting Full pivoting 1 10 10 0 0.1 9 0 4 6.2 CS 205A: Mathematical Methods Linear Systems and LU 47 / 48
Pivoting by Swapping Columns (E k E 1 ) elimination A (P 1 P l ) (Pl P1 ) x permutations inv. permutations = (E k E 1 ) b A = LUP CS 205A: Mathematical Methods Linear Systems and LU 48 / 48