Liner Systems COS 33
Soving Liner Systems of Equtions Define iner system Singurities in iner systems Gussin Eimintion: A gener purpose method Nïve Guss Guss with pivoting Asymptotic nysis Tringur systems nd LU decomposition Speci mtrices nd gorithms: Symmetric positive definite: Choesky decomposition Tridigon mtrices Singurity detection nd condition numers
Grphic interprettion x 4, 3
Liner Systems 3 3 33 3 3 3 3 3 3 x x x x x x x x x + + + + + + + + + 3 3 33 3 3 3 3 x x x
Liner Systems Sove for x given Ax, where A is n n n mtrix nd is n n coumn vector Cn so tk out non-squre systems where A is m n, is m, nd x is n Overdetermined if m>n: more equtions thn unknowns Cn ook for est soution using est squres Underdetermined if n>m: more unknowns thn equtions
Singur Systems A is singur if some row is iner comintion of other rows Singur systems cn e underdetermined: or inconsistent: 6 4 5 3 + + x x x x 6 4 5 3 + + x x x x
Grphic Interprettion Singur with infinite soutions Singur with no soution
Ner-Singur or I-Conditioned
Why not just invert A? xa - BUT: Inefficient Prone to roundoff error In fct, compute inverse using iner sover
Sove y hnd 3x + x 8 -x + x x + 8x 4 x 3 -x + * 3 x 4
Gussin Eimintion Fundment opertions:. Repce one eqution with iner comintion of other equtions. Interchnge two equtions 3. Re-e two vries Comine to reduce to trivi system (identity) Aterntive: tringur system + ck-sustitution Simpest vrint ony uses # opertions, ut get etter stiity y dding # (prti pivoting) or # & #3 (fu pivoting)
Nïve Gussin Eimintion Sove: x 4x + 3x + 5x 7 3 Ony cre out numers form teu or ugmented mtrix : 4 3 5 7 3
Nïve Gussin Eimintion Given: 4 3 5 7 3 ) Eimintion: reduce this to system of form????? ) Bck-sustitution: Sove for x, then pug in to sove for x
Nïve Gussin Eimintion: Forwrd eimintion stge 4. Define f / (here, f ). Repce nd row r with r (f * r ) 3 5 7 3 Here, repce r with r * r ' ' 3 7
Forwrd eimintion pseudocode for k to n- { // Loop over rows for i(k+) to n { // Loop over rows eneth k th } } fctor ik ik / kk for j k to n { // Loop over eements in the row ij ij fctor ik * kj // Updte eement }
Outcome of forwrd eimintion x + x + 3 x 3 + K + n x n x + 3 x 3 + K + n x n 33 x 3 + K + 3n x n 3...... (n ) (n ) nn x n n
Bck-sustitution Pseudocode x n n / nn for i (n-) to descending { sum i for j (i+) to n { } sum sum ij * x j } x i sum / ii
Questions?
Wht coud go wrong? for k to n- { // Loop over rows for i(k+) to n { // Loop over rows eneth k th } } fctor ik ik / kk for j k to n { // Loop over eements in the row ij ij fctor ik * kj // Updte eement }
Wht coud go wrong? x n n / nn for i (n-) to descending { sum i for j (i+) to n { } sum sum ij * x j } x i sum / ii
Sm pivot eement exmpe.3x + 3.x..x +.x. After pivot, eqution ecomes -9999x -6666 Sove for x /3 Sove for x (. 3 (/3)) /.3 x -3.33 or. or.33 (depending on # digits used to represent /3)
Prti Pivoting Swp rows to pivot on rgest eement possie (i.e., put rge numers in the digon):.3x + 3.x. ecomes.x +.x..x +.x..3x + 3.x.
Prti pivot ppied.x +.x..3x + 3.x. Fctor.3/., so Eqution ecomes.9997 x -.9998 Sove for x /3 Sove for x (. * (/3)) /. x.333 or.3333 or.333333 (depending on # digits used to represent /3)
Fu Pivoting Swp rgest eement onto digon y swpping rows nd coumns More ste, ut ony sighty Critic: when swpping coumns, must rememer to swp resuts!
Questions on Gussin Eimintion?
Compexity of Gussin Eimintion Forwrd eimintion: /3 * n 3 + O(n ) (tripe for-oops yied n 3 ) Bck sustitution: n + O(n)
Big-O Nottion Informy, O(n 3 ) mens tht the dominnt term for rge n is cuic More precisey, there exist c nd n such tht if running time c n 3 n > n This type of symptotic nysis is often used to chrcterize different gorithms
LU Decomposition
Tringur Systems re nice! Lower-tringur: 4 3 44 43 4 4 33 3 3
Tringur Systems Sove y forwrd sustitution 4 3 44 43 4 4 33 3 3 x
Tringur Systems Sove y forwrd sustitution 4 3 44 43 4 4 33 3 3 x x
Tringur Systems Sove y forwrd sustitution 4 3 44 43 4 4 33 3 3 33 3 3 3 3 x x x
Tringur Systems If A is upper tringur, sove y cksustitution 3 3 33 4 4 34 44 5 5 35 45 55 3 4 5 x 5 5 55
Tringur Systems If A is upper tringur, sove y cksustitution 3 3 33 4 4 34 44 5 5 35 45 55 3 4 5 x 4 4 44 45 x 5
Tringur Systems Both of these speci cses cn e soved in O(n ) time This motivtes fctoriztion pproch to soving ritrry systems: Find wy of writing A s LU, where L nd U re oth tringur Ax LUx Ld Uxd Time for fctoring mtrix domintes computtion
Soving Ax with LU Decomposition of A A x L U L d d Ux d x
A LU 3 3 3 3 33 3 3 u 33 u u u u u 3 3 33 More unknowns thn equtions! Let ii (Dooitte s method) or et u ii (Crout s method)
Dooitte Fctoriztion for LU Decomposition 3 3 3 3 33 3 3 u u u u u u 3 3 33 U is resut of forwrd eimintion step of Guss L eements re the fctors computed in forwrd eimintion! e.g. f / nd 3 f 3 3 /
Dooitte Fctoriztion 3 3 For i..n 3 3 33 For j..i u For j i+..n 3 ji ji 3 j ji k ji i k u ii jk jk u u u ki ki u u u u u 3 3 33
Dooitte Fctoriztion Interesting note: # of outputs # of inputs, gorithm ony refers to eements of A, not Cn do this in-pce! Agorithm repces A with mtrix of nd u vues, s re impied 3 3 u33 Resuting mtrix must e interpreted in speci wy: not regur mtrix Cn rewrite forwrd/cksustitution routines to use this pcked -u mtrix u u u u u 3 3
LU Decomposition Running time is / 3 n 3 Independent of RHS, ech of which requires O(n ) ck/forwrd sustitution This is the preferred gener method for soving iner equtions Pivoting very importnt Prti pivoting is sufficient, nd widey impemented LU with pivoting cn succeed even if mtrix is singur (!) (ut ck/forwrd sustitution fis )
Mtrix Inversion using LU LU depend ony on A, not on Re-use L & U for mutipe vues of i.e., repet ck-sustitution How to compute A -? AA - I (n n identity mtrix), e.g. Use LU decomposition with,, 3
Questions on LU Decomposition?
Working with Speci Mtrices
Tridigon Systems Common speci cse: Ony min digon + ove nd eow 4 3 44 43 34 33 3 3
Soving Tridigon Systems When soving using Gussin eimintion: Constnt # of mutipies/dds in ech row Ech row ony ffects others 3 3 33 43 34 44 3 4
Running Time n oops, 4 mutipy/dds per oop (ssuming correct ookkeeping) This running time hs fundmenty different dependence on n: iner insted of cuic Cn sy tht tridigon gorithm is O(n) whie Guss is O(n 3 ) In gener, nded system of ndwith w requires O(wn) storge nd O(w n) computtions.
Symmetric mtrices: Choesky Decomposition For symmetric mtrices, choose UL T (A LL T ) Perform decomposition 3 3 3 3 33 3 3 33 3 3 33 Ax LL T x Ld L T xd
Choesky Decomposition 3 3 3 3 3 3 3 3 3 3 + + 33 3 3 33 3 3 33 3 3 3 3
Choesky Decomposition ii i k jk ik ij ji i k ik ii ii 33 3 3 33 3 3 33 3 3 3 3
Choesky Decomposition This fis if it requires tking squre root of negtive numer Need nother condition on A: positive definite i.e., For ny v, v T A v > (Equiventy, positive eigenvues)
Choesky Decomposition Running time turns out to e / 6 n 3 mutipictions + / 6 n 3 dditions Sti cuic, ut ower constnt Hf s much computtion & storge s LU Resut: this is preferred method for soving symmetric positive definite systems
Running time revisited
Running Time Is O(n 3 ) the Limit? How fst is mtrix mutipiction? c c c c c c c c + + + + 8 mutipes, 4 dds, right? (In gener n 3 mutipies nd n (n-) dds )
Running Time Is O(n 3 ) the Limit? Strssen s method [969] 6 3 4 5 3 7 5 4 7 6 5 4 3 ) )( ( ) )( ( ) ( ) ( ) ( ) ( ) )( ( M M M M c M M c M M c M M M M c M M M M M M M + + + + + + + + + + + + c c c c Voker Strssen
Running Time Is O(n 3 ) the Limit? Strssen s method [969] c c c c Uses ony 7 mutipies (nd whoe unch of dds) Cn e ppied recursivey! M M M M M M M c c c c 3 4 5 6 7 ( ( ( ( ( M M M M ( 3 + + ( + M M + M + + M 4 5 4 )( ) ) )( )( ) ) M + M + + 5 3 + ) ) ) + M + M 7 6
Running Time Is O(n 3 ) the Limit? Recursive ppiction for 4 hf-size sumtrices needs 7 hf-size mtrix mutipies Asymptotic running time is Ony worth it for rge n, ecuse of ig constnt fctors ( those dditions ) Sti, prcticy usefu for n > hundreds or thousnds Current stte of the rt: Coppersmith-Winogrd.376... gorithm chieves O( n ) Not used in prctice O( n og 7 ) O( n.8 )
Running Time Is O(n 3 ) the Limit? Simir su-cuic gorithms for inverse, determinnt, LU, etc. Most cuic iner-ger proems ren t! Mjor open question: wht is the imit? Hypothesis: O(n ) or O(n og n)
Singurity nd Condition Numer
A ner-singur system
Detecting singurity nd ner-singurity Grph it! (in or 3 dimensions) Does A A - I (identity)? Does (A - ) - A? Does Ax? Does (A - ) c (A - ) c for compiers c, c? Are ny of LU digons (with pivoting) ner-zero?
A ner-singur system
Condition numer Cond(A) is function of A Cond(A), igger is d Mesures how chnge in input is propogted to chnge in output x x cond(a) A A E.g., if cond(a) 45 then cn ose og(45).65 digits of ccurcy in x, compred to precision of A
Computing condition numer cond(a) A A - where M is mtrix norm M mx j n M inf is often esiest to compute n, M i ij n mx i n j M ( λ ) / mx (using rgest eigenvue of A T A) Different norms give different vues, ut simir order of mgnitude ij
Usefu Mt functions \ : mtrix division e.g. x A\ cond: mtrix condition numer norm: mtrix or vector norm cho : Choesky fctoriztion u : LU decomposition uness you rey need the inverse!) rnk: # of inery independent rows or coumns det: determinnt trce: sum of digon eements nu: nu spce inv: inverse (don t use
Other resources Heth interctive demos: http://www.cse.iinois.edu/iem/iner_equtions/gus sin_eimintion/ http://www.cse.iinois.edu/iem/iner_equtions/con ditioning/ http://www.mth.ucsd.edu/~mthf/spring/l /L.shtm Good reding on how iner systems cn e used in we recommendtion (Pge Rnk) nd economics (Leontief Modes)