Matrices and Linear Algebra

Transcription

1 Volume FOXES TEAM Tutoril on Numericl Anlysis with Mtri.l Mtrices nd Liner Alger

2 TUTORIAL ON NUMERICAL ANALYSIS WITH MATRIX.XLA Mtrices nd Liner Alger

3 TUTORIAL FOR MATRIX.XLA Inde Aout this Tutoril...5 Mtri.l...5 Liner System...6 The Guss-Jordn lgorithm...7 The pivoting strtegy...8 Integer clcultion... Severl wys to use the Guss-Jordn lgorithm... Solving non-singulr liner system... Solving m simultneous liner systems... Inverse mtri computing... Determinnt computing... Liner independence checking... Non-singulr Liner system... 4 Round-off errors...4 Full pivoting or prtil pivoting?...5 Solution stility...7 The Condition Numer...9 Comple systems... Aout the comple mtri formt... Determinnt... Gussin elimintion... Ill-conditioned mtri...4 Lplce's epnsion...7 Simultneous Liner Systems... 8 Inverse mtri... 8 Round-off error...9 Homogeneous nd Singulr Liner Systems... Prmetric form... Rnk nd Suspce...4 Generl Cse - Rouché-Cpelli Theorem... 6 Homogeneous System Cses...7 Non Homogeneous System Cses...8 Tringulr Liner Systems... 9 Tringulr fctoriztion...9 Forwrd nd Bckwrd sustitutions...9 LU fctoriztion...4 Overdetermined Liner System... 4 The norml eqution...4 QR decomposition...44 SVD nd the pseudo-inverse mtri...45 Underdetermined Liner System Prmetric Liner System Crmer's rule...49 Block-Tringulr Form... 5 Liner system solving...5

4 TUTORIAL FOR MATRIX.XLA Computing the determinnt...5 Permuttions...5 Eigenvlue Prolems...5 Severl kinds of lock-tringulr form...5 Permuttion mtrices...5 Mtri Flow-Grph...5 The score-lgorithm...5 The Shortest Pth lgorithm...59 Limits in mtri computtion... 6 Sprse Liner Systems... 6 Filling fctor nd mtri dimension...6 The dominnce fctor...6 Algorithms for sprse systems...6 Sprse Mtri Genertor...65 How to solve sprse liner systems...66 How to solve tridigonl systems...7 Eigen-prolems... 7 Eigenvlues nd Eigenvectors... 7 Chrcteristic Polynomil... 7 Roots of the chrcteristic polynomil...74 Cse of symmetric mtri...74 Emple How to check the Cyley-Hmilton theorem...76 Eigenvectors Step-y-step method...77 Emple - Simple eigenvlues...77 Emple - How to check n eigenvector...78 Emple - Eigenvlues with multiplicity...79 Emple - Eigenvlues with multiplicity not corresponding to the numer of eigenvectors...8 Emple - Comple Eigenvlues...8 Emple - Comple Mtri...8 Emple - How to check comple eigenvector...8 Similrity Trnsformtion Fctoriztion methods...85 Eigen prolems versus resolution methods Jcoi trnsformtion of symmetric mtri...86 Orthogonl mtrices...88 Eigenvlues with the QR fctoriztion method...89 Rel nd comple eigenvlues with the QR method...9 Comple eigenvlues of comple mtri with the QR method...9 How to test comple eigenvlues...9 How to find polynomil roots with eigenvlues...94 Rootfinder with QR lgorithm for rel nd comple polynomils...94 The power method...95 Eigensystems with the power method...98 Comple Eigensystems...99 How to vlidte n eigen system... How to generte rndom symmetric mtri with given eigenvlues... Eigenvlues of tridigonl mtri... Eigenvlues of tridigonl Toeplitz mtri )... Generlized eigen prolem...8 Equivlent symmetric prolem...8 Equivlent symmetric prolem...9 Digonl mtri... References...6

5 4

6 Chpter Aout this tutoril Aout this Tutoril Mtri.l Mtri.l is n Ecel dd-in tht contins useful functions nd mcros for mtri nd liner Alger: Norm. Mtri multipliction. Similrity trnsformtion. Determinnt. Inverse. Power. Trce. Sclr Product. Vector Product. Eigenvlues nd Eigenvectors of symmetric mtri with Jcoi lgorithm. Jcoi's rottion mtri. Eigenvlues with QR nd QL lgorithm. Chrcteristic polynomil. Polynomil roots with QR lgorithm. Eigenvectors for rel nd comple mtrices Genertion of rndom mtri with given eigenvlues nd rndom mtri with given Rnk or Determinnt. Genertion of useful mtri: Hilert's, Houseolder's, Trtgli's, Vndermonde's Liner System. Liner System with itertive methods: Guss-Seidel nd Jcoi lgorithms. Guss Jordn lgorithm step y step. Singulr Liner System. Liner Trnsformtion. Grm-Schmidt's Orthogonliztion. Mtri fctoriztions: LU, QR, QH, SVD nd Cholesky decomposition. The min purpose of this document is to show how to work with mtrices nd vectors in Ecel, nd how to use mtrices for solving liner systems. This tutoril is written with the im to tech how to use the Mtri.l functions nd mcros. Of course it speks out mth nd liner lger, ut this is not mth ook. You rrely find here theorems nd demonstrtions. You cn find, on the contrry, mny emples tht eplin, step y step, how to rech the result tht you need. Just stright nd esy. And, of course, we spek out Microsoft Ecel ut this is not tutoril for Ecel. Tips nd tricks for Ecel cn e found ion mny Internet sites. This tutoril is divided into two prts. The first prt is the reference mnul of Mtri.l. The second prt eplins with prcticl emples how to solve sic topics out mtri theory nd liner lger. 5

7 Chpter Liner Systems This chpter eplins how to solve liner systems of equtions with the id of mny emples. They cover the mjor prt of cses: systems with single s well s with, infinitely mny solutions, or none t ll. Severl lgorithms re shown: Guss-Jordn, Crout's LU fctoriztion, SVD Liner System Emple. Solve the following 44 liner system A Where A nd re: Squre mtri. If the numer of unknowns nd the numer of equtions re the sme, the system hs surely one solution if the determinnt of the mtri A is not zero. Tht is, if A is non-singulr. In tht cse we cn solve the prolem with the SysLin function. The determinnt cn e computed with the MDet function or with Ecel's uilt-in function MDETERM. 6

8 The Guss-Jordn lgorithm The Guss nd Guss-Jordn lgorithms re proly the most populr pproches for solving liner systems. Functions SysLin nd SysLinSing of Mtri.l use this method with pivoting strtegy. Ancient, solid, efficient nd - lst ut not lest - elegnt. The min gol of this lgorithm is to reduce the mtri A of the system A to tringulr or digonl mtri with ll digonl elements equl to y using few row opertions: liner comintion, normliztion, nd echnge. Let's see how it works Emple: The following system hs the solution ; ;, s you cn verify it y direct sustitution Let's egin to uild the complete mtri (4) with the mtri coefficients nd the constnt vector (gry) s shown on the right. Our gol is to reduce the mtri coefficients to the identity mtri. Choose the first digonl element ; it is clled the "pivot" element. Normliztion step: if pivot nd pivot then divide ll first row elements y the vlue of the pivot, 4.. Liner comintion: if then sustitute the second row with the difference etween the second row itself nd the first row multiplied y Liner comintion: if then sustitute the second row with the difference etween the second row itself nd the first row As we cn see the first column hs ll zeros ecept for the digonl element, which is. Repeting the process for the second column - with pivot - nd for the third column - with pivot - we will perform the mtri "digonliztion"; the lst column will contin, t the end, the solution of the given system In Ecel, we cn perform these tsks y using the power of rry functions. Below is n emple of the resolution of system y the Guss-Jordn lgorithm Note tht ll the rows re otined y rry opertions {...}. You must insert them with the CTRL+SHIFT+ENTER key sequence. Properly clled Guss lgorithm Properly clled Guss-Jordn lgorithm 7

9 We see in the lst column the solution (- ; ; ). The formuls used for computing ech row re shown on the right Swp rows If one pivot is zero we cnnot normlize the corresponding row. In tht cse we will swp the row with nother row tht hs no zero in the sme position. This opertion does not ffect the finl solution t ll; it is equivlent to reordering the lgeric equtions of the given system Emple: The following system hs the solution 5 ; ; 7. Note tht the first pivot,, is zero. We cnnot normlize this row In this cse we swp the first row with the second one. Now the new pivot is nd the normliztion cn e done. Note tht the second row now hs the element, so we simply leve tht row unchnged. The liner comintion isn't needed in this cse The pivoting strtegy Pivoting cn lwys e performed. In the ove emple we hve echnged one zero pivot with ny other non-zero pivot in order to continue the Guss lgorithm. But there is nother reson for which the pivoting method is dopted: to minimize round off errors. Pivoting cn reduce round off errors The Gussin elimintion lgorithm cn hve lrge numer of opertions. If we count the opertions for the resolution of one system of n simultneous equtions, we will discover tht it requires of the order of n / computer opertions, i.e.., dditions, sutrctions, multiplictions, nd divisions. So, if the numer of equtions nd unknowns doules, the numer of opertions increses y fctor of 8. If n, then there re more thn two million opertions! Certinly, one might egin to worry out the ccumultion of round off errors. One method to reduce such round off errors is to void division y smll numers, nd this is known s row pivoting or prtil pivoting, the strtegy of the Gussin elimintion lgorithm. 8

10 Let's see the following remrkle emple of system Its solution is (, ) (, ), s we cn esily verify y sustitution If we pply the Guss-Jordn lgorithm, with numericl precision of 5 digits, we hve: The pivot The solution hs n error of out E-7 On the contrry, if we simply echnge the order of the lgeric equtions, so tht the second row ecomes the first one, we hve Pivot >> The solution is now much etter, with n error of less thn E-5 As we cn see, this little trick cn improve the generl ccurcy. The stndrd Guss-Jordn lgorithm lwys serches the element elow it for the mimum solute vlue, to e used s pivot. If tht mimum vlue is greter then tht of the current pivot, then the row of the pivot nd the row of the mimum vlue re echnged. Not ll elements cn e used s pivot echnge. In the mtri to the right we could use s pivot only the element, 4, 5, 6 (yellow cells). For emple: if 6 m(, 4, 5, 6 ) then rows 6 nd re swpped, nd the old element 6 ecomes the new pivot

11 Full Pivoting In order to etend the re in which to serch for mimum pivot we could echnge rows nd columns. But when we swp two columns, the corresponding unknown vriles re lso echnged. So, in order to reuild the finl solution in the originl given sequence, we hve to perform ll the permuttions, in reverse order, tht we hve mde. This mkes the finl lgorithm it more complicted, ecuse we now hve to store ll columns permuttions. The full pivoting method etends the serch re for the mimum vlue For emple, if the pivot is element, then the lgorithm serches for the solute miml vlue in the yellow re elow nd to the right of. If mimum vlue is found t 56, then rows 5 nd re swpped nd, therefter, columns 5 nd re echnged The functions SysLin nd SysLinSing of Mtri.l use the Guss-Jordn lgorithm with full pivoting strtegy Integer clcultion In the ove emples we hve seen tht the Guss elimintion steps introduce non-integer numers - nd round off errors -, even if the solutions nd coefficients of the system re integers. Is there wy to void such deciml round off errors nd preserve the glol ccurcy? The nswer is yes, ut in generl, only for integer mtrices. This method - vrint of the originl Guss-Jordn pproch - is very similr to the one tht is sometimes performed mnully y students. It is sed on the "minimum common multiple" MCM (lso LCM Lest Common Multiple) nd it is conceptully very simple Assume tht we hve the following two rows: the pivot row, nd the row tht hs to e reduced. Pivot is - 6 The element to set zero is 4 mcm MCM(6, 4) Multiply the first row y mcm / /4 And the second row y mcm / /( 6) < pivot row; multiply for 4 < for reducing; multiply for - 8 now, dd the two rows < the first row remins unchnged 9 48 < dd the first row to the second row In this wy we cn reduce row only using integer numers Let's see how it works, step y step, in the function GJ_setp of Mtri.l.

12 Note the rd prmeter setting the integer lgorithm. If "flse", the opertions will use stndrd floting point opertions using rel (i.e., not necessrily integer) numers. Only the lst step cn introduce deciml numers; the previous steps re lwys ect. Unfortuntely, this method cnnot e dopted in generl, even for mtrices contining only integers, ecuse the vlues grow t ech step nd cn ecome lrge enough to cuse overflow The function used re: {GJstep(A5:D7,,True)} inserted in the rnge A9:D {GJstep(A9:D,,True)} inserted in the rnge A:D5 {GJstep(A:D5,,True)} inserted in the rnge A7:D9 {GJstep(A7:D9,,True)} inserted in the rnge A:D {GJstep(A:D,,True)} inserted in the rnge A5:D7 The ove emple cn e quickly reproduced. After inserting the function in the Tip rnge A9:D, give the CTRL+C commnd to copy the rnge still selected; highlight cell A nd pste the new mtri with the instruction CTRL+V. Repet this simple step still you rech the finl identity mtri. The sought solution will e in the lst column. This sequence shows how to do it. Given complete system mtri in rnge B:E4, select the rnge A6:E8, just elow the given mtri (leving free row for seprtion) Insert the rry function GJstep with the CTRL+SHIFT+ENTER key sequence nd the given prmeter You should see the modified mtri fter the first step. Leve the selected rnge nd give the copy commnd (CTRL+C) Select cell B, under the st step mtri. Mke sure tht the rnge elow is empty. Now, simply give the pste commnd (CTRL+V) nd the nd step mtri will pper Repeting the ove steps you cn get ll the intermedite Guss-Jordn mtrices, either in floting or in integer mode (t your option).

13 Severl wys to use the Guss-Jordn lgorithm The mtri reduction method cn e used in severl wys. Here re some sic cses: Solving non-singulr liner system A The complete mtri ( 4) is At the end, the lst column is the solution of the given system; the originl mtri A is trnsformed into the identity mtri. Solving m simultneous liner systems AX B... m m m... m m m The complete mtri ( +m) is:... m m m... m m m At the end, the solutions of the m system re found in the lst m columns of the complete mtri Inverse mtri computing This prolem is similr to the one ove, ecept tht the mtri B is the identity mtri. In fct, y definition: A A I AX I X A AX I The complete ( 6) mtri is:

14 A At the end, the inverse mtri is found in the lst columns of the complete mtri Determinnt computing For this prolem we need only reduce the given mtri to the tringulr form. A t fter which the determinnt cn e computed redily s t t t t t Det( A) t t t Liner independence checking A linerly independent set of vectors S {v, v...v n } is this in which no vector is comintion of the others. Guss lgorithm cn evidence how mny liner dependent vectors there re in given set. For tht simply perform the tringulriztion of the mtri in which the columns re the vectors of the set. v v v v 4 v v, v v 4 v v, v v 4 v v, v v v v v v 4 v v v v 4 v v v v 4 v v v v The numer of zero rows t the ottom of the tringulrized mtri coincides with the linerly dependent vectors: i.e., one zero row, one dependent vector; two zero rows, two dependent vectors, nd so on. Of course, no zero row mens tht the columns of the mtri form linerly independent set.

15 Non-singulr Liner system The function SysLin finds the solution of non-singulr liner system using the Guss-Jordn lgorithm with full pivoting strtegy. Emple: solve the following mtri eqution The solution is A () A - () You cn get the numericl solution in two different wys. The first is the direct ppliction of the formul (); the second is the resolution of the simultneous liner system () Emple: Find the solution of the liner system hving the following A (6 6) nd (6 ) We solve this liner system with oth methods: y using Ecel's MINVERSE nd our SysLin function. In oth cses we find the unitry solution (,,,,, ) (Note tht the lgeric sum of terms in ech row is equl to the corresponding constnt term ) Note lso tht the methods give similr - ut not equl - results, ecuse their lgorithms re different. In this cse oth solutions re very ccurte ( E-5) ut this is not lwys true. Round-off errors Sometimes, the round-off errors decrese the otinle mimum ccurcy Look t the following system: The ect solution is, gin, the unitry solution (,,,,, ). In order to mesure the error, we use the following formul ABS(-ROUND(, )) where is one pproimte solution vlue 4

16 The totl error is clculted with AVERAGE(H:H6) totl error for SysLin function AVERAGE(J:J6) totl error for MINVERSE function As we cn see the totl errors of these solutions re more thn thousnd times greter tht tht in the previous emple. Sometimes, round-off errors re so lrge tht they cn give totlly wrong results. Look t this emple. A As we cn esily see y inspection, the mtri is singulr ecuse the first nd lst columns re equl. So there is no solution for this system. But if you try to solve this system with the MINVERSE function you will get totlly different (nd clerly wrong) result. This error is prticulrly sneky ecuse, if we try to compute the determinnt, we lso get wrong, nonzero result MDETERM(A) -.8 As the lgorithms used y Ecel nd Mtri.l re not equl, we cn compute n lterntive solution with SysLin nd the determinnt with MDet. In this cse, the full pivot strtegy of Guss-Jordn is used, nd gives us the right nswer. Full pivoting or prtil pivoting? The full-pivoting strtegy reduces the round-off errors, so we might epect tht its ccurcy is greter thn with prtil-pivoting method. But this is not lwys true. Sometimes it cn 5

17 hppen tht the full strtegy gives n error similr to or even greter thn the one otined y the prtil strtegy. In Mtri.l we cn perform the prtil Guss-Jordn lgorithm using the didctic function GJstep. Emple: Solve the following liner system. The mtri is the inverse of the 66 Trtgli mtri. The ect system solution is the vector [,,, 4, 5, 6] A Let's see how oth lgorithms - full nd prtil pivoting - work. As we cn see, in tht prolem, prtil pivoting is somewht more ccurte thn full pivoting. Why, then, complicte the lgorithm with full pivoting? The reson is tht the Guss-Jordn method, with full pivoting, is generlly more stle for lrge vriety of mtrices. Moreover, its round-off error control is more efficient. And the frequency of ctstrophic mistkes, such s in the erlier comprison of MINVERSE nd SysLin, is gretly reduced with full-pivoting strtegy. Look t this emple: Solve the following system A Solving with the Guss-Jordn lgorithm with either prtil or full pivoting we note in this cse loss of ccurcy of more thn thousnd times for prtil pivoting. Note tht, in these prolems, we hve not inserted the results given y Ecel's MINVERSE function, ecuse we will ignore tht lgorithm: in long series of testes, we hve found tht its results resemle those otined y prtilly pivoting lgorithm). 6

18 We cn oserve tht, in generl, prtil pivoting ecomes inefficient for mtrices tht hve lrge vlues in their right side. In tht cse, the round-off errors grow shrply. Full pivoting voids this rre - ut hevy - loss of ccurcy. Solution stility Sometimes, coefficients of liner system cnnot e known ectly. Often, they derive from eperimentl results, nd cn therefore e ffected y eperimentl errors. We re interested in investigting how the system solution chnges with such errors. Mny importnt studies hve demonstrted tht the solution ehvior depends on the mtri of system coefficients. Some mtrices tend to mplify the errors of the coefficients or the constnt terms, so the solution will e very different from tht of the "ect" system. When this hppens we cll it n "ill-conditioned" or "unstle" liner system. Emple: show tht the following liner system, with the Wilson mtri, is very unstle The solution of the ect system is (,,,). Now give some perturtions to the constnt terms. For simplicity we give ' + with. The solution of the pertured system is now A ' ' ' + Defining the system sensitivity coefficient s S ( %) / ( %) ( / ) / ( / ) We find S 4. 7

19 A high vlue of S mens high instility. In fct in this system, for smll perturtion of out.% of the constnt terms, we hve the solution -.,,.5,., which is completely different from the ect one,,,, Note tht Det Even worse stility is found in the following liner system A For very smll perturtion of out.% of the constnt term, the system solution vlues re moved fr wy from the point (,,, ) Note the very high sensitivity coefficient S of this prolem, nd the wide spred of the solution point, even for very smll perturtions. Note lso tht, in oth prolems, the determinnt ws unitry (Det ). So we cnnot discover the instility simply y considering the determinnt. 8

20 The Condition Numer One populr mesure of instility mtri uses its eigenvlues S λ λ m / λ min But, unfortuntely, eigenvlues re not very esy to compute A more prcticl inde is sed on the singulr vlue decomposition (see the SVDD ) function). Etrcting the lrgest nd smllest singulr vlues of the digonl mtri D we define the mesure of instility, commonly clled the condition numer, s: κ d m / d min For the ove mtri the eigenvlues re So we hve: λ S λ 4. / while the SVD gives κ 4.9 / It is lso possile to compute directly the condition numer with MCond nd MpCond functions Aprt theory, the condition numer hs useful, prcticl mening: in system solving, it indictes how mny significnt digits will e lost. See this emple Tking the lst Wilson system, we pertur the vector with smll rndom error: ' ( + ε R) where ε E- nd R is uniform rndom vrile etween nd. i i For ech set of vlues, we register the verge error of the solution otined with the formul A - As we cn see the verge error of is out E-7, just 6 digits less then the precision of the vector. The precision lekge roughly corresponds to the deciml log of condition numer pκ -log (κ) -6 9

21 Comple systems Comple systems re very common in pplied science. Mtri.l hs dedicted function SysLinC to solve them. We shll lern how it works with prcticl emple from Network theory. Emple - Anlysis of lttice network. For ll nodes, find the voltges nd phse ngles, t the frequencies: f, 5,, nd 4 Hz. R R R Components vlues G C C C R Ω C.5 µf R Ω C. µf R Ω C. µf G.5 sin(π f t) We use the nottion jω t v ( t) V sin( ω t + θ ) V Ve Vre + The Nodl Anlysis provides the solution through the following comple mtri eqution [ ] I Y V () where the rel mtricies G nd B re clled conductnce nd susceptnce respectively; they form the comple dmittnce mtri Y. These mtrices depend on the ngulr frequency ω π f Using the worksheet, the prolem cn e solved y clculting, first of ll, the frequency ω, the two rel mtrices G nd B, nd the input current vector. Then, we uild the comple system (). jv im SysLinC provides the vector solution in comple form; to convert it into mgnitude (modulo) nd phse we hve used the formuls V im ( V ) + ( ) re Vim, V tn Vre Note tht we hve dded n imginry column to the current vector, even though the input currents re purely rel quntities. Comple mtrices nd comple vectors must e lwys e specified with oth their rel nd imginry prts. Consequently they must lwys hve n even numers of columns. In the ove emple there re mny Ecel formuls tht we couldn't shown for clrity. To more fully eplin the emple, copy the following formuls (in lue) in your worksheet. V

22 See lso the function MAdm for dmittnce mtri. Emple - Solve the following comple system j z y z j y j z y j j ) ( ) ( ) ( The system is equivlent to the following comple mtri eqution + + j j z y j j j ) ( With the SysLinC function it is simple to find the solution of such comple mtri system. We hve only to seprte the rel nd imginry prts.

23 Aout the comple mtri formt Mtri.l supports different comple mtri formts: ) split, ) interlced, ) string ) Split formt ) Interlced formt ) String formt Ech formt hs its dvntges nd drwcks. In the split formt the comple mtri [ Z ] is split into two seprte mtrices: the first one contins the rel vlues, nd the second the imginry vlues. This is the defult formt In the interlced formt, ech comple vlue is written in two djcent cells, so tht single mtri element occupies two cells. The numer of columns is the sme s in the first formt, ut the vlues re interlced: one rel column is followed y n imginry column nd so on. This formt is useful when elements re returned y comple functions s, e.g, y the Xnumers.l dd-in The lst formt is the well known comple rectngulr formt. Ech element is written s string "+i" so tht squre mtri is still squre. It ppers to e the most compct nd intuitive formt, ut this is true only for integer vlues. For long deciml vlues the mtri elements ecome illegile. We should lso point out tht the elements, eing strings, cnnot e formted s other Ecel numers, or even used in susequent computtions without conversion from tet strings to numers.

24 Determinnt In contrst to the solution of liner system, the mtri determinnt chnges with the reduction opertions of the Guss-Jordn lgorithm. In fct the finl reduced mtri is the identity mtri tht hs lwys determinnt. But the determinnt of the originl mtri cn e computed with the following simple rules When we multiply mtri row y numer k, the determinnt must e multiplied y the sme numer When we echnge two rows, the determinnt chnges its sign ut retins its mgnitude Gussin elimintion With these simple rules it is esy to clculte the mtri determinnt. It is sufficient to trck of ll pivot multiplictions nd rows swppings performed during the Guss-Jordn process There lso nother rule, useful to reduce the computing effort. A tringulr mtri nd digonl mtri with the sme digonl hve the sme determinnt So, in order to compute the determinnt, we cn reduce the given mtri to tringulr mtri insted of digonl one, sving hlf of the computtion effort. This is clled the Guss lgorithm or Gussin elimintion. The determinnt of digonl mtri is the product of ll elements det( A) det The determinnt of tringulr mtri is the product of ll elements And: det( A) det det( A) det The emple elow shows how to compute, step y step, the determinnt with the Guss lgorithm 4 A - - Det(A)? -

25 4 R R + *R (*) A - Det(A) Det(A) - 4 R R + ( 4)*R A - Det(A) -8 Det(A) (*) The formul R R + *R is compct wy for descriing the following opertions: ) Multiply the nd row for. ) Add the nd row nd the st row ) sustitute the result to the nd row 4 A 9-8 < swp Det(A) 8 Det(A) - < swp 4 R R + *R A4 9-8 Det(A4) 4 Det(A) - 4 Det(A4) 4 Det(A) A4 9-8 Det(A4) 4*9*(-) -7 - The finl mtri A4 is tringulr. So its determinnt is redily computed s -7 But it is lso: Det(A 4 ) 4 Det(A) Sustituting, we hve: -7 4 Det(A) Det(A) -4 / 7 - Ill-conditioned mtri Of course there re functions such s MDet in Mtri.l nd MDETERM in Ecel to compute the determinnt of ny squre mtri. Both re very fst nd efficient, covering most cses. But, sometimes, they cn fil ecuse of the round-off error introduced y the finite precision of the computer. It usully hppens for lrge mtrices ut, sometimes, even for smll mtrices. Look t this emple. Compute the determinnt of this simple ( ) mtri Both functions return very smll, ut non-zero vlue, quite different from ech other. If you repet the clcultion with nother numericl routine in it operting system you will get similr results. The given mtri is singulr nd its determinnt is. We cn esily verify this y hnd with ect frctionl numers, or y using the GJstep function with integer lgorithm, s shown 4

26 elow < swp < swp Det(A) - Det(A) Det(A) 57 Det(A) < swp < swp Det(A) Det(A) Det(A4) 5749 Det(A) Det(A5) -554 Det(A) Det(A6) Det(A) The lst row is ll zero. This mens tht the mtri is singulr nd its determinnt is zero. Det(A6) > Det(A) In this cse it ws esy to nlyze the mtri, ut for lrger mtri do you know wht would hppen? Before one ccepts ny results - especilly for lrge mtrices, one hs to perform some etr tests, such s the singulr vlue decomposition. Emple. Compute the determinnt of the following ( ) sprse mtri Select the ove mtri nd pste it in worksheet strting from the cell A. Using the Ecel 5

27 MINVERSE we get the determinnt of out Det(A).459 If we try to chnge one element y little mount - for emple, the element K5 from. to. - we get complete different result Det(A) -.. Note tht the determinnt even chnges its sign. It would e sufficient for suspecting of lrge round-off error. In fct, if we compute the determinnt y the function MDet (tht uses the Guss lgorithm with full-pivot) with hve the result Det(A). This mens tht the mtri is singulr. We cn check tht result with the SVD lgorithm. Using the function SVDD we get the singulr vlue mtri: the minimum vlue, less then E-5, fully confirms tht the mtri is singulr. 6

28 Lplce's epnsion Epnsion y minors is nother technique for computing the determinnt of given squre mtri. Although efficient for smll mtrices (prcticlly for n, ), techniques such s Gussin elimintion re much more efficient when the mtri ecomes lrge. Lplce's epnsion ecomes competitive when there re rows or columns with mny zeros. The epnsion formul is pplied to ny row or column of the mtri. The choice is ritrry. For emple, the epnsion long the first row of mtri ecomes. n (+ j) A ( ) A j j A A + A j where A ij re the minors, tht is the determinnt of the su mtri etrcted from the originl mtri eliminting the row i nd the column j. The minors re tken with sign + if the sum of (i+j) is even; or with the minus sign if (I+j) is odd. Mny uthors cll the term: (-) (i+j) A ij cofctor. Let's see how it works with n emple Emple - Clculte the determinnt of the given mtri with the Lplce s epnsion We use the function MEtrct to get the minor su mtri; we use lso the INDEX function to get the ij element Completing the worksheet with the other minors nd the cofctor terms we hve Tip. We cn use the row (or column) epnsion in order to minimize the computing effort. Usully we choose the row or column with the lrgest numer of zeros (if ny). 7

29 Simultneous Liner Systems The function SysLin cn give solutions for mny liner systems hving the sme incomplete coefficient mtri nd different constnt vectors. Emple: solve the following mtri eqution Where: A X B () A B The solution is X A - B () You cn get the numericl solution in two different wys. The first is the direct ppliction of the formul (); the second is the resolution of the simultneous liner system () From the point of view of ccurcy, oth methods re sustntilly the sme; in terms of efficiency, the second is etter, especilly for lrge mtrices Inverse mtri Simultneous systems solving is used to find the inverse of mtri. In fct, if B is the identity mtri, we hve: A X I X A - I A - You hve the function MINVERSE in Ecel or the function MInv in Mtri.l to invert squre mtri. Emple: find the inverse of the 4 4 Hilert mtri Hilert mtrices re known clss of ill-conditioned mtrices, very esy to generte: (i, j) /(i+j -) / / /4 / / /4 /5 / /4 /5 /6 /4 /5 /6 /7 The inverse of Hilert mtri is lwys integer. So, if ny decimls pper in the result, we cn e sure tht they re due to round off errors, nd we cn consequently estimte the ccurcy of the result. You cn esily generte these mtrices y hnd or with the function MHilert 8

30 Round-off error As you cn see, Ecel hides the round-off error nd the result seems to e ect. But this is not true. In order to show the error without formtting the cells with or more decimls we cn use this simple trick:: etrct only the round-off error from ech ij vlue y the following formul: Error ROUND( ij, ) - ij Applying this method to the ove inverse mtri, we see tht there re solute round-off errors from E- to E-. There is nother method to estimte the ccurcy of the inverse mtri: multiplying the given mtri y its pproimte inverse we get "ner" identity mtri. The off-digonl vlues mesure the errors. If we compute the men of their solute vlues we hve n estimtion of the round-off error. The "digonliztion" ccurcy mesures the glol error due to the following three step: Glol error Input mtri error + Inversion + multipliction The first step needs n eplntion. Ecel cn show frctionl numer s ect s, e.g., / or /7. But, ctully, these numers re lwys ffected y trunction errors of out E-5. Other clsses of mtrices, such s Trtgli's mtrices, void the input trunction errors, ecuse they re lwys integer. 9

31 Trtgli's mtrices Trtgli's mtrices re very useful ecuse they re esy to generte ut - this is very importnt - the mtri nd its inverse re lwys integer. This comes in hndy for mesuring round-off errors. Trtgli mtrices re defined s Here is 66 Trtgli mtri j for j...n (ll in the first row) i for i...n (ll in the first column) ij Σ j (i-) j for j.. n nd its inverse As we cn see, oth mtrices re integer. Any errors in the inverse mtri must e regrded s round-off errors, nd re immeditely detected. In the emple elow we evlute the glol ccurcy of the inverse of the 6 6 Trtgli mtri with two different functions Ecel occsionlly compute A - even if mtri is singulr. If this hppens, your solution will e wrong. Let's see this emple: Emple: find the inverse of the following mtri As we hve seen in previous emple, the given mtri is singulr. So, its inverse doesn't eist. However, if we try to compute the inverse we hve the following result

32 Tip. You should lwys emine the determinnt efore ttempting to clculte the inverse. If the determinnt is close to zero, you should try to verify the solution with other methods. For instnce, you cn lwys try to solve the inverse y the function MInv (in this cse, with the integere option), or y GJstep function, or with SVD (see lter). How to void decimls An inverse mtri is not lwys integer; usully it contins decimls. If the given mtri is integer, we cn otin the frctionl epression of its inverse with this little trick Emple Note the compct formt of mtri multipliction y sclr {A6*E:G4} in the lst mtri instruction Multiplying the inverse y the determinnt we get mtri B of integer vlues. Thus, the inverse cn e put in the following frctionl form A B det( A)

33 Homogeneous nd Singulr Liner Systems A liner system A with A n (n m) mtri nd with we cll homogeneous liner system.. Such system lwys hs the trivil solution. But we re interested in knowing if the system lso hs other solutions. Assume A is squre mtri of the following system We note tht the lst row cn e otined y multiplying the first row y -. So, hving two rows tht re linerly dependent, the given mtri is singulr, with zero determinnt. One of the two rows cn e eliminted; we choose to eliminte the lst row, nd otin the following system The system of liner equtions () epresses ll the solutions of the given system, n infinite numer of them. Geometriclly speking it is line in the spce R It cn e lso e regrded s liner trnsformtion tht moves generic point P(, y, z) of the spce into nother point P'(, y, z) of the suspce. In this cse, the suspce is line, nd the dimension of this suspce is R z y z y 7 ' If we ssume, on the contrry, s independent prmeter, the other vriles y, z cn e epressed s functions of the "independent" prmeter Tht is represented y the liner trnsformtion t the right z y z y 7 7 ' This mtri trnsformtion is useful for finding the prmetric form of the liner function (mpping function) z y z y z y z y z y One of the three vriles cn e freely chosen nd it cn e regrded s new independent vrile. Assume, for emple, z s the independent prmeter; the other vriles, y cn then e epressed s function of the "independent" prmeter z + + z y z y 5 4 z y z 7 () + z y z y 5 4 z y 7 7

34 Prmetric form The liner trnsformtions of the ove emple give reltions etween points in spce. A common form for hndling this reltion is the prmetric form. It is esy to pss from the trnsformtion mtri to its prmetric form ' Hving the trnsformtion mtri, we 7 7 serch for the vrile tht hs in t y y the digonl element, z in this cse. y t z z Setting z t, nd performing the multipliction, we hve the prmetric z t function Geometriclly specking the prmetric function is line with the direction vector: D r 7 D r () () Note tht 6 is the norm of the first vector You cn study the entire prolem with the function SysLinSing of Mtri.l. Here is n emple: SysLinSing solves singulr liner system, returning the trnsformtion mtri of the solution, if one eists. The determinnt is clculted only to show tht the given mtri is singulr. It is not used in the clcultion. SysLinSing utomticlly detects if mtri is singulr or not. If the mtri is not singulr (Det ), the function returns ll zeros. From the trnsformtion mtri we cn etrct the direction vector y normliztion of the third column of mtri B. To get the norm of the vector we hve used the function MAs. Note tht oth epression must e inserted s rry functions { } In D spce, the function represents line pssing trough the origin, hving for direction the vector D, s shown in the figure.

35 Rnk nd Suspce In the ove emple we hve seen tht, if the mtri of homogeneous system is singulr, then there re n infinite numer of solutions of the system; those solutions represent suspce. After tht we hve found solution, nd we hve seen tht the suspce ws line nd its dimension ws. Is there wy to know the dimension of the suspce without resolving the system? The nswer is yes, knowing the rnk of the mtrices. But we hve to sy tht this is esy only for low mtri dimensions; it ecomes very difficult for high mtri dimensions. The rnk of squre mtri is the mimum numer of independent rows (or columns) tht we cn find in the mtri. For mtri the possile cses re collected in the following tle Independent rows Rnk Liner System Solution Suspce Null Line Plne The function MRnk of Mtri.l clcultes the rnk of given mtri. In the following emple we clculte the determinnt nd the rnk of three different mtrices Note tht the determinnt is lwys when the rnk is less then the mtri dimension n. Solving homogeneous systems with the given mtrices, we will generte in D spce respectively the following suspces: null spce, line, nd plne. Let's test the lst mtri, solving its homogeneous system. Consequently, the trnsformtion mtri hs two columns, indicting tht the suspce hs dimensions, thus is plne. In order to get the prmetric form of the plne we oserve tht the trnsform mtri: vriles y nd z hve oth the digonl element (, ). These cn e ssumed to e independent prmeters. 4

36 Let y t nd z s, then we hve t + s y y y t z z z s Eliminting oth prmeters we get the norml equtions of the plne y + z + y z The liner eqution () epress ll the infinite solutions of the given system. Geometriclly speking it is plne in the spce R () Rnk for rectngulr mtri Differently form the determinnt, the rnk cn e computed lso for non-squre mtri. Emple: find the rnk of the following 5 mtri By inspection we see tht there re independent rows nd independent columns. In fct, column c is otined multiplying the first column y ; column c4 c + c; nd column c5 c c. So the rnk is: rnk One populr theorem - due to Kronecker - sys tht if the rnk r, then ll the squre sumtrices (p p) etrcted from the given mtri, hving p > r, re ll singulr In other words: ll mtrices etrcted from the mtri in the ove emple hve determinnt. You cn enjoy finding yourself ll the mtrices of dimensions. Here re 5 of them

37 Generl Cse - Rouché-Cpelli Theorem Given liner system of m equtions nd n unknowns n n n m + m mn... n... n A( m n)... m m... mn m () The mtri A is clled the coefficient mtri or incomplete mtri... n +... n B( m, n )... m m... mn The mtri B is clled complete mtri or ugmented mtri If the column only contins zeros, the system is clled homogeneous In order to know if the system () hs solutions, the following, fundmentl theorem is useful ROUCHÈ-CAPELLI THEOREM : A liner system hs solutions if, nd only if, the rnks of mtrices A nd B re equl Tht is: rnk(a) rnk(b) solution Among rnks, numer of equtions nd numer of unknowns eist importnt reltions. The following tle reviews possile cses: 6 for homogeneous systems, nd 6 for full system.. 6

38 Homogeneous System Cses 4 Cse Rnk of incomplete mtri A Non homogeneous system solution Emple rnk(a) m n Trivil solution (,,..) + y y S(,) rnk(a) m <n n-m solutions + trivil solution rnk(a)< m <n n-r solutions + trivil solution 4 rnk(a)< m n n-r solutions + trivil solution 5 rnk(a) n <m Trivil solution (,,..) + y y y S(,) 6 rnk(a)<n <m n-r solutions + trivil solution 4 This tle, very cler nd well-orgnized, is due to Mrcello Pedone 7

39 Non Homogeneous System Cses Cse Rnk of incomplete mtri A Non homogeneous system solution Emple + y rnk(a) m n One solution S(,) y rnk(a) m <n n-m solutions rnk(a)< m <n n-r solutions If r(b)r(a) 4 rnk(a)< m n n-r solutions se r(b)r(a) y ) 4y y ) 4y S ( y, y) ( soluzioni) incomptiile r( A) r( B) 5 rnk(a) n <m One solution If r(b)r(a) 6 rnk(a)<n <m n-r solutions If r(b)r(a) 8

40 Tringulr Liner Systems Solving tringulr liner system is simple, nd very efficient lgorithms eist for this tsk. Therefore, mny methods try to decompose the full system into one or two tringulr systems y fctoriztion lgorithms. Tringulr fctoriztion Suppose tht, for the liner system A () you hve gotten the following fctoriztion A LU () where L is lower-tringulr nd U upper-tringulr. Tht is: α α α α 4 α α α 4 α α 4 β α 44 β β β β β β β β β In tht cse, we cn split the liner system () into two systems: A (LU) L (U ) Setting: y U we cn write: L y () U y (4) The tringulr systems () nd (4) cn now e solved with very efficient lgorithms Forwrd nd Bckwrd sustitutions The method proceeds in two steps: t the first, it solves the lower-tringulr system () with the forwrd-sustitution lgorithm; then, with the vector y used s constnt terms, it solves the uppertringulr system (4) with the ck-sustitutions lgorithm. Both lgorithms re very fst. Let' see how it works Hving the following fctoriztion LU A, solve the liner system A A L R In Mtri.l we cn use the function SysLinTtht pplies the efficient forwrd/ckwrd lgorithm to solve tringulr systems. This function hs n optionl prmeter to switch the lgorithm to the upper (Typ "U") or lower (Typ "L") tringulr mtri. If omitted, the function utomticlly finds the mtri type 9

41 The originl system is roken into two tringulr systems A L y U y We cn prove tht the vector (,, ) is the solution of the originl system A LU fctoriztion This method, sed on Crout's fctoriztion lgorithm, splits squre mtri into two tringulr mtrices. This is very efficient nd populr method to solve liner systems nd to invert mtrices. In Mtri.l this lgorithm is performed y the MLU function. This function returns oth fctors in n (n n) rry. But there re sme things tht should e pointed out. We my elieve tht, once we hve the LU decomposition of A, we cn solve s mny liner systems s we wnt, simple chnging the vector. This is not completely true. Look t this emple.. A where: If we compute the LU fctoriztion we hve: A Note tht you must select (6) cells if you wnt to get the fctoriztion of () mtri The Crout lgorithm hs returned the following tringulr mtrices: L U Now solve the system () nd (4) in order to hve the finl solution We hve L y () U y (4) y L - U - y The ect solution of the originl system () is (,, ), ut the LU method hs given 4

42 different result. Why? Wht's hppened? The fct is tht LU lgorithm does not give the ect originl mtri A, ut new mtri A' tht is row permuttion of the given one. This is due to the prtil pivoting strtegy of Crout's lgorithm. You simple prove it y multiplying L nd U. So the correct fctoriztion formul is: where P is permuttion mtri The process to solve the system is therefore: A PLU ' P T (5) L y ' (6) U y (7) We hve shown tht only the informtion of the two fctors L nd U insufficient to solve the generl system. We lso need the P mtri. But how cn we get the permuttion mtri? This mtri is provided y the lgorithm itself t the end of the fctoriztion process. Most LU routines do not give us the permuttion mtri, ecuse formul (5) is pplied directly to the vector pssed to the routines. But the concept is sustntilly the sme: for solving system with LU fctoriztion we need, in generlly, three mtrices P, L, nd U. The originl system is roken into two tringulr systems A ' P T L y ' U y The permuttion mtri cn e otined y compring the originl A mtri with the mtri otined from the product A' LU. Let' see how. The se vectors u, u, u re: u, u, u We emine now the mtri rows of the two mtrices A' nd A. The row of A' comes from row of A, p u The row of A' comes from row of A, p u The row of A' comes from row of A, p u So the permuttion mtri will e: P ( p, p, p) ( u, u, u ) Clerly this process cn e very tedious for lrger mtrices. Fortuntely the permuttion mtri is 4

43 supplied y the function MLU s the third, optionl prt of its output. For mtri you should therefore select rnge of 9 (rther thn the usul 6) columns to see the permuttion mtri. Tht gives the decomposition A P L U. MLU(A) returns ( L, U, P) rrys Emple - Perform the ect LU decomposition for 55 Trtgli mtri If we form the mtri product P L U (here the MProd function is useful) we otin finlly the given originl mtri. (Note tht the lst mtri P must e the first of the mtri product) 4

44 4 Overdetermined Liner System If the mtri hs more rows thn columns, then the liner system is sid to e overdetermined. Often in n overdetermined system, there is no solution tht stisfies ll the rows ectly, ut there re solutions such tht the residul r A is vector of "smll vlues" tht re within working ccurcy. The solution minimizing the norm of the residul vector r r r r T i i is the lest squres solution of the liner system. In this cse the system hs unique solution specified y the lest squres criterion. Emple. Solve the following liner system with the lest squres criterion The norml eqution One wy to resolve the given liner system is y trnsforming the rectngulr system mtri into squre mtri using the so clled norml eqution trnsformtion. c B A A A A T T The norml mtri B A T A is squre nd symmetric. So the lst system cn e resolved with the usul methods (B -, LR, LL, Guss, etc.). The solution of the norml system is lso the lest squres solution of the overdetermined system B c Solving the norml system we find the solution [ 7, 5, ] The residul vector cn e esily computed s

45 Note tht only the lst eqution is ectly stisfied ( residul zero). Remrk. Mny ooks wrm us of using the norml eqution for solving these prolems. They point out tht, generlly, the trnsformed system is worse conditioned thn the originl system nd so the numericl solution my e error prone. This is conceptully true in generl. But we should not emphsize this spect too much. We hve seen tht, for systems of low-to-moderte size this method gives resonly good solution ccurcy. This method is lso quick nd esy to pply. Another dvntge of this method is tht the trnsformed mtri is symmetric positive definite, nd thus we cn dopt severl efficient lgorithms (e.g. Cholesky decomposition) to solve the system. Lst ut not lest, if the originl mtri is integer, the norml mtri is still integer. QR decomposition Another wy to resolve rectngulr liner system is performing the QR fctoriztion of the mtri system. As know, the QR fctoriztion cn e pplied lso to rectngulr mtri. The trnsformtion is: A Q R R Q R Q T Rememer tht R is tringulr nd Q is orthogonl nd unitry so Q Q The given rectngulr liner system is trnsformed into tringulr liner system tht cn e solved efficiently with the ck-sustitution lgorithm. Let's see. T Now the ( ) tringulr liner system cn esily e solved Note tht we hve only used the first rows of the R mtri returned y the QR fctoriztion lgorithm in order to set the ( ) system mtri. This method is in generl more ccurte nd stle thn the norml eqution for solving lrge liner systems. On the other hnd, the QR method requires - for m >> n - out twice s much work s the norml eqution. We note lso tht integer vlues re never conserved y the QR fctoriztion 44

46 SVD nd the pseudo-inverse mtri The most generl wy for solving n overdetermined liner system is A The mtri A + is clled the "pseudo-inverse" of A nd, for squre mtri, coincides with the inverse A -. The pseudo-inverse lwys eists, whether or not mtri is squre or hs full rnk. For rectngulr mtri A(n m), it is defined s A + + A T T ( A A) A We note tht the norml mtri (A T A) ppers in this definition We cn void computing the norml mtri directly y using the singulr vlue decomposition A T UDV where, setting p min(n, m), U is (n p) orthogonl 5 mtri, V is n (m p) orthogonl mtri nd D is (p p) digonl mtri. For semplicity ssume here n > m. In tht cse D nd V re oth squre with dimension (m m). Multiplying oth sides y U T nd rememering tht: U T U I, we hve. The mtri Becuse T T T T U UDV U DV U T DV is squre so, tking its inverse, we hve. V T DV V T U T, we hve finlly: T T T ( DV ) U ( V ) V D U A T + Therefore, the pseudo-inverse cn e computed y the following stle formul V D In Mtri.l this computtion is performed y the function MPseudoinv A + U T T D U T Note tht the pseudo-inverse of (5 ) mtri is ( 5) mtri. The solution is the product of the pseudo-inverse nd the vector 5 The terms orthogonl here implies the concept of column-orthogonl: A mtri A (n m), with n m, hving ll its columns mutully orthogonl is clled column-orthogonl mtri. 45

47 46 Underdetermined Liner System If the mtri hs less rows thn columns, then the liner system is sid to e underdetermined. If the rnk of the incomplete nd the complete mtri is equl then there re infinite solutions tht stisfie the given system In tht cse the mtri equtions A or A define n implicit Liner Function - lso clled Liner Trnsformtion - etween the vector spces, tht cn e put in the following eplicit form d C y + () where C is the trnsformtion mtri nd d is the known vector; C is squre mtri hving the sme columns of A, nd d the sme dimension of Emple. Find the solutions (if ny) of the following ( ) system The given rectngulr system cn e conceptully trnsformed into squre singulr system simply dding zero row (for emple, t the ottom) 4 4 The rnk of this system is, therefore there re infinite solution tht cn e put in the form () The solutions, in tht cse, cn e esily find y hnd or y SysLinSing Note tht is not necessry to dd the zero row ecuse the function utomticlly does it. The solutions cn e written, fter the sustitution t, s: t y y y + + t y t y t y 5 4 Note tht the prmetric form is not unique: sustituting, for emple the epression (t -)/4, we get nother prmetric form representing the sme suspce. Emple. Find the solutions (if ny) of the following ( 4) system

48 47 The rnk r of the mtri A nd the ugmented mtri [A, ] re equl Therefore the system hs surely infinite solutions. The mtri [C, d] returned y SysLinSing is tht cn e written in prmetric form, fter the sustitution t, 4 s, s: + 4 s t y y y y + s y s y t y s t y 4 Note tht, s m 4, r, the suspce genertes y the solutions hs dimension: m - r ; nd therefore there re two prmeters in the solution set. Emple. Find the solutions (if ny) of the following ( 4) system This system, pprently very similr to the ove one, cnnot e solved ecuse the rnk r of the mtri A nd the ugmented mtri [A, ] re different. In this cse SysLinSing would return "?" Minimum module solution As we hve seen, n undetermined liner system hve generlly infinite solutions. We wnder if, mong the infinite solutions, there is one hving the minimum module. For homogenous systems this solution surely eists ecuse the trivil solution hs the minimum module. The non-homogeneous cse is more interesting. Reclling the emple + + 4

49 we hve found in previous emple tht ll its solutions cn e represented y the following prmetric eqution. [ 4 t, 5t +, t] T Computing the squre module of : + + ( 4t ) ( 5t + ) + t 4 t t + 5 Tking its derivtive, nd solving, we hve: d dt (4 t t + 5) 84 t t 7 Thus for t /7 the solution * [/7, 9/7, /7] hs minimum module For lrge systems this method ecomes quite difficult, ut we cn otin the miniml module solution of the undetermined system A in very quick wy y the following mtri eqution where the squre mtri C (A A T ) In the previous emple we hve nd the finl solution is T A T C - 7 C C * The following worksheet shows possile solution rrngement 48

50 Prmetric Liner System Sometimes the system mtri my contin prmeter, for emple "k", nd we my hve to study the system solutions s function of this prmeter. Generlly speking this is not truly numericl prolem, nd the mtri cnnot e inverted or fctorized with the usul numericl methods. This prolem cn e solved using symolic computtionl systems or, lterntively, y hnd. The function MDetPr in Mtri.l computes the prmetric determinnt for mtrices of low dimension. It returns the determinnt D(k) s polynomil in the vrile k. Then, with the id of the Crmer's rule, we cn otin the solutions of the prmetric system in the form of polynomils frctions. Crmer's rule Given liner system: [A] [,,... n ] The single element i of the solution vector, cn e found tking the frction of the determinnts of two mtrices: the first mtri is otined from the system mtri replcing the column i with the vector ; the second mtri is the system mtri itself. Tht is, in formuls: D i det [,... i-,, i+... n ] D det [,... i-, i, i+... n ] i D i / D Repeting for i,...n, we find the solution vector. Emple. Solve the following system contining the rel prmeter k k k 7 + k + k 7 5 -k 7 k For the first, we uild the 4 mtrices nd compute theirs determinnts The zeros of the determinnt D -+k +k cn e found y the PolyRoots function. Two roots re comple, nd one is rel: k. Thus the system hs solutions for k, tht re: 7k + 7k + k + k, 66 + k + k + k k 7k + k + k 49

51 Block-Tringulr Form Squre sprse mtrices, i.e., mtrices with severl zero elements, cn under certin conditions e put in useful form clled lock-tringulr (or Jordn s form ) y simple permuttions of rows nd columns 5 A A A -9 7 The lock-tringulr form sves lot of computtionl effort for mny importnt prolems of liner lger: liner system, determinnts, eigenvlues, etc. We hve to point out tht ech of these tsks hs computing cost tht grows pproimtely with N. Thus, reducing for emple the dimension to N/, the effort will decrese 8 times. Clerly it s gret dvntge. Liner system solving For emple, the following (6 6) liner system A It could e written s A A c where the vector c is given y: c A Prcticlly, the originl system (6 6) is split into two ( ) su-systems Computing the determinnt Determinnt computing lso tkes dvntge of the lock-tringulr form For emple, the determinnt of the following (6 6) mtri is given y the product of the determinnts of the two ( ) mtrices A nd A. 5

52 Permuttions Differently form the other fctoriztion lgorithms (Guss, LR, etc.), the lock-tringulr reduction uses only permuttions of rows nd columns. Formlly permuttion cn e treted s similrity trnsformtion. For emple, given (6 6) mtri, echnging rows nd 5, followed y echnging columns nd 5, cn e formlly (ut only formlly!) written s. B P T A P, where the permuttion mtri is P (e, e 5, e, e 4, e, e 6 ) A P P T A P Remrk. Mtri multipliction is very epensive tsk tht should e voided whenever possile; we use insted the direct echnge of rows nd columns or, even etter, the echnge of their indices. Note tht the similrity trnsform keeps the originl eigenvlues. Consequently the eigenvlues of the mtri A re the sme s those of the mtri B Eigenvlue Prolems The eigenvlue prolem tkes dvntge of the lock-tringulr form. For emple, the following (6 6) mtri A hs the eigenvlues: λ [-7, -,,,, 5 ] A λ A λ A -7 λ The set of eigenvlues of the (6 6) mtri A is the sum of the eigenvlue set of A [,, -7 ] nd the eigenvlue set of A [-,, 5 ]. Severl kinds of lock-tringulr form Up to now the mtrices tht we hve seen re only one kind of lock-tringulr form; ut there re mny other schemes hving locks with mutully different dimensions. At lst, ll locks cn hve 5

53 unitry dimension s in tringulr mtri. Below re shown some emples of lock-tringulr mtrices (locks re yellow) Remrk. The effort of reduction is high when the dimension of the mimum lock is low. In the first mtri the dimension of the mimum lock is ; in the second mtri it is ; in the third mtri the dimension is, showing the est-effort reduction tht would e possile. On the contrry, the lst two mtrices give quite poor effort reduction. Permuttion mtrices Is it lwys possile to trnsform squre mtri into lock-tringulr form? Unfortuntely not. The chnce for lock-tringulr reduction depends of course on the zero elements. So only sprse mtrices could e lock-prtitioned. But this is not sufficient. It depends lso on the configurtion of the zeros in the mtri. Two importnt prolems rise:. To detect if mtri cn e reduced to lock-tringulr form. To otin the permuttion mtri P Severl methods hve een developed in the pst for solving these prolems. A very populr one is the Flow-Grph method. Mtri Flow-Grph Following this method, we drw the grph of the given mtri following these simple rules: the grph consists of nodes nd rnches the numer of nodes is equl to the dimension of the mtri the nodes, numered from to N, represent the elements of the first digonl ii for ll elements ij we drw n oriented rnch (rrow) from node-i to node-j Complicted? Not relly. Let s hve look t this emple. Given the (4 4) mtri A The flow-grph G(A) ssocited, looks like the following (see the mcro Grph Drw for utomtic drwing) 5

54 4 where: node is linked to nodes,, 4; node is linked to node 4; node is linked to nodes,, 4; node 4 is linked to node. We oserve tht from node there is no pth linking to node or to node The sme hppens if we strt from node 4 It is sufficient to sy tht the grph is not strongly connected Flow-Grph rule. If it is lwys possile for ech node to find pth going through ll other nodes, then we sy tht the grph is strongly connected An importnt theorem of Grph Theory sttes tht if the flow-grph G(A) is strongly connected, then the ssocited mtri is not reducile to lock-tringulr form, nd vice vers. On the contrry, if the flow-grph G(A) is not strongly connected then there lwys eists permuttion mtri P tht reduces the ssocited mtri to lock-tringulr form. Syntheticlly: G(A) strongly connected mtri A irreducile G(A) not strongly connected mtri A lock reducile This pproch is quite elegnt nd very importnt in Grph theory. But from the point of view of prcticl clculus it hs severl drwcks: it ecomes lorious for lrger mtrices the softwre coding is quite complicted it does not provide directly the permuttion mtri P In the ove emple, we oserve tht for P [ e, e 4, e, e ], the similrity trnsform gives locktringulr form B P T A P A P P T A P For mtrices lrger thn (4 4) the effort of serching for nd testing ll possile permuttions grows shrply. For emple, it requires much work for mtrices like the following one. For this reson the flow-grph method ecomes prcticlly useless for mtrices of dimension (7 7 ) or higher The score-lgorithm In this chpter we shll introduce heuristic technique for efficiently reducing sprse mtri to lock-tringulr form. The method is oth simple nd very efficient, nd cn e pplied lso to medium-to-lrge mtrices. It consists of n itertive process hving s its min gol to group zeros ner the upper-right corner of the mtri using only rows nd columns echnges. This lgorithm ws first implemented s n utomtic progrm, ut thnks to its simplicity it cn lso 5

55 e performed y hnd, t lest, for low-to-modertely dimensioned mtrices. Let s see how it works Given, e.g., the (6 6) mtri shown just ove, we egin y initilizing the permuttion vector e e e e 4 e 5 e 6 The min gol is to ring to the upper tringulr (grey) re) the lrgest possile numer of zeros. Let s egin to serch ll non-zero elements ove the first digonl. The serching must strt from the first row nd from right to left: thus from the element 6 ; if zero, we jump to the neighoring element 5 nd so on till we hve reched. Then we repet long the second row, from 6 to. And so on till the lst row In this emple, the first non-zero element is 5 ; Let s find, if eists, the first zero on the sme row, eginning from left to right. The first is the element. We shll echnge columns nd 5 nd, therefter, rows nd 5 After the permuttion (, 5), the mtri will e the following: A P P T A P We oserve the zero grouping close to the upper-right corner Now the first non-zero element strting from the right is 4. The first, strting from left, is. Thus we permute nd 4 After permuttion, 4 we hve: A P P T A P All zeros re now positioned in the upper-tringulr re. The mtri is prtitioned in two ( ) 54

56 locks. The process ends. The finlly permuttion mtri is e e5 e4 e e e6 As shown, with only permuttions we were le to reduce (6 6) mtri to lock-tringulr form. We hve to emphsize tht we worked only y hnd. This method lso keeps good efficiency with lrger mtrices. Let s hve look t nother emple. Reduce, if possile, the following (6 6) mtri The first element, from right, is: 6 The first element, from left, is:. So the pivot columns re nd 6 The first element, from right, is: 4 The first element, from left, is:. So the pivot columns re nd 4 The first element, from right, is: The first element, from left, is:. So the pivot columns re nd. Finlly we get the lock-tringulr mtri The mtri hs een lock-prtitioned: There re locks ( ) nd one lock ( ) We oserve tht this lgorithm does not provide ny informtion out the success of the process. It simply stops itself when there re no more elements to permute. At the end of the process, if the resulting mtri is in lock-tringulr form, then the originl mtri is reducile. Otherwise, it mens tht the originl mtri is irreducile nd its flow grph is strongly connected. The Score Function The mtrices used up to now hd ll zero elements completely filled moved into the upper-tringle re. Now let s see wht hppens if the mtri hs more zeros thn those strictly necessry for lock prtitioning (spurious zeros). In tht cse not ll permuttions will e useful for grouping zeros. Some of them will e useless, nd some others even worse. Thus, it is necessry to mesure the goodness of ech permuttion. By simple inspection it is esy to select the good permuttions from d permuttions. But in n utomtic process it is necessry to choose function for evluting the permuttion goodness: the score- function is the mesure dopted in this lgorithm. 55

57 The score function counts the zeros in the upper tringle re (grey) efore (A) nd fter (B) the permuttion, returning the difference. score w( i, j) w( i, j) B A The score will e positive if the permuttion will e dvntgeous; otherwise it will e negtive or null. The zeros do not ll hve the sme weight: the zeros nerest to the upper-right corner hve higher weight, ecuse mtri filled with zeros close to the upper-right corner is etter thn one with zeros close to the first digonl. etter worse Aprt from this concept, the weight function w(i,j) is ritrry. One function tht we hve tested with good result is the following ij w i, j) ( n i + ) j ( ij Weight function for n (n n) mtri. For ech recognized permuttion, the lgorithm mesures the score. If positive, the permuttion is performed, otherwise the permuttion is rejected nd the lgorithm continues to find new permuttion. After some loops the disposition of zeros will rech the mimum score possile; every other ttempt of permuttion will produce negtive or null score. So the lgorithm will stop the process. Some emples Now let s see the lgorithm in prcticl cses A P T A P P [e5, e6, e, e, e4, e] Accepted permuttions 6 Rejected permuttions 4 56

58 A P T A P P [ e7, e, e, e, e8, e, e4, e6, e9, e5 ] Accepted permuttions 9 Rejected permuttions A P T A P P [ e, e7, e5, e8, e, e, e6, e4, e9, e ] Accepted permuttions 7 Rejected permuttions A

59 P T A P P [ e7, e9, e, e, e6, e6, e8, e, e, e, e, e, e, e4, e8, e5, e4, e7, e9, e5 ] Accepted permuttions 8 Rejected permuttions 7 As we cn see, lso for lrger mtrices the numer of permuttions remins quite limited. Regrding this, nd the fct tht the permuttion is much fster then ny other rithmetic opertion in floting point, we cn guess the high speed of this lgorithm In Ecel, with Mtri.l, it is very esy to study the mtri permuttions. A simple rrngement of (6 6) mtrices is shown in the following emple. We hve used the function MPerm. When you chnge the permuttion numers, lso the permuttion mtri chnges nd, consequently the finl, trnsformed mtri 58

60 The Shortest Pth lgorithm The ove lgorithm does not sy if the mtri is irreducile. For tht the shortest-pth mtri, uilt y the Floyd's lgorithm, comes in hndy. In Mtri.l you cn perform this y the function PthFloyd or y the mcro "Mcros>Shortest Pth" Emple. Sy if the given mtri is reducile The shortest-pth mtri show the presence of empty elements. For emple, the element is null, mening tht there is no pth reching node from node. This is sufficient for sying tht the given mtri is not strongly connected nd thus, reducile. Emple. Prove tht, on the contrry, the following mtri is irreducile The shortest-pth mtri is dense, mening tht every node cn e reched from ny other. By definition, the given mtri is strongly connected nd thus, irreducile 59

61 Limits in mtri computtion One recurrent question out mtri computtion is: - wht is the mimum dimension for mtri opertion, for emple for the determinnt, or for inversion? Well, the right nswer should e: it depends. Mny fctors, such s hrdwre configurtion, lgorithm, softwre code, operting system nd - of course - the mtri itself, contriute to limit the mimum dimension. One sure thing is tht the limit is not fied t ll. In the pst, the min limittion ws memory nd evlution speed, ut nowdys these fctors no longer constitute limit. We cn sy tht, for the stndrd PC, the min limittion is due to the -it rithmetic nd to the mtri itself. Suppose you hve dense mtri (n n) with its elements ij rndomly distriuted from -k to k. With this hypothesis the determinnt grows roughly s: Log( D ) n Log(k) +.7 n n Log(k) where Log is deciml logrithm, n is the dimension of the mtri, k its m vlue In it doule precision the m vlue llowed is out E+, E-. So if we wnt to void the overflow/underflow error, we must constrin: n Log(k) () If we plot this reltion for ll points (k, n) we hve the re for computing (lue re in the grph elow). On the other hnd, the dngerous error re is the remining (white) re 5 5 n Limit of mtri computing 5 overflow n Log(k) 5 computing k.e+.e+.e+5.e+7.e+9.e+.e+.e+5 How does it work? Simple. If you hve to compute the determinnt of (8 8) mtri hving vlues no lrger thn, the point (, 8) flls into the lue re; so you will e le to performs this opertion. On the contrry, if you hve (8 8) mtri hving vlues up to E+7, the point (E+7, 8) flls within the white re; so you will proly get n overflow error From this grph we see tht mtrices of dimension (5 5) or less, cn e evluted for ll vlues, while mtrices of size ( ) or more cn e computed only if their vlues re less tht Of course this result is vlid only for generic, dense mtrices tht re not ill-conditioned. If the mtri is ill-conditioned you could get n overflow/underflow error even for low-to- /moderte mtri dimensions. Fortuntely, there re lso specil kinds of mtrices tht cn e evluted even if the constrint () is flse. We spek out digonl, tridigonl, sprse, lock mtrices, etc. We hve to sy tht voiding the overflow error is not sufficient to get good result. We hve to tke cre, especilly for lrge mtrices, of the round-off errors. They re very tricky nd difficult to detect. Sometime the result of inverting lrge mtri is tken s vlid even if it is completely wrong! 6

62 Sprse Liner Systems We hve seen tht finite rithmetic nd memory storge oth limit the mimum dimension of the mtri, nd thus the ssocited liner system. In pre-7 Ecel, for emple, the solute mimum dimension for liner system would e out (5 5). This limittion is due to the mimum numer of the spredsheet columns. But rrely we cn solve such lrge systems ecuse with 5 digits finite rithmetic the round-off errors often overwhelms the results. There is sitution tht llows one to successfully solve lrger systems, of dimension greter thn 5. It hppens when the systems mtri is sprse. A system of liner equtions is clled sprse if only reltively few of its mtri elements [ ij ] re nonzero. If we store only these vlues, we cn sve lrge mount of storge. For emple, ( ) mtri with % nonzero elements requires only 9, cells of storge, just out the sme s dense (95 95) mtri. Of course we hve to choose new rrngement to store these vlues. In the pst, severl ingenious nd efficient schemes, tightly relted to the hrdwre/softwre of the mchine, were developed for this purpose. Here we dopt the sprse coordinte formt (or Yle scheme) This scheme is surely not one of the most efficientones, ut it is conceptully simple, compct nd dptle to spredsheet implementtion. Specificlly, the first columns contin the integer coordintes while the lst column contins the element vlues. The sprse mtri of the previous emple requires 9 rows nd columns for totl of 7. cells. We note tht this rry cn esily e rrnged in spredsheet while, on the contrry, its ssocited ( ) stndrd mtri cnnot e written, ecept with Ecel 7. The coordinte tet formt provides simple nd portle method to echnge sprse mtrices. Any lnguge or computer system tht understnds ASCII tet cn red this file formt with simple red loop. This mkes these dt ccessile not only to users in the Fortrn community, ut lso to developers using C, C++, Pscl, or Bsic environments. Filling fctor nd mtri dimension The filling fctor mesures how much "dense" mtri is. In this pper, the filling fctor is defined s F N N zero N zero numer of zero elements Tot N tot totl numer of mtri elements There is simple reltion etween the fctor F nd the mimum dimension of the system tht cn e solved in Ecel. Rememering tht the mimum numer of rows of the pre- 7 spredsheet re 6, we hve 6 F N N The corresponding limit in Ecel 7, with rows, is fctor of lrger. 8 F 6

63 The reltion of N m versus the filling fctor shows tht, for sprse mtrices hving. < F <.4 the m pre-7 dimension of the system mtri is out 4 < N m < 8 Tht is gret improvement with respect to the stndrd mtri formt F N N m F The following pictures show rndom sprse mtrices hving different filling fctors ( ) F. ( ) F. Usully, lrge sprse mtrices in pplied science hve fctor F less then. (%) The dominnce fctor Storing mtri system does not utomticlly men "solving" the system. As we hve seen in the previous chpters, the round off errors my overwhelm the finl result if the mtri is dly conditioned. For very lrge liner system the results cn e cceptle only if the system mtri is well conditioned. It hs een demonstrted tht this hppens for rowdigonl dominnt mtrices. A mtri is clled row digonl dominnt if ech digonl solute element ii is greter then the sum of the other solute elements of the corresponding row. Tht is, in formul form: n ii > ij for i,...n j, j i This criterion gurntees the convergence of itertive lgorithms such s those of Guss- Seidel nd Jcoi. Moreover, it ssures the complete Cholesky LL T fctoriztion, nd generl good ehvior ginst the propgtion of round-off error. The row dominnce criterion is sufficient ut not necessry. Tht mens tht lso non-dominnt mtrices my converge with resonle ccurcy. On the other hnd, there re mtrices stisfying this criterion ut in prctice converging very slowly. For these resons it is convenient to define row dominnce fctor mesuring how much mtri is "digonl dominnt". In this pper, it is defined, for non-empty row, s 6

64 D i ii di n di + S j ij i d S i ii i n j, j i ij The row dominnce fctor D i is lwys etween nd Cse Description D i The digonl element is zero: ii < D i <.5 The row is dominted: d i < S i D i.5 The row is indifferent: d i S i.5 < D i < The row is dominnt: d i > S i D i The row contins only the digonl element: S i Therefore, the ove criterion cn e simply epressed s: D i >.5 for i,...n With ll due cution, we define the sttistics D, D m, D M D n n D i i D min{ } D m{ } m D i M D i These re kind of mtri dominnce fctors summrizing the glol dominnce ehviors of the mtri itself. Note tht D cn e greter thn.5 even if some rows re less thn.5 or even. Algorithms for sprse systems Now we emine the lgorithms suitle for solving lrge sprse systems: they cn e direct nd itertive lgorithms. Direct lgorithm Most direct system-solving lgorithms operte trnsformtion on the system mtri nd thus chnge the numer of the zero elements. Unfortuntely, none of these lgorithm mintins the initil filling fctor. For emple, strting with ( ) sprse mtri with F 5 %, the verge ehvior of the most populr fctoriztion lgorithms re shows in this tle. Clerly, we should give our preference to those lgorithms tht minimize the filling fctor. Algorithm Finl mtri Guss F % LR F 46% LL T (Cholesky) F % QR F 75% The Guss lgorithm with prtil pivot nd ck sustitution still ppers to e the right choice for generl system. For symmetric dominnt systems, the Cholesky fctoriztion is preferle for its efficiency Those lgorithms hve computtionl effort proportionl to n, where n is the dimension of the liner system. The following grph shows two typicl fctoriztion-time curves 6 performed y the Guss lgorithm for solving sprse liner systems hving F % with incresing dimension. 6 Pentium 4,.8 GHz, 56 MB RAM 6

65 The time is mesured in seconds. The upper curve is otined for sprse system mtrices tht re uniformly distriuted, while the lowest curve is otined for mtrices concentrted round the first digonl. As we cn see, t the sme dimension, the ltter sve more thn % of the fctoriztion time. For symmetric sprse mtrices the Cholesky fctoriztion sves even more thn 5% sec nrrow spred N Itertive lgorithms But the truly strong reduction of effort is ehiited y itertive lgorithms like the Successive Reltion Guss-Seidel lgorithm or, etter yet, the ADSOR method (Adptive Successive Over-Reltion). When the system mtri is well-conditioned, for emple for digonl dominnt mtri, these methods converge to the solution with the est ccurcy possile, in very few itertions, typiclly less then steps. Unfortuntely, not ll sprse systems cn e solved y n itertive procedure. But when they cn, the time svings in fctoriztion re remrkle The following grph shows the fctoriztion time of direct method nd n itertive method for digonlly dominnt sprse liner systems (F %) of incresing dimension n Time (sec) Guss Time (sec) ADSOR sec ADSOR.8 Guss.6.4. N 4 5 We see tht the fctoriztion time remins less then one second even for very lrge systems. How cn we justify this rillint result? There re three fcts: ) Itertive lgorithms operte in very strightforwrd wy, using only mtrivector multiplictions; for sprse mtrices, this opertion is very efficient, requiring only F n elementry opertions (multiplictions + dditions). ) Itertive lgorithms do no trnsform the system mtri, so its sprse fctor F does not increse long the itertive process. ) The numer of steps Ns required for converging to fied precision is sustntlilly independent of the dimension; it mostly depends on the dominnce fctor of the mtri nd, for the ADSOR lgorithm, is usully less then 5-. The fctoriztion time Ti of n itertive lgorithm is proportionl to the numer of opertions for ech step, tht is Ti Ns F n. The elortion time Td of direct method is proportionl to n, i.e., Td n. Therefore the efficiency gin defined s G Td / Ti will e: G n / (Ns F). Tht gin is directly proportionl to the dimension nd inversely proportionl to the filling fctor. Emple, for rel lrge sprse mtri of n 4, with F %, the gin G 75. The gin reches more thn 5 if F is less then %. 64

66 Sprse Mtri Genertor Of course, sprse mtrices come from prolems, nd should not e generted. However sometimes we need to generte sprse mtri for lgorithm testing, time mesuring, etc. On the internet there re some resources tht cn generte mny type of mtrices, including sprse mtrices 7. Mtri.l lso hs little tool for generting sprse mtrices. Prmeters: Rndom sprse mtri [ij] is generted with the following constrints: M: vlue: upper limit of ij Min: vlue: lower limit of ij Dim: mtri dimension (n n) Dom: Dominnce fctor D, with < D < Fill: Filling fctor F, with < F < Spred: Spreding fctor S, with < S < Sym: check it for symmetric mtri Int: check it for integer mtri. Strting from: left-top mtri corner Output formt Coordintes: genertes (k ) mtri in sprse coordinte formt: [ i, j, ij ] Squre: genertes squre mtri [ ij ] This mcro cn output mtri in stndrd or coordinte formt. Of course the coordinte formt is the only possile one on pre-7 Ecel for mtrices greter then (56 56). Here re some ptterns generted for different prmeters F nd S F., S.5 F., S. F., S.6 F., S.5 F., S. F., S.6 7 NIST MtriMrket hs one of the most useful nd complete tools, clled "Deli", for generting wide rnge of mtrices with severl output formts: 65

67 How to solve sprse liner systems Assume tht you hve to solve sprse ( ) system A, where the system mtri "A" is in the rnge A: GR nd the vector "" is in the rnge GT:GT First we nlyze the dominnce. Select one cell inside the mtri, for emple A; cll the mcro "Mcros > Sprse mtri Opertions..." from the menu, nd select the opertion "Dominnce" The mcro returns the dominnce fctors of ech row nd the verge, the m nd the min of ll dominnce fctors. In this cse we hve otined D min >.5 with n verge of D.66. This indictes tht the system is digonlly dominnt nd wellconditioned. We cn use oth itertive nd direct methods Select one cell inside the mtri, for emple A; cll the mcro "Sprse mtri Opertions..." from the menu, nd select the opertion "System (Guss)" The input A mtri is lredy filled with the system mtri. Move the cursor inside the field "vector " nd select the rnge GT:GT. Tip: You cn select only the first cell GT nd then click the smrt selector t the right: the correct rnge will now e selected utomticlly. But mke sure tht the vector is surrounded y empty cells. Then choose the output rnge, nd click "Run" After while (9 seconds in this emple), the mcro returns the solution vector of the system with very high glol reltive ccurcy (E-4). Now we solve the sme prolem with the itertive lgorithm ADSOR. The procedure is the sme s ove, ecept tht we hve to set the itertion limit (the defult is 4). This lgorithm returns the vector solution nd, in ddition, the numer of itertions performed, the verge reltive error, nd the reltion fctor used. In this emple, only.5 sec nd itertions re een necessry to rech n ccurcy of out E-5. As we cn see the fctoriztion time is much shorter thn with the direct Guss method. The Guss method should e utilized only when the sprse mtri is not dominnt, or the digonl hs some nonzero elements. 66

68 How to get the true dimensions When the system is very lrge we necessrily hve to dopt the coordinte formt. For emple, ssume to hve the mtri system A in the first three columns in the rnge A:C4779. The coordinte formt does not show directly the dimensions of the mtri. To void errors it is necessry to get the dimensions of sprse mtri written in coordinte form, i.e (rows columns). For tht, it is convenient to use the mcro tsk "Dimension", which serches for the mimum numer of rows nd columns; in ddition it returns the filling fctor of the mtri itself How to nlyze the dominnce Before solving lrge system we hve to nlyze the conditioning of the system mtri in order to choose the lgorithm nd to understnd if there is chnce of otining n cceptle result. If the mtri is digonlly dominnt (D min >.5), itertive lgorithms converge to the solution. The dominnce ssures lso n ccurte result. For tht, use the mcro tsk "Dominnce", tht computes the dominnce fctor Di of ech i th row nd, in ddition, computes the sttistics: verge, m, nd min. A mtri is totlly row-dominnt if D min >.5. In this emple we hve.48 < D min <.5, so the mtri is not totlly row-dominnt. Becuse D min >, ll rows hve digonl nonzero elements nd this is the only necessry condition for using the itertive ADSOR method. The totl dominnce is sufficient condition ut it is not necessry; the ADSOR lgorithm cn often converge lso for "qusi-dominnt" mtrices. Solving Sprse System in coordinte formt Assume to hve the system mtri in the rnge A:C4779 nd the vector "" in the rnge D:D. Select one cell inside the mtri, for emple A. Cll the mcro "Sprse mtri Opertions..." from the menu, nd select the opertion "System (ADSOR)" The input A mtri is lredy filled with the system mtri A:C4779. Move the cursor to the field "vector " nd select the rnge D:D. The time for solving ( ) system is out sec The mcro outputs the solution vector plus some useful informtion, such s the numer of itertions, the estimted reltive error, nd the reltion fctor. Note tht the Guss lgorithm would need out sec to solve this system Note lso tht this system cnnot e solved directly in pre-7 Ecel 67

69 How to check the result A quick wy for testing liner system solution is to compute the residuls vector: r - A We hve to point out tht low residuls vector does not utomticlly men n ccurte solution, ut it is lwys good nd chep test. In the previous emple we hve the sprse mtri A in A:C4779, the vector in D:D nd the solution in G:G. First of ll, we form the product A, putting the result in the rnge I:I. For this tsk we cll the mcro "Sprse mtri opertions", selecting the product opertion A*. The mtri-vector product is very fst opertion on sprse mtrices. After tht, we compute the residul vector r s the difference etween the vector nd the product A*. We cn compute the difference etween two vectors simply y selecting the rnge K:K nd inserting the rry function {E:E-I:I} with the ctrl+shift+enter keys sequence. Or, lterntively, y using the mcro "Mtri opertions", selecting the "sutrction" tsk The result is in the rnge K:K The reltive residul error cn e computed s Erres r / The norm cn e computed with the MAs function or with the Ecel formul SQRT(SUMSQ(K:K)) 68

70 Solving Sprse System with Guss Often the liner system cnnot e solved with the fst ADSOR lgorithm. This hppens, for emple, when the system mtri hs some zeros on the first digonl, or hs low dominnce fctor. In these cses we hve to go ck to the Guss reduction lgorithm, dpted for sprse mtrices For emple, ssume to hve system with some digonl zero elements. The dominnce fctor nlysis gives us the following fctors Dvg.95 Dm.4 Dmin The presence of zero digonl elements is reveled y D min. In tht cse we cnnot dopt ADSOR nd we hve to use the Guss lgorithm. Alwys rememer to check the result ecuse, in tht cse, the round-off error my completely oscure the solution otined. How to improve the dominnce In some cses the dominnce of liner system cn e improved simply reordering the equtions. For emple, the following system is not digonl dominnt But it ecomes digonl dominnt simply echnging the nd nd 4 th equtions. Of course for lrge system the mnul rows echnging is prohiitive. For this tsk comes useful the mcro "Dominnce improving". Strting from the system mtri A nd the vector, the mcro tries to improve the dominnce y rows echnging nd returns new system mtri A nd new vector. Using the system of the ove emple we get the following new mtri. The dominnce fctors re now: Dvg.4 Dm.45 Dmin. As we cn see the verge dominnce is improved ut the est result is tht no zero element ppers in the digonl (Dmin > ). Tht system cn e efficiently solved with itertive lgorithms. Note tht ADSOR con converge to the solution even if the system is not row-dominnt 69

71 The reltion prmeter w The convergence of itertive methods cn e improved introducing reltion prmeter ω Thus, the itertion schem, clled SOR (Successive Over-Reltion) cn e modified s: (k+) ( - ω) (k) + ω GS (k+) where GS is the vector generted y the Guss-Seidel lgorithm. Usully is < ω <. Generlly, it is not simple to find the dptive prmeter for the fstest convergence. In the ADSOR (ADptive Successive Over-Reltion) the prmeter is chosen y the lgorithm itself. Emple. Appling the ADSOR lgorithm to the following system, we hve the solution with n error of less then E-4, in out 8 itertions. We note lso tht this result is reched with the reltion prmeter ω.7 If we repet the clcultion using the Guss-Seidel lgorithm (ω ) we need out twice s mny itertions. The following grph shows the ccelerting effect of the reltion prmeter.. E-6 Residul Error w ADSOR GS E-8 E- w.7 E- steps E

72 How to solve tridigonl systems Tridigonl systems re suclss of sprse systems. Thnks to their prticulr structure they cn e efficiently written in very compct -column formt. The first column contins the lower sudigonl; The second column contins the digonl The third column contins the upper sudigonl The spce sving is evident. Note tht the first element of the first column nd the lst element of the third column do not relly eist. Usully they re set to zero, ut their vlues re irrelevnt ecuse the mcro does not red them. Lrge liner tridigonl systems cn e solved efficiently using the mcro "Sprse Mtri Opertion" In this emple the system mtri is contined in A:C, nd the vector is in E:E As we cn see, only. sec is sufficient for solving ( ) system. Usully the ccurcy is very high for dominnt system. 7

73 WHITE PAGE 7

74 Chpter Eigen-prolems This chpter eplins how to solve common prolems involving eigenvlues nd eigenvectors, with the id of mny emples nd different methods. Eigen-prolems Eigenvlues nd Eigenvectors These prolems re very common in mth, physics, engineering, etc. Usully they consist of solving the following mtri eqution A λ () where A is n n n mtri, nd the unknowns re l nd, respectively clled eigenvlue nd eigenvector. Rerrnging eqution () we hve: ( A λ I) () This homogeneous system cn hve non-trivil solutions if its determinnt is zero. Tht is: A λ I () Chrcteristic Polynomil The left-hnd side of () is n n th degree polynomil in λ, clled chrcteristic polynomil - whose roots re the eigenvlues of the mtri A. For () mtri, the system () ecomes: λ λ λ Computing the determinnt we hve eqution () in epnded form λ ( + ) λ + det( A) 7

75 For () mtri, the system () ecomes: λ λ λ nd its chrcteristic eqution () ecomes λ + ( + + ) λ ( + + ) λ + det( A) With lrger mtri the difficulty of computing the chrcteristic polynomil grows shrply;. Fortuntely there is very efficient wy to compute the polynomil coefficients, using the Newton- Girrd recursive formuls. In Mtri.l we cn get these coefficients with the function MChrPoly. Roots of the chrcteristic polynomil Aprt from the nd degree cse, finding the roots of polynomil needs numericl pproimtion methods. Mtri.l hs the function PolyRoots tht finds ll roots - rel or comple - of given rel polynomil, using the Siljk+Ruffini methods. This function is suitle for generl polynomils up to 6 th or 7 th degree. When possile, the function uses the Ruffini method for finding smll integer roots. There is lso the function PolyRootsQR for finding ll polynomil roots. It uses the efficient QR lgorithm nd it is dpted for polynomils up to th or th degree. For comple polynomils there is the similr function PolyRootsQRC Cse of symmetric mtri Symmetric mtrices ply fundmentl role in numericl nlysis. They hve feture of gret importnce: Their eigenvlues re ll rel. Or, in other words, its chrcteristic polynomil hs only rel roots. Another importnt reson for using symmetric mtrices is tht there re mny strightforwrd, efficient, nd lso ccurte lgorithms for solving their eigen-systems; this is much more complicted for symmetric mtrices. Tip. There is nice, closed formul for generting symmetric (n n) mtri hving the first n nturl numers s eigenvlues i i i j ( i + ) n 4i + n n + i j n i j Below re the first such mtrices for n,, 4, 5, 6, 8 74

76 eigenvlues:, 7/ / eigenvlues:,, / 6/ -/ -/ 5/.5.5 eigenvlues:,,, eigenvlues:,,, 4, / 4/ / / / eigenvlues:,,, 4, 5, 6 4/ 9/ / / -/ / / / -/ -/ / / / -/ -/ / -/ -/ / -4/ -/ -/ -/ -4/ / eigenvlues:,,, 4, 5, 6, 7,

77 Emple How to check the Cyley-Hmilton theorem Regrding the chrcteristic polynomil P(λ) n importnt theorem, known s Cyley-Hmilton s theorem - sttes tht the ny squre mtri A verifies its chrcteristic polynomil. Tht is, in formul: P(A) O (where O is the null mtri) The ove mtri eqution cn e formlly otined y sustituting the vrile λ with the mtri A. Let s see how to test this sttement with prcticl emple in Ecel. Given the following ( ) mtri 9 - A Its chrcteristic polynomil is: P( λ) 6 λ + 6λ λ After sustituting A for λ we hve P ( A) 6 I A + 6 A A Evluting this formul y hnd is quite tedious, ut it is very esy in Ecel. Let s see the following spredsheet rrngement using the function MPow Note tht we hve inserted the P(A) formul s n rry function {.} Of course it is lso possile to compute the mtri powers A, A with the mtri product. 76

78 Eigenvectors Logiclly speking, once we hve found n eigenvlue we cn solve the homogeneous system () in order to find the ssocite eigenvector. Normlly for ech rel eigenvlues with multiplicity one, there is only one eigenvector. For multiplicity, we will find two eigenvectors or even only one. Step-y-step method The method eplined ove is generl nd is vlid for ll kind of mtrices. It is known to every mth student, nd it is very populr. For this resons it is eplined in this chpter, despite its intrinsic inefficiency. As we cn see in the following prgrphs, there re other methods tht cn compute oth eigenvectors nd eigenvlues t the sme time in very efficient nd fst wy. They re suitle for lrger mtrices, while the step-y-step method cn e pplied to mtrices of low dimension (usully from, up to 55). But, didcticlly speking, this method is still vlid, nd it cn help when other methods fil or rise douts. The step-y-step method, is composed of the following steps:. Compute the coefficients of the chrcteristic polynomil. Find their roots, tht is, the mtri eigenvlues λ i. For ech root λ i uild the mtri A λ i I 4. Find the ssocited eigenvector i y solving the homogeneous system Let's see how it works with some emples Emple - Simple eigenvlues Find ll eigenvlues nd ssocited eigenvectors of the following mtri ( A λ I ) i i i For tsk ) we use the function MthChrPoly; for tsk ) we use the function PolyRoots; tsk ) is performed with the MIde function which returns the identity mtri.finlly, tsk 4) uses the function SysLinSing to find solution of the singulr system. 77

79 For the given mtri, we hve found the eigenvlues nd eigenvectors t the right Eigenvector - Eigenvlues λ λ λ 7 Emple - How to check n eigenvector Once we hve found the eigenvectors, we cn esily verify them y simple mtri multipliction. u A i i u If is n eigenvector, the vector u must e ectly λ multiple of the vector, s we cn see in the worksheet elow i λ i Eigenvectors re not unique. It is esy to prove tht ny multiple of n eigenvector is lso n eigenvector. This mens tht if (-,, -) is n eigenvector, other possile eigenvectors re: Mtri Eigenvlue Eigenvectors λ By convention, mthemticins tke the eigenvector with norm, tht is:. In tht cse it is clled the eigenversor. Following this rule the eigenvector mtri ecomes s we cn see t right Sometimes, in order to void floting numers, we normlize only the smllest vlue of the vector; for tht, we divide ll vlues y the GCD The SysLinSing function dopts this solution. If you wnt to get the eigenversors you hve to do it mnully. 78

80 Emple - Eigenvlues with multiplicity Find ll eigenvlues nd ssocited eigenvectors of the following mtri For the given mtri we hve found two roots: λ, m λ, m. With n eigenvlue with multiplicity, we get one eigenvector; while with the second eigenvlue with multiplicity, we get two eigenvectors Tip: The ccurcy of multiple roots is in generl lower thn tht of singulr root. For this reson, the SysLinSing function sometimes cnnot return ny solution. In those cses, try to set the SysLinSing prmeter MError to less then E-5, depending on the eigenvlue ccurcy (usully for root with m., we set MError E-) In the ove emple the numer of eigenvectors corresponds ectly to the eigenvlue multiplicity. But this is lwys vlid? Does the eigenvlu multiplicity gives the dimension of the eigenvector suspce? Unfortuntely not. There re cses in which the multiplicity doesn't' t correspond to the ssocited eigenvectors. Lets' see the following emple. 79

81 Emple - Eigenvlues with multiplicity not corresponding to the numer of eigenvectors Find ll eigenvlues nd ssocited eigenvectors of the following mtri - - For the given mtri the chrcteristic polynomil is: λ + 4λ 4λ Tht hs two roots: λ, m λ, m. With the eigenvlue with multiplicity, we get one eigenvector; with the second eigenvector, with multiplicity, we get only one eigenvector, not two. Emple - Comple Eigenvlues Sometimes it hppens tht not ll roots of the chrcteristic polynomil re rel. In tht cse, the eigenvectors ssocited with these comple eigenvlues re comple too. Find ll eigenvlues nd ssocited eigenvectors of the following mtri A The chrcteristic polynomil is: λ + λ 46λ + 5 The eigenvlues re λ, λ 5+ j, λ 5 j Mtri.l does not contin SysLinSing for solving comple singulr system, ut we cn derive rel system from the originl comple one: Seprting oth eigenvlues nd eigenvectors in their rel nd imginry prts: λ λ re + jλ im re + j im 8

82 the homogeneous liner system, ecomes ( A ( λ + jλ ) I )( + j ) ( A λi) re im re im Rerrnging: (( A λ I) + λ I ) + j( λ I + ( A λ I) ) re re im im im re re im The ove comple eqution is equivlent to the following homogeneous system ( A λrei ) re + λimi im λimi re + ( A λrei) im ( A λrei) λimi λimi ( ) A λ rei Let's see how to rrnge solution in Ecel The 6 6 homogeneous system mtri is uilt in four su-mtrices. re im The solution of the homogeneous system returned y SysLinSing is conceptully divided in two prts: the upper prt contins the rel prts of the eigenvectors; the lower prt holds the imginry prts of the sme eigenvectors. Sustituting the conjugted eigenvlues we find conjugted eigenvectors. The cse of rel eigenvlue is the sme s in the ove emple, so we do not repet the process. Rther, we wnt to show here how to rrnge check for comple eigenvectors. 8

83 Emple - Comple Mtri Mtri.l hs severl functions developed for solving the eigen prolem for comple mtrices of moderte dimension. Following the step-y-step method previous seen, we need the following functions: MChrPolyC - computes the comple coefficient of the chrcteristic polynomil PolyRootsQRC - computes the roots of comple polynomil MEigenvecInvC - computes the eigenvectors of comple mtri 4+j -4j 4+5j 5-4j +j +j -j -+4j 4+j -+j +6j -j --j -j -j A possile rrngement is shown in the following worksheet. Note tht the given mtri hs distinct eigenvlues: rel nd comple This mens tht its eigenvectors re distinct nd we cn use the inverse itertion lgorithm for finding them. Note lso tht, in generl, rel eigenvlue does not correspond to rel eigenvector. Curiously the only rel eigenvector corresponds to the imginry eigenvlue λ j 8

84 Emple - How to check comple eigenvector Given the mtri A nd one of its eigenvlues λ, prove tht the vector is n eigenvector A λ 5+j re im The test cn e rrnged s in the following worksheet We hve used the function M_MAT_C of Mtri.l for comple mtri multipliction. Note tht we hve to insert the imginry prt of the mtri ecuse those comple functions lwys require oth prts: rel nd imginry. There is lso nother wy to directly compute the eigenvector of given eigenvlue: the functions MEigenvec nd MEigenvecC of Mtri.l return the eigenvector ssocited with their eigenvlues; the first function works for rel eigenvlues, nd the second for comple eigenvlues. See the chpter "Function Reference" of Vol. for detils In the following rrngement we hve used MEigenvecC for clculting the ssocited eigenvectors, nd MMultsC for otining the comple sclr product Of course the finl result is equivlent 8

85 Similrity Trnsformtion This liner trnsformtion is very importnt ecuse it leves eigenvlues unchnged. Let's see how it works. Giving squre mtri A nd second squre mtri B we generte third mtri C with the formul: C B - A B We sy: C is the similrity trnsform of A y mtri B Similrity trnsformtions ply crucil role in the computtion of eigenvlues, ecuse they leve the eigenvlues of mtri unchnged. Thus, eigenvlues of A re the sme s those of C, for ny mtri B It cn e esily demonstrted tht det(c - λ I) det(a - λ I) In fct, rememering tht I B - B, we cn write: det(c - λ I) det(b - A B - λ I) det(b - A B - λ B - B) But, rerrnging, we hve det(b - A B - λ B - B) det(b - (A B - λ B)) det(b - (A - λ I) B)) det(b - ) det (A - λ I) det (B) det (A - λ I) det(b - ) det (B) det (A - λ I) Emple - verify tht the similrity-trnsformed mtri of A y the mtri B hs the sme eigenvlues. To prove tht eigenvlues re the sme it is sufficient tht the chrcteristic polynomils of A nd B re equls. For computing the trnsformed mtri we cn use the function MBAB of Mtri.l. But, of course we cn use, the stndrd formul s well. MMULT(MMULT(MINVERSE(E:G5),A:C5),E:G5) For computing the coefficients of the chrcteristic polynomil we hve used the function MChrPoly 84

86 Fctoriztion methods The hert of mny eigensystem routines is to perform sequence of similrity trnsformtions until the resulting mtri is nerly digonl within smll error. A (P ) - A (P ) A (P ) - A (P ) A (P ) - A (P )... A n (P n ) - A n- (P n ) A n λ D n D λ λ Where D is digonl Eigenvlues of digonl mtri re simply the digonl elements; ut, ecuse they re equl to the mtri A for the similrity property, we hve found lso the eigenvlues of the mtri A. We found this strtegy in lgorithms such s Jcoi' itertive rottions, QR fctoriztion, etc. Note: This itertive method does not converge for ll mtrices. There re severl convergence criteri. One of the most populr sys tht convergence is gurnteed for the clss of symmetric mtrices. Eigen prolems versus resolution methods In the ove prgrph we hve spoken out the generl method for resolving eigen-prolems. It strts form the chrcteristic polynomil, nd uilds the solutions step-y-step. It is vlid for ny kind of mtri, with rel or comple eigenvlues. Unfortuntely, this method cn e used only for mtrices with low dimensions. When the mtri size is lrger thn, this method ecomes quite tedious, long, nd inefficient. To overcome this, mny lgorithms hve een developed. Generlly, they clculte ll eigenvlues nd eigenvectors y efficient itertive methods. The price is tht those methods re not generl ut re specilized for prticulr types of mtri clsses. Very efficient lgorithms eist for the symmetric mtri clss, ut the sme lgorithms cnnot work, for emple, with comple eigenvlues mtrices. So, for specific eigen-prolem, we hve to nlyze which method cn e pplied. Mtri.l offers severl different methods; their rnges of ppliction re summrized in the following tle Method Rel eigensystem Symmetric rel mtri Rel mtri Comple eigensystem Rel mtri Comple mtri Jcoy yes no no no QR fctoriztion yes yes yes yes Power yes yes no no Chrcteristic polynomil yes yes yes yes Inverse itertion yes yes yes yes Singulr system yes yes yes yes There re lso specil, highly efficient lgorithms for tridigonl nd Toeplitz mtrices. 85

87 Jcoi trnsformtion of symmetric mtri For rel symmetric mtrices, Jcoi's method is convergent, nd gives oth eigenvlues nd eigenvectors. It consists of sequence of orthogonl similrity trnsformtions, ech of them clled Jcoi rottion - is just plne rottion tht nnihiltes one of the off-digonl elements. Referring to the prgrph "Fctoriztion methods", this method gives us two mtrices: D (eigenvlues) nd U (eigenvectors), eing: lim A n n λ λ n lim P P... P n P U Emple - Solve the eigenprolem for the following symmetric 55 mtri n n We note how clen this method is. Just plin nd strightforwrd! By defult, oth functions use itertions to rech this highly ccurte result. Sometimes, for lrger mtrices, you my need to increse this limit, otherwise you my hve to ccept lower precision. Tip. Jcoi's lgorithm returns eigenvlues in the min digonl. If you like to etrct them in vector, the function MDigEtr comes in hndy. 86

88 Emple - Compute the first steps A, A,... A6 of Jcoi's lgorithm nd study the convergence of the previous emple Ech step of Jcoi's rottion method mkes zero the two highest off-digonl vlues. At susequent steps these zeros cnnot e preserved, ut the off digonl elements re getting lower nd lower step y step. The digonliztion error indictes this convergence, slow ut ineorle, to zero For symmetric mtri, convergence is lwys gurnteed. In our emple, fter 5 steps, we hve n verge digonliztion error of only out. 87

89 Orthogonl mtrices The eigenvector mtri returned y the Jordn lgorithm is "orthogonl" with ech vector hving norm ; tht is, n "orthonorml" mtri Indicting the sclr product with the symol the norml nd orthogonl conditions re: δ i j i j i j ij 4 Orthogonl mtrices hve lso other interesting fetures. If U is orthogonl, we hve U - U T If U is lso orthonorml; we hve det(u) In other words, the sclr product of vector with itself must e ; for ny other vector it must e. ( δ ij is clled Kroneker's symol) Py ttention: the second sttement is not invertile. There re mtrices with det tht re not orthogonl t ll. det The mtri t the left, for emple, hs det (unitry) ut is not orthogonl. Also, ll the Trtgli mtrices, encountered in the previous chpters, hve lwys det, ut they re never orthogonl. Emple - verify the orthogonlity of the eigenvector mtri of the ove emple ProdScl To verify, we cn clculte the sclr cross product of ech pir of columns with the help of the function ProdScl. But this will tedious for lrge mtri. It is fster to use the identity U U T I, s shown in the ove worksheet. Tip. Often, mtri product genertes round-off errors, s in this cse. We cn sweep them up with the function MMopUp 88

90 Eigenvlues with the QR fctoriztion method Another populr lgorithm to find ll eigenvlues of mtri is the QR fctoriztion method. Its hert is the following fctoriztion of mtri A: A Q R where Q is orthonorml nd R is upper tringulr This fctoriztion is lwys possile; you cn esily perform such fctoriztion in Mtri.l with the function MQR. This method pplies the following steps:. Fctorize the given mtri A Q R. Multiply the two fctors R nd Q otining new mtri A R Q. Fctorize the new mtri A Q R nd then repet steps nd We hve the itertive process, strting with A: A Q R A R Q A Q R A R Q A Q R A R Q A p Q p R p A p+ R p Q p If the eigenvlues ll hve distinct solute vlues: λ > λ > λ >...> λ n nd A is symmetric, then the mtri A p converges to digonl form, where the elements re the eigenvlues of A With the function MQRiter it is very esy to test how this process works. Emple - clculte the first nd steps of the QR lgorithm for the following symmetric mtri hving the eigenvlues,,, 4, We use the function MQRiter to perform the first steps of the QR lgorithm. The convergence to the digonl form is evident, nd ecomes closer fter itertions. Note the eigenvlues,,, 4, 5 ppering in the digonl When the given mtri is not symmetric the method works the sme; only the finl mtri is tringulr insted of digonl. See the following emple. 89

91 Emple - clculte the first nd steps of the QR lgorithm for the following symmetric mtri hving the eigenvlues,,, 4, We use the function MQRiter for performing the first step of the QR lgorithm. The convergence t the tringulr form is evident nd ecomes more close fter itertions. Note the eigenvlues,,, 4, 5 ppering in the digonl Does the QR method lwys converge? There re cses - very rre indeed - where the lgorithm fils. This hppens for emple when the eigenvlues re equl nd opposite. Let's see this emple Emple - The following () mtri hs the eigenvlues λ 9, λ 9, λ 8. Applying the QR method we get In this simple cse QR fils (we note the two -9 off-digonl elements). It ws not le to find the two opposite eigenvlues ± 9, ut it hs found only the 8 one. Note tht, under the sme conditions, the Jcoi lgorithm finds ll the eigenvlues, ectly. 9

92 Rel nd comple eigenvlues with the QR method Strting from the simple QR method shown ove, more generl QR lgorithm ws developed with importnt improvements - shifting for rpid convergence, Hessenerg reduction, etc. The result is very roust nd efficient generl QR lgorithm 8 tht cn find comple nd rel eigenvlues of ny rel mtri. This tsk is performed y the function MEigenvlQR of mtri.l Emple: find ll eigenvlues of the given symmetric mtri As previous shown, this mtri hs the first 8 nturl eigenvlues,,, 4, 8 We use MEigenvlQR to find ll eigenvlues in very strightforwrd wy The function cn lso return comple eigenvlues. Let s see this emple This mtri hs rel nd 4 comple conjugte eigenvlues, 4, ± j, ±.5j Note how clen, esy nd fst is the eigenvlue computtion, even in this cse 8 Mtri.l uses the routines HQR nd ELMHES derived from the Fortrn 77 EISPACK lirry 9

93 Comple eigenvlues of comple mtri with the QR method The function MEigenvlQRC performs the comple implementtion of the QR lgorithm for generl comple mtri Emple. Find the eigenvlues of the following mtri 5 4 j A 5 4 j 77 9 j 5 8 j j 4 9 This function ccepts lso the compct rectngulr input formt "+j" Note tht the roots re lwys returned in split formt How to test comple eigenvlues This test is conceptully very esy. We hve only to compute the determinnt of the chrcteristic mtri A λ I For this tsk the functions MChrC nd MDetC re useful When the mtri size ecomes lrger, round-off errors my msk the finl result, nd the eigenvlue check my e not so esy nd strightforwrd. Just to give you n ide of the prolem, let's see the following emple Emple. Given the following ( ) rel mtri, prove tht is n eigenvlue We cn rrnge worksheet test like tht 9

94 If we compute the determinnt of the mtri A λ I, we see, surprisingly, tht it is much more thn zero. Wht is wrong? The fct is tht we hve computed the determinnt with 5 digits floting point rithmetic nd the round-off errors hve msked the finl true result. If we repet the computtion in integer mode, for emple, with the function MDet with the prmeter IMode True, we get the correct result Note tht, in generl, we cn hve non-integer mtrices or we cn hve non-integer eigenvlues, so we cn not lwys use the trick of ect integer computing. Pertured eigenvlue method. In tht cse we should study the ehvior of the determinnt round the given eigenvlue. We cn dd rndom little increment ε to the eigenvlue, registering the corresponding solute vlue of the determinnt. With the id of the ove functions, this process ecomes quite hndy. For emple, giving incrementl steps from E-4 to., we cn esily get the following tle nd plot DET E+ E+9 E+7. E-4 E- E- E-8 E-6.. 9

95 How to find polynomil roots with eigenvlues In previous emple we hve shown how to compute eigenvlues y polynomil roots. Sometimes the contrry hppens: we hve to find polynomil roots y eigenvlue methods. Emple - Find ll the roots of the given 4 th degree polynomil We need to get mtri hving s its chrcteristic polynomil the given polynomil. The compnion mtri is wht we need. It cn e esily uilt y hnd or - even etter - y the function MCmp When we hve the mtri, we cn pply method to find the eigenvlues. As the mtri is symmetric, we choose the QR method. Eigenvlues re lso the roots of the given polynomil. Rootfinder with QR lgorithm for rel nd comple polynomils The QR method is so roust nd efficient tht it is implemented in the rootfinder function PolyRootsQR nd PolyRootsQRC of Mtri.l Thnks to its efficiency, it is especilly dpt for higher degree polynomil. Let see this emple In the left th degree polynomil ll roots re rel. The right th degree polynomil hs oth comple nd rel roots with doule multiplicity. In the first cse the generl ccurcy is out E- 9; in the second one is out E-6. Even in this difficult cse the QR lgorithm returns sufficient 94

96 pproimtion of ll the roots It is the min dvntge of this method, tht it hs : good stility for ll roots configurtions nd voids the disstrous ccurcy loss, chrcteristic of other rootfinding lgorithms. The function PolyRootsQRC works in similr wy for comple polynomils. Emple. find the roots of the following polynomil i ( ) The power method The power method cn find the dominnt rel eigenvlue - the eigenvlue tht hs the highest solute vlue - nd its ssocited eigenvector of rel mtri. This ncient method, still very populr, hs some dvntges: It is conceptully simple in its first proposition; It is roust; It works with oth rel symmetric nd symmetric mtrices It hs n importnt didctic mening With the mtri reduction method it cn itertively find ll rel eigenvlues nd eigenvectors But let us egin to understnd the hert of the lgorithm: For the ske of simplicity we will ssume mtri with independent eigenvectors,, nd dominnt eigenvlue λ, i.e., λ > λ > λ. Tke n ritrry vector v - clled the strting vector - nd clculte the Ryleigh quotient (rtio) with the formuls: Iterting, we hve: v v Av T v v T v v Av r.. v r v T v T v T vn vn+ vn+ Avn r T vn vn Under certin conditions, the rtio converges to the dominnt eigenvlue for n >> nd the ssocited eigenvector cn e otined y the formuls: n lim r λ lim vn( λ ) n n We shll see how it works in prcticl cse Emple - Anlyze the convergence of the power method for the following mtri The mtri hs three seprte eigenvlues: λ, λ, λ 95

97 Let's see how to rrnge the worksheet. First of ll, insert the formuls s indicted to the left; then, select the pproprite rnge nd drg it to the right to iterte the formuls. Assume the strting vector to e v (,, ) Insert the formuls in column E Select the rnge E:E nd drg it to right As we cn oserve, the convergence to the dominnt eigenvlue λ nd its ssocited eigenvector (,, ) is slow ut evident. Rescling. We note lso first drwck of this method:; the vlues of vector v ecome lrger step fter step. This could cuse n overflow error for higher numer of steps. To void this, the lgorithm is modified y inserting vector-rescling routine fter fied numer of steps. v9 v v9 v - -E-4 E rescling dividing for The vlue of the rescling fctor is not very importnt; the mgnitude is the min thing. Note lso tht the Ryleigh rtio is not ffected y rescling Finding non-dominnt eigenvlues. Once the dominnt eigenvlue λ nd its ssocited eigenvector re found, we my wnt to continue to compute the remining eigenvlues. Compute the normlized vlue of nd the new mtri A : u / A A - λ u u T The mtri A hs the eigenvlues:, λ, λ. Now, the dominnt eigenvlues of A is λ Therefore we cn pply the power method once more. 96

98 Emple - reduce the mtri A of the previous emple with the eigenvlue λ nd eigenvector (,, ). Repet the power method to find the dominnt eigenvector λ The mtri A is the new reduced mtri. It should hve ll the eigenvlues of the originl mtri A, ecept λ. Let's see. Repeting the power method we will find its dominnt eigenvlues. Choosing (,, ) for strting vector, we hve something like this: As we cn oserve, the convergence to dominnt eigenvlue λ nd its ssocited eigenvector (-,.5, -.75) is slow ut evident. After 5 steps the error is less thn out E-6 The process power method + mtri reduction cn e iterted for ll eigenvlues. We hve to relize tht, since the computed eigenvlues re pproimtions, round-off errors will e introduced in the net itertion steps; the lst eigenvlue could e ffected y considerle round-off error. In generl, the mtri reduction (or mtri defltion) method ecomes more inccurte s we clculte more eigenvlues, ecuse round-off error is introduced in ech result nd ccumultes s the process continues. Does the power method lwys converge? Although it hs worked well in the ove emples, we must sy tht there re cses in which the method my fil. There re siclly three cses: The mtri A is not digonlizle; tht mens tht it does not hve n linerly independent eigenvectors. Simple, of course, ut it is not esy to tell y just looking t A how mny eigenvectors there re. The mtri A hs comple eigenvlues The mtri A does not hve very dominnt eigenvlue. In tht cse the convergence is so slow tht the m itertion limit my hve to e etended 97

99 Eigensystems with the power method In Mtri.l the power method is implemented y two min functions: MEigenvecPow returns ll eigenvectors MtEigenvlues_pow returns ll eigenvlues Just simple nd strightforwrd. Let's see Emple - solve the eigenprolem for the following symmetric mtri The function MEigenvecPow hs second prmeter: Norm. If TRUE, the function returns normlized eigenvectors (defult FALSE). Becuse of the symmetry, the eigenvector mtri U is lso orthogonl. To prove it, simple check the reltion I U U T s shown it the ove worksheet. Emple: solve the eigenprolem for the following symmetric 66 mtri. This mtri hs eigenvlues,, 6, 9,, 5 The power method works lso for symmetric mtrices. In this emple we hve left the round-off errors to give n ide of the generl ccurcy. Eigenvlue errors re shown in the lst column. 98

100 Comple Eigensystems In Mtri.l the eigen prolem of generl comple mtri is solved with the id of the following min functions: MtEigenvlues_QRC returns ll the eigenvlues y the comple QR lgorithm MEigenvecInvC returns ll distinct eigenvectors y inverse itertion MEigenvecC returns the eigenvectors of ssocited eigenvlues Emple. Find eigenvlues nd eigenvectors of the following comple mtri +4j -+j +j 4-j -j -7+j -6-j --j +7j In this cse the eigenvlues re ll distinct, therefore we cn quickly otin the ssocited eigenvectors y the inverse itertion lgorithm Note tht the eigenvectors returned y the function MEigenvecInvC hve lwys unit solute mgnitude (norm ). For chnging the normliztion type we cn use the function MNormlizeC. When the eigenvlues re not ll distinct we cnnot use the inverse itertion ut insted should use the singulr system method performed y the MEigenvecC Emple. The following mtri hs only distinct eigenvlues:, nd j Note tht the eigenvlue λ j with multiplicity hs two ssocited eigenvectors returned in ( 4) rry. The eigenvlue λ hs one ssocited eigenvector returned in the lst ( ) rry 99

101 How to vlidte n eigen system Emple - Check the rel eigen system of the previous emple In order to test n eigenvector mtri U of given mtri A, we cn use the definition A U (λ u, λ u,... λ 6 u 6 ) But, efore testing, we show how to rrnge the eigenvector mtri in order to void non-integers. This is not essentil, ut it helps the visul inspection. First of ll, we egin with eliminting round off error y using the function MMopUp Now, for ech column, we choose the pivot, tht is, the solute minimum vlue, ecept the zeros. Multipling ech pivot y the corresponding eigenvector we otin new integer vector tht it is still n eigenvector The mtri on the left is otined y multiplying the originl mtri y its eigenvector mtri: A U. The mtri on the right is otined y multiplying ech eigenvector u i for its corresponding eigenvlue. Becuse the two mtrices re identicl, the eigensystem (eigenvectors + eigenvlues) is correct.

102 How to generte rndom symmetric mtri with given eigenvlues Mny times, for testing lgorithms, we need symmetric mtri with known eigenvlues For uilding this test mtri, the following simple method cn e useful First, we generte rndom (n ) vector, v Then we generte the Householder mtri H with the vector v We crete digonl ( n n) mtri D with the eigenvlues tht we wnt to otin. Finlly we mke Similrity Trnsformtion of mtri D y the mtri W. The result is symmetric mtri with the given eigenvlues. Emple: Suppose we wnt ( ) rndom symmetric mtri with eigenvlues (,, 4) Choose rndom vector v, like for emple: Build the ssocited Householder mtri H Set the digonl mtri D T v v H I v v / / / D / 4 / / / / / Perform the similrity trnsformtion of D y H A H 5/ 9 A H / 9 / 9 / 9 6 / 9 8/ 9 / 9 8/ 9 / 9 Note tht, in this cse, the inverse of H is the sme s H. The resulting mtri A hs the wnted eigenvlues (,, 4) If we wnt to void frctionl numers we cn multiply the mtri A y 9 nd get new symmetric mtri B 5 B 9 A The eigenvlues of B re now multiples of 9; thus 9, 8, 6 As we cn see, this method is generl, nd cn e very useful in mny cses: for testing lgorithms, formuls, suroutines, etc. In the dd-in Mtri.l, there re functions for generting Householder mtrices nd performing the Similrity Trnsform.

103 All these ctions re performed y the function MRndEigSym Eigenvlues of tridigonl mtri Tridigonl mtrices re very common in prcticl numericl computtion. These mtrices cn e hndled with ll methods shown efore, ut there re dedicted lgorithms, more efficient nd fster, to solve those specilized eigenvlues prolem. We hve to consider tht mny times prolem involving tridigonl mtrices hs quite lrge dimension. Also, the storge of tridigonl mtri should e considered. A generl full mtri requires 9 cells, ut for tridigonl one with the sme dimension we need to store only 9 cells, sving more thn 9%. Clerly, pying prticulr ttention to storge is quite importnt. Mtri.l contins the following specilized functions pplicle to tridigonl mtrices: MEigenvlQL finds ll rel eigenvlues with the QL lgorithm MEigenvecT computes the eigenvector of rel eigenvlue MtEigenvlTTpz finds ll eigenvlues for toeplitz tridigonl mtri All these function ccept the mtri either in stndrd (n n) form or in compct (n ) form For tridigonl mtrices there re severl useful lemms tht help us to find the eigenvlues One rule sys tht: If ll perpendiculr couples of elements hve the sme sign, thn the mtri hs only rel eigenvlues (The condition is sufficient.) So we cn pply the fst QL lgorithm to clculte ll 5 eigenvlues of the given mtri

104 In the following emple we hve computed ll eigenvlues nd the first 4 eigenvectors with very good pproimtion (out E-4) Note tht the eigenvectors returned y MEigenvecT re not normlized. Use for this tsk the MNormlize function. Eigenvlues of tridigonl Toeplitz mtri ) In numeric clculus it is common to encounter symmetric, tridigonl, toeplitz mtrices like the following. For this kind, there is nice close formul giving ll eigenvlues for mtrices of ny dimension. If the symmetric mtri hs n n dimension, eigenvlues re: λ k kπ + cos n + where k, n We mke the following oservtions: All eigenvlues re rel nd distinct when the mtri is symmetric All eigenvlues re symmetric round the point "" For n odd there eists the trivil eigenvlue λ All roots lie inside the intervl < λ κ < + Also the eigenvector mtri cn e written in compct closed form. U u u... u n If the symmetric mtri hs the n n dimension n n, the u u... u elements of the eigenvectors mtri re: n π u ik sin i k un un... unn n + where i, n, k, n

105 The unsymmetricl tridigonl toeplitz cse cn e led ck to the ove one. We distingue two cses: ) The su-digonls hve the sme sign. In tht cse we cn demonstrte tht ll roots re rel nd distinct. c A... c... c... c If the mtri hs the dimension n n, nd c >, the eigenvlues re: λ + k kπ c cos n + where k, n All roots lie within the intervl: c < λ k < + c ) The su-digonls hve different sign. In tht cse we cn demonstrte tht ll roots re comple conjugte for n even; for n odd there eists only one rel root, λ. c A... c... c... c If the mtri hs the dimension n n, nd c <, the eigenvlues re comple: kπ λ k + i c cos + iδ n + where k, n k All roots lie inside the segment: re k ) (λ c < im( λk ) < c Eigenvectors cn e computed y the following itertive lgorithm k λ k ( u c u ) u ik k ( i ) k ( i ) k where : k, n, i, n u k, uk k Emple Find ll eigenvlues of the following tridigonl toeplitz 8 8 mtri

106 We oserve tht the vlues of the su-digonls in the lower nd upper tringles hve the sme signs, so tht ll eigenvlues re rel nd distinct. They cn e otined y the following closed formul: λ + k kπ c cos n + for k,, 8 where,, c 4, n 8 giving the following 8 eigenvlues λ λ λ λ λ λ 6 8 λ λ All eigenvlues re contined into within the intervl ( 4, + 4) (6, 4) Emple Find ll eigenvlues of the following tridigonl toeplitz 7 7 mtri We oserve tht the su-digonl vlues hve different signs, nd tht the dimension n is odd, so tht ll eigenvlues re comple conjugte ecept one rel, trivil root t λ. The eigenvlues cn e otined from the following closed formul: kπ λ k + i c cos + iδ n + k for k,, 7 where,, c, n 7 giving the following 7 eigenvlues. rel im λ λ λ.8994 λ 4 λ λ 6 - λ Emple Find ll eigenvlues of the following tridigonl toeplitz 8 8 mtri 5

107 We oserve tht the su-digonl vlues hve different signs, nd the dimension n is even, so tht no rel eigenvlues eist, nd ll eigenvlues re comple conjugte. They cn e otined y the following closed formul: kπ λ k + i c cos + iδ n + k for k,, 8 where,, c, n 8 giving the following 8 eigenvlues. rel im λ λ λ λ λ λ 6 - λ λ Emple Find ll eigenvlues of the following tridigonl toeplitz 8 8 mtri We oserve tht the mtri is symmetric so ll eigenvlues re rel nd distinct. They cn e otined y the following closed formul: kπ λk + cos n + for k,, 8 where,, c, n 8 giving the following 8 eigenvlues λ λ λ - λ

108 λ λ 6 - λ λ All eigenvlues re contined in the intervl (, + ) ( 4, ) We oserve tht they re ll negtive The eigenvector mtri cn e otined in very fst wy using the formul u ij π sin i j n + sin( α) sin(α ) U... sin(8α ) sin(α ) sin(4α )... sin(6α ) sin(8α ) sin(6α )... sin(64α ) Tht gives the following pproimte eigenvector mtri Note tht the column-vectors re orthogonl. 7

109 Generlized eigen prolem The mtri eqution A λ B () where A nd B re oth symmetric mtrices, nd B is positive definite, is clled generlized eigen prolem. Equivlent symmetric prolem This prolem is equivlent to: (B - A) λ C λ () In generlly C is not symmetric even when A nd B re. Emple: trnsform generlized eigen-prolem into stndrd eigen prolem, where the mtrices A nd B re A B In the following worksheet we hve clculted the mtri C B - A As we cn see, the mtri C is not symmetric even if A nd B re oth symmetric. In order to clculte the eigenvlues we hve, efore, etrcted the chrcteristic polynomil with the function MthChrPoly; then pproimted its roots with the function PolyRoots. The pproimte eigenvlues re: λ.77 λ. λ.9444 To solve the eigenvectors we cn now follow the step-y-step method shown in the previous emples. But, we cn lso trnsform the given generlized prolem into symmetric one. Let's see how. 8

110 Equivlent symmetric prolem Given the following mtri eqution A λ B () where A nd B re oth symmetric mtrices nd B is positive definite. In the previous prgrph we hve seen how to trnsform this prolem into stndrd eigenprolem y setting C B - A. But C is not symmetric. Mny lgorithms only work well for symmetric mtrices. By contrst, there is no eqully stisfctory lgorithm for the symmetric cse. So, it is etter to convert the prolem into symmetricl mtri, y the Cholesky's decomposition B L L T () Where L is tringulr mtri. Sustituting () into () nd multiplying the eqution y L -, we get: L - A λ (L - L) L T L - A λ L T And, ecuse I (L T ) - L T (L - ) T L T, we cn write: L - A (L - ) T L T λ L T L - A λ L T After setting the uiliry mtri: W equl to L -, nd the uiliry vector d to L T, we hve W A W T d λ d D d λ d () Eqution () is the new eigen prolem where D W A W T is symmetric Eigenvlues of prolem () re equivlent to () while the originl eigenvectors cn e otined from the eigenvectors d y the following formul: d L T ( L T ) - d ( L - ) T d W T d Tht is, eigenvectors of () cn e otined y multiplying eigenvectors of () y the uiliry mtri W. Mtri.l contins everything you need to solve generlized eigen prolems: Cholesky decomposition cn e done y the function MCholesky; eigenvectors nd eigenvlues of symmetric mtrices cn e clculted with Jcoy itertive rottions performed y the two functions MEigenvlJcoi nd MEigenvecJcoi. Thus, let's see how to rrnge worksheet for solving generlized eigen-prolem, ssuming the mtrices A nd B of the previous emple. The following worksheet contins ll formuls shown efore. Formuls used for ech mtri re written in lue, under the mtri itself. 9

111 Digonl mtri The cse in which the mtri B is digonl is prticulrly simple ecuse L is digonl too nd cn e computed y simple squre root. Also the L - is quite simple: just tke the inverse of ech digonl element. B L L

112 Emple - How to get mode shpes nd frequencies for structure with multiple degrees of freedom 9 Emple - Our prolem is n emple of the "generlized" eigenprolem: k φ ω m φ () where k nd m re oth symmetric positive definite mtrices. In this specific cse they were: Stiffness mtri k: Mss mtri m: This prolem is equivlent to "stndrd" eigenprolem: (m - k) φ ω φ C φ ω φ The prolem is tht C is not symmetric. One cn work round this prolem y converting the prolem to symmetric one using the Cholesky decomposition m L L T where L is tringulr mtri. In cse like ours, where m is digonl, the L mtri is lso digonl, with ech term of L eing the squre root of the corresponding term in m. Define new mtri W s: W L - Multiplying eqution () y W, one gets: W k W T (L T φ) ω (L T φ) or, more concisely, where D v ω v () D W k W T () The eigenvlues for eqution () re identicl to those of eqution (), nd the eigenvlues of eqution () cn e otined esily from the eigenvlues of eqution (): φ ( L T ) - v W v (4) So here is wht you do: Strting with k nd m, mke L ; then W ; nd then D. 9 This emple comes from true prolem proposed to me y Dougls C. Sthl of the Architecturl Engineering nd Building Construction of the Milwukee School of Engineering. Becuse it seems to me very interesting lso for other people, I decide to pulish it in this tutoril, in the version rrnged y Doug nd me.

113 Clculte the eigenvlues nd eigenvectors for D, with the functions mteigenvlue_jcoi nd mteigenvector_jcoi contined in the dd-in MATRIX. Allow for numer of itertions lrger thn 4. These eigenvlues re the ones you wnt. These re the correct squred frequencies for our prolem. The eigenvectors must e converted using eqution 4. They re the correct mode shpes for our prolem. The eigenvectors re lredy orthonormlized. Emple - Seven inerti torsion system This emple shows how to solve lrger torsion system with good ccurcy. Assume to hve the following torsion system eqution K φ ω M φ () where the mtrices K nd M re M Thnks to Anthony Grci

114 K Tip. Scling the given mtri for suitle fctor my increse the computing ccurcy y severl orders. In this cse we divide the K mtri for fctor 6. The eigenvlues re proportionlly scled y the sme fctor. In fct, multiplying oth sides of eqution () y the sme scling fctor, we hve: -6 K φ -6 ω M φ K' φ λ M φ where K' -6 K nd ω 6 λ K' The Cholesky fctoriztion of M cn e computed esily ecuse it is digonl mtri L [ (m ) /, (m ) /,... (m 77 ) / ] The uiliry mtri is the inverse of the L mtri; ut lso in this cse, it is very esy to compute the inverse, s W L - [ /L, /L,... /L ] Now we compute the mtri [D][W][K'][W] T Note tht W T W ecuse W is digonl. y the function MProd

115 Applying the Jcoy lgorithm or, even etter, the QL lgorithm, to the symmetric tridigonl mtri [D], we get ll its rel eigenvlues. Multiplying them y the fctor 6, we finlly hve the eigenvlues of the given torsion system The eigenvectors of D my e computed y the Jcoi lgorithm or y the inverse itertion Here we hve used the function MEigenvecJcoi Multiplying the Vd mtri y the uiliry W mtri we find the eigenvectors of the given system tht cn e normlized s we like y the function MNormlize 4