Methods to Invert a Matrix

Transcription

1 Lecture 3: Determts & trx Iverso ethods to Ivert trx The pproches vlble to fd the verse of mtrx re extesve d dverse. ll methods seek to solve ler system of equtos tht c be expressed mtrx formt s for the ukows coted the vector {x},.e., [ ]{ x } { b } { x } [ ] { b } The methods used to ccomplsh ths c be loosely grouped to the followg three ctegores: methods tht explctly clculte {x} methods tht mplctly clculte {x}, d tertve methods tht clculte {x} Of course hybrd methods exst tht re combtos of two or methods the ctegores lsted bove.

2 Lecture 3: Determts & trx Iverso Cosder the followg lst of methods whch s ot comprehesve:. Explct methods for sprse mtrces cludes Crmer s rule whch s specfc cse of usg self djot mtrces. Vrtos clude:. Guss elmto - Row echelo form; ll etres below ozero etry the mtrx re zero - Bress lgorthm; every elemet computed s the determt of [] - Tr-dgol mtrx lgorthm; specl form of Guss elmto b. Guss-Jord elmto. LDU decomposto mplct method tht fctors [] to product of lower d upper trgulr mtrces d dgol mtrx. Vrtos clude. LU reducto specl prllelzed verso of LDU decomposto lgorthm - Crout mtrx decomposto s specl type of LU decomposto 3. Cholesky LDL decomposto mplct method tht decomposes [], whe t s postvedefte mtrx, to the product of lower trgulr mtrx, dgol mtrx d the cojugte trspose. Frotl solvers used fte elemet methods b. Nested dssecto for symmetrc mtrces, bsed o grph prttog c. mum degree lgorthm d. Symbolc Cholesky decomposto

3 Lecture 3: Determts & trx Iverso 5. Itertve methods:. Guss-Sedel methods Successve over relxto (SOR) Bck ft lgorthms b. Cojugte grdet methods (CG) used ofte optmzto problems Noler cojugte grdet method Bcojugte grdet method (BCG) Bcojugte grdet stblzed method (BCGSTB) Cojugte resdul method c. Jcob method d. odfed Rchrdso terto e. Geerlzed mml resdul method (GRES) bsed o the rold terto f. Chebyshev terto vods er products but eeds bouds o the spectrum g. Stoe's method (SIP: Strogly Implct Procedure) uses complete LU decomposto h. czmrz method. Itertve refemet procedure to tur ccurte soluto more ccurte oe 6. Levso recurso for Toepltz mtrces 7. SPIE lgorthm hybrd prllel solver for rrow-bded mtrces Detls d dervtos re preseted for severl of the methods ths secto of the otes. Cocepts ssocted wth determts, cofctors d mors re preseted frst.

4 Lecture 3: Determts & trx Iverso The Determt of Squre trx squre mtrx of order ( x mtrx),.e., [ ] possesses uquely defed d sclr tht s desgted d s the determt of the mtrx, or merely the determt O L det [ ] Observe tht oly squre mtrces possess determts.

5 Lecture 3: Determts & trx Iverso Vertcl les d ot brckets desgte determt, d whle det[] s umber d hs o elemets, t s customry to represet t s rry of elemets of the mtrx [ ] det geerl procedure for fdg the vlue of determt sometmes s clled expso by mors. We wll dscuss ths method fter gog over some groud rules for opertg wth determts. O L

6 Lecture 3: Determts & trx Iverso Rules for Opertg wth Determts Rules pertg to the mpulto of determts re preseted ths secto wthout forml proof. Ther vldty s demostrted through exmples preseted t the ed of the secto. Rule #: Iterchgg g y row (or colum) of determt wth ts mmedte djcet row (or colum) flps the sg of the determt. Rule #: The multplcto of y sgle row (colum) of determt by sclr costt s equvlet to the multplcto of the determt by the sclr. Rule #3: If y two rows (colums) of determt re detcl, the vlue of the determt s zero d the mtrx from whch the determt s derved s sd to be sgulr. Rule #4: If y row (colum) of determt cots othg but zeroes the the mtrx from whch the determt s derved s sgulr. Rule #5: If y two rows (two colums) of determt re proportol,.e., the two rows (two colums) re lerly depedet, the the determt s zero d the mtrx from whch the determt s derved s sgulr.

7 Lecture 3: Determts & trx Iverso Rule #6: If the elemets of y row (colum) of determt re dded to or subtrcted from the correspodg elemets of other row (colum) the vlue of the determt s uchged. Rule #6: If the elemets of y row (colum) of determt re multpled by costt d the dded or subtrcted from the correspodg elemets of other row (colum), the vlue of the determt s uchged. Rule #7: The vlue of the determt of dgol mtrx s equl to the product of the terms o the dgol. Rule #8: The vlue for the determt of mtrx s equl to the vlue of the determt of the trspose of the mtrx. Rule #9: The determt of the product of two mtrces s equl to the product of the determts of the two mtrces. Rule #0: If the determt of the product of two squre mtrces s zero, the t lest oe of the two mtrces s sgulr. Rule #: If m x rectgulr mtrx s post-multpled by x m rectgulr mtrx B, the resultg squre mtrx [C] [][B] of order m wll, geerl, be sgulr f m >.

8 Lecture 3: Determts & trx Iverso I Clss Exmple

9 Lecture 3: Determts & trx Iverso Cosder the th order determt: ors d Cofctors [ ] det The m th order mor of the th order mtrx s the determt formed by deletg ( m ) rows d ( m ) colums the th order determt. For exmple the mor r of the determt s formed by deletg the th row d the r th colum. Becuse s th order determt, the mor r s of order m d cots m elemets. I geerl, mor formed by deletg p rows d p colums the th ordered d determt t s ( p) th order mor. If p, the mor s of frst order d cots oly sgle elemet from. F th t t th t th d t t t l t f f t d From ths t s esy to see tht the determt cots elemets of frst order mors, ech cotg sgle elemet. O L

10 Lecture 3: Determts & trx Iverso Whe delg wth mors other th the ( ) th order, the desgto of the elmted rows d colums of the determt must be cosdered crefully. It s best to cosder cosecutve rows j, k, l, m d cosecutve colums r, s, t, u so tht the ( ) th, ( ) th, d ( 3) th order mors would be desgted, respectvely, s j,r, jk,rs d jkl,rst. The complemetry mor, or the complemet of the mor, s desgted s N (wth subscrpts). Ths mor s the determt formed by plcg the elemets tht le t the tersectos of the deleted rows d colums of the orgl determt to squre rry the sme order tht they pper the orgl determt. For exmple, gve the determt from the prevous pge, the N 3 3 N 3 3,3 3 33

11 Lecture 3: Determts & trx Iverso The lgebrc complemet of the mor s the sged complemetry mor. If mor s obted by deletg rows, k, l d colums r, s, t from the determt the mor s desgted the complemetry mor s desgted kl, rst N kl, rst d the lgebrc complemet s desgted k l L r s t ( ) N kl, rst The cofctor, desgted wth cptl letters d subscrpts, s the sged ( ) th mor formed from the th order determt. Suppose the tht the ( ) th order mor s formed by deletg the th row d j th colum from the determt. The correspodg cofctor s j ( ) j j

12 Lecture 3: Determts & trx Iverso Observe the cofctor hs o meg for mors wth orders smller th ( ) uless the mor tself s beg treted s determt of order oe less th the determt from whch t ws derved. lso observe tht whe the mor s order ( ), the product of the cofctor d the complemet s equl to the product of the mor d the lgebrc complemet. We c ssemble the cofctors of squre mtrx of order ( x mtrx) t )to squre cofctor mtrx,.e., [ ] C O L So whe the elemets of mtrx re deoted wth cptl letters the mtrx represets mtrx of cofctors for other mtrx.

13 Lecture 3: Determts & trx Iverso I Clss Exmple

14 Lecture 3: Determts & trx Iverso Rules for Opertos wth Cofctors The determt for three by three mtrx c be computed v the expso of the mtrx by mors s follows: det [ ] Ths c be cofrmed usg the clssc expso techque for 3 x 3 determts. Ths expresso c be rewrtte s: det 3 [ ] or usg cofctor otto: det [ ]

15 Lecture 3: Determts & trx Iverso Rule #: determt my be evluted by summg the products of every elemet y row or colum by the respectve cofctor. Ths s kow s Lplce s expso. Rule #3: If ll cofctors row or colum re zero, the determt s zero d mtrx from whch they re derved s sgulr. Rule #4: If the elemets row or colum of determt re multpled by cofctors of the correspodg elemets of dfferet row or colum, the resultg sum of these products re zero.

16 Lecture 3: Determts & trx Iverso The djot trx The djot mtrx s the mtrx of trsposed cofctors. If we hve th order mtrx [ ] O [ ] O ths mtrx possess the followg mtrx of cofctors ths mtrx possess the followg mtrx of cofctors [ ] C O

17 Lecture 3: Determts & trx Iverso d the djot of the mtrx s defed s the trspose of the cofctor mtrx dj C [ ] T [ ] [ ] We wll show the ext secto tht fdg the verse of squre mtrx c be ccomplshed wth the followg expresso: O [ ] dj [ ] For 3x3mtrx 3 ths s kow s Crmer s rule.

18 Lecture 3: Determts & trx Iverso Drect Iverso ethod Suppose x mtrx s post multpled by ts djot d the resultg x mtrx s detfed s [P] [ P ] The elemets of mtrx [P] re dvded d d to two ctegores,.e., elemets tht t le log the dgol p p p

19 Lecture 3: Determts & trx Iverso d those tht do ot p p p p p The elemets of [P] tht le o the dgol re ll equl to the determt of [] (see Rule # d recogze the Lplce expso for ech dgol vlue). Note tht the odgol elemets wll be equl to zero sce they volve the expso of oe row of mtrx wth the cofctors of etrely dfferet row (see Rule #4)

20 Lecture 3: Determts & trx Iverso Thus p p p d 0 p p p p p

21 Lecture 3: Determts & trx Iverso whch leds to or P dj 0 0 [ ] [ ] [ ] [] I Whe ths expresso s compred to the t s evdet tht [] I [ ] dj [ ] [ I ] [ ] [ ] [ ] dj The verse exsts oly whe the determt of s ot zero,.e., whe s ot sgulr. [ ]

22 Lecture 3: Determts & trx Iverso I Clss Exmple

23 Lecture 3: Determts & trx Iverso The drect verso method preseted bove s referred to s brute force pproch. From computtol stdpot the method s effcet (but doble) whe the mtrx s qute lrge. There re more effcet methods for solvg lrge systems of ler equtos tht do ot volve fdg the verse. Geerlly these pproches re dvded to the followg two ctegores: Drect Elmto (ot verso) ethods (LDU decomposto, Guss elmto, Cholesky) Itertve ethods (Guss-Sedel, Jcob) We wll look t methods from both ctegores.

24 Lecture 3: Determts & trx Iverso Drect Elmto ethods Elmto methods fctor the mtrx [] to products of trgulr d dgol mtrces,.e., the mtrx c be expressed s [ ] [ L] [ D] [ U ] Where [L] d [U] re lower d upper trgulr mtrces wth ll dgol etres equl to. The mtrx [D] s dgol mtrx. Vrtos of ths decomposto re obted f the mtrc [D] s ssocted wth ether the mtrx [L] or the mtrx [U],.e., [ ] [ L ] [ U ] where [L] d [U] ths lst expresso re ot ecessrly the sme s the mtrces detfed the prevous expresso.

25 Lecture 3: Determts & trx Iverso I expded formt [ ] u u u u u l l l O O O Th t [L] d [U] th d t t Dff th u l l l O O O The mtrces [L] d [U] ths decomposto re ot uque. Dffereces the my vrtos of elmto methods re smply dffereces how these two mtrces re costructed. I solvg system of ler equtos we c ow wrte s [ ] { } { } b x s [ ] { } [ ][ ] { } { } b x U L x

26 Lecture 3: Determts & trx Iverso If we let the [ U ]{ x} { y} [ L ][ U ] { x} [ L] { y} { b} whch s eser computto. Solvg ths lst expresso for ech y c be ccomplshed wth the followg expresso b l j j y j y,, L, l Wth the vector {y} kow, the vector of ukows {x} re computed from x y u j j j l x, L, The process for solvg for the ukow vector quttes {x} c be completed wthout computg the verse of [].

27 Lecture 3: Determts & trx Iverso The Guss Drect Elmto ethod The Guss elmto method begs wth forwrd elmto process d fds ukows by bckwrd substtuto. Here the decomposto of the x mtrx [] [ ] [ L ] [ U ] s ccomplshed s follows. For ech vlue of (where rges from to ) compute d u u j l l j jj j, L, l j j uk l jk j, L, k If t y stge the elmto process the coeffcet of the frst equto,.e., jj (ofte referred to s the pvot pot) or l jj becomes zero the method fls.

28 Lecture 3: Determts & trx Iverso I Clss Exmple

29 Lecture 3: Determts & trx Iverso Cholesky s Decomposto Drect Elmto ethod I ler lgebr, the Cholesky decomposto or Cholesky trgle s decomposto of Hermt, postve-defte df mtrx to the product of lower trgulr mtrx d dts cojugte trspose. It ws dscovered by dré-lous Cholesky for rel mtrces (s opposed to mtrces wth elemets tht re complex). Whe t s pplcble, the Cholesky decomposto s roughly twce s effcet s the LU decomposto for solvg systems of ler equtos. The dstgushg feture of the Cholesky decomposto s tht the mtrx [], whch s symmetrcl d postve-defte, c be decomposed dto upper d lower trgulr mtrces tht re the trspose of ech other,.e., [ ] [ L ] [ L ] T Ths c be loosely thought of s the mtrx equvlet of tkg the squre root. Note tht [] s postve defte mtrx f for ll o-zero vectors {z} the er product { z} T [ ]{ z} > 0 s lwys greter th zero. Ths s gurteed f ll the egevlues of the mtrx re postve.

30 Lecture 3: Determts & trx Iverso Oce the decomposto hs bee performed, the soluto of the system of equtos proceeds by forwrd d bckwrds substtuto the sme mer s the Guss elmto method. Coveece recurrece reltoshps for the Cholesky decomposto re s follows for ech successve colum ( th dex) l k ( l ) These expressos c be modfed d where the expressos re free of eedg to tke squre root f the prevous mtrx expresso s fctored such tht where g [D] s dgol mtrx. k j l jk lk k l j l j,, [ ] [ L ] [ D ] [ L ] T

31 Lecture 3: Determts & trx Iverso Recurrece reltoshps for the Cholesky LDL decomposto. They re expressed s follows for ech successve colum ( th dex) d l l j k j d k d kk ( l ) k dkkl jk lk j, L, d Wth [] decomposed to trple mtrx product the soluto to the system of equtos proceeds wth { b} [ ] { x} T [ L][ D][ L] {} x [ L] { y}

32 Lecture 3: Determts & trx Iverso g b j l j y j y,, L, l but ow (verfy for homework) x y d k kj j j l l x, L,

33 Lecture 3: Determts & trx Iverso I Clss Exmple

34 Lecture 3: Determts & trx Iverso Guss-Sedel : Itertve ethod For lrge systems of equtos the explct d mplct elmto methods (bckwrds d forwrds) c be dequte from memory storge perspectve, executo tme d/or roudoff error ssues. Itertve soluto strteges re used whe these problems rse. Solutos foud usg tertve methods c be obted wth forekowledge of the error tolerce ssocted wth the method. The most commo tertve scheme s the Guss-Sedel method. Here tl estmte s mde for the soluto vector {x}. Subsequet to ths guess ech equto the system of ler equtos s used to solve for (updte) oe of the ukows from prevous clculto. l For ler system of three equtos three ukows the updtg procedure c be expressed by the followg expressos m m m x x x 3 b b b 3 m m ( ) ( x ) ( ) ( x ) m m ( )( x ) ( )( x ) 3 3 m m ( )( x ) ( )( x ) Note tht m d (m-) represet the curret d prevous terto respectvely.

35 Lecture 3: Determts & trx Iverso I geerl the Guss-Sedel terto formt c be expressed s m m m ( x ) b ( k )( xk ) ( k )( xk ) k k The orgl equtos must be of form where the dgol elemets of [] cot oly ozero vlues. For stble structure, t or compoet wth well-defed dboudry codtos ths wll lwys be possble. For coveece vector of zeroes s ofte used s tl guess t the soluto {x}. Itertos re repeted utl the soluto coverges to close eough to the true soluto. Sce we very seldom kow the true soluto pror (why do the problem f we kow the swer) tertos cotue utl the ext swer obted s oly frctolly dfferet th the swer from the prevous terto. themtclly ths s expressed s wth ε mx,..., m m ( x ) ( x ) m ( x ) x ( 00) (%) ε < ς

36 Lecture 3: Determts & trx Iverso where ς ( q) (%) d q s the desred umber of ccurte sgfct fgures. y techques hve bee developed order to mprove covergece of the Guss- Sedel method. The smplest d the most wdely used s successve over-relxto method (SOR). I ths pproch ewly computed vlue m x s modfed by tkg weghted verge of the curret d prevous vlues the followg mer: m m m ( x ) β ( x ) ( β )( x ) updted Here β s the relxto or weghtg fctor. For over-relxto methods β rges betwee d. The ssumpto s tht usg curret vlue of m x wll slowly l coverge to the exct soluto wheres the modfcto of m x defed bove wll speed covergece log. Whe β s betwee 0 d the pproch s termed successve uder-relxto method Whe β s betwee 0 d the pproch s termed successve uder relxto method (SUR). Choce of β vlue s problem depedet.

37 Lecture 3: Determts & trx Iverso I Clss Exmple

38 Lecture 3: Determts & trx Iverso The Jcob Itertve ethod The Jcob method clcultes ll ew vlues o the bss of the followg tertve expresso m m ( x ) b ( k )( x k ) k k The Guss-Sedel method typclly coverges to solutos fster th the Jcob method, the Jcob terto scheme hs cert dvtges whe soluto of system of equtos s coducted wth prllel processg (multple computers solvg the system t the sme tme).

39 Lecture 3: Determts & trx Iverso The Cojugte Grdet Itertve ethod Cojugte grdet methods re the most populr for solvg lrge systems of ler equtos of the form x b [ ]{ } { } Here {x}s ukow vector d d{b} s kow vector d d[] s kow, squre, symmetrc, postve-defte mtrx. Cojugte grdet methods re well suted for sprse mtrces. If [] s dese the best soluto strtegy s fctorg [] d usg bck substtuto. s oted erler, mtrx s postve defte f for every ozero vector {z} { z } T [ ]{ z } > 0 Ths my me lttle t ths pot becuse t s ot very tutve de. Oe hs hrd tme ssemblg metl pcture of postve-defte mtrx s opposed to mtrx tht s ot postve defte. mtrx tht s postve defte s defed s Hermt mtrx d there re umber of dvtgeous qultes ssocted wth ths ctegory of mtrces.

40 Lecture 3: Determts & trx Iverso If mtrx s postve defte Hermt mtrx the: ll egevlues of the mtrx re postve; ll the ledg prcpl mors of the mtrx re postve; d there exsts osgulr squre mtrx [B] such hthtt T [ ] [ B] [ B] Qudrtc forms re troduced ext d we wll see how the cocept of postve defte mtrces ffects qudrtc forms. The followg sclr fucto ( x ) x bx c f s sd to hve qudrtc form sclr lgebr. If we chge the formulto slghtly such tht f ( x) x bx 0

41 Lecture 3: Determts & trx Iverso Extedg ths equto formt to t lest two vrbles leds to equtos of the form f ( x, x ) ( x x x x ) ( b x b x ) d geerl f j, j ( x ) j x x j j x x j Ths lst expresso suggests tht we c rewrte t mtrx formt s f Where [] s kow xmtrx mtrx, the vector {b} of kow quttes s xvector s s the vector of ukows {x}. Severl observtos c be mde. Frst, the lst two equtos both sdes of the equl sg represet sclr quttes. s quck check o the mtrx expresso let {x} d {b} represet 3 x vectors, d [] s 3 x 3 squre mtrx. If [] s symmetrc d postve-defte the the qudrtc fucto f s mmzed by the soluto of x b j, j T T ({} x ) {}[ x ]{} x {}{} b x [ ]{ } { } k b k x k

42 Lecture 3: Determts & trx Iverso Let s cosder exmple. If we hve the followg system of ler equtos to solve 3 x x x 6x 8 [ ]{ x } { b } the the soluto of ths system, the qudrtc form of the system, d the mmzto of the qudrtc form s depcted the followg three grphs 3 [ ] 6 { x } x x 8 { b }

43 Lecture 3: Determts & trx Iverso ({ x} ) 3( x ) 4xx 6( x ) x 8x f Cotour plot depctg level surfces of the qudrtc fucto f({x}) Surfce plot of the qudrtc fucto f T T ({} x ) {}[ x ]{} x {}{} b x

44 Lecture 3: Determts & trx Iverso Tkg the prtl dervtve of sclr vlued fucto tht depeds o more th oe vrble results prtl dervtves wth respect to ech vrble. Returg to the system wth two ukow vrbles f ( x, x ) ( x x x x ) ( b x b x ) the [ f ( x, x )] x [ f ( x, x ) ] x Ths suggest the followg mtrx formt [ f ( x, x )] ] x x x x x b [ f ( x, x )] x x x b b b

45 Lecture 3: Determts & trx Iverso Now set the lst result equl to the zero vector,.e., 0 x b 0 x b Recll from clculus tht tkg dervtves d settg the resultg expresso(s) equl to zero yelds locl mmum or locl mxmum. We c geerl restte ths mtrx formt s or * { 0} [ ]{ x} { b} * [ ]{ x} { b} Thus the mmum/mxmum pot o the qudrtc surfce f T T ({ x} ) { x} [ ]{ x} { b} { x} occurs t {x} * whch s cocdetly the soluto of the system of ler equtos we wsh to solve.

46 Lecture 3: Determts & trx Iverso We ow seek wy to obt the soluto vector {x} * tertve fsho. Grphclly the tertve soluto would strt t some rbtrry strtg pot (could be the org, but the fgure below the tertve process strts t pot defed by ozero tl vector 0 {x}). Before estblshg tertve soluto techque we defe resdul vector {r} d error vector {e} t the th step.

47 Lecture 3: Determts & trx Iverso Before estblshg tertve soluto techque we defe resdul vector {r} d error vector {e} t the th step. The error vector s expressed s { e} { x} { x} d ths s vector tht dctes how fr we re from the soluto. The resdul vector s defed s { r} { b} [ ] {} x d ths resdul vector dctes how fr the mtrx product [] {x} s from correct vector {b}. Substtuto of the expresso for the error vector to the expresso for the resdul vector leds to the followg reltoshp betwee the two vectors ( ) { r} { b} [ ] { e} { x} {} b [ ] {} e [ ]{} x { b } [ ] { e } { b } {} r [ ] {} e

48 Lecture 3: Determts & trx Iverso ore mporttly, the resdul vector should be thought of terms of the followg expresso ( ) { r} [ ] { x} { b} [ f ( x, L, x ) ] x [ f ( x, Lx )] x {} x Thought must be gve to ths expresso. Ths represets the dervtve of the qudrtc sclr equto evluted t tthe compoets of fthe terto t vector t tthe th step. Thus the resdul vector wll lwys pot the drecto of steepest descet,.e., towrds the soluto pot. Ths s extremely mportt o-ler problems where tertve soluto schemes esly bog dow.

49 Lecture 3: Determts & trx Iverso Now let s puse d thk bout the soluto vector {x} geerl terms. vector wth elemets c be wrtte terms of ler combto of bss vectors. Whe s equl to 3 (.e., Crtes three spce) we c express vector s ler combto of the three ut vectors (,0,0), (0,,0) d (0,0,). But geerl {} { } s x α For the Crtes three spce exmple {} { } α 3 s x t the left we see dcted tht { } { } { } α α α s s s t the left we see dcted tht the α s re sclg fctors. Ths s mportt de to ltch oto. We wll ot tret them s α α α α vector. α α 3

50 Lecture 3: Determts & trx Iverso Next cosder the soluto expressed s the followg vector sum 0 {} x {} x α { s } The fuchs le represets α { s }.e., ler combto of the blue vectors. Ech blue vector correspods to specfc vlue of. The lst gree vector s the soluto d the frst red vector represets the tl guess. 0{} x

51 Lecture 3: Determts & trx Iverso For the frst tertve step the lst fgure 0 { x} { x} α { s } Ths expresso c be geerlzed to the followg tertve form { x} { x} α { s } The terto c be cotued utl soluto s obted tht s close eough. The ssues bol dow to fdg the α s,.e., the sclg fctors s well s the bss vectors, e.e., thess s. Let s see f we c use the cocept of the drecto of steepest descet to defe the bss vectors d the scle fctor t ech step.

52 Lecture 3: Determts & trx Iverso Retur to the exmple d use the strtg vector 0 {x} (-, -). Our frst step should be the drecto of steepest descet towrds the mmum. Tht drecto s the sold blck le the fgure below The ext ssue s how fr log the pth of steepest descet should we trvel? If we df defe the step s or geerl { x} { x} α { r} {} x { x} α { r} the how bg should the step be? le serch s procedure tht chooses the α to mmze f log drecto. lso ote tht the resdul vector s ow beg used s bss vector the expressos bove.

53 Lecture 3: Determts & trx Iverso The followg fgure llustrtes wht we re tryg to do. The process s restrcted to selectg pot log the surfce of the fucto f d the ple tht defes the drecto of steepest descet from the pot (-, -). 0 {} {r} s we trverse log the le of steepest descet we c plot the mgtude d the drecto of the grdet to the surfce t y pot o the le. We stop serchg log the le whe the grdet (dervtve) vector s orthogol to the le of steepest descet. The sclg fctor α s ow defed. ew serch drecto {r} The ew (red) serch drecto s updted drecto of deepest descet d wll be perpedculr to the old (blue) drecto.

54 Lecture 3: Determts & trx Iverso From the geometry of the prevous fgure we c see tht resdul vector 0 {r} d the updted resdul vector {r} re orthogol. Thus the umercl vlue for α c be obted from the followg dot product tke betwee these two vectors 0 T 0 { r} { r} { b } [ ] { x } ({} b [ ] 0 T {} x ) 0{} r [ ] {} 0 T {} r 0{} r [ ] {} T 0 ( ) { r } T 0 ({} b [ ] ( {} x α {} r ) {} r 0 T T 0 ( { b } [ ] { x } ) { r } α ( [ ] { r } ) { r } 0 0 T 0 α ( r ) {} r 0 0 T 0 α ( r ) {} r or 0 r T 0 0 { } { r } α 0 {} r T [ ] 0 {} r

55 Lecture 3: Determts & trx Iverso The tertos there most bsc formt re s follows { } { } [ ] {} x b r { } { } {} [ ] {} r r r r T T α { } { } { } r x x α