Miimizatio of the quadatic test fuctio A quadatic fom is a scala quadatic fuctio of a vecto with the fom f ( ) A b c with b R A R whee A is assumed to be SPD ad c is a scala costat Note: A symmetic mati A is SPD (symmetic positive defiite) if A > o equivaletly if all the eigevalues of A ae positive he miimize of the fuctio f is give as the poit whee the gadiet of the fuctio is equal to zeo Diect calculatio gives the optimality coditios: f A b he statioay poits ae obtaied by solvig the liea system A b Note: his shows that the same that miimizes f() also seves as the solutio to the liea system of equatios A b ad the uiqueess of the solutio is guaateed by the SPD coditio he Method of Steepest Descet he method of steepest descet geeates a sequece of iteates appoimatig a local miimize of f() At iteatio a give poit is assumed to be a appoimatio of the eact solutio he eo is give by the esidual vecto: A b We ae at the poit How do we each? he idea is to take the diectio i which f() deceases most quickly (ie the gadiet) so the diectio of steepest descet is give by: d -f( ) ( A b) his is the easo fo the ame steepest descet o gadiet method hus we seach fo a bette solutio i the diectiod ie we have to detemie the coefficiet R such that the value of the eal fuctio he miimum of the fuctio h is eached at h( ) f( f ( )) mi A ad thus the et appoimatio of the solutio is give by mif We epeat this fo evey step takig the gadiet of f () i the et poit ad by fidig a ew step legth Eample:
We apply the Method of Steepest Descet to the fuctio f ( ) 4 4 with iitial poit X We fist compute the steepest descet diectio fom 8 4 f ( X ) 4 4 to obtai 4 f ( X ) f 4 We the miimize the fuctio 4 h( ) f(x f ( X )) f( ) f ( 4 4 ) 4 h( ) 4(-4 ) 4(-4 )(-4 ) (-4 ) by computig h ( ) 64 - his fuctio has a global miimum wheh( ) o 5 We theefoe set 4 X 5 4 Cotiuig the pocess we have 4 f ( X) f 4 ad by defiig - 4 h( ) f(x f ( X)) f( ) f (4 4 ) 4 h( ) 4(4 ) 4(4 )(-4 ) (-4 ) we obtai h ( ) - We have h( ) whe We theefoe set - 4 4 X 4 6 Repeatig this pocess yields X We ca see that the Method of Steepest Descet poduces a sequece of iteates X that is covegig to the stict global miimize of f ) at ) () ( (
Algoithm of Steepest Descet: o summaize we have obtaied a iteative algoithm fo steepest descet with the followig update step: Algoithm Stat with a abitay iitial guess to the solutio to Fo i b A( ) (a) ( i) i ( i) ( i) ( i) (b) A ( i) ( i) ( i ) ( i) ( i) ( i) (c) A b Eample: Pefom two iteatios of the steepest descet method statig fom the oigi towads fidig a miimize ad also detemie the optimal solutio aalytically he fuctio of iteest is f ( ) Solutio: f ca be epessed as f ) X AX b X c whee ( A I fact we kow that the miimize is X A b ad b / c / / / / 4 We will ow test the steepest descet algoithm at least the fist two steps theeof he mati A is positive defiite ad symmetic Fo steepest descet we set the statig poit: X he gadiet is f ( X ) / AX - b Now fo the fist step of the algoithm we fid f ( X ) / so
he Repeatig the pocess we fid so ad / 5 / 4 / / A 5 5/ 6 X X - 6 / 5/ /6 f ( X) / / 6 / 6 / / / 6 / 6 / / A 5 6 5 9 5/ 6 5 / 6 5/ 7 X X - 5/ 9 / 5/8 Eample: Coside the system 5 A 5 b 5 Fid the appoimate solutios of the liea system usig the Steepest Descet method with Solutio: he mati A is positive defiite ad symmetic Fo the Steepest Descet Method fist set the statig poit ad the compute the steepest descet diectio A b b So the we set 6 A
6 Repeatig the pocess we fid 4 A b ad 5 8 A the we have 7 45 54
his is a collectio of outies compaig diffeet iteative schemes fo appoimatig the solutio of a system of liea equatios Iteative methods Jacobi method he followig M-file shows how to use Jacobi method i MALAB: (Jacobi_with_tolm) fuctio [it]=jacobi_with_tol(abmaitetol) Jacobi_with_tolm solves the liea system A=b usig the Jacobi Method A mati iitial guess vecto b ight had side vecto maite maimum umbe of iteatios tol eo toleace solutio vecto it umbe of iteatios pefomed ele = if; Relative eo it = ; D = diag(diag(a)); N = D - A; = ; while (ele > tol) & (it< maite) it = it+; _ew = D \ (N*+b); ele = om(_ew-if)/om(_ewif); =_ew; ed Ruig M-file i commad widow: he followig MALAB commads compute the fist (it) Jacobi iteatios with iitial guess = : Fist we to defie all iput paamete: >> A=[7 - ; 8-4; - 4 -; -4-6] >> b= [-;;-;]; >> eact=a\b he eact solutio
eact = - - >> =[;;;] >> tol=^-5; >> maite=; >> [it]=jacobi_with_tol(abmaitetol) he output: he appoimate solutio ad the umbe of iteatios pefomed = -9999 9999-9999 it = 7 Iteative methods - Gauss-Seidel method he followig M-file shows how to use Gauss-Seidl method i MALAB: (Gauss_Seidl _tolm) fuctio [it]=gauss_seidel_tol(abmaitetol) Gauss_Seidel_tolm solves the liea system A=b usig the Gauss- Seidel Method A mati iitial guess vecto b ight had side vecto maite maimum umbe of iteatios tol eo toleace solutio vecto it umbe of iteatios pefomed
ele = if; Relative eo M = til(a); M = L + D N = M - A; N = -U = M - A = ; it=; while (ele > tol) & (it< maite) it = it+; _ew = M \ (N*+b); ele = om(_ew-if)/om(_ewif); =_ew; ed he followig MALAB commads compute the fist (it) Gauss-Seidl iteatios with iitial guess = fo the pevious eample: >> [it] =Gauss_Seidel_tol(Abtol) he output: he appoimate solutio ad the umbe of iteatios pefomed = - - it = 4 Iteative methods Steepest descet o gadiet method he followig M-file shows how to use Steepest descet o gadiet method i MALAB: (gadm) fuctio [it]= gad(abmaitetol) Steepest descet o Gadiet method Gauss_Seidel_tolm solves the liea system A=b usig the Gauss- Seidel Method A mati iitial guess vecto b ight had side vecto maite maimum umbe of iteatios tol eo toleace solutio vecto
it umbe of iteatios pefomed = A\b; Eact solutio ele = if; Relative eo = ; it=; while (ele > tol) & (it< maite) = A* - b; = - (om()^ / dot(a*)) * ; ele = om(-if)/om(if); it=it+; ed he output: he appoimate solutio ad the umbe of iteatios pefomed >> [it]= gad(abmaitetol) = - - it = 65 Iteative methods Summay Method Solutio otal Iteatio umbe Jacobi method ( -9999 9999-9999) 7 Gauss Seidel method (- - ) 4 Gadiet method (- - ) 65