Computational Tutorial of Steepest Descent Method and Its Implementation in Digital Image Processing

Transcription

1 Computatioal utorial of Steepest Descet Method ad Its Implemetatio i Digital Image Processig Vorapoj Pataavijit Departmet of Electrical ad Electroic Egieerig, Faculty of Egieerig Assumptio Uiversity, Bagkok, hailad <Pataavijit@yahoo.com> Abstract I the last decade, optimizatio techiques have etesively come up as oe of pricipal sigal processig techiques, which are used for solvig may previous itractable problems i both digital sigal processig (DSP) problems ad digital image processig (DIP) problem. Due to its low computatioal compleity ad ucomplicated implemetatio, the Gradiet Descet (GD) method [] is oe of the most popular optimizatio methods for problems, which ca be formulated as a differetiable multivariable fuctios. he GD method is ubiquitously used from basic to advaced researches. First, this paper presets the cocept of GD method ad its implemetatios for geeral mathematical problems. Net, the computatio of GD processes is show step by step with the aim to uderstad the effect of importat parameters (such as its iitial value ad step size) to the performace of GD. Later, the computatioal cocept of GD method for DIP problems [-5] is formulated ad the computatio of GD is demostrated step by step. he effect of the iitial value ad the step size to the performace of GD method i DIP is also preseted. Keywords: Gradiet Descet (GD) method, Digital Image Processig ad Digital Sigal Processig.. Geeral Itroductio of Gradiet Descet (GD) Method [] he estimatio of the multivariable = [ ],, K, m that miimize the error or the cost fuctio, F( ), is ofte foud i both DSP ad DIP problems. Whe F( ) is Differetiable, ca be estimated simply by λ F( ) (.) + = where is the estimated multivariable at th iteratio. F is the gradiet of fuctio F that ca be determied as F F F F F = = K M m m λ is the step size of GD method at the th iterative. (.) he parameter (so call the iitial value) ad λ i Eq. (.) directly ifluece the performace of GD as show i the followig sectio.

2 . he Eamples of GD o Solvig the Cove Multivariable Problem his sectio presets the GD computatio to determie that miimizes the followig error fuctio. = ( ) + ( ) 4 F (.) he relatioship betwee F() ad = [ ] is show i Figure ad the global miimum of F() is at = [ ], where F ( ) ( ) mi 4 = + =. 4 Figure. he error fuctio F( ) = ( ) + ( ) he search for givig miimum F() i GD ca be formulated as + = λ F F, is as follows.. he gradiet F(), F F( ) = F = (.) F

3 4 (( ) + ( ) ) F( ) = F = 4 ( ( ) + ( ) ) 3 ( 4 + ) F = F = ( )( ) 3 ( 4 + ) F = F = (.3) 4( ) I the simplest GD implemetatio, the step size, λ, is set as a costat. he covergece of GD is guarateed if < λ < /L, where L is the Lipschitz costat. hus, the simple implemetatio comes at a cost of higher computatioal time.. Estimatio of optimal from GD Method at. = his sectio presets the computatioal steps of GD method whe λ =. ad = [ ]. At =, F 3 ( 4 + ) F 4( ) ca be foud as follows. λ = ad [ ] = (.3) 3 ( 4 + ) F = 4( ) 3 F( ) = (.4) Accordig to Eq. (.), at the et iteratio ( = ) ca be estimated as follows. + = λ F( ) (.) ( F) = λ (.5) 3 =..3 = (.6) 4 F( ) at is F ( ) = ( ) + ( ) = 6 ad F( ) at is 4 F ( ) = (.3 ) + (.3 ) = 8.7. he graph of F( ) is show i the Figure ad it is obvious that the miimum poit of F( ) is at = [ ]. From the result i able, we observed that F( ) decreased whe the umber of iteratio icreased.

4 ˆ F( ˆ ) M ˆ F ˆ F d = F ( ˆ ) λ ˆ = ˆ +λd M M M M M M M able. Summary of GD Method for fidig the miimum poit of Eq. (.) at. λ = ad = [ ] here are may termiatio criteria for GD. Eamples of the popular termiatio criteria are as follows. he absolute value of F (), F( ). For eample, the GD method i this sectio will be termiated, if F ( ) <.. hus, the error fuctio calculated at each step ad the optimum is givig F( ) <.. he magitude of F( ), F( ) termiated, if F( ) <.. F optimum is givig F <. F is. For eample, the GD method will be is calculated at each step ad ad the.

5 he umber of iterative,. For eample, the GD method will be termiated, if >. It is the cosidered that is the optimum solutio. he progress of the GD method for estimatig at. is show i Figure. λ = ad = [ ] Figure. he progress of the GD method for estimatig atλ =. ad [ ] =. Estimatio of optimal from GD Method at.5 λ = ad = [ ] he step-size parameter, λ, is oe of the key parameters that directly ifluece the performace of GD. If λ is too small, GD is slow to coverge ad the umber of iteratio,, required to fid the optimum is very high. O the other had, if λ is too large, GD method may be ustable ad diverge. For the high coverget rate, λ should be adaptive at each iteratio; however, the calculatio for λ may be too comple to estimate. Furthermore, λ is based o F(). For eample, λ=. may be too small i some problem; however, it may be too large i other problems. For easy implemetatio, λ, is fied as a costat ( λ = λ ). Cosequetly, i order to uderstad the ifluece of λ to GD, the progress of the GD method at λ =.5 is ivestigated ad show i the able. By comparig ables ad, we ca coclude that F() decreased more rapidly at λ =.5 tha λ =.. For eample, if the GD method will be

6 termiated, whe F <.3. GD method with λ =.required iteratios ( = ), whereas, GD at λ =.5 required oly 6 iteratios ( = 6 ). ˆ F( ˆ ) ˆ F ˆ F d = F ( ˆ ) λ ˆ = ˆ +λd + M M M M M M M M able. Summary of GD Method for fidig the miimum poit of Eq. (.) at λ =.5 ad [ ] = he progress of the GD method for estimatig at.5 is show i Figure 3. λ = ad = [ ]

7 Figure 3. he progress of the GD method for estimatig atλ =.5 ad [ ] =.3 Estimatio of optimal from GD Method at.5 λ = ad = [ ] 3 3 he iitial estimated value,, is aother parameter that directly iflueces the performace of GD. If is set properly, GD method will coverge rapidly ad give the global optimal value; however, if is set improperly, GD method coverges slowly ad the result may be the local optimal poit (o-optimal solutio). I order to uderstad the ifluece of to the performace of GD, the progress of the GD method for.5 λ = ad = [ ] 3 3 is show i the able 3. By comparig ables ad 3, we ca coclude that F() with = [ ] more rapidly tha F() with = [ ] 3 3 decreased. For eample, if the GD method will be termiated, whe F ( ) <.3. GD method with = [ ] ( 6 = ), whereas, GD at = [ ] 3 3 required iteratios ( = ). required 6 iteratios

8 ˆ F( ˆ ) ˆ F ˆ F d = F ( ˆ ) λ ˆ = ˆ +λd + M M M M M M M M able 3. Summary of GD Method for fidig the miimum poit of Eq. (.) at.5 λ = ad = [ ] 3 3 he progress of the GD method for estimatig at.5 = 3 3 is show i Figure 4. From the three eperimets i Sectios.-.3, GD at λ =.5 ad = [ ] λ = ad [ ] has the highest coverget rate (coverge with the least computatioal time). he eperimets i Sectios. ad. demostrated that differet λ provided differet coverget rate. he larger λ led to the higher coverget rate but at the risk of istability. Fially, the eperimets i Sectios. ad.3 demostrated the effect of iitial value to the coverget rate. Differet led to differet progress of the GD method.

9 Figure 4 he progress of the GD method for estimatig ( λ =.5 ad [ ] = 3 3 ) 3. GD Method i Digital Image Processig (DIP) he applicatio of optimizatio i DIP [-5] ca be foud i may applicatios icludig image restoratio [6], image super resolutio recostructio (SRR) [7] ad compressive sesig [8]. he origial image is defied as [ ] =,, K, m, which is a m-dimesio vector, ad the observed image (or y y, y, K, yl, which is a l-dimesio vector (l m ). I geeral, we assume that ad y are related via the followig equatio y= A+ (3.) corrupted image) is defied as = [ ] y a a K a m y a a a m K = +, (3.) M M M O M M M yl al al K alm m l where A is a degradatio process or observed process (blurrig process, wrappig process, geometric trasform process ad/or dow samplig process), which is defied as a matri; is a system oise that cotamiates y. However, the effect of is ot cosidered ad set to the zero vector i this paper due to the page limitatio.

10 Assume that y ad A are kow. he problem is to fid that miimize = [ ] F = y A. I GD, the fidig of the estimated origial image ˆ ˆ ˆ ˆ,, K, m ca be formulated as follows. + = λ F (.) where F ca be determied as follows. F = y A (3.3) ( A )( y A ) F = F = y (3.4) herefore + = λ F( ) (.) + = + λ A y A (3.5) Eq. (3.5) ca be epressed i the vector/matri format as follows., +, a a K a m y a a K a m,,, a a a m y a a a + m, λ K = + K M M M M O M M M M O M M m, + m, a l al K alm yl al al K alm m, (3.6) I DIP, the dimesio of ca be i the scale of several hudred thousads; cosequetly, A may be too large to directly fid the iverse of A. Sice the iverse of A is ot required i Eq. (3.5), the optimal ca be foud eve though the dimesio of is very large. he eamples of applyig GD i DIP is provided i Sectio he Eamples of GD o Solvig the DIP problem his sectio presets the GD computatio to determie the estimated origial image,, from the observed image, y. Assume that ad y are related by the followig equatio.. y= A, (4.) where he degradatio process matri: A = ad he observed image: y =

11 4. Estimatio of from the Iverse of the Degradatio Process Matri I the degradatio process questio, the origial image ca be determied directly by usig the iverse of A as follows. y= A (4.) herefore A y= A A A y = or = A y (4.) = = = (4.3) I practice, the estimatio of form the iverse of A is impossible because the size of A is too large. 4. Estimatio of from the GD Method with Zero Vector as his sectio presets the GD computatio to determie that miimizes the F = y A. he fidig of is formulated as follows. where + = + λ A y A, (3.5) λ is the step size of GD method that is set as a costat λ =.5 ) i this eample; is the iitial estimated image. At =, ca be foud as follows. + = + λ A ( y A ) (3.5) = + λ A ( y A ) = + (.5) = + (.5) = [ ]

12 = + (.5) = + = At =, ca be foud as follows. + = + λ A ( y A ) (3.5) = + λ A ( y A ) (.5) 6 = = + (.5) = + (.5) = F( ) at, ad are ( ) ( y A ) ad F = = 47.8 F = y A = 376., F = y A = 56., respectively. It ca be observed that F decreased whe icreased. he optimum ca be foud by applyig GD for a umber of times. he progress of the GD method for estimatig i this sectio is show i able 4. From the table, it was foud that i GD coverged to = [ ]. However, because is 4-D vector, it is impossible to sketch the relatioship betwee F( ) ad as i Sectio. Istead, the sectio sketches the relatioship betwee F( ), λ ad as show i Figure 5. he figure cofirmed that F( ) decreased whe icreased. F() decreased more rapidly whe λ was higher.

13 y ( y A ) ( ) A y A λ + ( ) ( y A ) F = M M M M M M M M able 4 Summary of GD Method for fidig the optimum solutio i Sectio 4. (.5 λ = ad = [ ] ) 33.6

14 Figure 5 he relatioship betwee F = ( ) y A, λ ad give that [ ] = Estimatio of from the GD Method with Good Iitial Vector I the same maer as Sectio, a iitial estimated value,, is aother importat parameter that directly impacts the performace of GD i both the rate of covergece ad the fial optimal solutio. Geerally, i DIP is set accordig to y by the followig equatio. = A y (4.4) i Eq. (4.4) ofte yields the better performace tha the zero vector. he better performace is measured i term of the coverget rate ad the possibility of fidig the global optimal poit. hus i this eperimetal sectio ca be epressed as follows = = (4.6) he graph showig F( ), λ ad at = [ ] show i Figure 6. From this figure, we ca observe that GD at = ) fidig the solutio faster tha at [ ] is = A y led GD

15 Figure 6 he relatioship betwee F = ( ) y A, λ ad give that ( = [ ] ) 5. Coclusio ad Discussio GD is widely used i both DSP ad DIP. his paper shows how to implemet F. Several computatio GD method i solvig the differetiable cost fuctio, eamples are give i order to make the paper easier to uderstad. he effect of learig step size ad the iitial estimated value o the coverget rate ad the quality of the fial solutio is ivestigated. I geeral, the efficiecy of GD depeds o the accuracy of the mathematical F i the closed model, F( ). If F( ) is complicated, it is difficult to fid form. However, if F( ) is iaccurate, GD method may give the sub-optimal solutio. Cosequetly, F( ) must be the first to be determied ad must be carefully modeled. 6. Referece I would also like to epress my sicere gratitude to Asst. Prof. Dr. Supataa Auethavekiat (Departmet of Electrical Egieerig, Faculty of Egieerig, Chulalogkor Uiversity) for their very helpful commets o various aspects of my work ad comprehesive reviews o my research paper. 7. Referece

16 [] Mokhtar S. Bazaraa ad C. M. Shetty, Noliear Programmig: heory ad Algorithm, Joh Wiley & Sos (SEA) Pte Ltd, 99. [] R. C. Gozalez ad R. E. WoodGDigital Image Processig, Addiso-Wesley Publishig Compay, 99 [3] Vorapoj Pataavijit, Mathematical Aalysis of Stochastic Regularizatio Approach for Super-Resolutio Recostructio, AU Joural of echology (AU J..), Assumptio Uiversity (ABAC), Bagkok, hailad, Vol., No. 4, pp , April 9. ( [4] M. R. Baham ad A. K. KatsaggeloGDigital Image Restoratio, IEEE Sigal Processig Magazie, Vol. 4, Issue, pp. 4 4, March 997. [5] L. P. Kodi, D. A. Scriber, J. M. Schuler, A Compariso of Digital Image Resolutio Ehacemet echiques, i Proceedig o SPIE AeroSese Coferece (Ifrared ad Passive Millimiter-wave Imagig SystemGDesig, Aalysis, Modelig, ad estig), Orlado, FL, pp. -9, April. [6] Korkamol hakulsukaat, Wilaipor Lee ad Vorapoj Pataavijit, A Eperimetal Performace Aalysis of Image Recostructio echiques uder Both Gaussia ad No-Gaussia Noise Models, he 4th Iteratioal Coferece o Kowledge ad Smart echologies (KS-), Faculty of Iformatics, Burapha Uiversity, Choburi, hailad, July,. [7] V. Pataavijit ad S. Jitapukul, A Loretzia Stochastic Estimatio for A Robust Iterative Multiframe Super-Resolutio Recostructio with Loretzia-ikhoov Regularizatio, EURASIP Joural o Applied Sigal Processig (EURASIP JASP) : Special Issue o Super-Resolutio Ehacemet of Digital Video, Hidawi Publishig Corporatio, Article ID 348, May 7, pp. -. [8] Pham Hog Ha, Wilaipor Lee ad Vorapoj Pataavijit, he Novel Frequecy Domai ikhoov Regularizatio for a Image Recostructio Based o Compressive Sesig with SL Algorithm, Proceedig of he Nith Aual Iteratioal Coferece of Electrical Egieerig/Electroics, Computer, elecommuicatios ad Iformatio echology (ECI-CON ), ECI Associatio hailad, Hua Hi, hailad, May. (IEEE Xplore)