An FPGA Implementation of the Two-Dimensional Finite-Difference Time-Domain (FDTD) Algorithm

Transcription

1 An FPGA Implmntation of th Two-Dimnsional Finit-Diffrnc Tim-Domain (FDTD) Algorithm Wang Chn, Panos Kosmas, Miriam Lsr, Cary Rappaport Dpartmnt of Elctrical and Computr Enginring Northastrn Univrsity, Boston, MA 5 {wchn,pkosmas,ml,rappaport}@c.nu.du ABSTRACT Undrstanding and prdicting lctromagntic bhavior is ndd mor and mor in modrn tchnology. Th Finit- Diffrnc Tim-Domain (FDTD) mthod is a powrful computational lctromagntic tchniqu for modlling th lctromagntic spac. Th D FDTD burid objct dtction forward modl is mrging as a usful application in min dtction and othr subsurfac snsing aras. Howvr, th computation of this modl is complx and tim consuming. Implmnting this algorithm in hardwar will gratly incras its computational spd and widn its us in many othr aras. W prsnt an FPGA implmntation to spdup th psudo-d FDTD algorithm which is a simplifid vrsion of th D FDTD modl. Th psudo-d modl can b upgradd to D with limitd modification of structur. W implmnt th psudo-d FDTD modl for layrd mdia and complt boundary conditions on an FPGA. Th computational spd on th rconfigurabl hardwar dsign is about 4 tims fastr than a softwar implmntation on a.ghz PC. Th spdup is du to piplining, paralllism, us of fixd point arithmtic, and carful mmory architctur dsign. Kywords FPGA, FDTD, hardwar implmntation, Finit-Diffrnc Tim-Domain, hardwar acclration.. INTRODUCTION Undrstanding and prdicting lctromagntic bhavior is mor and mor ndd in ky lctrical nginring tchnologis such as cllular phons, mobil computing, lasrs and photonic circuits [8]. Sinc th Finit-Diffrnc Tim- Domain (FDTD) mthod provids a dirct, tim domain solution to Maxwll s Equations in diffrntial form with rlativly good accuracy and flxibility, it has bcom a powrful mthod for solving a wid varity of diffrnt lctromagntics problms. Th FDTD mthod was not widly usd until Prmission to mak digital or hard copis of all or part of this work for prsonal or classroom us is grantd without f providd that copis ar not mad or distributd for profit or commrcial advantag and that copis bar this notic and th full citation on th first pag. To copy othrwis, to rpublish, to post on srvrs or to rdistribut to lists, rquirs prior spcific prmission and/or a f. FPGA 4, Fbruary -4, 4, Montry, California, USA. Copyright 4 ACM /4/...$5.. th past dcad whn computing rsourcs improvd. Evn today, th computational cost is still too high for ral-tim application of th FDTD mthod. Much ffort has bn spnt on softwar acclration rsarch and popl hav usd larg workstations or paralll computr arrays to calculat th FDTD algorithm in softwar. Howvr, th ral-tim application of th FDTD algorithm nds much fastr spd. Although application spcific intgratd circuits (ASICs) provid much fastr spd, popl hsitat to apply th FDTD algorithm to ASICs du to thir high dvlopmnt cost. Rcntly, as high capacity fild-programmabl gat arrays (FPGAs) hav mrgd, popl hav bcom intrstd in rconfigurabl hardwar implmntations of th FDTD algorithm for fastr calculation and ral-tim applications.. Rlatd Work Th first us of FPGA tchnology for th FDTD mthod application is dscribd in [8]. This papr analyzs th bnfits of hardwar implmntation for th FDTD mthod, dscribs a on-dimnsional FDTD cll in hardwar and succssfully simulats th on-dimnsional FDTD. Th algorithm is transfrrd to custom FPGA-basd hardwar using a piplind bit-srial arithmtic architctur. Intgr arithmtic is usd sinc th on-dimnsional opration dos not rquir complx calculation. A clls on dimnsional dsign is implmntd and runs at 7.7Mhz. Th rsults show that hardwar dsign acclrats th on-dimnsional algorithm by svral ordrs of magnitud compard to th softwar dsign. Boundary conditions ar not considrd in this dsign and th changing matrial cofficints ar also not considrd. Thus this rsarch is significantly lss complicatd than our rsarch. Th scond and most rcnt approach is a thr dimnsional dsign using floating-point arithmtic as dscribd in [9]. This papr succssfully prsnts th first thr dimnsional FDTD acclrator in physical hardwar. Boundary conditions and changing matrial cofficints ar considrd. It also proposs a high lvl architctur of th FDTD hardwar systm and dscribs th computational data path and th functionality of ach modul. -bit floating-point arithmtic is usd to provid rlativly high prcision. Howvr, th floating-point arithmtic and high lvl architctur wakn th acclrator. In addition, th input and output mmory intrfac ar not sparatd, and no piplining is considrd in this dsign. Bcaus of th slow mmory intrfac, complx floating-point rprsntation and th lack of piplining, this dsign xhibits poor prformanc. Th hard-

2 war dsign only runs at 4MHz and th rsult is 9 tims slowr than softwar dsign running on a.ghz computr. In contrast, our dsign is significantly fastr. W prsnt a rconfigurabl hardwar implmntation of th psudo-d FDTD burid objct dtction forward modl. This FDTD modl was dvlopd by Panos Kosmas and Dr. Cary Rappaport of Northastrn Univrsity for us in rsarch on subsurfac snsing of land mins via ground pntrating radar [4]. Th rsulting prformanc of th hardwar implmntation is prsntd and compard with th softwar implmntation. Th psudo-d FDTD dsign running at 7MHz shows approximatly 4 tims spd up ovr th fastst softwar implmntation running on a.ghz PC.. BACKGROUND. Th Finit-Diffrnc Tim-Domain (FD- TD) Mthod Aftr K. Y first introducd th FDTD mthod in 966, popl bgan to raliz its accuracy and flxibility for solving lctromagntic problms []. Th FDTD mthod provids a dirct tim-domain solution of Maxwll s Equations in diffrntial form by discrtizing both th physical rgion and tim intrval using a uniform grid. Bcaus this mthod can solv Maxwll s quations on any scal with almost all kinds of nvironmnts, it has bcom a powrful mthod for solving a wid varity of diffrnt lctromagntic problms [4]. Th diffrntial form of Maxwll s quations and constitutiv rlations can b writtn as. E = B t σm H M () H = D t + σ E + J () D = ρ () B = ρ m (4) B = µ H (5) D = ɛe (6) In Equation ()-(6), th following symbols ar dfind: E: lctric fild D: lctric flux dnsity H: magntic fild B: magntic flux dnsity J: lctric currnt dnsity M: quivalnt magntic currnt dnsity σ : lctric conductivity σ m: quivalnt magntic loss ɛ: lctric prmitivity µ: magntic prmability Th basic ida of Y s algorithm is to discrtiz both th physical rgion and th tim intrval of th diffrntial-form thr-dimnsional Maxwll s quations shown abov on uniform grids. Th spac cll siz and th tim intrval ar pr-dfind. Th spac and tim drivativs ar approximatd by th th cntral diffrnc approximation. Thn th spac and tim modls ar stablishd so th algorithm can updat lctromagntic fild valus tim stp by tim stp from two parts: fild valus calculatd in prvious tim stps and fild valus in adjacnt spac clls. This modl maks it possibl for th utilization of modrn computation rsourcs to solv Maxwll s quations, and crats a nw ara of lctromagntic scintific rsarch. Figur : Th gomtrical rprsntation of th D Y cll Th D grid shown in Figur, namd th Y-Cll [], is vry hlpful for undrstanding th discrtizd lctromagntic spac. Th Y-Cll is a small cub, it can b tratd as on singl cll pickd from discrtizd modl spac. x, y, z ar th thr dimnsions of this cub. W us (i, j, k) to dnot th point whos ral coordinat is (i x, j y,k z) in th modl spac. Instad of placing thr lctric fild componnts E and thr magntic fild componnts H in th cntr of ach cll, th E and H componnts ar intrlacd so that vry E componnt is surroundd by four circulating H componnts, and vis vrsa. Th E and H componnts ar intrlacd not only in spac but also in tim. As illustratd in Figur [], all th E componnts ar updatd at N t and all th H componnts ar updatd at (N + ) t. In mor dtail, th E componnts in th N-th tim stp ar updatd and stord in mmory using prviously stord H data, and thn th H componnts in th (N + )-th tim stp ar updatd and stord in mmory also using th E data which wr just calculatd. This calculation can b itratd through tim stps so th lctromagntic fild valus can b updatd at any tim intrval. Figur : Diagram shows how th Y algorithm intrlacs th E and H componnts in spac and tim 4

3 This algorithm is not only accurat and flxibl, its most appaling charactristic is its simplicity of computation and structur. All th oprations in th algorithm ar additions, subtractions and multiplications, which ar asy to comput and cost rlativly fw rsourcs in a hardwar implmntation. Th structurs of th six quations in th algorithm ar similar, making it possibl to apply various mthods to spdup th algorithm, such as hardwar implmntation and paralll computing. What s mor, th algorithm simply itrats th sam procdurs for ach tim stp in th tim domain, making th computation rgular. A complt FDTD modl is shown in Figur 4 in th nxt sction.. Burid Objct Dtction Forward Modl Th FDTD mthod was not widly usd until th past dcad whn computing rsourcs improvd. Evn today, th computational cost is still too high for ral-tim application of th FDTD mthod. To solv this problm, w targt a rconfigurabl hardwar implmntation of th D FDTD burid objct dtction forward modl. Y Z Transmitting Antnna Objct Min Rciving Antnna Figur : D FDTD burid objct dtction forward modl spac As shown in Figur, this modl approximats a plan wav snt from ground pntrating radar with a 45 incidnc angl, which is thn fd into a thr-dimnsional spac grid and propagatd through an air-soil intrfac. Th wav is thn rflctd and scattrd from th boundary of th burid objcts and capturd by rciving antnnas. Th ultimat goal of this rsarch is to dvlop a land min dtctor whos dtction is basd on th scattrd signals rcivd at th rciving antnnas [6] [7]. This forward modl has th sam structur as Y s FDTD algorithm shown in Figur 4. Th xcitation sourc in this modl ar th signals snt from th ground pntrating radar. Th boundary condition algorithm usd in this modl is th nd ordr Mur-typ Absorbing Boundary Condition [4] sinc it has bn shown to b mor ffctiv in th disprsiv soil mdia for small angls of incidnc. This modl is computationally intnsiv. Th modl spac is discrtizd to up to millions of computational clls. For ach of th clls, th FDTD algorithm updats all its paramtrs at vry tim stp. Svral hours may b ndd to simulat hundrds tim stps to achiv usful information. What s mor, th backward modl, whos task is using th forward modl s output data to dtct th burid mins, nds to run th forward modl itrativly to gt th final rsult. Thus th spd of th forward modl is critical to th ral-tim application of th backward dtcting dvic. X. ALGORITHM ANALYSIS AND D SIM- PLIFICATION Bfor hardwar implmntation, th targt algorithm is carfully studid in ordr to undrstand its logical structur, data rang, input/output and mmory hirarchy. This analysis is tightly connctd to th hardwar dsign. Most of th hardwar architctur w dscrib in Sctions 5 and 6 is basd on th analysis in this sction.. Algorithm Structural Analysis As shown in Figur 4, th FDTD algorithm starts with initializing all its paramtrs and building th modl spac. Evry cll has associatd it with thr lctric filds, thr magntic filds and svral matrial proprty paramtrs. Each cll will b spcifid as soil, air, or othr objcts, including burid objcts w want to dtct. Th modl spac is built and stord in mmory aftr initialization. Th xcitation sourcs ar lctromagntic wavs snt from ground pntrating radar. Th sourc data ar inputs from outsid of th algorithm and ar loadd to th sourc clls in th modl spac at vry tim stp. All th lctric and magntic filds data will b rad from mmory, calculatd and stord back to th sam mmory location in vry tim stp. Th calculation includs updating lctric fild data, updating boundary conditions and updating magntic fild data. Th boundary condition part consists of spcial algorithms daling with th unit clls locatd on th boundary of th modl spac. For ths clls, som of th adjacnt clls may not xist, so spcial mthods ar ndd to updat thm appropriatly. Ths thr algorithms ar not indpndnt. Both th lctric and magntic updating algorithms nd ach othr s formr rsults as input, whil th boundary condition updating algorithm nds both formr and currnt lctric and magntic fild data. Diffrnt subalgorithms will b usd for diffrnt data, dimnsions and matrials. Ths calculations ar computationally intnsiv and tak th most tim in th FDTD algorithm. Th updating procdur is rpatd until th lapsd tim incrass to th rquird valu. Elctromagntic data on th rcivr clls will b stord as output. In addition, w can output th whol modl spac at any tim stp.. Simplification and D Structur Th D FDTD algorithm consums so many hardwar rsourcs that it cannot fit into our currnt FPGA chip. As th first stp of th dsign, w simplifid th D algorithm to a D modl. Th D simplifid modl spac is not a ral D spac. It kps most of th cross sctional structurs of th D modl. Th Y dimnsion of th D modl spac is flattnd so th modl spac in th Y dimnsion is rducd toclls,j =andj =, as shown in Figur 4. All th clls in j = ar th D modl spac w want, whil all th clls in j = ar kpt for algorithmic structural purposs. Ex, Ez and Hy no long xist in this D modl sinc w chos th D TM wav modl [5]. So only on lctrical fild updating algorithm and two magntic fild updating algorithms ar lft aftr th simplification. Th boundary condition updating algorithm is gratly simplifid sinc th boundary of th D modl only has 4 dgs compard to th dgs and 6 facs in th D modl. But sinc most of structurs ar kpt without modification, this D modl can b upgradd to D latr with limitd ffort. W call this th psudo-d FDTD modl spac. 5

4 Simplify Y Y Antnna Z Z Antnna Min Min Rcivr Rcivr Initialization Initializ paramtrs of modl spac and tim stp Build paramtrs of soil and burid objct Load all th EM spac data into mmory X X t = n Excitation Calculat E Fild Updat Exs fild Updat Eys fild Updat Ezs fild Extrior Boundary Conditions Boundary of EYX Boundary of EZX Boundary of EZY t = n +.5 Calculat H Fild Updat Hxs fild Updat Hys fild Updat Hzs fild Tim ovr? Boundary of EXY Boundary of EXZ Boundary of EYZ Ys End n = n + No, Go to Nxt Tim Stp Figur 4: Flow diagram of th D FDTD algorithm and th D simplification Th upgrad from D to D modl can b don in thr stps: Add thr mor fild updating algorithms which hav th sam structur as th original thr algorithms. Upgrad th boundary condition updating algorithm. Dsign a mor advancd mmory intrfac to fit th nw cor procssing moduls. Sinc this modl can b xtndd to D latr, this is a good first stp for hardwar implmntation.. Data Dpndncy Analysis Th lctric fild, boundary conditions and magntic fild updating algorithms ar procssd in sris in th FDTD modl structur shown in Figur 4. Sinc paralllism is vry important for spdup, th data dpndncis of ach updating algorithm ar analyzd for possibl paralllism. Th lctric updating algorithm nds th valus from prvious tim stp s lctric fild and magntic fild calculations as input. In contrast, th magntic fild updating algorithms nd th currnt tim stp s lctric fild data as input, which ar not availabl until th lctric fild updating algorithm is xcutd. If w want to procss th lctric and magntic fild updating algorithms in paralll, data dpndncis hav to b considrd and solvd by starting th magntic fild updating algorithm two rows aftr th lctric fild updating algorithm. Thus, although th updating algorithms ar not symmtrically paralll, th dsign spd is still almost doubld. This ralizabl paralll structur is shown in Figur 5. Bcaus th boundary condition updating algorithm dos not procss all th clls in th modl spac, it is not ncssary to parallliz th boundary condition updating algorithm with th othr two. Mor paralllism can b xtractd sinc thr is no dpndncy among th fild updating algorithms for all clls in on row. If hardwar rsourcs ar availabl, two or mor clls Initialization Initializ paramtrs of modl spac and tim stp Build paramtrs of soil and burid objct Load all th EM spac data into mmory Rows Calculat Eys Fild Excitation Extrior Boundary Conditions Boundary of EYX Calculat Hxs Fild Tim ovr? Boundary of EYZ Ys End Calculat Hzs Fild n = n + No, Go to Nxt Tim Stp Figur 5: Structural diagram of th paralllld psudo-d FDTD algorithm can b procssd at th sam tim. If th dsign can procss M clls at th sam tim, dsign spd can b M tims fastr. For ach additional cll of paralllism, thr mor instanc of th updating implmntation ar ndd, so th incras in th hardwar cost is proportional to th incras in th dsign spd. W call this paralllism paralllism by cll rplication as distinguishd from paralllism by piplining which was dscribd in th prvious paragraph. Also bcaus of th data dpndncis, paralllism will mt its limit. Th optimal dsign procsss M clls at th sam tim whr M is th numbr of clls in on row. Mor than M-way paralllism will introduc difficulty for paralllism by piplining. 4. DATA QUANTIZATION Th data in this rsarch ar valus of normalizd lctromagntic fild valus that rang btwn - and and tnd to b accurat to at most on part in,. Th original Fortran program uss 64-bit doubl floating point rprsntation which provids high rsolution and dynamic rang. But floating-point rprsntation is significantly mor xpnsiv than fixd-point rprsntation in spd and cost whn implmntd in hardwar. W choos a fixd-point rprsntation hr sinc it can sav spac and incras th spd of our dsign without sacrificing accuracy. All th input data nds to b quantizd to th slctd fixd-point rprsntation. Th hardwar dsign also nds to b spcifid to th slctd bit-width. Suitabl data bit-width is chosn according to rlativ rror analysis. 4. Rlativ Error Analysis Sinc all th lctromagntic fild data ar btwn - and, th rsolution of this fixd-point rprsntation is dcidd by th bit-width w us aftr th binary point. Th longr th bit-width aftr th binary point, th highr th 6

5 rsolution so th smallr th rror. Howvr, longr bitwidth data costs mor hardwar rsourcs. So w nd to study th algorithm carfully to pick a suitabl bit-width with rlativly small rror. Floating-point to fixd-point quantization crats rrors. Th rrors com from two major factors: arithmtic rror from floating to fixd-point transformation and additional rror cratd by th calculation from pics of data which originally hav rror. Sinc w cannot avoid th scond typ of rror xcpt by changing th algorithm structur, w nd to considr th two typs of rror as a whol. Th rlativ rror mthod compars all th floating-point data rsults and th corrsponding fixd-point data rsults for th sam modl spac. Th rlativ rrors ar calculatd according to th following quations. Th avrag rlativ rror ovr th tim priod rflcts th diffrnc of th fixdpoint dsign from th original floating-point program. W will choos a suitabl bit-width with th bst rlativ rror/cost ratio. Absolut rror = floating point data fixd point data (7) floating point data fixd point data Rlativ rror = floating point data (8) Th floating-point data is producd by th original Fortran program which runs in 64-bit floating-point rprsntation. Th fixd-point dsign is simulatd with C cod. Th C cod uss long intgrs to simulat th fixd-point rprsntation and uss a spcial program to calculat long intgr multiplication. Th multiplir has diffrnt structurs for diffrnt data bit-widths, and th fixd-point C cod changs accordingly. Th C cod dsign mulats th dsign w implmnt in hardwar. So th rsult of this C cod is xactly th sam as th hardwar output. Thus, this C cod is also usd for hardwar rsults vrification. Magnitud Magnitud Floating-point lctromagntic data on rcivr (Hxs) Tim Stp Floating-point lctromagntic data on rcivr (Hxs) Tim Stp Figur 6: An xampl of th xprimntal data sts - lctromagntic data rcivd on two rciving antnnas Th lctromagntic fild data rcivd by th ground pntrating radar is usd as our xprimntal data st. A sampl of th xprimntal data sts ar shown in Figur 6. Thr ar two sts of rcivrs in th currnt psudo-d FDTD modl, so thr ar two sts of data for comparison. W calculat th avrag rlativ rror btwn floatingpoint and fixd-point data from ach rcivr for diffrnt bit-widths. Whn w slct N bits aftr th binary point, th fixd-point data is prcis to N. All signals whos magnitud is blow N will b tratd as in th fixdpoint dsign. This diffrnc can b indistinguishabl to human ys if N is big nough. But if w calculat rlativ rror, it can b as big as. So whn w calculat rlativ rror, w avoid th starting tim stps which hav xtrmly small data by slcting a spcific tim fram, normally starting from tim stp 7. For th sam sourc data, th Fortran cod and C cod ar xcutd to produc sts of rcivr rsults for diffrnt bit-widths. Thn th avrag rlativ rrors ovr th tim priod ar calculatd according to Equations 7 and 8. Tabls and show th avrag abso- Bits aftr Avrag absolut rror Avrag absolut rror binary point of E fild of M fild bits bits bits bits bits bits Tabl : Avrag absolut rror btwn fixd-point arithmtic and floating-point arithmtic on diffrnt bit-width Bits aftr EY(R) EY(R) HX(R) HX(R) Sourc binary point Tim Stp (7-8) (7-8) (7-8) (7-8) (7-8) bits bits bits bits bits bits Tabl : Avrag rlativ rror (%, inprcntag) btwn fixd-point arithmtic and floating-point arithmtic on diffrnt bit-width lut and avrag rlativ rrors of th D fr spac modl. W tstd th bit-width rang from bits to 8 bits. Th first four columns of Tabl corrspond to th lctromagntic fild data rcivd from th two rcivrs, whil th last column is th rlativ rror of th sourc data. Only th floating to fixd-point transformation rror occurs for th sourc data bcaus th sourc data ar not usd for calculations. Thus, th rlativ rror in th sourc data column is th smallst. It rflcts th pur rror btwn th floating-point and fixd-point rprsntations. Th rsults data hav largr rlativ rror sinc many mor rrors ar cratd in th calculations. Although th absolut rror is vry diffrnt btwn lctric fild data and magntic fild data, th rlativ rrors ar rlativly consistnt for both typs of fild data. Thus, th rlativ rror is a good mtric for choosing suitabl bit-widths. Th rlativ rrors ar plottd in Figur 7. Th rlativ rrors of both lctric fild data and magntic fild data dcras as bit-widths incras. Howvr, th rat of dcras slows as th bit-widths incras. Considring both th rlativ rror and th bit-width cost, 6-bits aftr th binary 7

6 Avrag rlativ rror (%) Elctric Fild Valu at R Elctric Fild Valu at R Magntic Fild Valu at R Magntic Fild Valu at R Sourc Data 4bits 5bits 6bits 7bits 8bits Bit-width Figur 7: Rlativ rror btwn fixd-point arithmtic and floating-point arithmtic for diffrnt bitwidths point is a good choic. Th avrag absolut rror for this rprsntation is on th ordr of 7 for magntic fild data and on th ordr of 4 for lctric fild data; th avrag rlativ rror is around.%. Thus this rprsntation satisfis th accurat rquirmnt which is on part in,. 4. Fixd-point Quantization According to th figurs and analysis in this sction, w chos a fixd-point rprsntation with 6 bits aftr th binary point for all variabls. Th fix-point data structur is shown in Figur 8. Th lftmost 4 bits ar sign bit and data bits rprsnting th intgr bfor th binary point in sign-magnitud rprsntation. Th rmaining bits starting from th 5th bit rprsnt th valus aftr th binary point. Th intgr bits ar usd bcaus thr ar som paramtrs with valu biggr than. Sinc almost all th data and paramtrs ar lss than 7, bits bfor th binary point is sufficint. Th fw pics of data with valu largr than 7 ar tratd as spcial cass in th dsign. All th input data will b quantizd to th slctd fixd-point rprsntation, and th hardwar dsign is spcifid using th slctd bit-width. S A AA. B BBBBBBBBBBBBBBBBBBBBBBBBB - -6 bits 6 bits Figur 8: Signd fixd-point rprsntation 5. MEMORY INTERFACE W analyzd th FDTD algorithm s structur, data dpndncis and I/O in Sction. Also w analyzd th algorithm s data rang and quantizd th data into a fixd-point rprsntation in Sction 4. Basd on ths analysis, w dsignd th hardwar architctur and implmntd th customhardwardsignonanfpga.thcorofthfdtd dsign is th lctromagntic fild updating implmntations which hav dp connctions with th mmoris. Ths cor updating implmntations nd to rad from and writ back to mmoris continuously. Th data xchang volum btwn cor implmntations and mmoris is larg, so a mmory intrfac modul nds to b dsignd to mak th cor implmntations work wll. Th cor implmntations ar dsignd as piplins (s Sction 6). To mak th piplins run smoothly, th mmory intrfac should fd spcific data into th piplins vry clock cycl. Th shortr th clock cycl, th lss tim th piplins will tak to complt, so th fastr th whol dsign. Whil rducing th lngth of th clock cycl is on of th ky ways to spd up th dsign, a task which is slowr than th othrs will crat a bottlnck for th piplin, thus rducing th piplin spd. Th rad/writ procsss of mmory tak mor tim than th arithmtic componnts in many cass, spcially for slow mmoris. Th fastst mmory should b usd in th piplins to liminat th bottlnck. Howvr, fast mmory is limitd in siz so that only a small amount of lctromagntic fild data can b stord thr, whil othr data nds to b stord in slowr mmoris. So a mmory intrfac which xploits th distribution of fast and slow mmoris is dsignd. 5. Mmory Hirarchy Th mmory hirarchy in this dsign has thr lvls: Th slowst mmory locatd in th host PC Th fast on-board mmory locatd on th Firbird TM FPGA board Th fastst on-chip mmory (BlockRAM) intgratd in th FPGA chip Mmory in PC Mmory in PC PC HOST PCI BUS Simulatd Elctromagntic Spac, On-Board MEMORY Sourc 5 Data On-Board MEMORY,4 On-Board MEMORY Mmory Intrfac BlockRAM Mmory Intrfac BlockRAM DESIGN Elctric Fild Piplin Modul Magntic Fild Piplin Modul Boundary Conditions Modul FPGA FIREBIRD BOARD Figur 9: Structural diagram of th hardwar organization and mmory hirarchy Figur 9 shows this structur clarly. Th mmory accss spd btwn host PC and FPGA board via th PCI intrfac is quit slow. It cannot provid data as fast as ndd by th cor updating piplins insid th FPGA chip. So th mmory on th host PC is not frquntly usd xcpt at th bginning and nd of th dsign. Th dsign loads th modl spac data from th host PC to th on-board mmory and data will b loadd back to mmory on th PC at th nd of th dsign. On-board mmory is rlativly fastr sinc it communicats dirctly with th FPGA chip. In ordr to achiv optimal spd, as long as thr is adquat on-board mmory spac, th lctromagntic fild and modl spac data ar all stord thr. W hav 5 on-board mmoris (4 x 64Mbyts, x Mbyts SDRAM) on th Firbird TM FPGA board and w us all of thm. Four mmoris ar usd for mmory updating and th last on is usd to stor th modl spac matrial paramtrs and sourc fild data. BlockRAMs ar fast mmory units intgratd insid FPGA chips. Although th on-chip BlockRAMs ar th fastst mmory in our dsign, thr is much lss BlockRAM availabl than on-board mmory. Th 6555 bits of BlockRAM 8

7 on our chip can only fit a modl spac of at most 78 clls, sinc ach cll has pics of fild data associatd with it and ach fild data taks bits of BlockRAM. Th modl spac with only 78 clls is too small to b practical, spcially for thr dimnsional modl spac. So w cannot load all th modl spac data into BlockRAMs. Instad, Block- RAMs ar usd to build a mmory intrfac modul which rads from and writs to on-board mmory continuously and fds data to th cor implmntations. 5. Mmory Intrfac Modul Rading and writing to on-board mmory incurs svral clock cycls dlay. Th situation is mor critical whn svral rad and writ instructions hav to b xcutd at th sam tim. If w intgrat a on-board mmory accss into our cor piplins, th unprdictability and slow rspons will mak th whol dsign unstabl. Also, all th cor updating piplins nd to rad svral data for ach updating procss continuously. If all th rquird data cannot b providd in on clock cycl, piplin throughput will b gratly rducd and dsign will b much slowr. For xampl, lctric fild updating piplin nd to rad 6 fild data for ach procss, among which 4 fild ar from sam mmory block. Whil normal dual-portd mmory block can only provid data ach clock cycl, which mans th dsign can only run in half spd, w can build a mmory intrfac with duplicat BlockRAM mmory blocks to achiv th full spd of 4dataachclockcycl. ON-BOARD MEMORIES EM FIELD DATA, ON-BOARD MEMORIES MATERIALS PARAMETER 5 Input BlockRAMs Mmory Intrfac A C B D CORE PROCESSING ic t n g a M l ld u d i o F M l a ic l r u t ld c d i o l F E M d lin ip P d n li i p P Boundary Rsult Rsult A B C D Ouput BlockRAMs Mmory Intrfac FPGA CHIP Figur : Structural diagram of th mmory intrfac modul Figur shows th structur of th mmory intrfac modul, which is composd of BlockRAMs and control logic. Th mmory intrfac modul is split into two parts: input and output. Th four on-board mmoris which ar usd to stor lctromagntic fild data ar split into two parts rspctivly. Th rason to split th input and output mmory intrfac is that w can rad from on-board mmoris, updating fild data and writ to on-board mmoris at th sam clock cycl to maximum th piplin throughput. For xampl, whil w rad M clls fild data from on-board mmoris to input BlockRAM intrfac, th cor updating piplins should rad M clls fild data from input Block- RAM intrfac and writ th rsult to output BlockRAM intrfac, and at th sam tim, w nd writ M clls fild data from output BlockRAM intrfac to on-board mmoris. If w don t sparat th input and output intrfac, th Rsult ON-BOARD MEMORIES EM FIELD DATA,4 BlockRAM block intrfac nd to b accssd four tims pr clock cycl, which is not supportd by our dual-portd BlockRAM block. Th sam rason, th on-board mmoris nd to b sparatd sinc thy ar singl-portd. Thus th sparation of th input and output mmory intrfac maks th maximum piplin throughput ralizabl. Also, th on-board mmoris ar organizd in a swapping mchanism to spd up th dsign. Th on-board mmoris and which stor th prvious tim stp s data ar connctd to th input part only, whil th othr two on-board mmoris and 4 ar connctd to th output part. Th input part of th mmory intrfac modul rads th prvious tim stp s data from th on-board mmoris and. Aftr th cor updating piplins, th output part writs th currnt tim stp s rsult back to th on-board mmoris and 4. In th nxt tim stp, th connction of th mmory intrfac modul and on-board mmoris will b swappd. Th on-board mmoris and 4 which stor th currnt tim stp s data will b connctd to th input mmory intrfac modul and th on-board mmoris and will b rst and connctd to th output modul to stor th nxt tim stp s rsult. Sinc prvious tim stps s rsult data ar currnt tim stps input data, th advantag of this swapping mchanism is that w no longr nd to copy whol modl spac data from output mmory to input mmory in th nd of vry tim stp. Four duplicatd mmory groups A, B, C, D ar assignd on ach sid of th modul. Ths duplicatd mmory groups stor idntical data and updat data simultanously. Th rason w nd four duplicatd mmory groups is that vry dual-portd BlockRAM can only b rad or writtn onc by cor updating piplins in ach clock cycl sinc th onboard mmory accss alrady consum on port of ths BlockRAMS. Whil th cor updating piplins nd svral data as input at ach clock cycl, four duplicatd groups of BlockRAM ar usd to minimiz th mmory accss tim and incras th piplin throughput. Each mmory group has four idntical mmory slots. Th mmory intrfac modul loads on row of lctromagntic fild data from on-board mmory to on BlockRAM slot at a tim. On row mans M clls in an M N modl spac. Th cor dsign rads on row of filds data from th input BlockRAM slots, calculats and writs updatd dat to th output BlockRAM slots at th sam tim. Th input data ar loadd to th scond BlockRAM slot thn and th procsss continus. Th first BlockRAM slot will not b ovrwrittn until th fifth row of data has to b loadd. Th input procsss itrat using four BlockRAM slots. Sinc th cor updating piplins only nd th adjacnt cll s fild data as input, th four BlockRAM slots which stor thr rows of fild data ar adquat. Bcaus thr is no limit on th column lngth on th modl spac, w can simulat a much biggr modl spac by introducing BlockRAM mmory intrfac. With 78 clls rprsntabl, th maximum row width is approximatly 7 clls sinc 4 mmory groups ar ndd for both input and output intrfacs and 4 rows of fild data ar stord in ach mmory group. A 7 N modl spac can fit in this dsign, with N any positiv intgr. Th mmory intrfac modul rads and outputs data from th on-board mmory, so th cor updating piplins communicat with th BlockRAMs only. Th intrfac modul sacrifics som spd to achiv a mor stabl structur. 9

8 Thr procsss Mmory to BlockRAM, Data updating and BlockRAM to Mmory ar synchronizd in th modul. It wasts tim whn ths procsss wait for on anothr. Also, th mmory architctur placd an uppr limit on th dsign spd that th whol dsign will not run fastr thn th on-board mmory accss spd. Of cours, a mor stabl dsign is mor important. And sinc th cor piplins ar sparatd from on-board mmory by this modul, w do not nd to considr th mmory intrfacs in latr minor modifications. This mmory intrfac modul is vry xpandabl to adapt largr data xchang volum. For xampl, Sinc th Data updating procss is th critical path in th currnt dsign, w can doubl th cor updating piplins to spdup th dsign. Mmory intrfac modul can b simply doubld and thr mor cor updating piplins nd to b duplicatd. As long as th FPGA spac and BlockRAM is sufficint, Data updating procss can b parallld mor until currnt architctur rach its spd limit whn Mmory to BlockRAM and BlockRAM to Mmory procsss bcam th critical path of th dsign. Th mmory intrfac modul has to b changd accord to th incrasd input rquirmnts in th thr dimnsional dsign. But th basic ida of a sparat mmory intrfac modul rmains th sam. 6. HARDWARE ORGANIZATION Our hardwar dsign is basd on th structural diagram of th paralllld psudo-d FDTD algorithm prsntd in Figur 5. Th targt FPGA hardwar dvic w usd is a Firbird Board from Annapolis Micro Systms with 88Mbyts on-board mmory. It uss a Xilinx VIRTEX-E XCVE FPGA with ovr.5 million systm gats, 9 slics and 6555 bits of on-chip BlockRAM. 6. Data Flow Diagram Th data flow diagram of th hardwar dsign is shown in Figur. Most componnts hav bn introducd in th prvious sctions. For ach tim stp, th BlockRAM mmory intrfac modul accsss th on-board mmoris and fds th ncssary data to th cor updating piplins. Th sourc data will b rad from th fifth on-board mmory and addd to th spcific clls. Th connction of th Block- RAM mmory intrfac modul and th on-board mmoris will b swappd in th nxt tim stp whn th currnt tim stp s output data bcom th nxt tim stp s input data. 6. Piplining and Paralllism Th cor updating implmntations ar all piplind to spd up th dsign. Thy hav similar structurs but th input and output data ar diffrnt. Each of th piplins consists of two -bit multiplirs and svral addrs and subtractors. Sinc th multiplirs ar much biggr and mor tim consuming than th addrs and subtractors, th bottlnck in ths piplins is th multiplirs and th rad/writ procsss. W us piplind-multiplirs which tak 6 stps to complt. Thus, th dlay of on stag in th piplin can b rducd by a factor of 6, substantially spding up th piplin. Possibl paralllism in our dsign is introducd in Sction. As shown in Figur, w procss thr cor piplins in paralll to spd up th dsign. As pointd out in Sction, th magntic fild updating piplins nd to b pro- Elctric Fild Mmory Piplin Eys Piplin Hxs Elctric Fild Mmory Magntic Fild Mmory BlockRam Piplin Hzs BlockRam Magntic Fild Mmory Matrials and Sourc Data Mmory 4 Piplin Boundary BlockRam Sourc Addr Figur : Data flow diagram of th hardwar dsign cssd two rows aftr th lctric fild updating piplins. So th Hxs and Hzs fild updating piplins will b dlayd for two rows. Idally, th mor paralllism, th gratr th spd. As long as thr is nough rsourcs on th FPGA, w can procss mor piplins in paralll to spd up th dsign. In th currnt FPGA chip, it is possibl to us 6 piplins instad of piplins to doubl th procssing spd. 7. RESULTS COMPARISON Th rsults of th psudo-d fixd-point C cod ar usd for vrification of both th simulation rsults and th hardwar implmntation rsults. This sction dscribs th comparison of th implmntation of th FDTD algorithm with both th fixd-point C cod and th floating-point Fortran cod. Th prformanc of th dsign is also analyzd. Figur compars on tim stp s magntic fild modl spac rsults from both th fixd-point C cod and th FPGA dsign. Ths two sts of data ar xactly th sam. This rsult shows th hardwar dsign corrctly implmnts th targt fixd-point algorithm. Figur compars on tim stp s magntic fild modl spac rsults from both th FPGA dsign and th floating-point Fortran cod. Th avrag absolut rror btwn ths two rsults is.8 7 with th avrag rlativ rror around.5%. This imag comparison shows that th hardwar implmntation prsntd in this thsis givs th lctromagntic rsults satisfying th accuracy rquirmnts of 4. Th modl spac imags hr ar psudo-colord, maning that ach valu in ach imag is assignd a color from a color tabl. Th actual color usd to rprsnt th modl spac fild data carris no information. 7. Prformanc Th prformanc rsults of th softwar and hardwar implmntations ar shown in Figur 4. For a D modl spac, th hardwar dsign running at 7MHz and taks.45 sconds to procss tim stps. Th fixdpoint C cod running on a Pntium4.GHz CPU taks.75 sconds and th floating-point Fortran cod running on sam PC taks 4. sconds. Th hardwar implmntation has 4 tims spd up compard to fixd-point softwar

9 Softwar Fixd-point Hardwar Magntic fild rsult Magntic fild rsult Tim stp Tim Tim stp Figur : Magntic fild modl spac rsults: th fixd-point C cod and th hardwar dsign on chip Softwar Floating-point Magntic fild rsult Tim stp Hardwar Magntic fild rsult Tim stp Figur : Magntic fild modl spac rsults: th floating-point Fortran cod and th hardwar dsign on chip Excuting Tim (Scond) Prformanc Rsult A B C A Softwar Floating-point ~~ 4.s Fortran cod at. GHz PC B Softwar Fixd-point ~~.75s C cod at. GHz PC C Hardwar ~~.45s Dsign working at 7MHz Modl spac * clls Itrat tim stps Figur 4: Prformanc rsults - Softwars vs. FPGA Hardwar running on a.ghz PC. Th psudo-d FDTD hardwar implmntation runs at a clock spd of at most 7 MHz on th Firbird TM board. It taks 46% of th slics and 54% of th BlockRAMs in th currnt FPGA chip. 8. CONCLUSION Th goal of this papr is to invstigat th possibility of hardwar implmntation of th FDTD algorithm. As a rsult, a psudo-d FDTD hardwar architctur has bn proposd and implmntd. Th FDTD algorithm has bn carfully studid to find an fficint hardwar architctur. Th data rang of th FDTD algorithm has bn analyzd and fixd-point quantization has bn applid to th hardwar implmntation. This rsarch shows th proposd hardwar implmntation xploits th algorithm proprtis and th capabilitis of hardwar rsourcs in ordr to achiv high computational spd. Th computational spd of th custom hardwar implmntation rprsnts a spdup of 4 tims compard to th fastst softwar implmntation running on.ghz PC. This rsult indicats that th FPGA hardwar implmntation can achiv significant spd up for th FDTD algorithm. Th structur and data dpndncy of th FDTD algorithm has bn analyzd to xploit piplining and paralllism. Th piplind and paralll architctur has bn introducd and a mmory intrfac modul has bn dsignd to facilitat high lvl piplining and paralllism. It is shown that high gat-capacity FPGAs ar amnabl for xploiting mor paralllism whil th limits of paralllism is also introducd. A psudo-d FDTD modl has bn cratd by simplifying th D FDTD algorithm to D. Th structur and data dpndncis of both th D and D modls ar sufficintly similar that a latr D FDTD modl can b dsignd basd on th currnt D modl structur. Th data rang of th FDTD algorithm has bn studid for fixd-point quantization. As long as th data rrors btwn th floating and fixd-point implmntations fall insid a satisfactory rang, th fixd-point quantization rducs th computational cost and acclrats th dsign by spding up vry arithmtic componnt in th piplin architctur. Th tchniqus usd in this rsarch, including analysis of piplining and paralllism, carful mmory hirarchy dsign and fixd-point arithmtic analysis and dsign, can asily b applid to similar, high prformanc FPGA dsigns. Synthsis rsults shows that a ralistic psudo-d FDTD dsign on a Xilinx VIRTEX-E XCVE FPGA taks 46% of th slics and 54% of th BlockRAMs. This rsult indicats that highr lvls of paralllism may nd a mor advancd FPGA chip. 9. FUTURE WORK Focusing on th burid objct dtction forward modl, w nd to add a disprsiv soil mdia option to our D FDTD modl. A simpl pol conductivity modl [7] is usd to updat th lossy disprsiv soil mdia. This algorithm nds th prvious thr tim stps fild data as input, so it is mor complicatd than th updating algorithms usd in th currnt standard FDTD modl. Also, w will implmnt PML Absorbing Boundary Conditions (ABC) as an altrnativ option to th currnt Murtyp ABC. Th currnt Mur-typ ABC is suitabl for our burid objct dtction modl whil PML ABC is mor popular and asir to implmnt. Th most intrsting futur rsarch lis in upgrading th psudo-d FDTD hardwar dsign to thr dimnsional spac. On th basis of currnt D prliminary work, th upgrad of th FDTD dsign to D can b don in thr stps: Add thr mor fild updating algorithms which hav th sam structur as th original thr algorithms. Upgrad th boundary condition updating algorithm. Rdsign a mor advancd mmory intrfac to fit th nw cor procssing moduls. From a brif stimat, th D dsign will b 4-5 tims biggr than currnt dsign in FPGA slic usag. First, two mor lctric fild updating algorithms, which consum mor slics than magntic updating algorithms, will tripl th dsign siz. Scond, th nd ordr boundary conditions will

10 b mor complicatd and may doubl th dsign siz. Th BlockRAM mmory block consumption is proportional to th siz of ach layr. Whil BlockRAM usag is K M in a M N D modl, BlockRAM siz is K M M in a M M N D modl spac, which mans th siz of Block- RAM is rlatd to th numbr of clls in ach layr of th D spac. So th D modl nds M tims mor BlockRAMs. As long as w kp a similar mmory intrfac modl structur, th running spd of th D modl should b similar to th currnt D modl, although th hardwar structur will b much mor complicatd. Also, mor paralllism can b introducd as long as thr is nough FPGA spac. Th whol dsign s spd rachs architctural limits whn th Mmory to BlockRAM and th BlockRAM to Mmory procsss bcom th critical path of th dsign. Th ultimat goal of this rsarch is to dvlop a D FDTD custom hardwar implmntation that can gratly acclrat th FDTD algorithm for ral-tim applications. Th D FDTD implmntation can b usd as a forward modl acclrator in th burid objct dtction projct in CnSSIS, and it can also b applid to many othr aras. Th fast dvlopmnt of FPGA tchnology maks D FDTD implmntation possibl for ral-tim computational lctromagntics. Finit-Diffrnc Tim-Domain (FDTD) Mthod, Procdings of th ACM/SIGDA tnth intrnational symposium on Fild-programmabl gat arrays. pp [9] J. P. Durbano, F. E. Ortiz, J. R. Humphry, D. W. Prathr, and M. S. Mirotznik, Hardwar Implmntation of a Thr-Dimnsional Finit-Diffrnc Tim-Domain Algorithm, IEEE Antnnas and Wirlss Propagation Lttrs, VOL.,. ACKNOWLEDGEMENT This rsarch was supportd in part by CnSSIS, th Cntr for Subsurfac Snsing and Imaging Systms, undr th Enginring Rsarch Cntrs Program of th National Scinc Foundation (Award Numbr EEC-99868).. REFERENCES [] K. S. Kunz, R. J. Lubbrs, Th Finit Diffrnc Tim Domain Mthod for Elctromagntics, CRC Prss, 99. [] A. Taflov, Advancs in Computational Elctrodynamics: Th Finit-Diffrnc Tim-Domain Mthod, Artch Hous, Inc., 998. [] K. Y, Numrical solution of initial boundary valu problms involving Maxwll s quations in isotropic mdia, IEEE Trans. Antnnas and Propagation, 6 (966), pp. 7. [4] P. Kosmas, Y. Wang, and C. Rappaport, Thr-Dimnsional FDTD Modl for GPR Dtction of Objcts Burid in Ralistic Disprsiv Soil, SPIE Arosns Confrnc, Orlando, FL, April, pp. 8. [5] B. Yang and C. Rappaport, Rspons of Ralistic Soil for GPR Applications with Two Dimnsional FDTD, IEEE Transactions on Goscinc and Rmot Snsing, Jun, pp [6] W. Wdon, and C. Rappaport, A Gnral Mthod for FDTD Modling of Wav Propagation in Arbitrary Frquncy-Disprsiv Mdia, IEEE Transactions on Antnnas and Propagation, vol. 45, March 997, pp [7] C. Rappaport, S. Wu and S. Winton, FDTD Wav Propagation Modling in Disprsiv Soil Using a Singl Pol Conductivity Modl, IEEE Transactions on Magntics, vol. 5, May 999, pp [8] R. N. Schnidr, L. E. Turnr, M. M. Okoniwski, Application of FPGA Tchnology to Acclrat th