This Unit: Dynamic Scheduling. Can Hardware Overcome These Limits? Scheduling: Compiler or Hardware. The Problem With In-Order Pipelines

Transcription

1 This Uit: Damic Schedulig CSE 0 Computer Sstems Architecture Damic Schedulig Slides origiall developed b Drew Hilto (IBM) ad Milo Marti (Uiversit of Peslvaia) App App App Sstem software Mem CPU I/O Code schedulig To reduce pipelie stalls To icrease ILP (is level parallelism) Two approaches to schedulig Last Uit: Static schedulig b the compiler This Uit: Damic schedulig b the hardware Schedulig: Compiler or Hardware Compiler + Potetiall large schedulig scope (full program) + Simple hardware fast clock, short pipelie, ad low power Low brach predictio accurac (profilig?) Little iformatio o memor depedeces (profilig?) Ca t damicall respod to cache misses Pai to speculate ad recover from mis-speculatio (h/w support?) Hardware + High brach predictio accurac + Damic iformatio about memor depedeces + Ca respod to cache misses + Eas to speculate ad recover from mis-speculatio Fiite bufferig resources fudametall limit schedulig scope Schedulig machier adds pipelie stages ad cosumes power Ca Hardware Overcome These Limits? Damicall-scheduled processors Also called out-of-order processors Hardware re-schedules iss withi a slidig widow of VoNeuma iss As with pipeliig ad superscalar, ISA uchaged Same hardware/software iterface, appearace of i-order Icreases schedulig scope Does loop urollig trasparetl Uses brach predictio to uroll braches Examples: Petium Pro/II/III (-wide), Core (-wide), Alpha (-wide), MIPS R0000 (-wide), Power (-wide) Basic overview of approach The Problem With I-Order Pipelies 9 0 addf f0,f f F D E+ E+ E+ W mulf f,f f F d* d* D E*E*E*E*E*W subf f0,f f F p* p* D E+ E+ E+ W What s happeig i ccle? mulf stalls due to data depedece OK, this is a fudametal problem subf stalls due to pipelie hazard Wh? subf ca t proceed ito D because mulf is there That is the ol reaso, ad it is t a fudametal oe Maitaiig i-order writes to reg. file (both write f) Wh ca t subf go ito D i ccle ad E+ i ccle? A Word About Data Hazards Real is sequeces pass values via registers/memor Three kids of data depedeces (where s the fourth?) Read-after-write (RAW) True-depedece R E G M E M add r,r r sub r,r r or r,r r st r [r] ld[r] r Write-after-read (WAR) Ati-depedece add r,r r sub r,r r or r,r r ld[r] r st r [r] Write-after-write (WAW) Output-depedece add r,r r sub r,r r or r,r r st r [r] st r [r] Ol oe depedece betwee a two iss (RAW has priorit) Focus o RAW depedeces WAR ad WAW: less commo, just bad amig luck Elimiated b usig ew register ames, (ca t reame memor!)

2 Fid the RAW, WAR, ad WAW depedeces add r r, r sub r r, r ad r r, r xor r0 r, r or r r0, r mult r r0, r Raw iss: Code Example True Depedecies add r,r r sub r,r r mul r,r r div r, r False Depedecies add r,r r sub r,r r mul r,r r div r, r True (real) & False (artificial) depedecies Divide is idepedet of subtract ad multipl iss Ca execute i parallel with subtract Ma registers re-used Just as i static schedulig, the register ames get i the wa How does the hardware get aroud this? Approach: (step #) reame registers, (step #) schedule 9 Step #: Register Reamig To elimiate register coflicts/hazards Architected vs. Phsical registers level of idirectio Names: r,r,r Locatios: p,p,p,p,p,p, Origial mappig: r p, r p, r p, p are available MapTable FreeList r r r p p p p,p,p, p p p p p p p p p p p p,p, p, add r,r,r sub r,r,r mul r,r,r div r,,r add p,p,p sub p,p,p mul p,p,p div p,, Reamig: coceptuall write each register oce + Removes false depedeces +Leaves true depedeces itact! Whe to reuse a phsical register? After overwritig is doe I$ B P Step #: Damic Schedulig D add p,p,p sub p,p,p mul p,p,p div p,, is buffer regfile Read Table P P P P P P Yes Yes add p,p,p Yes Yes Yes sub p,p,p ad div p,, Yes Yes Yes Yes Yes mul p,p,p Yes Yes Yes Yes Yes Yes Istructios fetch/decoded/reamed ito Istructio Buffer AKA istructio widow or istructio scheduler Istructios (coceptuall) check read bits ever ccle Execute whe read S D$ 0 Out-of-order Pipelie Buffer of istructios Fetch Decode Reame Dispatch Issue Reg-read Execute Writeback Commit I-order frot ed Out-of-order executio REGISTER RENAMING

3 Register Reamig Algorithm Data structures: maptable[architectural_reg] phsical_reg Free list: get/put free register Algorithm: at decode for each istructio: is.phs_iput = maptable[is.arch_iput] is.phs_iput = maptable[is.arch_iput] is.phs_to_free = maptable[arch_output] ew_reg = get_free_phs_reg() is.phs_output = ew_reg maptable[arch_output] = ew_reg At commit Oce all older istructios have committed, free register put_free_phs_reg(is.phs_to_free) xor r, r r sub r, r r addi r, r r r r r r p p p p p p p0 xor r, r r sub r, r r addi r, r xor p, p xor r, r r sub r, r r addi r, r xor p, p p r p r p r p r p p p0 r p r p r p r p p p0 xor r, r r sub r, r r addi r, r xor p, p p xor r, r r sub r, r r addi r, r xor p, p p add p, p r p r p r p r p p0 r p r p r p r p p0 9

4 xor r, r r sub r, r r addi r, r xor p, p p xor r, r r sub r, r r addi r, r xor p, p p r p r p r p r p p0 r p r p r p r p0 0 xor r, r r sub r, r r addi r, r xor p, p p sub p, p xor r, r r sub r, r r addi r, r xor p, p p sub p, p r p r p r p r p0 r p r p r p r p0 xor r, r r sub r, r r addi r, r xor p, p p sub p, p xor r, r r sub r, r r addi r, r xor p, p p sub p, p addi, r p r p r r p0 r p r p r r p0

5 xor r, r r sub r, r r addi r, r xor p, p p sub p, p addi, xor r, r r sub r, r r addi r, r xor p, p p sub p, p addi, r p r p r r p0 r r p r r p0 Out-of-order Pipelie Buffer of istructios Fetch Decode Reame Dispatch Issue Reg-read Execute Writeback Commit DYNAMIC SCHEDULING Have uique register ames Now put ito ooo executio structures 9 Dispatch Reamed istructios ito ooo structures Re-order buffer (ROB) Holds all istructios util the commit Issue Queue U-executed istructios Cetral piece of schedulig logic Cotet Addressable Memor (CAM) (more later) Issue Queue Holds u-executed istructios Tracks read iputs Phsical register ames + read bit AND to tell if read Is Ip R Ip R Dst Age Read? 0

6 Dispatch Steps Allocate IQ slot Full? Stall Read read bits of iputs Table -bit per preg Clear read bit of output i table Istructio has ot produced value et Write data i IQ slot xor p, p p sub p, p addi, Dispatch Example Issue Queue Is Ip R Ip R Dst Age Read bits p p p p p p Dispatch Example Dispatch Example xor p, p p sub p, p addi, Read bits p p xor p, p p sub p, p addi, Read bits p p Issue Queue Is Ip R Ip R Dst Age xor p p p 0 p p p p Issue Queue Is Ip R Ip R Dst Age xor p p p 0 add p p p p p p Dispatch Example Dispatch Example xor p, p p sub p, p addi, Read bits p p xor p, p p sub p, p addi, Read bits p p Issue Queue Is Ip R Ip R Dst Age xor p p p 0 add p p sub p p p p p p Issue Queue Is Ip R Ip R Dst Age xor p p p 0 add p p sub p p addi --- p p p p

7 Executio (ooo) stages Select read istructios Sed for executio Wakeup depedets Out-of-order pipelie Issue Reg-read Execute Writeback Damic Schedulig/Issue Algorithm Data structures: Read table[phs_reg] es/o queue) (part of issue Algorithm at schedule stage (prior to read registers): foreach istructio: if table[is.phs_iput] read && table[is.phs_iput] read the is is read select the oldest read istructio table[is.phs_output] = read 9 Issue = Select + Wakeup Select N oldest, read istructios N=, xor N=, xor ad sub Note: ma have executio resource costraits: i.e., load/store/fp Is Ip R Ip R Dst Age xor p p p 0 add p p sub p p addi --- Read! Read! Issue = Select + Wakeup Wakeup depedet istructios CAM search for Dst i iputs Set read Also update read-bit table for future istructios Is Ip R Ip R Dst Age xor p p p 0 add p p sub p p addi --- Read bits p p p p p p 0 Select/Wakeup oe ccle Depedets go back to back Next ccle: add/addi are read: Issue Is Ip R Ip R Dst Age add p p addi --- Register Read Whe do istructios read the register file? Optio #: after select, right before execute (Not doe at decode) Read phsical register (reamed) Or get value via bpassig (based o phsical register ame) This is Petium, MIPS R0k, Alpha stle Phsical register file ma be large Multi-ccle read Optio #: as part of issue, keep values i Issue Queue Petium Pro, Core, Core i

8 OOO executio (-wide) OOO executio (-wide) xor RDY add sub RDY addi p p p p 9 p p xor add RDY sub addi RDY xor p, p p sub p, p p p p p 9 p p OOO executio (-wide) OOO executio (-wide) xor add sub addi add p, p addi, p p p p 9 p p xor, p sub xor add sub addi p p p p 9 p p add _, 9 addi _, p 0 OOO executio (-wide) OOO executio (-wide) p p xor add sub addi p p p 9 p p 0 0 xor add sub addi p p p 9 p p

9 Note similarit to i-order OOO executio (-wide) p p p p 9 p p Multi-ccle operatios Multi-ccle ops (load, fp, multipl, etc.) Wakeup deferred a few ccles Structural hazard? Cache misses? Speculative wake-up (assume hit) Cacel exec of depedets Re-issue later Details: complicated, ot importat Re-order Buffer (ROB) All istructios i order Two purposes Mispredictio recover I-order commit Maitai appearace of i-order executio Freeig of phsical registers RENAMING REVISITED Reamig revisited Overwritte register Freed at commit Restore i map table o recover Brach mis-predictio recover Also must be read at reame xor r,r r add r,r r sub r,r r addi r, r r p r p r p r p p p0 9

10 xor r,r r xor p, p [p] add r,r r sub r,r r addi r, r xor r,r r xor p, p p [p] add r,r r sub r,r r addi r, r r p r p r p r p p p0 r p r p r p r p p0 0 xor r,r r xor p, p p [p] add r,r r add p, p [p] sub r,r r addi r, r xor r,r r xor p, p p [p] add r,r r [p] sub r,r r addi r, r r p r p r p r p p0 r p r p r p r p0 xor r,r r xor p, p p [p] add r,r r sub r,r r sub p, p [p] [p] addi r, r xor r,r r xor p, p p [p] add r,r r sub r,r r sub p, p [p] [p] addi r, r r p r p r p r p0 r p r p r r p0

11 xor r,r r xor p, p p [p] add r,r r sub r,r r addi r, r sub p, p addi, [p] [p] [p] xor r,r r xor p, p p [p] add r,r r sub r,r r addi r, r sub p, p addi, [p] [p] [p] r p r p r r p0 r r p r r p0 ROB ROB etr holds all ifo for recover/commit Logical register ames Phsical register ames Istructio tpes Dispatch: isert at tail Full? Stall Commit: remove from head Not completed? Stall Recover Completel remove wrog path istructios Flush from IQ Remove from ROB Restore map table to before mispredictio Free destiatio registers 9 Recover example bz r loop bz p, loop [ ] xor r, r r xor p, p p [p] sub r, r r addi r, r sub p, p addi, [p] [p] [p] Recover example bz r loop bz p, loop [ ] xor r, r r xor p, p p [p] sub r, r r addi r, r sub p, p addi, [p] [p] [p] r r p r r p0 r p r p r r p0 0

12 Recover example bz r loop bz p, loop [ ] xor r, r r xor p, p p [p] sub r, r r sub p, p [p] [p] Recover example bz r loop bz p, loop [ ] xor r, r r xor p, p p [p] [p] r p r p r p r p0 r p r p r p r p p0 bz r loop xor r, r r Recover example bz p, loop xor p, p p [ ] [p] Recover example bz r loop bz p, loop [ ] r p r p r p r p p p0 r p r p r p r p p p0 What about stores Stores: Write D$, ot registers Ca we reame memor? Recover i the cache? No (at least ot easil) Cache writes urecoverable Stores: ol whe certai Commit Commit xor r, r r xor p, p p [p] sub r, r r addi r, r sub p, p addi, [p] [p] [p] At commit: istructio becomes architected state I order Ol whe istructios are fiished Free overwritte register (wh?)

13 xor r,r r add r,r r sub r,r r addi r, r Freeig over-writte register xor p, p p sub p, p addi, [p] [p] [p] [p] Commit Example xor r,r r xor p, p p [p] add r,r r sub r,r r addi r, r sub p, p addi, [p] [p] [p] Before xor: r p After xor: r p Iss older tha xor reads p Iss ouger tha xor read p (util ext r-writig istructio) At commit of xor, o older istructios exist No oe else eeds p free it! r r p r r p0 9 Commit Example xor r,r r add r,r r sub r,r r addi r, r xor p, p p sub p, p addi, [p] [p] [p] [p] Commit Example add r,r r sub r,r r addi r, r sub p, p addi, [p] [p] [p] r r r r r p p p0 p r r r r r p p p0 p p 0 Commit Example Commit Example sub r,r r addi r, r sub p, p addi, [p] [p] addi r, r addi, [p] r r r r r p p p0 p p p r r r r r p p p0 p p p p

14 Out of order pipelie diagrams Stadard stle: large ad cumbersome Chage laout slightl Colums = stages (dispatch, issue, etc.) Rows = istructios Cotet of boxes = ccles For our purposes: issue/exec = ccle Igore preg read latec, etc. Load-use, mul, div, ad FP loger Out of order pipelie diagrams Istructio ld [p] p add p, p p xor p, p p ld [] Buffer of istructios Fetch Decode Reame Dispatch Issue Reg-read Execute Writeback Commit Out of order pipelie diagrams Out of order pipelie diagrams Istructio Istructio ld [p] p add p, p p xor p, p p ld [] ld [p] p add p, p p xor p, p p ld [] -wide Ifiite ROB, IQ, Pregs Loads: ccles Ccle : Dispatch xor ad ld Out of order pipelie diagrams Out of order pipelie diagrams Istructio Istructio ld [p] p ld [p] p add p, p p add p, p p xor p, p p xor p, p p ld [] ld [] Ccle : Dispatch xor ad ld st Ld issues -- also ote WB ccle while ou do this (Note: do t issue if WB ports full) Ccle : add ad xor are ot read d load is issue it 9

15 Out of order pipelie diagrams Out of order pipelie diagrams Istructio Istructio ld [p] p ld [p] p add p, p p add p, p p xor p, p p xor p, p p ld [] ld [] Ccle : othig Ccle : add ca issue Ccle : st load ca commit (oldest istructio & fiished) xor ca issue 90 9 Out of order pipelie diagrams Out of order pipelie diagrams Istructio Istructio ld [p] p ld [p] p add p, p p add p, p p xor p, p p xor p, p p ld [] ld [] Ccle : add ca commit (oldest istructio & fiished) Ccle : xor ad ld ca commit (-wide: ca do both at oce) 9 9 Out of order pipelie diagrams Istructio ld [p] p add p, p p xor p, p p ld [] Buffer of istructios HANDLING MEMORY OPS Fetch Decode Reame Dispatch Issue Reg-read Execute Writeback Commit 9 9

16 Damicall Schedulig Memor Ops Compilers must schedule memor ops coservativel Optios for hardware: Hold loads util all prior stores execute (coservative) Execute loads as soo as possible, detect violatios (aggressive) Whe a store executes, it checks if a later loads executed too earl (to same address). If so, flush pipelie Lear violatios over time, selectivel reorder (predictive) Before Wrog(?) ld r,(sp) ld r,(sp) ld r,(sp) ld r,(sp) add r,r,r //stall ld r,0(r) //does rsp? st r,0(sp) add r,r,r ld r,0(r) ld r,(r) //does r+sp? ld r,(r) st r,0(sp) sub r,r,r //stall sub r,r,r st r,(r) st r,(r) 9 Loads ad Stores Istructio fdiv p,p p st p [p] st p [p] ld [] Ccle : Ca ld[] execute? (wh or wh ot?) 9 Loads ad Stores Loads ad Stores Istructio Istructio fdiv p,p p fdiv p,p p st p [p] st p [p] st p [p] st p [p] ld [] ld [] Aliasig (agai) p? p? Suppose p ad p!= Ca ld[] execute? (wh or wh ot?) 9 99 Memor Forwardig Stores write cache at commit Commit is i-order, delaed b all istructios Allows stores to be udoe o brach mis-predictios, etc. Loads read cache Earl executio of loads is critical Forwardig Allow store load commuicatio before store commit Coceptuall like reg. bpassig, but differet implemetatio Wh? Addresses ukow util execute Forwardig: Store Queue Store Queue Holds all i-flight stores CAM: searchable b address Age logic: determie ougest matchig store older tha load Store executio Write Store Queue Address + Data Load executio Search SQ Match? Forward Read D$ address address load positio Store Queue (SQ) age Data cache data i value data out head tail 00 0

17 Load schedulig Store Load Forwardig: Get value from executed (but ot comitted) store to load Load Schedulig: Determie whe load ca execute with regard to older stores Coservative load schedulig: All older stores have executed Some architectures: split store address / store data Ol require kow address Advatage: alwas safe Disadvatage: performace (limits out-of-orderess) ld [r] r ld [r] r add r,r r st r [r] ld [r] r ld [r] r add r,r r st r [r] // loop cotrol here With coservative load schedulig, what ca go out of order? 0 0 ld [p] p ld [p] p ld [p] p ld [p] p add p,p add p,p st [p] st [p] ld [p] ld [p] ld [p] ld [p] add, p add, p st p [p] st p [p] wide, coservative schedulig issue load per ccle loads take ccles to complete Ccle : Dispatch iss #, # wide, coservative schedulig issue load per ccle loads take ccles to complete Ccle : Wh do t we issue #? 0 0 ld [p] p ld [p] p ld [p] p ld [p] p add p,p add p,p st [p] st [p] ld [p] ld [p] ld [p] ld [p] add, p add, p st p [p] st p [p] wide, coservative schedulig issue load per ccle loads take ccles to complete Ccle : Wh do t we issue #? Wh do t we issue #? wide, coservative schedulig issue load per ccle loads take ccles to complete Ccle : Wh do t we issue #? 0 0

18 ld [p] p ld [p] p ld [p] p ld [p] p add p,p add p,p st [p] st [p] ld [p] ld [p] ld [p] ld [p] add, p add, p st p [p] st p [p] wide, coservative schedulig issue load per ccle loads take ccles to complete Ccle : Fiall some actio! wide, coservative schedulig issue load per ccle loads take ccles to complete Ccle : Gettig somewhere ld [p] p ld [p] p ld [p] p ld [p] p add p,p add p,p st [p] st [p] 9 ld [p] ld [p] ld [p] ld [p] 9 add, p add, p st p [p] st p [p] wide, coservative schedulig issue load per ccle loads take ccles to complete Ccle : Etc... wide, coservative schedulig issue load per ccle loads take ccles to complete Ccle 9: Etc... 0 ld [p] p ld [p] p ld [p] p ld [p] p add p,p add p,p st [p] 9 st [p] 9 ld [p] ld [p] ld [p] 9 ld [p] 9 add, p add, p st p [p] st p [p] wide, coservative schedulig issue load per ccle loads take ccles to complete Ccle : Yaw wide, coservative schedulig issue load per ccle loads take ccles to complete Ccle : Stretch

19 ld [p] p ld [p] p ld [p] p ld [p] p add p,p add p,p st [p] 9 st [p] 9 ld [p] ld [p] ld [p] 9 ld [p] 9 add, p add, p st p [p] st p [p] wide, coservative schedulig issue load per ccle loads take ccles to complete Ccle : Zzzzzz wide, coservative schedulig issue load per ccle loads take ccles to complete Ccle : -wide ooo = -wide iorder I am goig to cr. ld [p] p ld [p] p add p,p st [p] 9 ld [p] ld [p] 9 add, p st p [p] Load Speculatio Speculatio requires two thigs.. Detectio of mis-speculatios How ca we do this? Recover from mis-speculatios Squash from offedig load Saw how to squash from braches: same method wide, coservative schedulig issue load per ccle loads take ccles to complete What was # waitig for?? Ca I speculate? Load Queue flush? Detects ld orderig violatios store positio Execute load: write addr to LQ load queue (LQ) Also ote a store forwarded from head address Execute store: search LQ Youger load with same age addr? tail Did t forward from ouger store? Data Cache SQ head tail Store Queue + Load Queue Store Queue: hadles forwardig Writte b stores (@ execute) Searched b loads (@ execute) Read SQ whe ou write to the data cache (@ commit) Load Queue: detects orderig violatios Writte b loads (@ execute) Searched b stores (@ execute) Both together Allows aggressive load schedulig Stores do t costrai load executio 9

20 ld [p] p ld [p] p ld [p] p ld [p] p add p,p add p,p st [p] st [p] ld [p] ld [p] ld [p] ld [p] add, p add, p st p [p] st p [p] wide, aggressive schedulig issue load per ccle loads take ccles to complete Ccle : Speculativel execute # before the store (#). wide, aggressive schedulig issue load per ccle loads take ccles to complete Ccle : Speculativel execute # before the store (#). 0 ld [p] p ld [p] p add p,p st [p] 9 ld [p] 9 ld [p] 0 add, p 9 0 st p [p] 9 0 wide, aggressive schedulig issue load per ccle loads take ccles to complete Fast forward: ccles faster Actuall ooo this time! Aggressive Load Schedulig Allows loads to issue before older stores Icreases out-of-orderess + Whe o coflict, icreases performace - Coflict squash worse performace tha waitig Some loads might forward from stores Alwas aggressive will squash a lot Ca we have our cake AND eat it too? Predictive Load Schedulig Predict which loads must wait for stores Fool me oce, shame o ou fool me twice? Loads default to aggressive Keep table of load PCs that have bee caused squashes Schedule these coservativel + Simple predictor Makes bad loads wait for all older stores: ot great More complex predictors used i practice Predict which stores loads should wait for Out of Order: Widow Size Schedulig scope = ooo widow size Larger = better Costraied b phsical registers (#preg) ROB roughl limited b #preg = ROB size + #logical registers Big register file = hard/slow Costraied b issue queue Limits umber of u-executed istructios CAM = ca t make big (power + area) Costraied b load + store queues Limit umber of loads/stores CAMs Active area of research: scalig widow sizes Usefuless of large widow: limited b brach predictio 9% brach mis-predictio rate: i 0 braches, i 00 iss

21 Out of Order: Beefits Allows speculative re-orderig Loads / stores Brach predictio Schedule ca chage due to cache misses Differet schedule optimal from o cache hit Doe b hardware Compiler ma wat differet schedule for differet hw cofigs Hardware has ol its ow cofiguratio to deal with Static vs. Damic Schedulig If we ca do this i software wh build complex (slow-clock, high-power) hardware? + Performace portabilit Do t wat to recompile for ew machies + More iformatio available Memor addresses, brach directios, cache misses + More registers available Compiler ma ot have eough to schedule well + Speculative memor operatio re-orderig Compiler must be coservative, hardware ca speculate But compiler has a larger scope Compiler does as much as it ca (ot much) Hardware does the rest Out of Order: Top Thigs to Kow Register reamig How to perform it ad how to recover it Commit Precise state (ROB) How/whe registers are freed Issue/Select Wakeup: CAM Choose N oldest read istructios Stores Write at commit Forward to loads via SQ Loads Coservative/aggressive/predictive schedulig Violatio detectio via LQ Summar: Damic Schedulig Damic schedulig Totall i the hardware Also called out-of-order executio (OoO) Fetch ma istructios ito istructio widow Use brach predictio to speculate past (multiple) braches Flush pipelie o brach mispredictio Reame to avoid false depedecies Execute istructios as soo as possible Register depedecies are kow Hadlig memor depedecies more trick Commit istructios i order Athig strage happes pre-commit, just flush the pipelie Curret machies: 00+ istructio schedulig widow 9