CO Algorithms: Brief History

Size: px

Start display at page:

Download "CO Algorithms: Brief History"

Benjamin Hood
5 years ago
Views:

1 Piyush Kumar Departmet of Computer Sciece Advisor: Joseph S.B. Mitchell CO Algorithms: Brief History Frigo, Leiserso, Prokop, Ramachadra (FOCS 99) Cache Oblivious Algorithms Thesis Harold Prokop s Thesis Beder, Demaie, Farch-Coltu (FOCS 00) Cache Oblivious B-TreesB Arge, Beder, Demaie et.al. (STOC02) CO Priority Queue 1

2 Talk Outlie 4 4! Workstatios SUN UltraSparc 2: UltraSparc 16kB L1, 512kB L2. SGI Visual Workstatio 540: Quad-Petium III 32kB L1, 1024kB L2. Dell Precisio: Dual-Petium III 32kB L1 512kB L2. IBM ThikPad 600: Petium II 32kB L1, 256kB L2. Compaq Presario: AMD K6-III 64kB L1, 256kB L2, 1024kB L3. How ca we write portable code that rus efficietly o differet multilevel cachig architectures? 2

3 3 Itel Itaiums Itel Itaiums Matrix Multiplicatio (MM) Matrix Multiplicatio (MM) = = k kj ik ij b a c 1 C c c c c c c c c c B b b b b b b b b b A a a a a a a a a a =

4 Cache-Aware MM BLOCK-MULT (A,B,C,) 1 for forii 1 to to/s 2 do do for forjj 1 to to/s 3 do for k 1 to to/s 4 do do ORD-MULT (A (A ik,b ik,b kj,c kj,c ij,s) ij,s) s s s s [HK81] Cache-Aware MM s s BLOCK-MULT (A,B,C,) Oracle?! 1 for forii 1 to to/s 2 do do for forjj 1 to to/s 3 do for k 1 to to/s 4 do do ORD-MULT (A (A ik,b ik,b kj,c kj,c ij,s) ij,s) Tue s so that A ik, B kj, ad C ij s = Θ Z just fit ito cache ( ) If > s, the Q 3 = Θ( L Z ). 3 2 ( ) = Θ ( s) ( s L) ) Optimal [HK81]. 4

5 Two-Level Three-Level Cache s t s Oe parameter voodoo parameter per cachig per level! cachig level! BLOCK-MULT (A,B,C,) BLOCK-MULT (A,B,C,) 1 1 for for i 1 i 1 1 1to to/s /s 2 2 do do for for j 1 j to to/s /s 3 3 do do for forkk to to/s /s 4 4 do do for ORD-MULT for i ORD-MULT 2 i (A (A ik,b ik,b kj,c kj,c ij,s) ij,s) to tos s /t /t 5 5 do do for forj 2 j 2 1 1to tos s /t /t 66 do do for for k k to tos s /t /t 77 do do for ORD-MULT fori (A ORD-MULT 3 i (A ik,b ik,b kj,c kj,c ij,t) 3 11to tot t /u /u 8 ij,t) 8 do do for forj 3 j 3 1 1to tot t /u /u 99 do do for fork k to tot t /u /u do do ORD-MULT (A ORD-MULT (A ik,b ik,b kj,c kj,c ij,u) ij,u) Recursive Matrix Multiplicatio Divide ad coquer o matrices. C 11 C 12 = A 11 A 12 B 11 B 12 C 21 C 22 A 21 A 22 B 21 B 22 = A 11 B 11 A 11 B 12 A 21 B 11 A 21 B 12 + A 12 B 21 A 12 B 22 A 22 B 21 A 22 B 22 8 multiplicatios of (/2) (/2) matrices. 1 additio of matrices. 5

6 ( 3 )-Matrix Multiplicatio (,) (,) time / / 3 3 [i [i aosecods] iterative algorithm recursive algorithm 450-MHz AMD K6-III processor with with 32kB 32kB L1-cache, 64kB 64kB L2-cache, ad ad 1MB 1MB L3-cache. [double precisio umbers] Experimets: MM Liux Athlo 1Ghz/1Gb/g++ -O Time i Secods Size of Matrix Loop Base = bytes Base = 32 bytes 6

7 Experimets: MM Liux/Itaium/2GB/g++ -O Time i secods Size of Matrix loop Base = 32 bytes Base = bytes Recursive Traspose Partitio 1. matrix Partitio i matrix i 4 i submatrices 4 A, submatrices A, B, A, B, C, B, C, ad C, D ad D Recursively 2. traspose Recursively A. traspose A. A Recursively 3. traspose Recursively ad traspose swap ad B swap ad B C. ad C. C Recursively 4. traspose Recursively D. traspose D. D. A T C C T B T D D T ( ( 2 2 / L) L) cache cache misses, misses, which which is is optimal. optimal. Used Used as as a a subroutie subroutie i i our our optimal optimal cache-oblivious FFT FFT [HK81]. [HK81]. Harald Prokop 18 Oct

8 Code: The IPlace Loop Code : Co Traspose 8

9 Experimets: MT Notebook, Widows 2k/512Mb/PIII 1GHz/g++ -O Time i Secods N loop co_traspose Experimets: MT Notebook, Widows 2k/512Mb/PIII 1GHz/g++ -O Time i secods N loop co_traspose 9

10 Experimets: MT Liux Athlo 1Ghz/1Gb/g++ -O Time i Secods N loop co_traspose Experimets: MT Liux Athlo 1Ghz/1Gb/g++ -O3/ Size =N x (P =100), tall matrices Time i secods M Not Iplace Loop co_traspose 10

11 Experimets: MT Liux Athlo 1Ghz/1Gb/g++ -O3/ Size = N x (P =1000) Time i secods M Not Iplace Loop co_traspose What wet Wrog? Blockig! Ad the loop was IPlace! 11

12 Experimets: MT Loop ot Iplace Liux Athlo 1Ghz/1Gb/g++ -O3/ Size = N x N Time i secods N loop Base = 2kb Base = 8b Experimets: MT Loop ot Iplace Notebook, Widows 2k/512Mb/PIII 1GHz/g++ -O Time i secods N Loop Base = 2k el Base = 8 el Base = 1el 12

13 Did we miss somethig? " #$%&'" " ($ )! " *$+, " -$, ".$ " /$ Chatterjee & Se HPCA 00 Did we miss somethig? 13

14 Static Searches,) ), ", What is a layout?,!, $8'), 14

15 Example of Va Emde Boas Cut 1 1, 2, 3, 4, 8, 9, 5, 10, 11, 6, 12, 13, 7, 14, 15 Aother View 15

16 Theoretical Guaratees? Cache Complexity Q() 9 O(log L ) Work Complexity W() 9 O(log ) 5! I Practice?? Widows otebook/512mb/piii 1Gz/256 byte odes Time i secods ^x odes Geeral Search Cache Oblivious 16

17 I Practice II 1.4 Widows otebook/512mb/piii 1Gz/32 byte odes Time i secods Number of Nodes 2^x Geeral Search Cache Oblivious I Practice III 3 Liux/Itaium/2GB/g++ -O3/ 48 byte odes 2.5 Time i secods ^x odes Geeral Search Cache Oblivious 17

18 I Practice! ),,': (Cache oblivious Practical Matrix operatio results) ;;: (Cache oblivious dictioaries) 8; ;8 (CO B-Trees) Talk outlie '12 < 18

19 (M,B) Ideal Cache Model (Z,L) Ideal Cache Model Z = Ω( L 2 ) P cache L Z/L Cache Lies Q mai memory Features: Two-level hierarchy. Cache of size Z. Cache-lie legth L. Fully associative. Optimal, omisciet replacemet. Measures: Work W. Cache misses Q. 19

20 Assumptios? =', " Optimal Cache Replacemet No Asymptotic loss Fully-associative LRU ca be used istead of optimal replacemet with o asymptotic loss of performace [ST85]. Fully-associative LRU caches ca be maitaied i ordiary memory with costat slowdow i expected performace. Cache Obliviousess Cache-oblivious algorithms aturally tue for varyig cache sizes. multiple levels of cache. Whe a subproblem fits ito a give level of cache, o further cache misses are icurred beyod those required to brig the subproblem itself ito the cache. A optimal cache-oblivious algorithm ca be made to ru optimally i the HMM [AACN87] ad SUMH [VN93] models 20

21 CO-Sortig!,! 12 ➅ 1 ;: >'2 Fuel Sort 1/3 2/3 54> N N 8', 1/3 N Iput Array Sorted Output 21

22 Harold s slide picture 2 3 buffers 2 3 Q ( ) = 13 = O Q ( ) + Q ( ) Merge ( 1+ ( L)( 1+ log ) ) Z Fuel Sort: k-mergersk!!?!@* 3 4 ' 3 A!,1 2 22

23 Fuel Sort: k-mergerk 2 k 3 k L 1 2 L 2 2k 3/ 2 4'!1! )' 2 R k 3 k L k B 3/ 2 '8 k Fuel Sort : Optimality :!, O( log ), O( 1+ (1 + logz )) L Agarwal ad Vitter show that there is a Boud o the umber of cache misses. Ω( log L Z L ( )) L 23

24 Distributio Sort 5"), >C8', ))!!8',,! Recursive Sortig of Subarrays iput elemets, partitioed ito cotiguous subarrays of of size :: Recursively sorted arrays: Order: Harald Prokop 18 Oct

25 Distributio Step Recursively sorted arrays: Buckets: Pivots: Distribute step? Order: Harald Prokop 18 Oct The Distributio Step +)), )! %45 B B, B... 1, 2 3 B q ',' 25

26 The Recursive Bucketig used SubArray1 )",( SubArray2 Buffer 1# Buffer 2 Recursive Sortig of Buckets After distributio step: Recursively sort each bucket. Harald Prokop 18 Oct

27 Some Aalysis ow 12 6 Talk outlie ' ) ", 7 27

28 Radomized CO Sortig CO-Sortig Experimets 8 (5)! D5 D 1!,) 2 28

29 I Practice: 2 Pass CO-Dissort CO-Distributio Sort secods Series1 Series ^x doubles 9(@#-9#/*E- Does this imply aythig?* 4,'#F), G;4,; #HHHF), I ' 8 29

30 Two Levels of Recursio CO-Distributio Sort Time i Secods Series1 Series ^x doubles 9(@#H9#H(- Talk outlie ' 0 30

31 What wet Wrog? What wet Wrog? Assumptios! 31

32 Is the model oversimplified? "', (Not fully associative) " ( Asymptotics hides disasters! ) D(Istructio Caches) =(ot tall),(coherece misses: Xeo) 8 5,(4Gb Limit)!(Ca icrese I/O speed) : ( Causes misses eve if problem fits ito cache ) 4Practice Elaborated Here What did I lear from it? J;8 %=! '", ) ) 4 I 1, ) 2 = 4 " ), 7 ', ' ' ), 32

33 Kow Optimal Results K Matrix Multiplicatio Matrix Traspose -poit FFT LUP Decompositio Sortig Searchig Results Kow 5,B 14 2 =8! " D 1 O( log B M B N ) B O( sort( V )) O( sort( V )) E O (( V+ )log2v + sort( E)) B O ( V + sort( E)) O ( sort( E) + log log2 2 V ) 33

34 Results Kow ",8' =D 14 2L =, '+ * Θ (1 + 2 N + B O( sort( V )) O(logB ) OPT O( sca( V )) O(log(B)) O( sort( N)) 3 N ) B M New Result ) '1:78;D4D2 6(* " $, '8 34

35 Publicatios Book : Algorithms for Memory Hierarchies Chapter: : Cache Oblivious Algorithms (Editors Meyer et.al.) Cache Oblivious Vorooi diagrams (with Edgar Ramos, I Progress) Miimum Eclosig Balls (with Joe Mitchell, Alper Yildirim) (i Aleex 03) < Also cache oblivious > Other Itroductios Chapter by Eric Demaie 35

36 Ackowledgemets! " #!$! % &$ ' ( ) Special thaks to MPI Saarbruecke where part of this work was doe. 36

Cache-Efficient Algorithms II

Cache-Efficient Algorithms II 6.172 Performace Egieerig of Software Systems LECTURE 9 Cache-Efficiet Algorithms II Charles E. Leiserso October 7, 2010 2010 Charles E. Leiserso 1 Ideal-Cache Model Features Two-level hierarchy. Cache