Toward Understanding Heterogeneity in Computing

Transcription

1 Toward Understandng Heterogenety n Computng Arnold L. Rosenberg Ron C. Chang Department of Electrcal and Computer Engneerng Colorado State Unversty Fort Collns, CO, USA {rsnbrg, ron.chang@colostate.edu}

2 Motvaton Goal to ncrease our understandng of heterogenety n computng platforms 2

3 Motvaton Goal to ncrease our understandng of heterogenety n computng platforms Heterogeneous computng platforms dfferent computng speeds 3

4 Motvaton Goal to ncrease our understandng of heterogenety n computng platforms Heterogeneous computng platforms dfferent computng speeds archtecturally balanced 4

5 Understandng Heterogenety Suppose we have n+ computers: the server C a cluster C comprsng n computers, C,, C n Heterogenety profle of C C can complete one unt of work n tme <,..., n >... 2 n 5

6 The Cluster-Explotaton Problem (CEP) C must complete as many unts of work as possble on cluster C wthn a gven lfespan of L tme unts 6

7 The Cluster-Explotaton Problem (CEP) C must complete as many unts of work as possble on cluster C wthn a gven lfespan of L tme unts A worksharng protocol a schedule that solves the CEP 7

8 Archtectural Parameters Fxed communcaton cost setup tme σ latency λ neglgble over a long lfespan 8

9 Archtectural Parameters and Sample Values Common parameters: transmsson rate τ (e.g. μsec. / work unt) output-to-nput length rato δ (= ) For computer, packagng rate π (e.g. μsec. / work unt) unpackagng rate π (e.g. μsec. / work unt) workload (work unts) w 9

10 Worksharng Protocols w C ( π + τ w ) C C n

11 Worksharng Protocols ( + π ) w C C w n ( π + τ)w n C n

12 Worksharng Protocols δw C ( π + τ δw ) C ( + π ) n w n C n 2

13 Worksharng Protocols C C δw n ( π + τ) δ n w n C n 3

14 The FIFO Protocol C sends work to C ( sends work to C 2 π + τ) w ( τ) w2 sends work to C 3 π + ( π + τ) w3 C wats processes ( + π ) w results ( π + τ δw ) C 2 wats processes ( + π ) w 2 2 results ( π + τ δw 2 ) 2 C 3 wats processes ( + π ) w 3 3 results ( π + τ δw 3 ) 3 (NOT TO SCALE) 4

15 The FIFO Protocol s Optmal Theorem [Adler-Gong-Rosenberg] Over any suffcently long lfespan L, for any heterogeneous cluster C no matter what ts heterogenety profle: FIFO worksharng protocols provde optmal solutons to the cluster-explotaton problem C s equally productve under every FIFO protocol,.e., under all startup orderngs 5

16 Let X = n = ( π The Work-Producton of FIFO + τ) + (+ π + πδ) π + τ τδ = ( π + τ) + (+ π + πδ j j ) 6

17 Let X = n = ( π The Work-Producton of FIFO + τ) + (+ π + πδ) π + τ τδ = ( π + τ) + (+ π + πδ j j ) Then, W = τδ + X L 7

18 8 The Work-Producton of FIFO = = = + + = + = n j j j B A B B A X, and B A let smplfy, To τδ πδ π τ π = = = n j j X ) ( ) ( ) ( ) ( Let πδ π τ π τδ τ π πδ π τ π

19 On Comparng Heterogenety Profles For any cluster C wth heterogenety profle P =,..., n 9

20 On Comparng Heterogenety Profles For any cluster C wth heterogenety profle P =,..., n C s homogeneous-equvalent computng rate (HECR) s where P c = ( ) max =,..., { ( ) X( P ) X( P) } 2

21 Heterogenety Profles Profle : when n = 8, n + =, whch spreads evenly n a range n 8 7 6,,,..., Number of Computers HECR Recall: faster cluster has smaller HECR value 2

22 Heterogenety Profles Profle 2 : when n = 8, =,,,..., Number of Computers HECR

23 Avg. Speed vs. Std-Dev of Speed 8 computers HECR Avg. Speed =.75 Avg. Speed =.5 Avg. Speed =.25 Std-Dev=.2 Std-Dev=. Std-Dev=.5 Randomly generate profles for each combnaton 23

24 Avg. Speed vs. Std-Dev of Speed 8 computers Std-Dev HECR Avg. Speed The probablty that these two groups have the same mean s 2 24

25 Avg. Speed vs. Std-Dev of Speed 8 computers Std-Dev HECR Avg. Speed Trals wth 6, 32 computers show smlar pattern 25

26 Speedng Up Clusters Optmally under FIFO Protocols Whch one computer should you speed up, f you can speed up only one? 26

27 Speedng Up Clusters Optmally under FIFO Protocols Whch one computer should you speed up, f you can speed up only one? We study two varants of ths queston 27

28 Speedng Up Clusters Optmally under FIFO Protocols For convenence, - let cluster have heterogenety profle where C let and j > be two computer ndces n P =<,..., n >, 28

29 29 Fxed and Proportonal Speed-up Fxed-speedup scenaro by a fxed amount n j j j j n j j j P P φ φ,...,,,,...,,,,...,,...,,,,...,,,,..., ) ( ) ( = = n φ <

30 3 Fxed and Proportonal Speed-up Fxed-speedup scenaro (by a fxed amount ) n j j j j n j j j P P φ φ,...,,,,...,,,,...,,...,,,,...,,,,..., ) ( ) ( = = Proportonal-speedup scenaro by a relatve amount n j j j j n j j j P P ψ ψ,...,,,,...,,,,...,,...,,,,...,,,,..., ] [ ] [ = = n φ < < ψ

31 Proposton for Fxed-Speedup Under the fxed-speedup scenaro, the most advantageous sngle computer to speed up s C s fastest computer 3

32 Terms for followng fgures Recall: work producton W = τδ + X L Work rato the rato of work producton after speedup to work producton before speedup Speedup computer the sngle computer that s sped up 32

33 Fxed-Speedup Scenaro Work rato <, /2, /3, /4> </2, /4, /6, /8> speedup computer φ = /6 33

34 Proposton for Proportonal-Speedup (Recall : A = π + τ, B = + π + πδ, and > j ) If If ψ > speedng up (faster) s better ψ < j j Aτδ Aτδ 2 / B C j 2 / B speedng up (slower) s better C 34

35 If ψ Proposton for Proportonal-Speedup (Recall : A = π + τ, B = + π + πδ, and > j ) speedng up C j (faster) s better 2 5 If ψ < Aτδ / B =. j j > Aτδ / B 2 =. speedng up (slower) s better C 5 Parameter A B wth coarse ( sec / task) tasks Rate μ second / work unt. second / work unt 35

36 If ψ Proposton for Proportonal-Speedup (Recall : A = π + τ, B = + π + πδ, and > j ) speedng up C j (faster) s better 2 5 If ψ < Aτδ / B =. j j > Aτδ / B 2 =. speedng up (slower) s better C 5 That s, t s more advantageous to speed up the faster one unless ether both computers are already very fast or the speedup factor s very large. 36

37 Proportonal-Speedup n Acton Round speedup computer 37

38 Proportonal-Speedup n Acton Round speedup computer 38

39 Proportonal-Speedup n Acton Round speedup computer 39

40 Proportonal-Speedup n Acton Round speedup computer 4

41 Proportonal-Speedup n Acton Round speedup computer 4

42 Proportonal-Speedup n Acton Round speedup computer 42

43 Proportonal-Speedup n Acton Round speedup computer 43

44 Proportonal-Speedup n Acton Round speedup computer 44

45 Proportonal-Speedup n Acton Round speedup computer 45

46 Proportonal-Speedup n Acton Round speedup computer 46

47 Proportonal-Speedup n Acton Round speedup computer 47

48 Proportonal-Speedup n Acton Round speedup computer 48

49 Proportonal-Speedup n Acton Round speedup computer 49

50 Proportonal-Speedup n Acton Round speedup computer 5

51 Proportonal-Speedup n Acton Round speedup computer 5

52 Proportonal-Speedup n Acton Round speedup computer 52

53 Proportonal-Speedup n Acton When all computers are very fast It s more advantageous to speed up the slower one 53

54 Proportonal-Speedup n Acton.8 Round speedup computer 54

55 Proportonal-Speedup n Acton.8 Round speedup computer 55

56 Proportonal-Speedup n Acton.8 Round speedup computer 56

57 Proportonal-Speedup n Acton.8 Round speedup computer 57

58 Proportonal-Speedup n Acton.8 Round speedup computer 58

59 Summary Two ways to measure computng power the X functon the HECR value 59

60 Summary Two ways to measure computng power the X functon the HECR value Standard devaton nfluences work producton 6

61 Summary Two ways to measure computng power the X functon the HECR value Standard devaton nfluences work producton Speedng up a fast computer n a cluster s almost always more advantageous than speedng up a slower one 6

62 Thank you Questons?

63 HECR values Number of Computers Profle Profle Profle : = n n + Profle 2 : = Recall: faster cluster has smaller HECR value 63

64 Avg. Speed vs. Std-Dev of Speed 6 computers Std-Dev HECR Avg. Speed

65 Avg. Speed vs. Std-Dev of Speed 32 computers Std-Dev HECR Avg. Speed