Institute of Computer Science Chair of Communication Networks Prof. Dr.-Ing. P. Tran-Gia A Hysteresis-Based Energy-Saving Mechanism for Data Centers Christian Schwartz, Rastin Pries, Phuoc Tran-Gia www3.informatik.uni-wuerzburg.de
Motivation Datacenter load varies over time Throughput 00:00 06:00 12:00 18:00 00:00 Time of Day 2 2
Motivation Datacenter load varies over time Throughput Power Consumption 00:00 06:00 12:00 18:00 00:00 Time of Day 3 3
Motivation Datacenter load varies over time Throughput Power Consumption Power Consumption 00:00 06:00 12:00 18:00 00:00 Time of Day Possible solution: Disable subset of servers Decreases power consumption Increases per job processing time Trade-Off required 4 4
Overview Motivation Model Architecture Queuing Model Numerical Algorithm Numerical Evaluation Parameters System behavior Optimization Conclusion 5 5
Architecture 6 6
Architecture 7 7
Architecture 8 8
Energy-Efficient Datacenter Model M 9 9
Energy-Efficient Datacenter Model M 10 10
Energy-Efficient Datacenter Model M 1 M μ 2 M μ n M μ n Servers 11 11
Energy-Efficient Datacenter Model M 1 M μ 2 M μ 2 n M μ n Servers 12 12
Energy-Efficient Datacenter Model M 1 M μ 2 M μ 2 n M μ 1 M μ 2 M μ m M μ n Servers m Servers 13 13
Energy-Efficient Datacenter Model M 1 M μ 2 M μ 2 1 n M μ 1 M μ 2 M μ m M μ n Servers m Servers 14 14
Energy-Efficient Datacenter Model M 1 M μ 2 M μ 2 1 n M μ n Servers 15 15
State Space 0, 0 1, 0 θ 1 1, 0 θ 1, 0 n, 0 n + θ 2 1, 0 n + θ 2, 0 µ 2µ (θ 1 1)µ θ 1 µ (θ 1 + 1)µ θ 1 µ θ 1, 1 θ 1 +1, 1 n + θ 2, 1 n + θ 2 +1, 1 n + m, 1 (θ 1 + 1)µ (θ 1 + 2)µ (n + θ 2 )µ (n + θ 1 + 1)µ (n + θ 2 + 2)µ (n + m)µ (n + m)µ i jobs in the system j = 0 reserved servers deactivated 1 reserved servers activated 16 16
State Space Queue length exceeds : Activate reserve servers 17 17
State Space Queue length exceeds : Activate reserve servers Number of busy servers goes below : Deactivate reserve servers 18 18
x(θ 1, 1) = x(0, 0) Closed Form Solution n+θ 2 +1 µ (θ1 1)! (1 µ θ 2 )θ 1 1 µ + a θ 2 n θ 1 k=0 k µ (n k 1)! 1+ µ θ 1 n θ 2 n!(n θ1 + 1)! 0, 0 1, 0 θ 1 1, 0 θ 1, 0 n, 0 n + θ 2 1, 0 n + θ 2, 0 µ 2µ (θ 1 1)µ θ 1 µ (θ 1 + 1)µ θ 1 µ θ 1, 1 θ 1 +1, 1 n + θ 2, 1 n + θ 2 +1, 1 n + m, 1 Closed form solutions are available Problems: Numerical instability (θ 1 + 1)µ (θ 1 + 2)µ (n + θ 2 )µ Efficiency (Recomputation of common terms) (n + θ 1 + 1)µ (n + θ 2 + 2)µ (n + m)µ (n + m)µ 19 19
Numeric Algorithm Using Macro States x(i, 0) = iµ x(i 1, 0) i [1, θ 1) 0, 0 1, 0 θ 1 1, 0 θ 1, 0 n, 0 n + θ 2 1, 0 n + θ 2, 0 µ 2µ (θ 1 1)µ θ 1 µ (θ 1 + 1)µ θ 1 µ θ 1, 1 θ 1 +1, 1 n + θ 2, 1 n + θ 2 +1, 1 n + m, 1 (θ 1 + 1)µ (θ 1 + 2)µ Recursive calculation of probabilities (n + θ 2 )µ (n + θ 1 + 1)µ (n + θ 2 + 2)µ (n + m)µ (n + m)µ 20 20
Numeric Algorithm Using Macro States x(i, 0) = iµ x(i 1, 0) θ 1 i x(θ 1, 1) i [θ 1,n] 0, 0 1, 0 θ 1 1, 0 θ 1, 0 n, 0 n + θ 2 1, 0 n + θ 2, 0 µ 2µ (θ 1 1)µ θ 1 µ (θ 1 + 1)µ θ 1 µ θ 1, 1 θ 1 +1, 1 n + θ 2, 1 n + θ 2 +1, 1 n + m, 1 (θ 1 + 1)µ (θ 1 + 2)µ Recursive calculation of probabilities (n + θ 2 )µ (n + θ 1 + 1)µ (n + θ 2 + 2)µ (n + m)µ (n + m)µ 21 21
Numeric Algorithm Using Macro States u i = iµ u i 1 v i = iµ v i 1 + θ 1 i x(i, 0) = x(θ 1 1, 0)u i x(θ 1, 1)v i i [θ 1,n] 0, 0 1, 0 θ 1 1, 0 θ 1, 0 n, 0 n + θ 2 1, 0 n + θ 2, 0 µ 2µ (θ 1 1)µ θ 1 µ (θ 1 + 1)µ θ 1 µ θ 1, 1 θ 1 +1, 1 n + θ 2, 1 n + θ 2 +1, 1 n + m, 1 (θ 1 + 1)µ (θ 1 + 2)µ Recursive calculation of probabilities (n + θ 2 )µ For terms depending on unknown probabilities Store coefficients of linear combination (n + θ 1 + 1)µ (n + θ 2 + 2)µ (n + m)µ (n + m)µ 22 22
Numeric Algorithm Using Macro States u i = u i 1 v i = v i 1 + θ 1 n x(i, 0) = x(θ 1 1, 0)u i x(θ 1, 1)v 1 i (n, n + θ 2 ] 0, 0 1, 0 θ 1 1, 0 θ 1, 0 n, 0 n + θ 2 1, 0 n + θ 2, 0 µ 2µ (θ 1 1)µ θ 1 µ (θ 1 + 1)µ θ 1 µ θ 1, 1 θ 1 +1, 1 n + θ 2, 1 n + θ 2 +1, 1 n + m, 1 (θ 1 + 1)µ (θ 1 + 2)µ Recursive calculation of probabilities (n + θ 2 )µ For terms depending on unknown probabilities Store coefficients of linear combination (n + θ 1 + 1)µ (n + θ 2 + 2)µ (n + m)µ (n + m)µ 23 23
Numeric Algorithm Using Macro States x(θ 1, 1) = θ 1 µ x(n + θ 2, 0) 0, 0 1, 0 θ 1 1, 0 θ 1, 0 n, 0 n + θ 2 1, 0 n + θ 2, 0 µ 2µ (θ 1 1)µ θ 1 µ (θ 1 + 1)µ θ 1 µ θ 1, 1 θ 1 +1, 1 n + θ 2, 1 n + θ 2 +1, 1 n + m, 1 (θ 1 + 1)µ (θ 1 + 2)µ Recursive calculation of probabilities (n + θ 2 )µ For terms depending on unknown probabilities Store coefficients of linear combination (n + θ 1 + 1)µ (n + θ 2 + 2)µ (n + m)µ (n + m)µ 24 24
Numeric Algorithm Using Macro States x(θ 1, 1) = θ 1 µ u n+θ 2 (1 + θ 1 µ )v n+θ 2 x(θ 1 1, 0) 0, 0 1, 0 θ 1 1, 0 θ 1, 0 n, 0 n + θ 2 1, 0 n + θ 2, 0 µ 2µ (θ 1 1)µ θ 1 µ (θ 1 + 1)µ θ 1 µ θ 1, 1 θ 1 +1, 1 n + θ 2, 1 n + θ 2 +1, 1 n + m, 1 (θ 1 + 1)µ (θ 1 + 2)µ Recursive calculation of probabilities (n + θ 2 )µ For terms depending on unknown probabilities Store coefficients of linear combination (n + θ 1 + 1)µ (n + θ 2 + 2)µ (n + m)µ (n + m)µ 25 25
Numeric Algorithm Using Macro States Recalculate x(i, 0) depending only on x(0, 0) i [θ 1,n+ θ 2 ] 0, 0 1, 0 θ 1 1, 0 θ 1, 0 n, 0 n + θ 2 1, 0 n + θ 2, 0 µ 2µ (θ 1 1)µ θ 1 µ (θ 1 + 1)µ θ 1 µ θ 1, 1 θ 1 +1, 1 n + θ 2, 1 n + θ 2 +1, 1 n + m, 1 (θ 1 + 1)µ (θ 1 + 2)µ Recursive calculation of probabilities (n + θ 2 )µ (n + θ 1 + 1)µ (n + θ 2 + 2)µ (n + m)µ For terms depending on unknown probabilities Store coefficients of linear combination and solve for missing probability (n + m)µ 26 26
Numeric Algorithm Using Macro States x(i, 1) = iµ x(x(i 1, 1) + x(n + θ 2, 0)) i (θ 1,n+ θ 2 ] 0, 0 1, 0 θ 1 1, 0 θ 1, 0 n, 0 n + θ 2 1, 0 n + θ 2, 0 µ 2µ (θ 1 1)µ θ 1 µ (θ 1 + 1)µ θ 1 µ θ 1, 1 θ 1 +1, 1 n + θ 2, 1 n + θ 2 +1, 1 n + m, 1 (θ 1 + 1)µ (θ 1 + 2)µ Recursive calculation of probabilities (n + θ 2 )µ (n + θ 1 + 1)µ (n + θ 2 + 2)µ (n + m)µ For terms depending on unknown probabilities Store coefficients of linear combination and solve for missing probability (n + m)µ 27 27
Numeric Algorithm Using Macro States x(i, 1) = iµ x(i 1, 1) i (n + θ 2,n+ m] 0, 0 1, 0 θ 1 1, 0 θ 1, 0 n, 0 n + θ 2 1, 0 n + θ 2, 0 µ 2µ (θ 1 1)µ θ 1 µ (θ 1 + 1)µ θ 1 µ θ 1, 1 θ 1 +1, 1 n + θ 2, 1 n + θ 2 +1, 1 n + m, 1 (θ 1 + 1)µ (θ 1 + 2)µ Recursive calculation of probabilities (n + θ 2 )µ (n + θ 1 + 1)µ (n + θ 2 + 2)µ (n + m)µ For terms depending on unknown probabilities Store coefficients of linear combination and solve for missing probability (n + m)µ 28 28
Numeric Algorithm Using Macro States x(i >n+ m, 1) = x(n + m, 1) 2 (n+m)µ 1 (n+m)µ 0, 0 1, 0 θ 1 1, 0 θ 1, 0 n, 0 n + θ 2 1, 0 n + θ 2, 0 µ 2µ (θ 1 1)µ θ 1 µ (θ 1 + 1)µ θ 1 µ θ 1, 1 θ 1 +1, 1 n + θ 2, 1 n + θ 2 +1, 1 n + m, 1 (θ 1 + 1)µ (θ 1 + 2)µ Recursive calculation of probabilities (n + θ 2 )µ For terms depending on unknown probabilities Store coefficients of linear combination and solve for missing probability Calculate waiting probability using geometric tail (n + θ 1 + 1)µ (n + θ 2 + 2)µ (n + m)µ (n + m)µ 29 29
Numeric Algorithm Using Macro States n+θ 2 x(0, 0) = ( i=1 x(i, 0) + n+m i=θ 1 x(i, 1) + x(i >n+ m, 1)) 1 0, 0 1, 0 θ 1 1, 0 θ 1, 0 n, 0 n + θ 2 1, 0 n + θ 2, 0 µ 2µ (θ 1 1)µ θ 1 µ (θ 1 + 1)µ θ 1 µ θ 1, 1 θ 1 +1, 1 n + θ 2, 1 n + θ 2 +1, 1 n + m, 1 (θ 1 + 1)µ (θ 1 + 2)µ Recursive calculation of probabilities (n + θ 2 )µ (n + θ 1 + 1)µ (n + θ 2 + 2)µ (n + m)µ For terms depending on unknown probabilities Store coefficients of linear combination and solve for missing probability Calculate waiting probability using geometric tail Calculate by normalization (n + m)µ 30 30
Overview Motivation Model Architecture Queuing Model Numerical Algorithm Numerical Evaluation Parameters System behavior Optimization Conclusion 31 31
Parameters Experiment setup for parameter study: Mean interarrival time: 10ms [1] Total number of servers: 100 Mean service time: 400 ms Energy consumption: [2] Busy 100% (122.46 Watt) Idle 58.6% Off 0.31% [1] T. Benson, A. Akella, and D. Maltz, Network Traffic Characteristics of Data Centers in the Wild in Internet Measurement Conference (IMC), Melbourne, Australia, November 2010 [2] Measured using PDUs on a Quad Core server 32 32
System Behavior 100 100 Mean Waiting Time (ms) 80 60 40 20 Power consumption if idle servers are never deactivated 80 60 40 20 Power Consumption (Percent) 0 20 40 60 80 0 Number of Baseline Servers n 33 33
System Behavior 100 100 Mean Waiting Time (ms) 80 60 40 20 Power consumption if idle servers are always deactivated 80 60 40 20 Power Consumption (Percent) 0 20 40 60 80 0 Number of Baseline Servers n 34 34
System Behavior 100 100 Mean Waiting Time (ms) 80 60 40 20 Waiting time achievable if servers are never shutdown 80 60 40 20 Power Consumption (Percent) 0 20 40 60 80 0 Number of Baseline Servers n 35 35
System Behavior Mean Waiting Time (ms) θ 1 = 3n 4 θ 2 = 3m 4 Power Consumption (Percent) Number of Baseline Servers n 36 36
System Behavior 100 100 Average Mean Waiting Time (ms) 80 60 40 20 E max E min θ 1 = n 2 θ 2 = m 2 80 60 40 20 Power Power Consumption (Percent) 0 E[W] 20 40 60 80 Number of of Baseline baseline Servers servers n n 0 37 37
System Behavior Mean Waiting Time (ms) θ 1 = n 4 θ 2 = m 4 Power Consumption (Percent) Number of Baseline Servers n 38 38
System Behavior Mean Waiting Time (ms) Power Consumption (Percent) Number of Baseline Servers n Acceptable waiting time can be achieved by selecting n=50 39 39
System Behavior Mean Waiting Time (ms) θ 1 = 30 Power Consumption (Percent) Server activation threshold θ 2 40 40
System Behavior Mean Waiting Time (ms) θ 1 = 10 Power Consumption (Percent) Server activation threshold θ 2 41 41
System Behavior Mean Waiting Time (ms) θ 1 = 48 Power Consumption (Percent) Server activation threshold θ 2 42 42
System Behavior Mean Waiting Time (ms) θ 1 = 48 Power Consumption (Percent) Server activation threshold θ 2 Selecting a high server deactivation threshold decreases power consumption. 43 43
Optimization Evaluate all parameter combinations 44 44
Optimization Evaluate all parameter combinations Filter for Pareto optima 45 45
Optimization Evaluate all parameter combinations Filter for Pareto optima Select area where acceptable Trade-Off exists 46 46
Simulation Simulation allows for a more realistic datacenter model Uses the OMNeT++ framework Arbitrary arrival processes (see Benson, et. al.) Server startup/shutdown takes time 47 47
Implementation OpenFlow Controller CPU Memory VM1 Power Consumption OpenFlow Switch VM2 VM3 DB (iscsi) 48 48
Implementation OpenFlow Controller CPU Memory VM1 Power Consumption OpenFlow Switch VM2 VM3 DB (iscsi) 49 49
Implementation OpenFlow Controller CPU Memory Power Consumption OpenFlow Switch VM1 VM2 VM3 DB (iscsi) 50 50
Implementation OpenFlow Controller CPU Memory Power Consumption OpenFlow Switch VM1 VM2 VM3 DB (iscsi) 51 51
Implementation OpenFlow Controller Wake on LAN CPU Memory Power Consumption OpenFlow Switch VM1 VM2 VM3 DB (iscsi) 52 52
Conclusion Introduced a mechanism for more energy efficient data centers Performed parameter studies Provided multi objective optimization approach 53 53
Conclusion Introduced a mechanism for more energy efficient data centers Performed parameter studies Provided multi objective optimization approach Questions? 54 54
Server (De)Activation in Multiple Stages Support the (de)activation of servers on POD scope Multiple levels of activation to manage different levels of load 55 55