Performance Modeling of Parallel Systems. A.J.C. van Gemund Delft University of Technology Delft, The Netherlands

Transcription

1 N 1 2 N par P β τ seq η P V ω T T -1 N 2 N 1 P... P Performance Modeling of Parallel Systems A.J.C. van Gemund Delft University of Technology Delft, The Netherlands

2 in4078, sheet # 1 Outline Part I Part II Part III Introduction: What is performance modeling about and what is so special about parallelism Performance Simulation: How to derive (performance) simulation models from parallel S/W and H/W Alternative Techniques: How to derive analytical performance models and what are the trade-offs involved

3 in4078, sheet # 2 Part I: Introduction What is performance modeling about and what is so special about parallelism performance modeling running example design space exploration modeling aspects synchronization model representation formalisms

4 in4078, sheet # 3 The Design Loop Performance feedback is important in system design problem synthesis program (π 1,π 2,...) + machine (µ 1,µ 2,...) T analysis

5 Sequential computation: y = a 0 + a 1 x + a 2 x 2 + a 3 x 3 Analytical performance model: T = 5fi mul + 3fi add in4078, sheet # 4 Performance Modeling Predict and diagnose the effect of system parameters N P.M. T = f(n,p) f(n,1) T N 2 N P... P Example:

6 in4078, sheet # 5 Challenges The result of performance modeling is f(ß 1 ;ß 2 ;:::;μ 1 ;μ 2 ;:::) Ideally, f has analytic and explicit form, is deterministic, has low solution complexity Often, however, f may represent simulation code or a system of equations, is stochastic, has exponential solution complexity Challenge is to select performance modeling method that optimally suits the application

7 in4078, sheet # 6 Our Running Example Scale sparse vector x by some factor alpha ( DSCAL ): for (i = 0; i < N; i++) if (x[i]!= 0) x[i] = alpha * x[i]; High-performance machine: 0 1 P-1 processors τ f memories τ m 0 1 M -1

8 in4078, sheet # 7 System Parameters Program: x (the input data set) N (vector length) Machine: P (# floating-point proc) fi f (average FLOP time) M (# memory banks) fi m (memory access time)

9 b = d N P in4078, sheet # 8 Parallelization Exploit data parallelism by (static) block partitioning of x: p: i: 0 l(1) u(1) 17 b e ; l(p) = pb ; u(p) = min(l + b 1;N) SPMD (Single Program Multiple Data) implementation: each processor executes same program, yet parameterized by the processor identifier (index p = 0;:::;P 1).

10 in4078, sheet # 9 Parallel Program for (i = l(p); i <= u(p); i++) if (x[i]!= 0) x[i] = alpha * x[i]; Why this example program? highly (data) parallelizable non-deterministic (sparse) work load potential work load imbalance simple, yet representative

11 in4078, sheet # 10 Parallel Machine 0 1 P-1 processors τ f memories τ m 0 1 M -1 Why this example machine? shared memory resources potential bottleneck instruction delays/resource contention non-deterministic simple, yet representative

12 in4078, sheet # 11 Design Space Exploration Design space exploration: select basic program/machine pair (incl. partitioning choice) for each pair derive f(n; P; : : :) (modeling step) explore parameter space (analysis step) by evaluating f(n; P; : : :) yielding T optimize parameters or modify program/machine pair Performance modeling essentially addresses modeling phase (e.g. deriving a queuing model) analysis phase (e.g. solving the queuing model)

13 in4078, sheet # 12 Modeling Aspects Modeling sequential computation is difficult enough due to work load non-determinism non-deterministic instruction delays non-deterministic branching, loop bounds Parallelism, however, introduces even more problems due to non-deterministic synchronization delay

14 fi f = Nfi f in4078, sheet # 13 Sequential Example 1 for (i = 0; i < N; i++) do a flop; 1 X N T = i=0 fi f ;N and therefore T deterministic straightforward modeling (no complex techniques needed)

15 ( x[i] = 0; fim m + fi f ; x[i] 6= 0: 2fi in4078, sheet # 14 Sequential Example 2 for (i = 0; i < N; i++) if (x[i]!= 0) x[i] = alpha * x[i]; 1 X N T = T i ; T i = i=0 fi f and fi m can be non-deterministic non-deterministic (data dependent) branching

16 in4078, sheet # 15 Parallel Example for (i = l(p); i <= u(p); i++) if (x[i]!= 0) x[i] = alpha * x[i]; Additional synchronization delays due to: load imbalance (slowest process finishes program) resource contention (memory access queuing)

17 in4078, sheet # 16 Task Graph Simple representation of parallelism and precedence relations computational task precedence constraint

18 in4078, sheet # 17 Synchronization Types condition synchronization (CS, static) mutual exclusion (ME, dynamic) or (ME) 7 8 T CS between 4 and 5 ME between 4 and T

19 in4078, sheet # 18 Synchronization Examples CS (or task synchronization ): data dependency, load imbalance, message-passing ME (or queuing ): critical sections (S/W), resource contention (sharing channels, memories, disks)

20 in4078, sheet # 19 Modeling Challenges in Parallel Computing basic instruction delays (cf. sequential computing) loops and branches (cf. sequential computing) CS delays (parallel computing) ME delays (parallel computing) Choice of modeling formalism is critical success factor

21 in4078, sheet # 20 Performance Modeling Approaches task graph model program (π 1,π 2,...) simulation model machine (µ 1,µ 2,...) queueing net model T = f (π,π,...,µ,µ,...) Petri net model modeling analysis

22 in4078, sheet # 21 Summary running example, modeling aspects parallelism means synchronization synchronization introduces additional complexity choice of modeling formalism important we shall choose simulation language as vehicle in Part II

23 in4078, sheet # 22 Part II: Performance Simulation How to derive performance simulation models from parallel S/W and H/W simulation as modeling vehicle example modeling language running example performance simulation simulation paradigms modeling of resources

24 in4078, sheet # 23 Simulation Design Simulation Model program (π 1,π 2,...) modeling machine (µ 1,µ 2,...) simulation T modeling pro: no a priori loss of modeling accuracy pro: timings, utilizations, throughputs, statistics con: computation-intensive procedure

25 in4078, sheet # 24 Simulation Language simulation language ß parallel programming language + notion of time in order to simulate time delays discrete event simulation languages: SIMULA-DEMOS, CSIM17,... we choose PAMELA (PerformAnce ModEling LAnguage) no syntax, didactical, analytical motivations process-algebraic approach

26 in4078, sheet # 25 process equations: Language Constructs (1) L = < process-expr-containing-task(i) > = < process-expr-containing-i-refs > task(i) time delay: L = delay(fi)! T = fi note: delay(100) will execute in e.g. 1 μs real time, simulated time delay (or virtual time), however 100 is data computations do not affect virtual time: = matadd(a; B; N) ; delay(n 2 fi f )! T = N 2 fi f L execution time delay may be considerable, however

27 in4078, sheet # 26 Language Constructs (2) sequential composition: task1 ; task2 (i = 1;N) task(i) task(1) ; :::; task(n) seq parallel composition: task1 k task2 (i = 1;N) task(i) task(1) k :::k task(n) par Note: implicit CS (barrier at exit) L = delay(2) ; fdelay(5) k delay(10)g! T = 12

28 in4078, sheet # 27 Language Constructs (3) conditional composition: if ( < boolean-expr > ) < process-expr > else process-expr > < iteration: while (< boolean-expr >) < process-expr >

29 in4078, sheet # 28 Simple Example Simple model of sequentially processing the example vector seq (i = 0; 17) if (x[i] 6= 0) f x[i] = x[i] Λ alpha ; delay(1) g Simulation result: T = 8

30 in4078, sheet # 29 Language Constructs (4) Condition synchronization (outside par): channel operators: wait(c), signal(c) wait(c): block until some other process signals c signal(c): set c, unblocking processes that wait on c E.g., (see example task graph): L = delay(fi 1 ) ; ff 1 k f 2 g ; delay(fi 8 ) f 1 = delay(fi 2 ) ; delay(fi 4 ) ; signal(c 45 ) ; delay(fi 6 ) f 2 = delay(fi 3 ) ; wait(c 45 ) ; delay(fi 5 ) ; delay(fi 7 )

31 in4078, sheet # 30 Language Constructs (5) Mutual exclusion (basic operators): counting semaphore operators P(s), V(s) P(s): block until some process Vs s > 0 ; decrement s V(s): increment s, possibly unblocking processes that P(s) E.g. (see example task graph): L = delay(fi 1 ) ; ff 1 k f 2 g ; delay(fi 8 ) f 1 = delay(fi 2 ) ; P(r) ; delay(fi 4 ) ; V(r) ; delay(fi 6 ) 2 = delay(fi 3 ) ; P(r) ; delay(fi 5 ) ; V(r) ; delay(fi 7 ) f where r = 1 initially r (resource has multiplicity 1).

32 in4078, sheet # 31 Language Constructs (6) Mutual exclusion (higher-order construct): use(r;fi) P(r) ; delay(fi) ; V(r) expresses usage of resource r for (virtual) time interval fi delay(fi) use(r;fi) ; r = 1 (infinite multiplicity) E.g., (see example task graph): L = delay(fi 1 ) ; ff 1 k f 2 g ; delay(fi 8 ) f 1 = delay(fi 2 ) ; use(r;fi 4 ) ; delay(fi 6 ) f 2 = delay(fi 3 ) ; use(r;fi 5 ) ; delay(fi 7 ) Default scheduling discipline is non-preemptive First Come First Served (FCFS) with non-deterministic conflict resolution.

33 in4078, sheet # 32 Modeling Example Recall the SPMD program: for (i = l(p); i <= u(p); i++) if (x[i]!= 0) x[i] = alpha * x[i]; Assume the following program-machine interface: flop(x; y): models the scalar floating point operation x * y including local register traffic move(a): models scalar access to memory address a loading of alpha + zero branch test + loop overhead ignored (first order approximation)

34 in4078, sheet # 33 Program Model Program part of the simulation model: par (p = 0;P 1) seq (i = l(p);u(p)) f move(addr x (i)) ; if (x[i] 6= 0) f x[i] = alpha Λ x[i] ; flop(x[i];alpha) ; move(addr x (i)) g g

35 in4078, sheet # 34 Simple Machine Model Assume: flop instruction delay independent of operands M = 1 Then flop(x; y) = delay(fi f ) move(a) = use(mem; fi m ) where initially the resource mem = 1. Substitute machine model in above program model

36 in4078, sheet # 35 Simulation Result For the example vector and fi f = 4fi m (fi f ;fi m deterministic), simulation yields p = 0: 0 1 flop flop 3 4 p = 1: 5 6 flop 6 7 flop 7 8 flop 8 9 p = 2: flop flop 14 p = 3: flop T CS: load imbalance ME: considerable memory contention slightly non-deterministic due to contention

37 in4078, sheet # 36 Alternative Machine Model Let f denote exact time delay M > 1. Assume low-order interleaved memory Then flop(x; y) = delay(f(x; y)) move(a) = use(mem a mod M ;fi m ) where initially the resources mem i = 1 Memory contention typically reduced by O(M)

38 in4078, sheet # 37 Performance Simulation Characterize data only by N and d (density): par (p = 0;P 1) seq (i = l(p);u(p)) f move(addr x (i)) ; (uniform(0; 1)» d) f if ; flop move(addr x (i)) g g where e.g. flop = delay(normal(fi f ;ff f ))

39 in4078, sheet # 38 Performance Simulation Results Speedup results for performance simulation model: psim static ideal 14 speedup (S) number of processors (P) N = 1000;d= 0:1;fi f = N 1;0:1 ;fi m = 0

40 in4078, sheet # 39 Performance Simulation pro: no original data computations (and allocations) needed (considerable savings in time and space) results more general (at the expense of being stochastic) interfaces to alternative models (task graphs, queuing models, Petri nets, Markov models) con: data modeling effort (identifying parameters such as d) interpretation can be tricky d (e.g., is merely setpoint) modeling accuracy (e.g., not all ifs can be modeled as Bernoulli processes)

41 in4078, sheet # 40 Performance Simulation Example Consider (x[i] uniform i.i.d.): for (i = 0; i < N; i++) if (x[i] > x[n-1]) task; Let N 0 denote execution frequency of task performance simulation (if (unif orm(0; 1) < 0:5) :::): μ N 0 = 0:5N; ff N 0 = 0:5 p N (B N;0:5 ) simulation, however (if (x[i] > x[n 1]) :::): μ N 0 = 0:5N; ff N 0 = N= p 12 (U 0;N ) Note: this also applies to other modeling techniques

42 in4078, sheet # 41 Consider the production line: Simulation Paradigms τ 0 τ 1 τ N-2 τ N-1 material 0... N-1 mach 0 mach 1 mach M-2 mach M-1 We distinguish: material-oriented modeling: describe the process from the material s point of view machine-oriented modeling: describe the process from the machine s point of view both models are equivalent

43 in4078, sheet # 42 Material-oriented Modeling map each individual material to a process each process describes sequence of machine usage map each machine to a passive resource constrain parallelism by ME machine usage Example (no handshaking): par (i = 0;N 1) seq (m = 0;M 1) use(r m ;fi m )

44 in4078, sheet # 43 Machine-oriented Modeling map each individual machine to a process each process describes material machining sequence ME machine usage implicit through process mapping Example (handshaking): par (m = 0;M 1) seq (i = 0;N 1) f recv(c m 1 ) ; delay(fi m ) ; send(c m ) g

45 in4078, sheet # 44 Message-Passing The functions send and recv used above: send(b) = P(b:room) ; V(b:data) = V(b:room) ; P(b:data) recv(b) where b:data = initially 0. send recv cell 0 cell 1 cell 2 cell 3 cell 4 buffer.room = 3 buffer.data = 2 synchronous communication: room = 0 (in above example) asynchronous communication: room > 0

46 in4078, sheet # 45 Simulation Paradigms What to choose? machine-oriented modeling ( send=recv): pro: more detailed (more synchronous) con: not analytic material-oriented modeling ( use ): con: less detailed (less synchronous) pro: analytic (under certain restrictions) PAMELA supports material-oriented paradigm in order to compile analytic models as well

47 Resource usage (ME): P/V, use in4078, sheet # 46 Modeling Resources Elements of modeling: Time delay: delay Synchronization (CS): par (implicit), wait/signal (explicit) In a material-oriented approach each delay is a resource usage: flop = use(cpu p ;fi f ) CPU treated as resource shared by process threads

48 in4078, sheet # 47 Example Let process i = 0;:::;N 1 be mapped on CPU map(i). Then: par (i = 0;N 1) f move(addr x (i)) ; if (uniform(0; 1)» d) f flop(map(i)) ; move(addr x (i)) g g where flop(p) = use(cpu p ;N fif ;ff f ) (note: resource cpu:pool > 1 would model dynamic scheduling)

49 in4078, sheet # 48 Summary simulation modeling using PAMELA (material-oriented) modeling principles: mapping programs to processes mapping machine parts to subroutines abstract, top down approach performance simulation: abstractions similar to other performance modeling techniques simulation paradigms and modeling of resources

50 in4078, sheet # 49 Part III: Alternative Techniques How to derive analytic performance models and what are the trade-offs involved drawback (performance) simulation Markov analysis task graph analysis queuing theory Petri net theory qualitative comparison

51 in4078, sheet # 50 Simulation Drawbacks Sources of non-determinism: workload, branching, ME Example task graph with ME (fi 1 4;7 8 = 1, fi 5;6 = N 100;10 ): rel. frequency execution time (T)

52 in4078, sheet # 51 Example Running example (N = 1000;d= 0:1;P = 10;fi m = 0): rel. frequency execution time (T) Conditional control flow and workload non-determinism. Per 500 point runs for 95% confidence interval of 2%.

53 in4078, sheet # 52 Analytic Techniques pro: exact solution (e.g. mean value of T ) pro: possibly parameterized con: additional model assumptions con: often exponential complexity Choosing between simulation and analytic technique generally depends on the level of abstraction required.

54 in4078, sheet # 53 Performance Modeling Approaches task graph model program (π 1,π 2,...) simulation model machine (µ 1,µ 2,...) queueing net model T = f (π,π,...,µ,µ,...) Petri net model modeling analysis

55 in4078, sheet # 54 Markov Modeling Model assumption: task delays must have neg.-exponential distribution (pdf = e t, memoryless or Markov property) Consider example task graph with ME with fi i exponentially ( distributed = fi 1 i i ). This maps to the CT Markov chain: 1 τ ,3 τ -1 2 τ ,6 3,4 5,6 τ τ -1 3 τ -1 4 τ -1 6 τ ,7 τ -1 5 τ τ τ -1 8 τ ,5 τ -1 2 τ ,7 τ -1 5 τ ,7 τ -1 7 τ -1 4 τ -1 τ τ -1 6

56 in4078, sheet # 55 Markov Analysis Let q ij denotes the transition rate of state i to j, where q ii = X j6=i q ij For the cyclic Markov chain (by inserting dashed feedback arc) the steady-state probability ß i of each state i is given by solving the linear equations : X 8i q ji ß j = 0 j under the constraint X ß i = 1 i

57 = T :665 = (= fi 1 = fi 8 ) = 8 ß 8 n 1 ß 1 n in4078, sheet # 56 Mean Execution Time We compute execution time from the chain s cycle time: Let pdf(fi i ) = i 1 e t=i ( i = i 1 ) For our example the solution for state (1) is ß 1 = 0: The transition rate out of state (1) is. ii = fi 1 1 q The average number of transitions (1)! (2; 3) per unit time n equals = fi 1 1 ß 1 1. It follows for the mean cycle time:

58 in4078, sheet # 57 Task Graph Modeling Modeling components: node! parallelism, (stochastic) time delay arc! CS Modeling restrictions no conditional control flow! stochastic delay approximations no ME! stochastic delay approximations Purpose: to study parallelism and (static) synchronization

59 in4078, sheet # 58 Task Graph Analysis Depending on the nature of the node delays: stochastic graphs: T distribution : path analysis mean value T : Markov analysis (see earlier) deterministic graphs: path analysis Path analysis principle: critical (or longest) path analysis, traversing the graph top to bottom in breadth-first order (B begin time, E end time): 8i : B i = max E j ; E i = B i + fi i j2pred(i)

60 in4078, sheet # 59 Path Analysis (1) Consider example task graph with CS where fi i deterministic: 1 B 1 = 0 E 1 = B 1 + τ B E B 2 = E 1 B 3 = E 1 B 4 = E 2 B 5 = E 3 max E 4 B 6 = E 4 B = 7 E 5 B = 8 E 6 max E 7 E 2 = B 2 + τ 2 E 3 = B 3 + τ 3 E 4 = B 4 + τ 4 E 5 = B 5 + τ 5 E 6 = B 6 + τ 6 E 7 = B 7 + τ 7 E 8 = B 8 + τ 8 = T Similar to 1 simulation run due to determinism.

61 in4078, sheet # 60 Path Analysis (2) Stochastic graph introduces complications: order statistics (see speedup plot): pdf(b i ) = Y pdf(e j )! B i > j2pred(i) applies only if E j are independent max E j j2pred(i) convolutions: pdf(e i ) = pdf(b i ) Λ pdf(fi i ) In practice: for non-sp graphs: heuristic bounding techniques SP graphs: E j can be considered independent: pdf product and convolution analysis for exponential distributions or mean value bounding techniques

62 in4078, sheet # 61 Definition of queuing center: Queuing Nets q s a r Examples: customers served at a counter queue server (m) r service time q clients in system r response time processes served by processor processes served by memory Purpose: to model and analyze queuing delay (synchronization delay due to ME)

63 q = ρ! ρ ; ρ = μ 1 in4078, sheet # 62 Queuing Analysis Example M/M/1 queue (pdf(a) = e t, pdf(d) = μe μt ): Markov analysis (see earlier) r = (q + 1) 1 μ (memoryless service time) (Little s Law) Mean Value Analysis (MVA): q = r (ρ =traffic intensity = u = server utilization) At 90% server utilization, response time is tenfold due to variance in arrival and service times

64 in4078, sheet # 63 Effect of Variance The effect of variance on queuing delay 10 8 a exp, s exp a exp, s det a det, s det clients in system (q) traffic intensity (rho) Variance of arrival and service time intervals causes queuing

65 = Nfi f Z P in4078, sheet # 64 Queuing Network Example Typically, closed queuing networks are used. Apart from queues a number of other components exist (e.g., infinite server) Running (M = 1;fi example ;fi m f exp. distr.): Z D MVA parameters: cpus P clients memory = (1 + d)nfi m D P Steady state needed (N! 1) + inability to model CS.

66 in4078, sheet # 65 Analysis Results Speedup results for queuing network model (using MVA): MVA psim psim (tm = 0) 14 speedup (S) number of processors (P) N = 1000;d= 0:1;fi f = 1;fi m = 0:02! MVA parameters Z = 100=P ; D = 22=P.

67 in4078, sheet # 66 Petri nets Definition of (timed) Petri net: t 1 p 1 p 3 after τ 1 t 1 p 1 p 3 token place p p 4 p p 4 (timed) transition marking µ = (1,3,1,0) marking µ = (0,1,2,2) Petri net covers both CS and ME Purpose: to study all aspects of concurrency

68 in4078, sheet # 67 Example Petri net A simple model of our running example (M = 1): τ m τ f P times replicated τm Model complexity: memory Queuing net» Petri net» Task graph» Markov chain

69 in4078, sheet # 68 Timed Petri net analysis reachability analysis! tree of possible markings bounded PN with exponential transition times: reachability! tree Markov chain ergodic Markov chain! steady state probability vector solution (see earlier) from certain marking prob. compute cycle time unbounded PN extended with inhibiter arcs (zero testing) have Turing power and are not analyzable

70 in4078, sheet # 69 Comparison Qualitative comparison of typical approaches: form CS ME DC analysis time cost acc. Sim CP polynomial ++ Psim + + +/- mult. CP polynomial + TG + - +/- CP linear/quadratic +/- QN - + +/- MVA polynomial +/- PN + + +/- Markov exponential + (note: TG deterministic, QN/PN exponential distr.) DC = data computation capability (e.g. branches) CP = critical path analysis

71 in4078, sheet # 70 Recent Research New research directions: parallel simulation approximate, hybrid techniques combining task graphs and queuing networks complexity reduction in Petri net analysis approximate, symbolic program analysis (extended task graph analysis)

72 in4078, sheet # 71 Summary drawbacks (performance) simulation brief overview of modeling with Markov chains, task graphs, queuing networks, timed Petri nets qualitative comparison performance modeling techniques recent research

73 in4078, sheet # 72 References and further reading: References (1) M. Ajmone Marsan, G. Balbo and G. Conte, Performance Models of Multiprocessor Systems. MIT Press, (Petri nets, queueing theory) G.R. Andrews and F.B. Schneider, Concepts and notations for concurrent programming, Computing Surveys, vol. 266, no. 24, 1983, pp (process synchronization) G.M. Birtwistle, Discrete Event Modelling on Simula. MacMillan, (simulation) W. Kreutzer, System simulation, programming styles and languages. Addison-Wesley, (simulation)

74 in4078, sheet # 73 References (2) References and further reading: R. Jain, The Art of Computer Systems Performance Analysis. Wiley, (statistics, simulation, queuing theory) E.D. Lazowska, J. Zahorjan, G.S. Graham and K.C. Sevcik, Quantitative System Performance. Prentice-Hall, (queueing theory) J.L. Peterson, Petri Net Theory and the Modeling of Systems. Prentice-Hall, (Petri nets, process synchronization)