Multi-threading model

Transcription

1 Multi-threading model High level model of thread processes using spawn and sync. Does not consider the underlying hardware. Algorithm Algorithm-A begin { } spawn Algorithm-B do Algorithm-B in parallel with this code { other stuff } sync wait here for all previous spawned parallel computations to complete { } end

2 Multi-threading model Many languages (e.g. Java) support the production of separately runnable processes called threads. Each thread looks like it is running on its own and the operating system shares time and processors between the threads. In the multi-threading model, the exact parallel implementation is left to the operating system

3 Multi-threading Model We look at some examples. Fibonacci Complexity measures See CLRS: Cormen, Lierseson, Rivest and Stein, Introduction to Algorithms (3rd edition). Chapter 27, Multithreaded Algorithms titles/sample/ chap27.pdf

4 Multi-treading Fibonacci Reminder: Recursive Fibonacci Algorithm FIB(n) 1: if n 1 return n 2: else 3: return FIB(n 1) + FIB(n 2)

5 Multi-treading Fibonacci Reminder: Recursive Fibonacci Algorithm FIB(n) 1: if n 1 return n 2: else 3: return FIB(n 1) + FIB(n 2) Parallel version Algorithm Par-FIB(n) 1: if n 1 return n 2: else 3: x = spawn Par-FIB(n 2) 4: y = Par-FIB(n 1) 5: sync 6: return x + y

6 Recursive Fibonacci. The recursion tree for FIB1(6) Figure : From CLRS Introduction to Algorithms Chapter 27 (downloaded)

7 Multi-threading Fibonacci

8 Complexity measures for multi-threading DAG: directed acyclic graph. Vertices are the circles for spawn, sync or procedure call. For a problem of size n: Span S or T (n). Number of vertices on the longest directed path from start to finish in the computation DAG. (The critical path). The run time if each vertex of the DAG has its own processor. Work W or T 1 (n). Total time to execute the entire computation on one processor. Defined as the number of vertices in the computation DAG T p (n). Total time to execute entire computation with p processors Speed up = T 1 /T p. How much faster it is Parallelism = T 1 /T. The maximum possible speed up

9 Example 1: Fibonacci Lets look at the answer first.(for details see pages of CLRS.) T 1 (n) = Θ(φ n ) where φ 1.62 (see page 776) T (n) = Θ(n). The critical path is proportional to Fib(n) Parallelism = T 1 /T = Θ(φ n /n). Let t = T 1 (n) (1.62) n be the (sequential) time. Then log t n log 1.62 = Θ(n). Parallelism = T 1 /T = Θ(t/ log t). Almost linear speed up relative to our chosen sequential algorithm.

10 Span and work

11 Back to Fib. FIB1(n) is exponential For Par-Fib(n) we have T (n) = max(t (n 1) + T (n 2)) + Θ(1) = Θ(n) The value (1.62) n is tricky. We can show T 1 (n) = Ω( 2 n ) T 1 (n) = T 1 (n 1) + T 1 (n 2) + Θ(1) 2T 1 (n 2) 2 2 T 1 (n 4) 2 3 T 1 (n 6) 2 n/2 T 1 (0) = 2 n Θ(1) where so T 1 (n) (1.4) n, an exponential run time

12 Example 2. Add up numbers S(n) = i.e. S(n) = n

13 Example 2. Add up numbers S(n) = i.e. S(n) = n Algorithm SUM1(n) 1: if n = 0 return 0 2: SUM1= 0 3: for i = 1,..., n do SUM1 = SUM : return SUM1

14 Example 2. Add up numbers S(n) = i.e. S(n) = n Algorithm SUM1(n) 1: if n = 0 return 0 2: SUM1= 0 3: for i = 1,..., n do SUM1 = SUM : return SUM1 How to make SUM look parallel? Recursive version!

15 Example 2. Add up numbers S(n) = i.e. S(n) = n Algorithm SUM1(n) 1: if n = 0 return 0 2: SUM1= 0 3: for i = 1,..., n do SUM1 = SUM : return SUM1 How to make SUM look parallel? Recursive version! Algorithm SUM(n) 1: if n = 1 return 1 2: else 3: return SUM(n/2) + SUM(n/2) Add up the first half and then the second half. Not very practical? But it has a good parallel counterpart

16 Example 2. Add up numbers Sequential recursion. Assume n is a power of 2, i.e.n = 2 m Algorithm SUM(n) 1: if n = 1 return 1 2: else 3: return SUM(n/2) + SUM(n/2)

17 Example 2. Add up numbers Sequential recursion. Assume n is a power of 2, i.e.n = 2 m Algorithm SUM(n) 1: if n = 1 return 1 2: else 3: return SUM(n/2) + SUM(n/2) Parallel version Algorithm Par-SUM(n) 1: if n = 1 return 1 2: else 3: x = spawn Par-SUM(n/2) 4: y = Par-SUM(n/2) 5: sync 6: return x + y

18 Example 2. Complexity comparison T 1 (n) = Θ(n) T (n) = m = log 2 n. Why? Ans: If n = 2 m then m = log 2 n T (n) = Θ(1) + max(t (n/2), T (n/2)) = Θ(1) + T (n/2) = (m 1)Θ(1) + T (n/2 m ) = mθ(1) Parellism= T 1 (n)/t (n) = Θ(n/ log 2 n) almost linear speed up compared to our initial algorithm

19 Example 3: Add up squares S(n) = n 2 = n 2 + (n 1) Algorithm SQUARE(n) 1: if n = 1 return 1 2: x =SQUARE(n 1) 3: y = n n 4: return x + y

20 Example 3: Add up squares S(n) = n 2 = n 2 + (n 1) Algorithm SQUARE(n) 1: if n = 1 return 1 2: x =SQUARE(n 1) 3: y = n n 4: return x + y Simple parallel version. Algorithm Par-SQUARE(n) 1: if n = 1 return 1 2: x = spawn Par-SQUARE(n 1) 3: y = n n 4: sync 5: return x + y

21 Example 3: Computation DAG

22 Example 3. Complexity comparison T 1 (n) = Θ(n) T (n) = Θ(1) + max(1, T (n 1)) = Θ(n) Pararellism= T 1 (n)/t (n) = Θ(1) No speed up over sequential algorithm. Bad parallel implementation

23 The bounds on speed up for p processors Speed up = T 1 /T p. In reality: How much faster does the program run with p processors? What are the bounds on T p (n)? Crude lower bound: T p T 1 /p. Why? Difficult to divide work perfectly between the p processors. i.e. pt p T 1 If p is very large this lower bound is inaccurate. Why? We need more accurate bounds

24 Greedy scheduling I A scheduler is greedy if it immediately allocates any free processor to an available tasks The greedy scheduling principle says that if a computation is run on p processors using a greedy scheduler then the total time T p is bounded by T p W p + S The span S measures the unavoidably sequential part of the algorithm

25 Greedy scheduling II The lower bound is ( ) W T p max p, S W /p allocates work equally to processors so they all finish at the same time, S is the span Thus ( ) W max p, S T p W p + S This means that if we increase the number of processors p so that W /p S we are wasting resources. The algorithm still takes time at least S

26 This material (and much more) is covered in Cormen, Lierseson, Rivest and Stein, Introduction to Algorithms (3rd edition) Chapter 27, Multithreaded Algorithms, downloadable from titles/sample/ chap27.pdf See also the free book at: (Sections 3.3.2, 3.4)