Parallel Vector Algorithms David A. Padua

Size: px

Start display at page:

Download "Parallel Vector Algorithms David A. Padua"

Jordan Bates
5 years ago
Views:

1 Parallel Vector Algorithms 1 of 32

2 Itroductio Next, we study several algorithms where parallelism ca be easily expressed i terms of array operatios. We will use Fortra 90 to represet these algorithms. Simplistic timig figures will be give i some cases for array machies. I these timigs, subscript computatios ad memory access/ commuicatios costs will be igored. 2 of 32

3 8.3 Time to execute a vector operatio Let us start with the simplest possible situatio. Cosider the followig geeric vector operatio: a(1:) # b(1:) Cosider also a array machie with P arithmetic uits. The executio time is: t parallel = - t # P where t # is the time to execute oe # operatio. 3 of 32

4 Reductios i Fortra 90 A typical reductio is sum(array) which returs,as we should expect, the sum of the elemets of a iteger, real, or complex array. It returs zero if array has size zero. Others iclude: all(mask) ay(mask) cout(mask) Returs the logical value.true. if all elemets of of the logical array mask are.true. or mask has size zero, ad otherwise returs the value.false. Returs the logical value.true. if ay of the elemets of the logical array mask is.true., ad returs the value.false. if o elemets are.true. or if mask has size zero. Returs the umber of.true. values i mask. maxval(array) Returs the maximum value of the elemets of a iteger or real array. 4 of 32

5 mival(array) Returs the miimum value of the elemets of a iteger or real array. product(array) returs the product of the elemets of a iteger, real, or complex array. It returs 1 if array has size zero. All these fuctios have a optioal argumet dim if this is preset, the operatio is applied to all rak-oe sectios that spa right through dimesio dim to produce a array of rak reduced by oe ad exteds equal to the extets i the other dimesios. For example, if a is a real array of shape [4,5,6], sum(a,dim=2) is a real array of shape [4,6] ad elemet (i,j) has value sum(a(i,:,j)). The fuctios maxval, mival, product, ad sum have a third optioal argumet, mask. If this is preset, it must have the same shape as the first argumet ad the operatio is applied to the elemets correspodig to true elemets of mask; for esample, sum(a,mask=a>0) sums the positive elemets of the array a. 5 of 32

6 Two other useful Fortra 90 fuctios. 1. spread(source,dim,copies) Returs a array of rhe same type as source but with rak icreased by oe over source. Source may be a scalar or a array. Dim ad copies are iteger scalars. The result cotais max(copies,0) copies of source, ad elemet (r 1,...,r +1 ) of the result is source (s 1,...,s ) where (s 1,...,s ) is (r 1,...,r +1 ) with subscript dim omitted (or source itself if it is a scalar). Example of use: a=spread(x,dim=2,copies=)+spread(x,1,) w=sum(abs(a),dim=1) 6 of 32

7 is equivalet to: do i=1, w(i)=0 do j=1, w(i)=w(i)+abs(x(i)+x(j)) ed do ed do 2. maxloc(array[,mask]) Returs a rak-oe iteger array of size equal to the rak of array. Its value is the subscript of a elemet of maximum value. 7 of 32

8 Time to Execute a Reductio Cosider a reductio such as: r = sum(a(1:)) = a(1) + a(2) + a(3) +... a() or, i geeral r = a(1) # a(2) # a(3) #... a() A sequece of log 2 vector operatios of legth /2, /4,..., 1 suffices to compute the reductio (assumig associativity of the # operatio). I the case of a array machie, there are two cases. First, if P < /2, ad if we follow the approach preseted i our discussio of reductios i OpeMP, we have: t parallel = - 1 P t + + ( P 1)t + 8 of 32

9 If the fial reductio ca also be doe i logarithmic time usig a reductio tree approach: I this case, the executio time is: t parallel = - 1 P t + + log P t + If P >= /2, the time is: t parallel = log t + 9 of 32

10 Parallel Prefix Cosider the followig loop: A(0)=0 DO I=1,N A(I)=A(I-1)+B(I) END DO The loop seems sequetial because each iteratio eeds iformatio o the value computed i the precedig iteratio. However, we ca use a parallel prefix approach to compute the value of vector A i parallel as follows: B(1) B(2) B(3) B(4)... B(N) B(1) B(1)+B(2) B(2)+B(3) B(3)+B(4)... B(N-1)+B(N) B(1) B(1)+B(2) B(1)+B(2)+ B(1)+B(2)+... B(N-3)+B(N-2) B(3) B(3)+B(4) +B(N-1)+B(N) 10 of 32

11 A parallel program implemetig this strategy uder the assumptio that N=2 k is: A(1:N)=B(1:N) DO I=0,K-1 A(2**I+1:N)=A(2**I+1:N)+A(1:N-2**I+1) END DO For a array machie with the umber of procesig uits P>=-1: t parallel = t + log As i the case of reductio, parallel prefix ca be applied to ay associative biary operatio. 11 of 32

12 Relative Performace How much faster does a program ru whe executed i parallel? Speedup: S P =T 1 / T P (1) T 1 : Executio time of the program o a sigle (scalar) processor. T P : Executio time o a parallel machie. Parallel programs may itroduce some redudacy to achieve higher parallelism. I a sequetial program, the goal is to miimize the total umber of operatios because this umber is directly related to the executio time. I a parallel program, this relatioship is ot direct. For this reaso a more hoest formula for speedup is: Speedup: S P =T 1 / T P (2) where T 1 is the best kow serial versio of the program. The speedup i (1) is kow as the parallel speedup. Assume a multiprocessor with P processors or a array machie with P processig elemets. The speedup ca be liear i P (that is, of the form k*p for k <= 1), logarithmic (that is, of the form k * log P), or it 12 of 32

13 ca have may other forms. I a real machie the speedup is seldom a ice fuctio of the umber of processors. I some cases the speedup is superliear; that is, the speedup is greater tha p for p processors. This happes whe, for example, each processor has its ow cache memory. I this way usig several processors also icreases the size of the cache memory. Aother case whe you ca get superliar speedup is i program performig some form of search operatio. Other importat measures iclude: 1. Efficiecy: E P =T 1 /PT P where P is the umber of processors if the target machie is a multiprocessor (assumig sigle-user mode) or the umber of processig elemets i a array processor. 2. Redudacy: R P = O P /O 1 where O P is the umber of operatios i the parallel program, ad O 1 is the umber of operatios i the best kow serial versio. 13 of 32

14 Examples of Speedup ad Efficiecy Cosider a(1:) + b(1:) The speedup, efficiecy, i a array machie: t parallel = - t P + t S + P = = - t P P The value of S P is P if is a multiple of P. t E + P = P = P - t P P - 14 of 32

15 E P is 1 if is a multiple of P. Otherwise it is < 1.The speedup, R P = 1 efficiecy, ad redudacy of the parallel prefix example o a array machie with P= are: S t + = = log t log t E + = = log t log R O ( 1) + ( 2) = = = O ( log 1) + 1 log 1 15 of 32

16 Matrix-Vector Multiplicatio I mathematical otatio: A 11 A 12 A 1 A 21 A 22 A 2 A m1 A m2 A m V 1 V 2 V = A 1i V i i = 1 A 2i V i i = 1 i = 1 A mi V i I Fortra: do i=1,m R(i) = 0 do j=1, R(i) = R(i) + A(i,j) * V(j) ed do ed do 16 of 32

17 The ier loop performs a dot product (or ier product) of two vectors. It ca be represeted i Fortra 90 as follows: do i=1,m R(i)=DOT_PRODUCT(A(i,1:),V(1:)) ed do The dot product is a vector multiplicatio (of legth, i this case) followed by a reductio. I a array machie or i a multiprocessor, the time if P> is: ( m( log t + + t * )) Alteratively, by iterchagig the loop headers, the program could be writte as follows: do j=1, do i=1,m R(i) = R(i) + A(i,j) * V(j) ed do ed do 17 of 32

18 This leads to the followig sequece of vector operatios: do j=1, R(1:m)=R(1:m)+A(1:m,j)*V(j) ed do 18 of 32

19 Matrix Multiplicatio 1. Ier product method. Matrix multiplicatio is usually writte: do i=1, do j=1, do k=1, C(i,j)=C(i,j)+A(i,k)*B(k,j) ed do ed do ed do The most direct traslatio of this program ito vector form is: do i=1, do j=1, C(i,j)=DOT_PRODUCT(A(i,1:),B(1:,j)) ed do ed do The time o a array machie or multiprocessor if P > is: ( t + log + t * ) 2 19 of 32

20 2. Middle-product method (-parallelism) This is obtaied by iterchagig the headers i the origial matrix multiplicatio loop. do j=1, do k=1, do i=1, C(i,j)=C(i,j)+A(i,k)*B(k,j) ed do ed do ed do The direct traslatio of this loop ito vector form is: do j=1, do k=1, C(1:,j)=C(1:,j)+A(1:,k)*B(k,j) ed do ed do 20 of 32

21 Alteratively, the headers could have bee exchaged i a differet order to obtai the loop: do i=1, do k=1, C(i,1:)=C(i,1:)+A(i,k)*B(k,1:) ed do ed do The time i a array machie is: ( t + + t * ) 2 21 of 32

22 3. Outer-product method ( 2 -parallelism) Aother iterchage of the loop headers produce: do k=1, do i=1, do j=1, C(i,j)=C(i,j)+A(i,k)*B(k,j) ed do ed do ed do To obtai 2 parallelism, the ier two loops should take the form of a matrix operatio: do k=1, C(1:,1:)=C(1:,1:)+A(1:,k) B(k,1:) ed do Where the operator represets the outer product of two vectors. Give two vectors a ad b, their outer product is a matrix Z such that Z i,j =a i b j. Notice that the previous loop is NOT a valid Fortra or Fortra 90 loop because is ot a valid Fortra character. 22 of 32

23 The outer product matrix i the loop above has the followig form: A 1k B k1 A 1k B k2 A 1k B k3 A 2k B k1 A 2k B k2 A 2k B k3 A 3k B k1 A 3k B k2 A 3k B k3 1 2 The two directios of replicatio This matrix is the elemet-by-elemet product of the followig two matrices: A k A k A k B k B k B k which are formed by replicatig A k =A(1:,k) ad B k =B(k,1:) alog the appropriate dimesios. This replicatio ca be achieved usig the Fortra 90 SPREAD fuctio discussed above: 23 of 32

24 spread(a(1:,k),dim=2,copies=)*spread(b(k,1:),dim=1,copies= )) The resultig loop is therefore: do k=1, C=C+SPREAD(A(1:,k),2,)*SPREAD(B(k,1:N,1,) ed do I a array machie with P> 2, the time would be: ( 2t copy log + t * + t + log ) where t copy is the time to copy a vector. The time to spread to copies is logarithmic as discussed i class. 24 of 32

25 4. 3 parallelism The product of two matrices, C=matmul(A,B), ca be computed by addig matrices of rak (,): C = A 11 B 11 A 11 B 12 A 11 B A 21 B 11 A 21 B 12 A 21 B A 12 B 21 A 12 B 22 A 12 B A 22 B 21 A 22 B 22 A 22 B A 13 B 31 A 13 B 32 A 13 B A 23 B 31 A 23 B 32 A 23 B of 32

26 These matrices of rak (,) ca be computed by multiplyig (elemet-by-elemet) two three-dimesioal arrays of rak(,,). The two three-dimesioal arrays are formed by replicatig A ad B alog differet dimesios as show ext:... A A A B B B B... This replicatio ca, agai, be achieved, with SPREAD. 26 of 32

27 Thus, give the followig three directios of replicatio: we ca start by computig a 3 temporary array T as follows: T(:,:,:)=SPREAD(A,DIM=3,NCOPIES=)*SPREAD(B,DIM=1,NCOPIES= ) The, the result is just C=SUM(T,DIM=2) I a array machie with P>= 3 processig uit, the time to compute C would be: ( 2t copy log + t * + t + ) 27 of 32

28 Multiplicatio by Diagoals A matrix A is baded if A ij =0 for i-j β 1, j-i β 2 : A 11 A 12 A 1, β A 22 A 23 A 2, β A β1, 1 A β 2 + 1, 0 A β1 + 12, 0 0 A 1, A, β1 + 1 A, For a small bad, for example, β 1 =β 2 =3, the algorithm discussed before for matrix-vector multiplicatio is ot efficiet. 28 of 32

29 A alterative is to do the product by diagoals: A 0 A 1 A β A A β 1 V After separatig the diagoals ito separate matrices, we get: A A A β A V V + + V V A β V 29 of 32

30 which ca be writte as follows: A 0 V A 1 V 2 β A β2 V + A 1 V 1 A β1 V β where V j =(V j,...,v ) ad V -j =(V 1,...,V -j ). Also, meas add the sorter vector to the first compoet of the loger oe, ad meas add the shorter vector to the last compoet of the loger oe. I Fortra 90 (except for the greek letters ad the subscripts): A 0 (1:)*V(1:) + (/ A 1 (1:-1)*V(2:),0. /) +... (/ A β1 (1:-β 1 )*V(β 1 +1:), (0., j=1,β 1 ) /) + (/ 0., A -1 (1:-1)*V(1:-1) /) +... (/ (0., j=1,β 2 ), A β2 (1:-β 2 )*V(1:-β 2 ) /) 30 of 32

31 31 of 32

32 32 of 32

Principle Of Superposition

Principle Of Superposition ecture 5: PREIMINRY CONCEP O RUCUR NYI Priciple Of uperpositio Mathematically, the priciple of superpositio is stated as ( a ) G( a ) G( ) G a a or for a liear structural system, the respose at a give