Lecture 23: I. Data Dependence II. Dependence Testing: Formulation III. Dependence Testers IV. Loop Parallelization V.

Transcription

1 Lecure 23: Array Dependence Analysis & Parallelizaion I. Daa Dependence II. Dependence Tesing: Formulaion III. Dependence Tesers IV. Loop Parallelizaion V. Loop Inerchange [ALSU 11.6, ] Phillip B. Gibbons : Parallelizaion 1

2 Le S i precede S j in execuion. I. Daa Dependence S 1 : a = 1; S 2 : b = a + 2; S 3 : a = c - d;... Flow (rue) dependence: S i compues a daa value ha S j uses. S 4 : a = b/c; S i δ S j E.g., S 1 δ S 2 and S 2 δ S 4 Ani dependence: S i uses a daa value ha S j overwries. S i δ a S j E.g., S 2 δ a S 3 Oupu dependence: S i compues a daa value ha S j overwries. S i δ o S j E.g., S 1 δ o S 3 and S 3 δ o S 4 Inpu dependence: S i uses a daa value ha S j also uses. S i δ i S j E.g., S 3 δ i S 4 (Unlike he oher 3, i is ypically safe o execue S i and S j in parallel) : Parallelizaion 2

3 Daa Dependence Graph Daa dependence in a program may be represened using a dependence graph G=(V,E), where he nodes V represen saemens in he program and he direced edges E represen dependence relaions. S 1 : a = 1; S 2 : b = a + 2; S 3 : a = c - d;... S 4 : a = b/c; d S 1 d S 2 S 3 d a d o d i d o S : Parallelizaion 3

4 Array Daa Dependence: Example 1 for i = 2 o 4 { S 1 : a[i] = b[i] + c[i] ; S 2 : d[i] = a[i] } S 1 2 i=2 i=3 i=4 S 2 2 S 1 3 S 2 3 S 1 4 S 2 4 d d d a[2] a[2] a[3] a[3] a[4] a[4] There is an insance of S 1 ha precedes an insance of S 2 in execuion and S 1 produces daa ha S 2 uses. S 1 is he source of he dependence; S 2 is he sink of he dependence. The dependence flows beween insances of saemens in he same ieraion (loop-independen dependence). The number of ieraions beween source and sink (dependence disance) is 0. The dependence direcion is =. S 1 δ = S 2 or S 1 δ 0 S : Parallelizaion 4

5 Array Daa Dependence: Example 2 do i = 2, 4 S 1 : a[i] = b[i] + c[i] S 2 : d[i] = a[i-1] i=2 i=3 i=4 2 S 2 1 S 2 S 1 3 S S 4 1 S 2 a[2] a[1] a[3] a[2] a[4] a[3] d d There is an insance of S 1 ha precedes an insance of S 2 in execuion and S 1 produces daa ha S 2 uses. S 1 is he source of he dependence; S 2 is he sink of he dependence. The dependence flows beween insances of saemens in differen ieraions (loop-carried dependence). The dependence disance is 1. The direcion is posiive (<). S 1 δ < S 2 or S 1 δ 1 S : Parallelizaion 5

6 Example 3 do i = 2, 4 S 1 : a[i] = b[i] + c[i] S 2 : d[i] = a[i+1] i=2 i=3 i=4 2 S 2 1 S 2 S 1 3 S 2 3 d a d a 4 S 4 1 S 2 a[2] a[3] a[3] a[4] a[4] a[5] There is an insance of S 2 ha precedes an insance of S 1 in execuion and S 2 uses daa ha S 1 overwries. S 2 is he source of he dependence; S 1 is he sink of he dependence. The dependence is loop-carried. The dependence disance is 1. The direcion is posiive (<). S 2 δ < a S 1 or S 2 δ 1 a S 1 Are you sure you know why i is S even hough S 1 appears before S 2 in 2 d S < 1 he code? a : Parallelizaion 6

7 Example 4: 2D Ieraion Space do i = 2, 4 do j = 2, 4 S: A[i,j] = A[i-1,j+1] An insance of S precedes anoher insance of S and S produces daa ha S uses. S is boh source and sink. The dependence is loopcarried. A[1,3] A[1,4] A[1,5] S[2,2] S[2,3] S[2,4] A[2,2] A[2,3] A[2,4] d A[2,3] A[2,4] A[2,5] S[3,2] S[3,3] S[3,4] A[3,2] A[3,3] A[3,4] d d d The dependence disance is (1,-1). S δ <,> S or S δ 1, 1 S S[4,2] A[3,3] A[3,4] A[3,5] S[4,3] S[4,4] A[4,2] A[4,3] A[4,4] : Parallelizaion 7

8 II. Dependence Tesing: Formulaion Consider he following perfec nes of deph d: do I1 L 1, U1 do I2 L2, U2 do I L, U d d d a(f(i),f (I),, fm (I)) 1 2 a(g (I), g (I),, gm (I)) 1 2 enddo enddo enddo array reference a(, fk (I),, subscrip posiion perfec means sep=1 ) subscrip expression I (I1,I 2,,Id ) L (L 1,L2,,Ld ) U (U 1,U2,,Ud) L U : Parallelizaion 8 Affine expressions: c 0 + c 1 I 1 + c 2 I c d I d for consans c 0, c 1,, c d

9 Problem Formulaion Dependence will exis if here exiss wo ieraion vecors k and j such ha L k j U and: f1(k) g1( j) and f (k) g ( j) and and f 2 m (k) g 2 m ( j) Tha is: and and and f(k) 1 g1(j) 0 f (k) g (j) 0 2 f m 2 (k) g m (j) : Parallelizaion 9

10 Problem Formulaion - Example do i = 2, 4 S 1 : a[i] = b[i] + c[i] S 2 : d[i] = a[i-1] Does here exis wo ieraion vecors i 1 and i 2, such ha 2 i 1 i 2 4 and such ha i 1 = i 2-1? Answer: yes; i 1 =2 & i 2 =3 and i 1 =3 & i 2 =4. Hence, here is dependence! The dependence disance vecor is i 2 -i 1 = 1. The dependence direcion vecor is sign(1) = < : Parallelizaion 10

11 Problem Formulaion - Example do i = 2, 4 S 1 : a[i] = b[i] + c[i] S 2 : d[i] = a[i+1] Does here exis wo ieraion vecors i 1 and i 2, such ha 2 i 1 i 2 4 and such ha i 1 = i 2 + 1? Answer: yes; i 1 =3 & i 2 =2 and i 1 =4 & i 2 =3. (Bu, bu!). Hence, here is dependence! The dependence disance vecor is i 2 -i 1 = -1. The dependence direcion vecor is sign(-1) = >. Is his possible? Yes: As an anidependence, no a rue dependence : Parallelizaion 11

12 Problem Formulaion - Example do i = 1, 10 S 1 : a[2*i] = b[i] + c[i] S 2 : d[i] = a[2*i+1] Does here exis wo ieraion vecors i 1 and i 2, such ha 1 i 1 i 2 10 and such ha 2*i 1 = 2*i 2 +1? Answer: no; 2*i 1 is even & 2*i 2 +1 is odd Hence, here is no dependence! : Parallelizaion 12

13 Problem Formulaion Dependence esing is equivalen o an ineger linear programming (ILP) problem of 2d variables & m+d consrains! An algorihm ha deermines if here exiss wo ieraion vecors k and j ha saisfies hese consrains is called a dependence eser. do I1 L 1, U1 do I2 L2, U2 do I L, U d d d a(f(i),f (I),, fm (I)) 1 2 a(g (I), g (I),, gm (I)) 1 2 enddo enddo enddo : Parallelizaion 13

14 Problem Formulaion Dependence esing is equivalen o an ineger linear programming (ILP) problem of 2d variables & m+d consrains! An algorihm ha deermines if here exiss wo ieraion vecors and ha saisfies hese consrains is called a dependence eser. k j k - j k - The dependence disance vecor is given by j. The dependence direcion vecor is give by sign( ). Dependence esing is NP-complee! A dependence es ha repors dependence only when here is dependence is said o be exac. Oherwise i is in-exac. A dependence es mus be conservaive; if he exisence of dependence canno be ascerained, dependence mus be assumed : Parallelizaion 14

15 Lampor s Tes. GCD Tes. Banerjee s Inequaliies. Generalized GCD Tes. Power Tes. I-Tes. Omega Tes. Dela Tes. Sanford Tes. ec III. Dependence Tesers : Parallelizaion 15

16 Lampor s Tes Lampor s Tes is used when here is a single index variable in he subscrip expressions, and when he coefficiens of he index variable in boh expressions are he same. A(, b *i c1, ) A(, b *i c, ) 2 The dependence problem: does here exis i 1 and i 2, such ha L i i 1 i 2 U i and such ha b*i 1 + c 1 = b*i 2 + c 2? i.e., i 2 i c b There is ineger soluion if and only if is ineger. 1 c 1 2? c 2 1 c b c1 c2 The dependence disance is d = b if d U i - L i d > 0 rue dependence d = 0 loop independen dependence d < 0 ani dependence : Parallelizaion 16

17 Lampor s Tes - Example do i = 1, n do j = 1, n S: a[i,j] = a[i-1,j+1] b*i 1 + c 1 = b*i 2 + c 2 i 1 = i 2-1? b = 1; c 1 = 0; c 2 = -1 c 1 b c 2 1 There is dependence. Disance (i) is 1. j 1 = j 2 + 1? b = 1; c 1 = 0; c 2 = 1 c 1 b c 2 1 There is dependence. Disance (j) is : Parallelizaion 17 S δ <,> S or S δ 1, 1 S

18 Lampor s Tes Anoher Example do i = 1, n do j = 1, n S: a[i,2*j] = a[i-1,2*j+1] b*i 1 + c 1 = b*i 2 + c 2 i 1 = i 2-1? b = 1; c 1 = 0; c 2 = -1 c 1 b c 2 1 There is dependence. Disance (i) is 1. 2*j 1 = 2*j 2 + 1? b = 2; c 1 = 0; c 2 = 1 c 1 b c There is no dependence.? There is no dependence! : Parallelizaion 18

19 Given he following equaion: n σ i=1 GCD Tes a i x i = c where a i and c are inegers an ineger soluion exiss if and only if: gcd a 1, a 2,, a n divides c Problems: ignores loop bounds gives no informaion on disance or direcion of dependence ofen gcd( ) is 1 which always divides c, resuling in false dependences : Parallelizaion 19

20 GCD Tes - Example do i = 1, 10 S 1 : a[2*i] = b[i] + c[i] S 2 : d[i] = a[2*i-1] Does here exis wo ieraion vecors i 1 and i 2, such ha 1 i 1 i 2 10 and such ha: or 2*i 1 = 2*i 2-1? 2*i 2-2*i 1 = 1? There will be an ineger soluion if and only if gcd(2,-2) divides 1. This is no he case, and hence, here is no dependence! : Parallelizaion 20

21 GCD Tes: Anoher Example do i = 1, 10 S 1 : a[i] = b[i] + c[i] S 2 : d[i] = a[i-100] Does here exis wo ieraion vecors i 1 and i 2, such ha 1 i 1, i 2 10 and such ha: or i 1 = i 2-100? i 2 - i 1 = 100? There will be an ineger soluion if and only if gcd(1,-1) divides 100. This is he case, and hence, here is dependence! Or is here? No: check loop bounds. Shows a limiaion of GCD : Parallelizaion 21

22 Dependence Tesing: Complicaions Unknown loop bounds: do i = 1, N S 1 : a[i] = a[i+10] Wha is he relaionship beween N and 10? Triangular loops: do i = 1, N do j = 1, i-1 S: a[i,j] = a[j,i] Mus impose j < i as an addiional consrain : Parallelizaion 22

23 More Complicaions User variables: do i = 1, 10 S 1 : a[i] = a[i+k] Same problem as unknown loop bounds, bu occur due o some loop ransformaions (e.g., loop bounds normalizaion). do i = L, H S 1 : a[i] = a[i-1] do i = 1, H-L S 1 : a[i+l] = a[i+l-1] : Parallelizaion 23

24 More Complicaions: Scalars do i = 1, N S 1 : x = a[i] S 2 : b[i] = x do i = 1, N S 1 : x[i] = a[i] S 2 : b[i] = x[i] privaizaion j = N-1 do i = 1, N S 1 : a[i] = a[j] S 2 : j = j - 1 do i = 1, N S 1 : a[i] = a[n-i] : Parallelizaion 24

25 IV. Loop Parallelizaion A dependence is said o be carried by a loop if he loop is he ouermos loop whose removal eliminaes he dependence. If a dependence is no carried by he loop, i is loop-independen. do i = 2, n-1 do j = 2, m-1 a(i, j) =... = a(i, j) b(i, j) = = b(i, j-1) : Parallelizaion 25 c(i, j) = = c(i-1, j)

26 Loop Parallelizaion A dependence is said o be carried by a loop if he loop is he ouermos loop whose removal eliminaes he dependence. If a dependence is no carried by he loop, i is loop-independen. Ouermos loop δ =,= do i = 2, n-1 do j = 2, m-1 a(i, j) =... = a(i, j) wih a non = direcion carries dependence! loop-independen δ =,< b(i, j) = = b(i, j-1) carried by loop j δ <,= : Parallelizaion 26 c(i, j) = = c(i-1, j) carried by loop i

27 Loop Parallelizaion The ieraions of a loop may be execued in parallel wih one anoher if and only if no dependences are carried by he loop! : Parallelizaion 27

28 δ =,< do i = 2, n-1 do j = 2, m-1 b(i, j) = = b(i, j-1) Loop Parallelizaion - Example i=2 i=3 i=n-2 fork i=n-1 join Ieraions of loop j mus be execued sequenially, bu he ieraions of loop i may be execued in parallel! Ouer loop parallelism : Parallelizaion 28

29 δ <,= do i = 2, n-1 do j = 2, m-1 b(i, j) = = b(i-1, j) Loop Parallelizaion - Example j=2 j=3 j=m-2 fork j=m-1 join i=i+1 Ieraions of loop i mus be execued sequenially, bu he ieraions of loop j may be execued in parallel! Inner loop parallelism (Vecorizaion, SIMD) : Parallelizaion 29

30 δ <,< do i = 2, n-1 do j = 2, m-1 b(i, j) = = b(i-1, j-1) Loop Parallelizaion - Example j=2 j=3 j=m-2 fork j=m-1 join i=i+1 Ieraions of loop i mus be execued sequenially, bu he ieraions of loop j may be execued in parallel! Why? Inner loop parallelism : Parallelizaion 30

31 V. Loop Inerchange Recall: Used o improve spaial localiy for i = 0 o N-1 for j = 0 o N-1 A[j][i] = i*j; for j = 0 o N-1 for i = 0 o N-1 A[j][i] = i*j; i j Hi Miss j i Assume row major order, N large, 4 elemens per cache line : Parallelizaion 31

32 Loop Inerchange Can also be used o improve he granulariy of parallelism! do i = 1, n do j = 1, n a[i,j] = b[i,j] c[i,j] = a[i-1,j] do j = 1, n do i = 1, n a[i,j] = b[i,j] c[i,j] = a[i-1,j] δ <,= δ =,< Inner loop parallelism Ouer loop parallelism : Parallelizaion 32

33 When Is Loop Inerchange Legal? j do i = 1,n do j = 1,n a[i,j] i δ δ,, < δ <, δ >, <, δ <, < δ <, > δ > do j = 1,n do i = 1,n a[i,j] δ >, δ >, > : Parallelizaion 33

34 When Is Loop Inerchange Legal? j Focus only on rue dependences (i.e., lexicographically posiive dependences) do i = 1,n do j = 1,n a[i,j] i δ δ,, < δ <, δ <, < δ <, > do j = 1,n do i = 1,n a[i,j] : Parallelizaion 34

35 When Is Loop Inerchange Legal? j do i = 1,n do j = 1,n a[i,j] i δ δ,, < δ <, δ <, < δ <, > do j = 1,n do i = 1,n a[i,j] : Parallelizaion 35

36 When Is Loop Inerchange Legal? j do i = 1,n do j = 1,n a[i,j] i δ δ,, < δ <, δ <, < δ <, > do j = 1,n do i = 1,n a[i,j] When is loop inerchange legal? when he inerchanged dependences remain lexiographically posiive! : Parallelizaion 36

37 Today s Class: Array Dependence Analysis & Parallelizaion I. Daa Dependence II. Dependence Tesing: Formulaion III. Dependence Tesers IV. Loop Parallelizaion V. Loop Inerchange Domain Specific Languages Monday s Class Warning: Projec Milesone Repors are due in one week : Parallelizaion 37