Parallel Computing Chapter 8 Dense Matrix Computation. Jun Zhang Department of Computer Science University of Kentucky

Transcription

1 Prllel Comuig Cher 8 Dese Mri Comuio Ju Zhg Derme of Comuer Sciece Uiversi of Keuck

2 8. Cegories of Mrices Dese mri: lmos ever elemes re ozero, or he locio of he zeros co be deeced esil d used efficiel i comuio Srse mri: locios of zero elemes re ko d such elemes re o sored or used i comuio Curre reserch focus: srse mrices Srucured or usrucured srse mrices: heher or o srse mri hs regulr srucures h m be eloied i sorge d comuio

3 8. Mig Mrices Sried riioig: mri is divided io grous of comlee ros or colums, ech grou is ssiged o oe rocessor. Mimum rocessors Block sried d (block) cclic sried Checkerbord riioig: mri is divided io smll squre or recgulr blocks d ech of hem is med o rocessor. Mimum rocessors More cocurrec m be eloied (block) cclic checkerbord

4

5 4

6 8. Mri Trsosiio Trsosig mri is o ierchge is ros d colums: T T A, Ai, j A j, i Noe h he digols re o moved. Assumig ui cos for ierchgig ir of mri elemes, he sequeil cos of rsosiio is: 8.. Checkerbord riioig A logicl mesh c be embedded io hsicl mesh or hercube (usig, e.g., MPI Cresi coordie) 5

7 6

8 Mri Trsosiio o Mesh A eleme belo he digol firs is moved u o he digol d he o he righ o is desiio rocessor A eleme bove he digol is moved do o he digol d he o he lef o is desiio rocessor No mjor commuicio cogesio Need schroizig commuicio ses Prllel ru ime: T P s 7

9 Mri Trsosiio o Hercube Recursive rsosiio lgorihm (RTA) Divide mri io four blocks d erform rsosiio Divide ech block io four smller blocks d erform rsosiio ih resec o he smll blocks. There re four ses of hem d he c be doe i rllel. Tol of log / ses, ech ih o commuicio hses d messge size / Prllel ru ime: T P ( s ) log 8

10 9

11 0

12

13 8.. Trsosiio ih Sried Priioig Block srucures c sill be used. A ll o ll ersolized commuicio is ivoked for he mri rsosiio The coss of ll o ll ersolized commuicio for differe rchiecures hve bee sudied. The messge size is /. Cos oiml for hercube ih cu hrough rouig. The rllel ru ime: T P s ( ) h log

14

15 8. Mri Vecor Mulilicio Comue Sequeil ru ime A * W b 8.. Ro ise Sriig Oe ro d oe eleme of b er rocessor. All o ll brodcs o disribue b o ll rocessors, folloed b locl comuio ih mulilicios d summios. Prllel ru ime: ll o ll brodcs (). Mulilicio d summio is ( ). Tol cos is ( ). This ssem is cosoiml 4

16 5

17 6

18 Usig < Processors / comlee ros d / elemes of b re sored i ech rocessor All o ll brodcs of messge size / is used o mke comlee b vilble o ll rocessors, folloed b locl comuio ih mulilicios d summios Prllel ru ime o hercube: ll o ll: log hich is cos oiml for s T P s ( / log )( O(). ), 7

19 Usig < Processors Prllel ru ime o mesh ih rroud: ll o ll: T ( ) ( / )( ) P s s ( ) hich is gi cos oiml for O(). We c see here is o much differece beee mesh d hercube rchiecures., Sclbili lsis Overhed fucio for hercube is T O s log, 8

20 Sclbili lsis (co.) Isoefficiec fucio is W KT, K Sere lsis, for erm, O s E E W K s log For erm, e hve W K Obviousl erm domies 9

21 Sclbili lsis (co.) Miulio so d I follos h so W W W K K K K ( (sequeil cos) ) 0

22 8.. Checkerbord Priioig Oe eleme er rocessor: rocessors c hold he mri A. The vecor is i he ls colum of rocessor..) oe o oe commuicio o lig he vecor log he ricile digol of he mri.) oe o ll brodcs o disribue he eleme mog rocessors i ech colum.) locl mulilicio ihi ech rocessor 4.) sigle ode ccumulio o obi he sum for he vecor

23

24

25 Prllel Ru Time ( rocessors) Three commuicio ses, oe o oe commuicio, oeo ll brodcs, sigle ode ccumulio, cos () o mesh d (log ) o hercube. The cos of he rllel ssem is ( ) d ( log ), resecivel. Neiher is cosoiml. Usig < rocessors Ech rocessor holds block of size ( / ) ( / ) elemes Three commuicio ses d locl comuios re sill eeded ih differe messge size. 4

26 Prllel Ru Time o Mesh ih Cu Through Rouig ihou Wrroud.) digol ligme: messge size Firs ro of rocessors s /.) columise oe o ll brodcs h / ( s / ) log( ) h.) roise sigle ode ccumulio cos he sme 4.) locl comuios cos / 5

27 Sclbili Alsis Prllel ru ime Is his rllel ssem cos oiml? Yes if T h s h s h s P ) log( ) ( ) log( ) ( ) ( log O 6

28 Sclbili Alsis (co.) Overhed fucio T O log Isoefficiec lsis, erm s s log h / erm W W K s log K log K log K log h erm W W K log K h / 7

29 Sclbili Alsis (co.) Degree of cocurrec of checkerbord riioig O( ) ( ) W ( ) Noe h / log ol for 65,56. Hece, he overll isoefficiec fucio is ( log ) 8

30 Crierio for Cos Oimli For cos oiml ssem, rllel cos is rooriol o sequeil cos log Igorig he double log erm, e hve Hece, subsiue his relioshi o he firs equio I follos h log log log log log O(log O log O( O( ) O(log ) ) ) 9

31 Mesh ih Sore d Forrd Rouig Prllel ru ime is Isoefficiec fucio is Much orse h cu hrough rouig Comrig Differe Priioigs Sried riioig ih rroud Checkerbord riioig ih cu hrough rouig T s P ) ( T s P ) ( T h s P log log 0

32 8.4 Mri Mulilicio C A B Algorihm ih rile loos: For i= 0 o do: For j = 0 o do: C[i,j] = 0 For k = 0 o do: C[i,j] = C[i,j] + A[i,k]*B[k,j] Ed do Ed do Ed do Comuio c be doe i elemeise or blockise. is relced b he umber of blocks

33 8.4. Simle Prllel Algorihm Checkerbord riioig of boh A d B. blocks of size ( / ) * ( / ). Processor P holds d i, A j i, j, Bi, j comues of he resulig mri. C i, j Comuig eeds ll d for C i, j A i, k B k, j 0 k All o ll brodcss for ech ro A ) d ech colum ) ( i,k ( B k, j Memor cos is high. Ech rocessor hs blocks of boh A d B. Ech rocessor eeds ( / ) memor, for ol of ( / ) memor.

34 Comuig Block of C Mri

35 Alsis for Hercube To ll o ll brodcs ses ih messge size / mog rocessors. ( Locl comuios, smll mri mri mulilicios Prllel ru ime ih rocessors Cos oiml if T s P log * s ( )) log O( ) 4

36 Alsis for Hercube (co.) The overll isoefficiec fucio is Mimum cocurrec is. Alsis for Mesh (sore d forrd) Commuicio cos Prllel ru ime ih rocessors Cos oiml if W K W ( s 8( ) ( / O( / TP s O( ) ) ) ) 5

37 8.4. Co s Algorihm This is memor efficie lgorihm. Tol memor requireme for Co s lgorihm is O( ). Performce o Hercube Mimum disce of ligme shifs is. O hercube ih cu hrough rouig, he cos is ( s log ) Ech sigle se shif i comue d shif cos s h 6

38 Iiil Disribuio of Mrices 7

39 Iiil Aligme d Secod Se 8

40 Third d Fourh Ses 9

41 8.4. Co s Algorihm (co.) Tol commuicio cos for shifig o mrices i ses is Locl comuio cos is Prllel ru ime ih rocessors Cos oiml if 40 s ) ( T s P ) ( O

42 8.4. Co s Algorihm (co.) Prllel Ru Time o Mesh Iiil ligme shif coss ( s ) The commuicio of comue d shif coss he sme. The ol rllel ru ime ih rocessors T P The Co s lgorihm is slighl more eesive h he simle lgorihm o boh hercube d mesh rchiecures, bu is more memor efficie 4 4 s 4

43 8.4. Fo s Algorihm Aoher memor efficie lgorihm O( ). Use oe o ll brodcs i ech ro of A, shif urd ech ro of B, ree imes. Performce o Hercube Messge size /, locl comuio / Oe o ll brodcs o hercube coss ( shifs d brodcss T P s s ) log log log 4

44 Firs d Secod Ses 4

45 Third d Fourh Ses 44

46 8.5 Solvig Lier Ssem A ssem of lier equios Wrie i mri form is d solved b Gussi elimiio d bck subsiuio 45,, 0,0,, 0,0 0 0, 0, 0 0, b b b b A,, 0 0, 0,

47 Elimiio Process 46

48 Algorihm for Gussi Elimiio For k=0 o do: For j=k+ o do: A[k,j] = A[k,j]/A[k,k] Ed do [k]=b[k]/a[k,k] A[k,k]= For i=k+ o do: For j=k+ o do: A[i,j]=A[i,j] A[i,k]*A[k,j] Ed do b[i]=b[i] A[i,k]*[k] A[i,k]=0 Ed do Ed do 47

49 Gussi Elimiio (co.) Sequeil cos of Gussi elimiio is /. Gussi elimiio does o sore he fcors h used o mulil he mri A. If e do sore hose fcors, e c rie A s fcored form (LU fcorizio, bes for m ssems ih he sme A) s L U Hece solvig A=b becomes solvig forrd elimiio d bck subsiuio A LU b L b U 48

50 Prllel Imlemeio ih Sried Priioig Suose ech rocessor holds oe ro of A. A he kh se, for rocessor k.) ll ozero elemes of he kh ro re divided b A[ k, k ].) oe o ll brodcs A[ k, j], k j o rocessors k+ o 0 Processors k+ o erform elimiio for A[ i, j] A[ i, j] A[ i, k ] A[ k, j] k i d k j 49

51 () Comuio (i) A[k,j]=A[k,j]/A[k,k] for k<j< (ii) A[k,k]= 50

52 (b) Commuicio Oe o ll brodcs of ro A[k,*] Ol he ls k colums 5

53 (c) Comuio (i) A[i,j]=A[i,j] A[i,k]*A[k,j] for k<i< d k<j< (ii) A[i,k]=0 for k<i< 5

54 Cos ih Sried Priioig Se : Divisios i he rocessor Pk, coss ( k ) Se : Oe o ll brodcs from Pk kes ( s ( k )) log Se : Locl comuio i he remiig k rocessors, ( k ) mulilicios d subrcios The ol cos of ierios is T ( k ) k 0 ( ) k 0 ( The ssem is o cos oiml, s he rocessor ime roduc is T ( log ) s log s ( ( k )) log ) log 5

55 Sriig ih feer h rocessors Processor idlig i Gussi elimiio c be llevied b usig cclicsried mig. 54

56 Gussi Elimiio ih Checkerbord Priioig Assume ech eleme is i rocessor The comuio is obviousl o cos oiml 55

57 () Comuio (i) A[k,j]=A[k,j]/A[k,k] for k<j< (ii) A[k,k]= (iii) A[i,j]=A[i,j] A[i,k]*A[k,j] 56

58 (b) Comuio (divisio) (i) A[k,j]=A[k,j]/A[k,k] for k<j< (ii) A[k,k]= (A[i,j]=A[i,j] A[i,k]*A[k,j]) 57

59 (c) Commuicio (for elimiio) (i) Oe o ll brodcs (colums) (ii) Sigle size messge (A[i,j]=A[i,j] A[i,k]*A[k,j]) 58

60 (d) Comuio (elimiio) (i) Perform elimiio i ech ro (ii) A[i,j]=A[i,j] A[i,k]*A[k,j] for k<i< d k<j< 59

61 Gussi Elimiio ih Checkerbord Priioig A ech se, oe o ll commuicio loe he rocessors i he sme ro, d oe o ll colum ise commuicio re erformed Boh commuicio ses cos O(log ) o hercube eork, d here re ses. So he rllel ssem cos is O( log ) The rllel ssem is o cos oiml 60

62 Pielied Commuicio d Comuio is Sried Priioig I ielied Gussi elimiio, ech rocessor erforms he folloig sequece of cios reeed uil ll ierios re comlee () If rocessor hs d desied for oher rocessors, i seds hose d o he rorie rocessor () If he rocessor c erform some comuio usig he d i hs, i does so () Oherise, he rocessor is o receive d o be used for oe of he bove cios 6

63 () Comuio (i) A[k,j]=A[k,j]/A[k,k] for k<j< (ii) A[k,k]= 6

64 (c) Comuio (i) A[i,j]=A[i,j] A[i,k]*A[k,j] for k<i< d k<j< (ii) A[i,k]=0 for k<i< 6

65 Cos of Pielied Commuicio d Comuio The iiiio of cosecuive ierios of he ouer loos is sered b cos umber of ses A ol of such ierios re iiied The ls ierio modifies ol he boom righ corer eleme of he coefficie mri, i comlees i cos ime I ech se, eiher O() elemes re commuiced beee direcl coeced rocessors, or divisio se is erformed o O() elemes of ro, or elimiio se is erformed o O() elemes of ro Ech of hese ses kes O() ime 64

66 Pielied Commuicio d Comuio ih Checkerbord Priioig The ides re similr o hose usig i he ielied versio ih sried riioig Commuicio d comuio c be ielied i severl s Commuicio d comuio ve fros move from he o lef corer o he boom righ corer Ech se kes cos ime There re such ses (fros) The rllel ru ime is O() Sice here re ^ rocessors re ivolved, he rllel ime cos is O(^), hich is cos oiml 65

67 66

68 67

69 68

70 69

71 Checkerbord ih feer h * rocessors Wih checkbord riioig d less h * rocessors, messge size m o be he sme i differe ros 70

72 Checkerbord ih feer h * rocessors Columise brodcs folloed b locl comuio fiishig elimiio rocess 7

73 Checkerbord ih cclic mig Beer lod blcig m be chieved b cclic mig. Hoever, commuicio overhed m icrese. A rde off is mchie deede 7

74 8.5. Pril Pivoig So, fr, e hve ssumed h he divisios i ech ro c be erformed ihou roblem. Th mes he mi digol elemes ii re lrge i bsolue vlues (relivel o off digol elemes). If ii is smll for he ih ro, e hve fid some (or ) h is lrger. ii For emle, if is smll, bu is lrge 0 ji,, 4 0,,,, 4, 0,,,, 4, 0,,,, 4, 0,4,4,4,4 4,

75 8.5. Pril Pivoig (co.) We c ierchge d (colum ivoig) Ro ivoig m be lso erformed , 4, 4, 4 4,4,,, 4,4,,, 4,4,,, 4,4 0 0, 0, 0, 4 0,4 0, 4, 4 4 4,4 4, 4, 4, 4,4,,, 4,4,,, 4,4,,, 0 4 0,4 0, 0, 0, 0

76 8.5. Pril Pivoig (co.) We c lso serch for he lrges eleme i boh he ih ro d he ih colum. This is he full ivoig sreg. Full ivoig is more robus bu is much more cosl h ril ivoig. For mos relisic licios, ril ivoig is sufficie o guree umericl sbili. Pril ivoig is srighforrd o imleme ih he simle lgorihms. Bu i is o efficie ih he ielied versios of Gussi elimiio. Wih checkerbord riioig, commuicio cos icreses subsill. A cclic mig ill lso icrese cos. 75

77 Pielied Bckrd Subsiuio The bckrd subsiuio is iherel sequeil, bu m be ielied Se ,4 4,4, 4,4,, 0 4 0,4 0, 0, 0, ,4 4,4, 4,4,, 0 4 0,4 0 0, 0, 0, 0

78 Pielied Bckrd Subsiuio (co.) Se Se ,,, 0 0, 0 0, 0, 0 ~ ~ ~ 4 4, 0 0, 0 0, 0 ~ ~ ~

79 Pielied Bckrd Subsiuio (co.) Se4 0 4 ~ 0 4 0, Comuiol cos is (). ses of cos comuio (oe ddiio d oe mulilicio) Commuicio cos is he sme s i he simle (o ielied) versio o hercube. ( ( ) log ) s 78

80 8.5.4 Differe Gussi Elimiio The versio of Gussi elimiio h e sudies is clled roorieed. Deedig uo our d srucure, Gussi elimiio c be imlemeed i colum orieed fshio. Sequeill, if he mri is sored ro b ro, he colum orieed fshio is more suible. Oherise ro orieed fshio should be used. I is lso sid h he colum orieed fshio is umericl more sble if LU fcorizio is erformed. Becuse i he columorieed fshio, he elemes of he L mri re ll smller h. 79