A note on the multiplication of sparse matrices

Transcription

1 Cent. Eur. J. Cop. Sci. 41) DOI: /s x Central European Journal of Coputer Science A note on the ultiplication of sparse atrices Research Article Keivan Borna 12, Sohrab Aboozarkhani Fard 1 1 Faculty of Matheatics and Coputer Science, Kharazi University, Tehran, Iran 2 School of Matheatics, Institute for Research in Fundaental Sciences IPM), P.O , Tehran, Iran. Received 07 Deceber 2012; accepted 28 Deceber 2013 Abstract: We present a practical algorith for ultiplication of two sparse atrices. In fact if A and B are two atrices of size n with 1 and 2 non zero eleents respectively, then our algorith perfors Oin{ 1 n, 2 n, 1 2 }) ultiplications and Ok) additions where k is the nuber of non zero eleents in the tiny atrices that are obtained by the coluns ties rows atrix ultiplication ethod. Note that in the useful case, k 2 n. However, in Proposition 3.3 and Proposition 3.4 we obtain tight upper bounds for the coplexity of additions. We also study the coplexity of ultiplication in a practical case where non-zero eleents of A resp. B) are distributed independently with unifor distribution aong coluns resp. rows) of the and show that the expected nuber of ultiplications is O 1 2 /n). Finally a coparison of nuber of required ultiplications in the naïve atrix ultiplication, Strassen s ethod and our algorith is given. Keywords: algoriths atrix ultiplication sparse atrices tiny atrices Versita sp. z o.o. 1. Introduction Let A be an n atrix and B a second atrix of size n p. Then the product AB is an p atrix whose entries are AB) i,j = n k=1 A ikb kj. In the following we recall soe alternate descriptions of atrix ultiplication; see [1, Section 2.4] for ore details: A ties coluns of B: The j-th colun of AB is the product of A and the j-th colun of B. Rows of A ties B: The i-th row of AB is the product of the i-th row of A and B. Coluns of A ties rows of B: AB is obtained as the su of coluns of A ties rows of B. That is, AB = A 1 B A n B n, where A i and B j stand for the i-th colun of A and the j-th row of B respectively. We also use this ethod for ultiplication in our algoriths. E-ail: borna@khu.ac.ir 1

2 A note on the ultiplication of sparse atrices In the following we give a very short review of soe algoriths for atrix ultiplication: The naïve atrix ultiplication algorith perfors On 3 ) operations using n 3 ultiplications and n 3 n 2 additions. Strassen [2] gave a divide-and-conquer algorith which runs in On 2.81 ) tie. For exaple, for two 2 2 atrices, the naïve ethod takes 8 ultiplications and 4 additions, while using the Strassen s ethod they can be ultiplied using only 7 ultiplications and 18 additions. Horowitz et al. [3, Section 2.4.4] gave an algorith that runs in O 1 n + 2 n) where the atrices are stored in sparse storage odel. Coppersith and Winograd [4] provided the fastest known atrix ultiplication algorith, with a coplexity of On 2.38 ). Yuster and Zwick [5] gave a new algorith that ultiplies A and B using Oin{ 1 2 ) n n 2+o1), 1 n, 2 n, n o1) }) algebraic operations ultiplications, additions and subtractions). In fact they split each of the given atrices into two dense and sparse atrices. Recall that an n n atrix is called sparse resp. dense) if the nuber of non zero eleents of it is On 1.37 ) resp. On 1.68 )). Then ultiply the dense parts using the fast dense rectangular atrix ultiplication algorith of Coppersith [6] and ultiply the sparse parts using the naïve sparse atrix ultiplication algorith. Finally they output the su of theses two parts. Note that if the given atrices are sparse, one has to avoid ultiplying zeros. Now ultiplying two sparse atrices using the atrix ultiplication algoriths of Coppersith and Winograd [4] for exaple, then it does not provide any iproveents over the non sparse atrix ultiplication, as their concern is to ultiply two atrices in general. Furtherore note that the result of Yuster and Zwick [5] is of theoretical iportance at least by now). In this paper we give a practical algorith for ultiplication of two sparse atrices using sparse storage odels. More precisely, let A and B be two sparse atrices of size n with 1 and 2 non zero eleents respectively then the coplexity of ultiplication of our algorith is Oin{ 1 n, 2 n, 1 2 }) and the coplexity of additions is Ok). This iproves the coplexity O 1 n + 2 n) entioned in [3, Section 2.4.4]. We also study the coplexity of ultiplication where non-zero eleents of A and B are distributed independently with unifor distribution aong coluns of A and rows of B) respectively. In fact we then show that the expected nuber of ultiplications is O 1 2 /n). Furtherore in Section 3.2 we obtain tight upper bounds for the coplexity of additions, k, as it is presented in the following: 1 2, 1, 2 n k 1 n, 1 n, 2 > n 2 n, 1 > n, 2 n and if 1, 2 > n we have { k 2 n, 2 αn, αn αn). in{ 2 αn, n}, 2 > αn where α = 1 /n. The organization of this paper is as follows. In Section 2 the ain results of the paper are given and in Section 3 the coplexity analysis of our algoriths are presented. Section 4 is devoted to soe conclusions and future works. Finally in Section 5 we give a coparision for the nuber of required ultiplications in the naïve atrix ultiplication, Strassen s ethod and our algorith. 2. Main results Let A and B be two square atrices of size n and with 1 and 2 non zero eleents. Our results could be generalized easily to rectangular atrices, but for the sake of siplicity we just present the for square atrices. We first recall two storage odels for sparse atrices; see [3, Section 2.3] for ore details. Note that these two sparse storage odels keep the row and colun nubers sorted respectively. Storage Model 1. In this odel we store any sparse atrix of size n in a atrix by + 1 rows and 3 coluns, where is the nuber of non zero eleents of it. 2

3 Keivan Borna, Sohrab Aboozarkhani Fard a) In the first row we store the triple row count, colun count and the nuber of non zero eleents. b) In the next rows we store the triple row nuber, colun nuber and value for each non zero eleent. Storage Model 2. This odel is essentially the sae as Storage Model 1 with a difference that we first store the colun nubers and then the row nubers. In the following our ain algorith for ultiplication of two sparse atrices is given. Our approach is essentially based on the Coluns ties Rows atrix ultiplication ethod which states that a ij b jk is the entry at row i and colun k of the j-th tiny atrix Main algorith If 2 n, ultiply A and B using Partial Algorith 1. Else ultiply A and B using Partial Algorith Partial algorith 1 Input: A and B two sparse atrices of size n and with 1 and 2 non-zero eleents which are stored in Storage Model 1 as Ā and B respectively. Output: The product C := AB which is a square atrix of size n) is given in its ordinary odel. Process: //Z: a atrix with k + 1 rows and 3 coluns, //i : index for non-zero eleents of A, //j : index for non-zero eleents of B, //u: index for non-zero eleents of tiny atrices, //t: counter of non-zero eleents of tiny atrices. 1 i t u 1 2 Z0, 0), Z0, 1)) n, n) 3 while i 1 do 4 j 1 5 while j 2 do 6 if Āi, 1) = Bj, 0) then 7 Zt, 0), Zt, 1), Zt, 2), t) Āi, 0), Bj, 1), Āi, 2) Bj, 2), t + 1) 8 k t 1 9 Z0, 2) k 10 while u k do 11 CZu, 0), Zu, 1)) CZu, 0), Zu, 1)) + Zu, 2) 2.3. Partial algorith 2 Let B be stored in Storage Model 1 as B, then the first colun of B is sorted. Using a one diensional array of size n, say W, for each i, let W [i] stand for the first row nuber whose first colun is i and zero if i does not appear in the first colun of B. For exaple if B is, then W is filled with the following values: W [0] = 1, W [1] = 0 and W [2] = 3. 3

4 A note on the ultiplication of sparse atrices Coputation of W We use the following algorith to copute W : Input: B a square atrix of size n with 2 non zero eleents. Output: W an array of length n initialized with zero. Process: 1 u B1, 0) 2 W u 1 3 while 2 i 2 do 4 if u Bi, 0) then 5 u Bi, 0) 6 W u i In the following we give an algorith to copute AB: Input: A and B two sparse atrices of size n and with 1 and 2 non-zero eleents where A is stored in Storage Model 2 as Ā and B is stored in Storage Model 1 as B. Output: The product C := AB which is a square atrix of size n) in its ordinary storage odel. Process: //Z: a atrix with k + 1 rows and 3 coluns, //i : index for non-zero eleents of A, //j : index for non-zero eleents of B, //u: index for non-zero eleents of tiny atrices, //t: counter of non-zero eleents of tiny atrices. 1 i t u 1 2 Z0, 0), Z0, 1)) n, n) 3 while i 1 do 4 W Āi, 0)) 5 if 0 then 6 while j 2 7 if Āi, 0) = Bj, 0) then 8 Zt, 0), Zt, 1), Zt, 2), t) Āi, 1), Bj, 1), Āi, 2) Bj, 2), t + 1) 9 k t 1 10 Z0, 2) k 11 while u k do 12 CZu, 0), Zu, 1)) CZu, 0), Zu, 1)) + Zu, 2) Reark 1. The ultiplication of two sparse atrices is not necessarily sparse. Thus the output of product C := AB is a square atrix of size n and so we have to store it in its ordinary storage odel. Reark 2. One can apply the Partial Algorith 1 when 2 > n. But the role of Partial Algorith 2 is to reduce the dependence of the ain algorith fro 2. This will iprove the functionality of our algorith as n grows. 4

5 Keivan Borna, Sohrab Aboozarkhani Fard 3. Coplexity analysis In the following for each 1 t n let ā t and b t denote the nuber of non zero eleents of the t-th colun of A and t-th row of B respectively Coplexity analysis of ultiplications The nuber of ultiplications in this algorith is n t=1 āt b t ; see [5] for ore inforation. On the other hand, the loops will run in O 1 2 ). Thus the total coplexity of ultiplications is Oin{ 1 n, 2 n, 1 2 }). Corollary 3.1. The particular case of atrix-vector product has cost O 1 ). This is because 2 n and for each i, b i = 0 or b i = 1. Thus the nuber of ultiplications is n āi b i n āi = 1 = On 1.37 ). A siilar proof shows that the vector-atrix product has cost O 2 ) = On 1.37 ). Recall that for a dense atrix A of size n n, its product Ax with an arbitrary input vector x has cost On 2 ). Furtherore in [7] the authors presented an Onlogn) algorith for coputing the atrix-vector product of a Pascal atrix and a vector. Note that a fast solution for the atrix-vector product has any applications in solving a syste of equations Ax = b where its solution is given by x = A 1 b Coplexity of ultiplications in a practical case Let A and B be two sparse atrices of size n with 1 and 2 non-zero eleents respectively. Assue that non-zero eleents of A resp. B) are distributed independently with unifor distribution aong coluns resp. rows) of the. For constructing A resp. B) we run the following Bernoulli rando test 1 resp. 2 ) ties. For each 1 i n define two rando variable ā i, b i that count the nuber of non-zero eleents of the i th colun of A and the i th row of B respectively. For ā i, put the first non-zero eleent randoly with unifor distribution in one of coluns in A. Then with probability 1/n this eleents locates in the colun i and with probability n 1)/n this eleent does not locate in the colun i of A. A siilar rando test can be applied for b i. Thus the rando distribution of ā i, P A, and of b i, P B, are binoial with the following probability functions for which we have 0 ā i 1 := in{ 1, n}, 0 b i 2 := in{ 2, n}: P A j) = Pā i = j) = P B j) = P b i = j) = 1 ) ) j 1 n 1 j n n ) ) j 1 n 1 j n n 2 ) 1 j, ) 2 j. Since ā i, b i are independent we have E[The nuber of ultiplications] = E[ = = = n ā i bi ] = n E[ā i ]E[ b i ] n 1 2 j.p A j). j.p B j) n 1 2 j.p A j). j.p B j) n 1 j. n 1 j. 1 2 n 1 j 1 ) ) j 1 n 1 n n ) 1 j ) n 1) 1 j 2. j j n 1 = On 1.74 ). 2 j 2. j 2 j ) n 1) 2 j n 2 ) ) j 1 n 1 n n ) 2 j 5

6 A note on the ultiplication of sparse atrices The second inequality is a siple siplification by Matheatica for exaple. We suarize this result in the following corollary. Corollary 3.2. The expected nuber of ultiplications in the product of two sparse atrices A and B where non-zero eleents of A resp. B) are distributed independently with unifor distribution aong coluns resp. rows) of the is On 1.74 ). In Section 5 we copare the required nuber of ultiplications in our algorith with those of the naïve and Strassen s ethods. The coplexity of additions in both Partial Algoriths 1 and 2 is Ok). A precise analysis of this issue is done in the following Coplexity analysis of additions Note that as we entioned k is the coplexity of additions. We can give upper and lower bounds for k as in the following: Proposition 3.3. Let A and B be two atrices of size n and with 1 and 2 non zero eleents respectively. Then upper and lower bounds for k are: 1 2, 1, 2 n k 1 n, 1 n, 2 > n 2 n, 1 > n, 2 n n n) 2 n), 1 > n, 2 > n and { n k )n, > n 2 0, n 2 Proof. Thus Let A = a ij ) and B = b ij ). Then ultiplying A and B by coluns ties rows ethod we have: a 11 a AB = [ ] 1t a. b 11 b 1n + + [ ] 1n. b t1 b tn + + [ ]. b n1 b nn. a n1 a nt a 11 b 11 a 11 b 1n a 1t b t1 a 1t b tn a 1n b n1 a 1n b nn AB = a n1 b 11 a n1 b 1n a nt b t1 a nt b tn a nn b n1 a nn b nn a nn Thus we obtained a su of n tiny atrices. Note that k is in fact the nuber of non zero eleents of these tiny atrices rows of Z). Lower bounds: Let = n 2 1 and p = n 2 2 be the nuber of zero eleents of A and B respectively. Since each zero in A will produce n zero in one of tiny atrices, we have n zero eleents in all tiny atrices. For exaple, if a 23 = 0, then the second row in the third tiny atrix is zero. Now if for each j, 1 j n, a ij and b jk are not zero siultaneously, then k = n 3 + p)n = )n n 3. Thus k )n n 3 when > n 2 and k 0 if n 2. Upper bounds: The axiu for k, the axiu nuber of non zero eleents in tiny atrices, is obtained if whenever soe colun of A is non zero say t), then the t-th row of B is non zero too. This is because then the t-th tiny atrix is non zero. We have the following upper bounds for k in different cases: 6

7 Keivan Borna, Sohrab Aboozarkhani Fard a) If 1, 2 n, then k 1 2. b) If 1 n and 2 > n, then k n 1. c) If 1, 2 > n, then k n n) 2 n). In order to see this when 1, 2 n, let 1 = n i and 2 = n j for soe i, j 0. Then as it was entioned the axiu for k is obtained if non zero eleents of A respectively B) are located in the t-th colun of A respectively row of B). Hence all tiny atrices except the t-th one becoe zero and in this tiny atrix, ni eleents vanish because of zeros in the t-th colun of A) and n i)j eleents vanish because of zeros in the t-th row of B). Hence we have n 2 ni n i)j = n i)n j) = 1 2 non zero eleents in Z totally. If 1 = n i for soe 0 i n and 2 > n, then one can argue siilarly to see that k n 2 ni = nn i) = n 1. Finally if 1, 2 > n, then let 1 = n + i and 2 = n + j for soe i, j > 0. Assue that n non zero eleents of A and B are in A t and B t respectively in the worth case). This will produce n 2 non zero eleents in the t-th tiny atrix. Then each of other i eleents, will produce at ost j non zero eleents. Hence k n 2 + ij = n n) 2 n). Reark 3. Note that the upper bound for k whenever 1, 2 > n is still large. For exaple let 1 = 3n and 2 = 2n i.e., i = 2n and j = n). Now assue that all eleents of A i, A j, A t and B i, B j be non zero. Thus by Proposition 3.3, k 3n 2, whereas we have only 2n 2 non zero eleents in the i-th and the j-th tiny atrices, i.e., k = 2n 2. It is obvious that if non zero eleents of A and B are located in different coluns and rows, then k 2n 2. In Proposition 3.4 we overcoe this proble. Proposition 3.4. Let A and B be two atrices of size n and with 1 > n and 2 > n non zero eleents respectively. Let α = 1 /n. Then the upper bound for k is: { 2 n, k 2 αn, αn αn). in{ 2 αn, n}, 2 > αn In particular, { 2 n, k 2 αn, n. in{ 1, 2 }, 2 > αn. That is, in the worth case, k n 2. Proof. Let 1 = n + i and 2 = n + j for soe i, j > 0. As it was entioned in the proof of Proposition 3.3, in the worth case, one can assue that n non zero eleents of A and B are in A t and B t respectively for soe t). This will produce n 2 non zero eleents in the t-th tiny atrix. In addition, If j α 1)n, then in fact α β 1)n j α β)n for soe 1 β α 1. Thus we have α β 1)n 2 further non zero eleents. Finally 0 j α β 1)n n and i α β 1)n n which yields at ost [j α β 1)n]n further non zero eleents. Hence for j α 1)n we have totally n 2 + α β 1)n 2 + [j α β 1)n]n = n 2 + jn = 2 n non zero eleents. 7

8 A note on the ultiplication of sparse atrices If j > α 1)n, then we have α 1)n 2 further non zero eleents. Finally j α 1)n > 0 and 0 i α 1)n n which gives at ost [i α 1)n]. in{j α 1)n, n} further non zero eleents. Hence for j > α 1)n we have totally n 2 + α 1)n 2 + [i α 1)n]. in{j α 1)n, n} = αn αn). in{ 2 αn, n} non zero eleents. Thus, { k 2 n, 2 αn, αn αn). in{ 2 αn, n}, 2 > αn Now let j > α 1)n. If in{ 2 αn, n} = n, then αn αn). in{ 2 αn, n} = 1 n, and if in{ 2 αn, n} = 2 αn, then αn αn). in{ 2 αn, n} = αn αn). 2 αn) < αn 2 + n. 2 αn) = 2 n, where the inequality is due to the fact that α 1)n 1 n < αn and so 0 1 αn < n. Hence for j > α 1)n, we have k n. in{ 1, 2 }. Exaple 4. Let 1 = 3n and 2 = 6n i.e., i = 2n and j = 5n) and assue that, in the worth case, all eleents of A i1, A i2, A i3 and B i1,, B i6 are non zero. Then k = 3n 2 and it is obvious that if non zero eleents of A and B are located in different coluns and rows, then k 3n 2. This result is also aditted by Proposition 3.4 applied for α = 3. As a corollary of Proposition 3.3 and Proposition 3.4 we have the following result: Corollary 3.5. In the worst case, k 2 n. In fact this upper bound is reachable only in the following three cases: i) 1 > 2, 2 > αn and 1, 2 > n. ii) 2 αn and 1, 2 > n. iii) 1 > n and 2 n. Exaple Let A = and B = Then n = 3, 1 = 3 and 2 = 6. By Proposition 3.3, k 9. One can note that in this situation in fact k = 0. But if one oves each of 1 s to the first row, it will produce three non zero in one of tiny atrices. As a result, if all eleents of the first row of B becoe non zero eg., by couting the first two rows of B), then k = 9. Now in order to apply Partial Algorith 2, note that Ā = , B = and W on B is W = ). Hence Z = ) which confirs the fact that AB = 0 and no ultiplications or additions are applied. 8

9 Keivan Borna, Sohrab Aboozarkhani Fard Exaple 6. Let A = and B = A. Then n =4, 1 = 2 = 10. Then the precise nuber of ultiplications and additions that our algorith does is 26 and the reported nubers are 40 and 36 respectively C := AB = Space coplexity Furtherore, Both storage odels 1 and 2 use ) ) = ) = Oax{ 1, 2 })=On 1.37 ) space. Since the product of two sparse atrices is not necessarily sparse, we have to use a further On 2 ) space for storing the output. Furtherore, we use 3k eory for coputing the nuber of non-zero eleents of tiny atrices, since k< 2 n we deduce that k = On 2.37 ). As a result the total space coplexity is On 2.37 ). 4. Conclusions and future work In this paper we iproved the coplexity of the algorith for ultiplication of two sparse atrices posed in [3]. In fact, when we use the sparse storage odel for storing input atrices, the required tie for ultiplication, for exaple via the algorith in [3, Section 2.4.4], exceeds the tie presented by the naïve algorith. In this paper we present an algorith that stores the input atrices in the sparse storage odel and the tie for ultiplication is less than the naïve and Strassen s algoriths. Furtherore, tight upper bounds for the coplexity of additions is presented. Studying our algorith for other alternatives to store sparse atrices for exaple [8]) is the subject of future work. 5. Coparision of our algorith with the naïve and Strassen s ethods The ai of this section is to copare the required nuber of ultiplications in our algorith, the naïve and Strassen s ethods. In Tables 1 and 2 the coluns N, S, O represent the required nuber of ultiplications in the naïve, Strassen s and our algorith, respectively. We have generated 100 pairs of sparse rando atrices each pair of different size) in the following two situations: 1. When each pair of the sparse atrices are uniforly distributed rando atrices: Figure 1. The average nuber of ultiplications in case 1) in 100 tests Using the functions sprand or sprandn) of MATLAB, one can generate 100 pairs of sparse uniforly or norally) distributed rando atrices. We have written a Java applet for coputing the average nuber of ultiplications that each of the three algoriths naïve, Strassen s and our algorith) are doing. The average nuber of ultiplications for the naïve atrix ultiplications is 30996, for Strassen is and for our algorith is 2226 as it is shown in Figure 1. This observation also shows that our algorith is doing the least nuber of ultiplications 9

10 A note on the ultiplication of sparse atrices Table 1. The nuber of ultiplications for 20 pairs of sparse unifor rando atrices Input Output Input Output No. No. size 1 2 N S O size 1 2 N S O Figure 2. The average nuber of ultiplications in case 2) for 100 tests and has the best perforance. Furtherore a benchark about ultiplication of 20 pairs of such atrices with different sizes is given in Table 1. This table can be read off as follows. For exaple when No. = 15, the required nuber of ultiplications in three algoriths for two sparse atrices A and B of size 48 with unifor distribution aong coluns of A and rows of B) with 778 and 915 non-zero eleents are , and respectively. Table 2. The nuber of ultiplications for 20 pairs of sparse rando atrices Input Output Input Output No. No. size 1 2 N S O size 1 2 N S O When there is no liit on the distribution of sparse atrices: In this case we generate 100 pairs of sparse rando atrices and copute the average nuber of ultiplications that each algorith is doing. Siilar to case 1) the results are presented in Figure 2 and Table??. For exaple when No. = 4, the required nuber of ultiplications in three algoriths for two rando sparse atrices of size 10

11 Keivan Borna, Sohrab Aboozarkhani Fard 42 with 103 and 98 non-zero eleents are 74088, and 227 respectively. Acknowledgeents The paper benefited fro the helpful coents and encourageents of Professor Gilbert Strang. The authors would like to thank hi very uch. The first author is also thankful to the National Elite Foundation of Iran for partial financial support. References [1] G. Strang, Introduction to Linear Algebra Wellesley-Cabridge Press, Wellesley, USA, 2003) [2] V. Strassen, Gaussian eliination is not optial, Nuer. Math. 13, , 1969 [3] A. Horrowithz, J. Sahny, Fundaentals of Data Structures Coputer Science Press, New York, 1983) [4] D. Coppersith, S. Winograd, Matrix ultiplication via arithetic progression, J. Syb. Coput. 9, , 1990 [5] R. Yuster, U. Zwick, Fast sparse atrix ultiplication, ACM T. Alg. 1, 2 13, 2005 [6] D. Coppersith, Rectangular atrix ultiplication revisited, J. Coplexity 13, 42 49, 1997 [7] Z. Tang, R. Duraiswai, N. Guerov, Fast algoriths to copute atrix-vector products for pascal atrices, Technical Reports fro UMIACS UMIACS-TR , 2004 [8] A. Björck, Block bidiagonal decoposition and least square probles, Perspectives in nuerical Analysis, Helsinki, May 27 29,