Math 304 (Spring 010) - Lecture Emre Mengi Department of Mathematics Koç University emengi@ku.edu.tr Lecture - Floating Point Operation Count p.1/10
Efficiency of an algorithm is determined by the total # of,,, required. Lecture - Floating Point Operation Count p./10
Efficiency of an algorithm is determined by the total # of,,, required. Crudeness in floating point operation (flop) count Lecture - Floating Point Operation Count p./10
Efficiency of an algorithm is determined by the total # of,,, required. Crudeness in floating point operation (flop) count Time required for data transfers is ignored. Lecture - Floating Point Operation Count p./10
Efficiency of an algorithm is determined by the total # of,,, required. Crudeness in floating point operation (flop) count Time required for data transfers is ignored. All of the operations,,, are considered of same computational difficulty. In reality, are more expensive. Lecture - Floating Point Operation Count p./10
Inner (or dot) product : Let f : R n R be defined as where a = [ f(x) = a 1 x 1 + a x +...a n x n = a T x ] T [ ] T a 1... a n R n and x = x 1... x n R n. Lecture - Floating Point Operation Count p.3/10
Inner (or dot) product : Let f : R n R be defined as where a = [ f(x) = a 1 x 1 + a x +...a n x n = a T x ] T [ ] T a 1... a n R n and x = x 1... x n R n. Pseudocode to compute f(x) f 0 for j = 1, n do f f + a j x j }{{} flops Return f Lecture - Floating Point Operation Count p.3/10
Inner (or dot) product : Let f : R n R be defined as f(x) = a 1 x 1 + a x +...a n x n = a T x where a = [ a 1... a n ] T R n and x = [ x 1... x n ] T R n. Pseudocode to compute f(x) f 0 for j = 1, n do f f + a j x j }{{} flops Return f Total flop count : flops per iteration for j = 1,...,n Total # of flops = n j=1 = n Lecture - Floating Point Operation Count p.3/10
Matrix-vector product : Let g : R n R m be defined as g(x) = Ax = x 1 A 1 + x A + + x n A n [ ] T where A = A 1... A n is an m n real matrix with [ ] T A 1,...,A n R m and x = x 1... x n R n. Lecture - Floating Point Operation Count p.4/10
Matrix-vector product : Let g : R n R m be defined as g(x) = Ax = x 1 A 1 + x A + + x n A n [ ] T where A = A 1... A n is an m n real matrix with [ ] T A 1,...,A n R m and x = x 1... x n R n. e.g. 1 1 0 1 3 1 1 = 1 3 1 0 1 +1 1 = 0 1 10 Lecture - Floating Point Operation Count p.4/10
Pseudocode to compute g(x) = Ax Given an m n real matrix A and x R n. g 0 (where g R n ) for j = 1, n do g g + x j A j }{{} m flops Return g Lecture - Floating Point Operation Count p.5/10
Pseudocode to compute g(x) = Ax Given an m n real matrix A and x R n. g 0 (where g R n ) for j = 1, n do g g + x j A j }{{} m flops Return g Above g + x j A j requires m addition and m multiplication for each j. Lecture - Floating Point Operation Count p.5/10
Pseudocode to compute g(x) = Ax Given an m n real matrix A and x R n. g 0 (where g R n ) for j = 1, n do g g + x j A j }{{} m flops Return g Above g + x j A j requires m addition and m multiplication for each j. Total flop count : m flops per iteration for j = 1,...,n Total # of flops = n j=1 m = mn Lecture - Floating Point Operation Count p.5/10
Inner product view of the matrix-vector product g(x) = Ax. g(x) = 6 4 Ā 1 x Ā x. Ā m x 3 7 5 = 6 4 a 11 x 1 + a 1 x + + a 1n x n a 1 x 1 + a x + + a nn x n. a m1 x 1 + a m x + + a mn x n 3 7 5 where A = 6 4 Ā 1 Ā. Ā m 3 7 5 and Ā 1,...,Ā m are the rows of A and a ij is the entry of A at the ith row and jth column. Lecture - Floating Point Operation Count p.6/10
Inner product view of the matrix-vector product g(x) = Ax. g(x) = 6 4 Ā 1 x Ā x. Ā m x 3 7 5 = 6 4 a 11 x 1 + a 1 x + + a 1n x n a 1 x 1 + a x + + a nn x n. a m1 x 1 + a m x + + a mn x n 3 7 5 where A = 6 4 Ā 1 Ā. Ā m 3 7 5 and Ā 1,...,Ā m are the rows of A and a ij is the entry of A at the ith row and jth column. e.g. 6 4 1 1 0 1 3 1 3 7 6 5 4 1 3 7 5 = 6 4 ()() + (1)( ) + ( )(1) (1)() + (0)( ) + ( 1)(1) (3)() + ( 1)( ) + ()(1) 3 7 5 = 6 4 0 1 10 3 7 5 Lecture - Floating Point Operation Count p.6/10
Pseudocode to compute g(x) = Ax exploiting the inner-product view Given an m n real matrix A and x R n. g 0 (where g R n ) for i = 1, m do for j = 1, n do g i g i + a ij x j }{{} flops Return g Lecture - Floating Point Operation Count p.7/10
Pseudocode to compute g(x) = Ax exploiting the inner-product view Given an m n real matrix A and x R n. g 0 (where g R n ) for i = 1, m do for j = 1, n do g i g i + a ij x j }{{} flops Return g Total flop count : flops per iteration for each j = 1,...,n and i = 1,...,m Total # of flops = m n i=1 j=1 = m i=1 n = mn Lecture - Floating Point Operation Count p.7/10
Matrix-matrix product : Given an n p matrix A and a p m matrix X. The product B = AX is an n m matrix and defined as b ij = Ā i X j = p a ik x kj k=1 where Āi is the ith row of A, X j is the jth column of X and b ij, a ij, x ij denote the (i, j)-entry of B, A and X,respectively. Lecture - Floating Point Operation Count p.8/10
Matrix-matrix product : Given an n p matrix A and a p m matrix X. The product B = AX is an n m matrix and defined as b ij = Ā i X j = p a ik x kj k=1 where Āi is the ith row of A, X j is the jth column of X and b ij, a ij, x ij denote the (i, j)-entry of B, A and X,respectively. e.g. 4 1 1 0 3 5 4 1 1 1 3 5 = 4 ( 1) + 1(1) (1) + 1( ) 1( 1) + 0(1) 1(1) + 0( ) 3 5 = 4 1 0 1 1 3 5 Lecture - Floating Point Operation Count p.8/10
Pseudocode to compute the product B = AX Given n p and p m matrices A and X. B 0 for i = 1, n do for j = 1, m do for k = 1, p do b ij b ij + a ik x kj }{{} flops Return g Lecture - Floating Point Operation Count p.9/10
Pseudocode to compute the product B = AX Given n p and p m matrices A and X. B 0 for i = 1, n do for j = 1, m do for k = 1, p do b ij b ij + a ik x kj }{{} flops Return g Total flop count : flops per iteration for each k = 1,...,p, j = 1,...,m and i = 1,...,n Total # of flops = P n i=1 P m j=1 P p k=1 = nmp Lecture - Floating Point Operation Count p.9/10
Big-O notation Floating Point Operation Count Lecture - Floating Point Operation Count p.10/10
Big-O notation Floating Point Operation Count The inner product a T x requires n = O(n) flops (linear # of flops). Lecture - Floating Point Operation Count p.10/10
Big-O notation Floating Point Operation Count The inner product a T x requires n = O(n) flops (linear # of flops). The matrix-vector product Ax for a square matrix A (with m = n) requires n = O(n ) flops (quadratic # of flops). Lecture - Floating Point Operation Count p.10/10
Big-O notation Floating Point Operation Count The inner product a T x requires n = O(n) flops (linear # of flops). The matrix-vector product Ax for a square matrix A (with m = n) requires n = O(n ) flops (quadratic # of flops). The matrix-matrix product AX for square n n matrices A and X (with m = n = p) requires n 3 = O(n 3 ) flops (cubic # of flops). Lecture - Floating Point Operation Count p.10/10
Big-O notation Floating Point Operation Count The inner product a T x requires n = O(n) flops (linear # of flops). The matrix-vector product Ax for a square matrix A (with m = n) requires n = O(n ) flops (quadratic # of flops). The matrix-matrix product AX for square n n matrices A and X (with m = n = p) requires n 3 = O(n 3 ) flops (cubic # of flops). The notation g(n) = O(f(n)) means asymptotically f(n) scaled up to a constant grows at least as fast as g(n), i.e. g(n) = O(f(n)) if there exists an n 0 and c such that g(n) cf(n) for all n n 0 Lecture - Floating Point Operation Count p.10/10
Big-O notation Floating Point Operation Count The inner product a T x requires n = O(n) flops (linear # of flops). The matrix-vector product Ax for a square matrix A (with m = n) requires n = O(n ) flops (quadratic # of flops). The matrix-matrix product AX for square n n matrices A and X (with m = n = p) requires n 3 = O(n 3 ) flops (cubic # of flops). The notation g(n) = O(f(n)) means asymptotically f(n) scaled up to a constant grows at least as fast as g(n), i.e. g(n) = O(f(n)) if there exists an n 0 and c such that g(n) cf(n) for all n n 0 Examples: n = O(n) as well as n = O(n ) and n = O(n 3 ) Lecture - Floating Point Operation Count p.10/10
Big-O notation Floating Point Operation Count The inner product a T x requires n = O(n) flops (linear # of flops). The matrix-vector product Ax for a square matrix A (with m = n) requires n = O(n ) flops (quadratic # of flops). The matrix-matrix product AX for square n n matrices A and X (with m = n = p) requires n 3 = O(n 3 ) flops (cubic # of flops). The notation g(n) = O(f(n)) means asymptotically f(n) scaled up to a constant grows at least as fast as g(n), i.e. g(n) = O(f(n)) if there exists an n 0 and c such that g(n) cf(n) for all n n 0 Examples: n = O(n) as well as n = O(n ) and n = O(n 3 ) n = O(n ) as well as n = O(n 3 ), but n is not O(n). Lecture - Floating Point Operation Count p.10/10