Estimating the Largest Elements of a Matrix

Transcription

1 Estimating the Largest Elements of a Matrix Samuel Relton samrelton.com blog.samrelton.com Joint work with Nick Higham nick.higham@manchester.ac.uk May 12th, 2016 Sam Relton (Univ. of Man.) Largest elements of A 12th May / 41

2 Defining the problem Applications Basic algorithm and theory Extensions and heuristics Numerical experiments Sam Relton (Univ. of Man.) Largest elements of A 12th May / 41

3 Defining the problem Applications Basic algorithm and theory Extensions and heuristics Numerical experiments Sam Relton (Univ. of Man.) Largest elements of A 12th May / 41

4 The problem We are interested in the max-norm A M = max a ij. i,j Sam Relton (Univ. of Man.) Largest elements of A 12th May / 41

5 The problem We are interested in the max-norm A M = max a ij. i,j Interesting features: Norm is not consistent: AB M A M B M is not always true. It is simple to compute when A is known explicitly. We only use Av for any v. Sam Relton (Univ. of Man.) Largest elements of A 12th May / 41

6 Key observation Our approach relies on a key observation: A M A 1, = max x 0 Ax / x 1. We will start from the mixed subordinate norm estimator derived independently by Boyd (1974) and Tao (1975). This was used by Gu and Miranian (2004) for A = B T C T for rank-revealing Cholesky but was not analyzed. Sam Relton (Univ. of Man.) Largest elements of A 12th May / 41

7 Key observation Our approach relies on a key observation: A M A 1, = max x 0 Ax / x 1. We will start from the mixed subordinate norm estimator derived independently by Boyd (1974) and Tao (1975). This was used by Gu and Miranian (2004) for A = B T C T for rank-revealing Cholesky but was not analyzed. More on this later. Things I won t mention here: Computing max i,j a ij without modulus is also possible. We focus on A R m n but complex matrices are supported. Sam Relton (Univ. of Man.) Largest elements of A 12th May / 41

8 Defining the problem Applications Basic algorithm and theory Extensions and heuristics Numerical experiments Sam Relton (Univ. of Man.) Largest elements of A 12th May / 41

9 Applications - Data mining Factor models in recommender systems (Amazon, Netflix etc.) need kbc T km with B, C rectangular matrices. Sam Relton (Univ. of Man.) Largest elements of A 12th May / 41

10 Applications - Data mining The largest values of BC T M give the strongest recommendations. For example, rows of B are user preferences for films. Action Romance Horror Comedy B = C describes how each film scores in these categories. C = Action Romance Horror Comedy Die Hard Love Actually The Ring Sam Relton (Univ. of Man.) Largest elements of A 12th May / 41

11 Applications - Data mining In general, finding the largest p entries of A T B is called the Maximum All-pairs Dot-product (MAD) search. Forming A T B is expensive and (due to fill-in) may not be possible, whereas calculating A T Bv is easy. Other applications: Link prediction in graphs. Text analysis. Sam Relton (Univ. of Man.) Largest elements of A 12th May / 41

12 Applications - Analysing complex networks What/who is the most important node of this graph? Sam Relton (Univ. of Man.) Largest elements of A 12th May / 41

13 Applications - Analysing complex networks What/who is the most important node of this graph? Clearly number 3 since he connects everyone else! Sam Relton (Univ. of Man.) Largest elements of A 12th May / 41

14 Applications - Analysing complex networks Between each pair of nodes we can define their communicability. It measures how easy it is to send a message from node i to node j. The communicability C(i, j) is a measure of the relative importance of each connection. If we use the matrix exponential exp(a) = k Ak /k!, then (Estrada & Hatano, 2008) C(i, j) = exp(a) i,j. Sam Relton (Univ. of Man.) Largest elements of A 12th May / 41

15 Applications - Analysing complex networks California road network. We want the p 1 most important edges without computing exp(a), which is dense. Sam Relton (Univ. of Man.) Largest elements of A 12th May / 41

16 Application - Optimization (CLIME Estimator) In the CLIME estimator (Constrained l-1 Inverse Matrix Estimator) we would like to solve min Ω 1 subject to Σ n Ω I λ n, where Ω R p p is the precision matrix (inverse covariance matrix). Sam Relton (Univ. of Man.) Largest elements of A 12th May / 41

17 Application - Optimization (CLIME Estimator) In the CLIME estimator (Constrained l-1 Inverse Matrix Estimator) we would like to solve min Ω 1 subject to Σ n Ω I λ n, where Ω R p p is the precision matrix (inverse covariance matrix). Computing Σ n Ω I involves a matrix product. Will become increasing expensive for larger p. Our new method only requires (Σ n Ω I )v and (Σ n Ω I ) T v. Sam Relton (Univ. of Man.) Largest elements of A 12th May / 41

18 Defining the problem Applications Basic algorithm and theory Extensions and heuristics Numerical experiments Sam Relton (Univ. of Man.) Largest elements of A 12th May / 41

19 Basic Algorithm - Theory Based on an iteration for A α,β by Boyd (1974), and Tao (1975). Our problem is reformulated as max F (x) := Ax subject to x S := { x : x 1 1 }. Sam Relton (Univ. of Man.) Largest elements of A 12th May / 41

20 Basic Algorithm - Theory Based on an iteration for A α,β by Boyd (1974), and Tao (1975). Our problem is reformulated as max F (x) := Ax subject to x S := { x : x 1 1 }. Since F and S are convex there exists a subgradient g such that F (v) F (u) + g T (v u), for all u, v S. Sam Relton (Univ. of Man.) Largest elements of A 12th May / 41

21 Basic Algorithm - Theory F (v) F (u) + g T (v u), for all u, v S. Each iteration of our algorithm follows the same procedure: 1. Given a current iterate u, pick a subgradient g. 2. Declare convergence if g is sufficiently small. 3. Move from the u to v, where v maximizes the above. Sam Relton (Univ. of Man.) Largest elements of A 12th May / 41

22 Basic Algorithm - Theory F (v) F (u) + g T (v u), for all u, v S. Each iteration of our algorithm follows the same procedure: 1. Given a current iterate u, pick a subgradient g. 2. Declare convergence if g is sufficiently small. 3. Move from the u to v, where v maximizes the above. After we pick a subgradient g, we need to find v S which maximizes g T (v u) (or equivalently g T v s.t. v 1 1). Clearly v dual (g) so it remains only to choose a subgradient. Sam Relton (Univ. of Man.) Largest elements of A 12th May / 41

23 Basic Algorithm - Theory After some manipulation we find that the set of all subgradients satisfies F A T dual (Ax), where dual (v) = e i and v i = v. Since we know that F A T dual (Ax), we can use the following iteration... Sam Relton (Univ. of Man.) Largest elements of A 12th May / 41

24 Basic Algorithm 1 x = n 1 e 2 for k = 1, 2,... 3 y = Ax 4 if k > 1 5 if y g, γ = g, quit, end 6 end 7 g = A T e i, where y i = y (smallest such i) 8 if g y, γ = y, quit, end 9 x = e j, where g j = g (smallest such j) 10 end Costs two matrix vector products per iteration. We can also extract the (i, j) position of the largest element. Sam Relton (Univ. of Man.) Largest elements of A 12th May / 41

25 Basic Algorithm - Example Here is an example of how the algorithm searches a matrix Sam Relton (Univ. of Man.) Largest elements of A 12th May / 41

26 Basic Algorithm - Example Here is an example of how the algorithm searches a matrix Sam Relton (Univ. of Man.) Largest elements of A 12th May / 41

27 Basic Algorithm - Example Here is an example of how the algorithm searches a matrix Sam Relton (Univ. of Man.) Largest elements of A 12th May / 41

28 Basic Algorithm - Example Here is an example of how the algorithm searches a matrix Note: This are the steps taken by Gaussian elimination with rook pivoting. Sam Relton (Univ. of Man.) Largest elements of A 12th May / 41

29 Basic Algorithm - Properties The algorithm has some interesting convergence properties. The estimate γ A M increases monotonically until convergence. The algorithm searches rows and columns sequentially If the elements of A are i.i.d random variables then the expected number of iterations is less than 1 + exp(1) 3 (Foster, GEPP). Sam Relton (Univ. of Man.) Largest elements of A 12th May / 41

30 Basic Algorithm - Properties The algorithm has some interesting convergence properties. The estimate γ A M increases monotonically until convergence. The algorithm searches rows and columns sequentially If the elements of A are i.i.d random variables then the expected number of iterations is less than 1 + exp(1) 3 (Foster, GEPP). Since the algorithm inspects m n rows/columns we can construct counterexamples. A class of examples where the γ/ A M = ɛ. A class of examples that n 1 steps for an n n input. Sam Relton (Univ. of Man.) Largest elements of A 12th May / 41

31 Basic Algorithm - Counterexample The following 6 6 matrix needs exactly 5 iterations to converge T = The algorithm searches rows/columns sequentially until 24. First iteration: column 1 and row 2. Second iteration: column 2 and row 3... Sam Relton (Univ. of Man.) Largest elements of A 12th May / 41

32 Defining the problem Applications Basic algorithm and theory Extensions and heuristics Numerical experiments Sam Relton (Univ. of Man.) Largest elements of A 12th May / 41

33 Algorithm enhancements There are a number of issues with the basic algorithm: only uses matrix vector operations (Level-2 BLAS). can only explore one column/row in each iteration. can only find the largest element (we want p 1 largest). We propose the following enhancements. Blocking the iterates. Deflating previous entries. Sam Relton (Univ. of Man.) Largest elements of A 12th May / 41

34 Blocked algorithm We want to produce a blocked version of this algorithm, meaning we replace the iterate x by X R n t. The advantages are that: the t columns can communicate to give better estimates. it allows use of level-3 BLAS. Sam Relton (Univ. of Man.) Largest elements of A 12th May / 41

35 Blocked algorithm We want to produce a blocked version of this algorithm, meaning we replace the iterate x by X R n t. The advantages are that: the t columns can communicate to give better estimates. it allows use of level-3 BLAS. There are a couple of small issues to deal with. How to choose the t starting vectors? What to do when we see the same column/row twice? Sam Relton (Univ. of Man.) Largest elements of A 12th May / 41

36 Blocked Algorithm - Starting vectors Using a matrix X R n t as our iterate means that t starting vectors are required. X (:, 1) = [1/n,..., 1/n] T. X (i, 2) = ( 1) i+1 (1 + (i 1)/(n 1)). The rest are random unit vectors. Sam Relton (Univ. of Man.) Largest elements of A 12th May / 41

37 Blocked Algorithm - Starting vectors Using a matrix X R n t as our iterate means that t starting vectors are required. X (:, 1) = [1/n,..., 1/n] T. X (i, 2) = ( 1) i+1 (1 + (i 1)/(n 1)). The rest are random unit vectors. This vector is recommended by Higham to pick out large rows of a matrix. Future work: Investigate how max-plus algebra can be used to choose starting vectors that are better than random. a b = max(a, b) a b = a + b Sam Relton (Univ. of Man.) Largest elements of A 12th May / 41

38 Blocked Algorithm - Replacing vectors Suppose on iteration number 3 we have X (:, i) = X (:, j) for i j. Multiplying them both by A would be a waste of computation. Solution: replace X (:, j) with a random unit vector which hasn t been seen in any previous iteration. This involves keeping track of all previous used unit vectors. Means that vectors can communicate to explore the search space. Sam Relton (Univ. of Man.) Largest elements of A 12th May / 41

39 Finding the p > 1 largest elements To find the p > 1 largest elements we need to keep track of the p largest elements encountered so far. Use block algorithm with t = ceil(αp) columns. Problem: all t columns of our block method try and converge to the same element. Sam Relton (Univ. of Man.) Largest elements of A 12th May / 41

40 Finding the p > 1 largest elements To find the p > 1 largest elements we need to keep track of the p largest elements encountered so far. Use block algorithm with t = ceil(αp) columns. Problem: all t columns of our block method try and converge to the same element. Solution: deflate the largest p entries found so far, i.e. work with the matrix p A p := A a ir,j r e ir ej T r. r=1 This means updating p entries of A p v for a given v each time. Note that this means A will change in each iteration. Sam Relton (Univ. of Man.) Largest elements of A 12th May / 41

41 Defining the problem Applications Basic algorithm and theory Extensions and heuristics Numerical experiments Sam Relton (Univ. of Man.) Largest elements of A 12th May / 41

42 Numerical Experiments Two main types of experiment. Random matrices Normal N(0, 1) distributed elements Inverse of such matrices Real-life application matrices MAD search Networks We track a number of metrics: ψ = γ/ A M which is in [0, 1] the underestimation ratio. π the number of iterations required. Machine details: 2x Intel i7 2.7GHz, 8GB RAM, Ubuntu 13.10, MATLAB 2014b. Sam Relton (Univ. of Man.) Largest elements of A 12th May / 41

43 Experiments on random matrices - p = 1 Using block algorithm for p = 1 we compute the metrics over 1000 random matrices of each type. We will use random matrices of the form: randn(100) inv(randn(100)) More types of matrix considered in the paper. Sam Relton (Univ. of Man.) Largest elements of A 12th May / 41

44 p = 1 randn(100) 1000 runs t ψ min ψ avg % exact π avg π max % improve Sam Relton (Univ. of Man.) Largest elements of A 12th May / 41

45 p = 1 inv(randn(100)) 1000 runs t ψ min ψ avg % exact π avg π max % improve Sam Relton (Univ. of Man.) Largest elements of A 12th May / 41

46 Real-life MAD search Recall the MAD search: Find the largest element of A T B. We will look at the following cases: MovieLens: m = 65, 133, n = 71, 567 nnz = 10, 000, 054 Form SVD with 150 singular values UΣV T = R then take A = (UΣ) T, B = V T. Sandia/ASIC 680k: n = 682, 862 nnz = 2, 638, 997 Find largest elements of A T A SNAP/as-Skitter: n = 1, 696, 415 nnz = 11, 095, 298 Find largest elements of A T A Sam Relton (Univ. of Man.) Largest elements of A 12th May / 41

47 MAD Search - MovieLens Parameters used: t = 10. Time to compute A T B and take maximum: 88 secs Time for new algorithm 0.4 secs: (found largest element) Speedup 200 Sam Relton (Univ. of Man.) Largest elements of A 12th May / 41

48 MAD Search - ASIC 680k Parameters used: t = 10. Time to compute A T A and take maximum: 1.6e4 secs Time for new algorithm 1.0 secs: (found largest element) Speedup 16,000 Sam Relton (Univ. of Man.) Largest elements of A 12th May / 41

49 MAD Search - as-skitter Parameters used: t = 10. Time to compute A T A and take maximum: 2.4e4 secs Time for new algorithm 3.6 secs: (found second largest element) Speedup 6,600 Sam Relton (Univ. of Man.) Largest elements of A 12th May / 41

50 MAD search - Summary Exact method (for comparison): Compute A T Ae i for i = 1 : n and store largest entries. Time (secs) Matrix ψ Exact Alg New Alg Speedup Movielens Sandia/ASIC 680k 1 1.6e , 000 SNAP/as-Skitter e , 600 Finding the p = 5 using deflation gets all elements correct. Sam Relton (Univ. of Man.) Largest elements of A 12th May / 41

51 Real-life networks Find the p = 10 strongest links in the following networks. SNAP/ca-AstroPh: n = 18, 772 nnz = 396, 160 SNAP/ca-CondMat: n = 23, 133 nnz = 186, 936 MUFC Twitter: n = 148, 918 nnz = 193, 032 exp(a)v computed using expmv (Higham and Al-Mohy). Exact method (for comparison): Compute exp(a)e i for i = 1 : n and store largest entries. Parameter α = 3 used meaning that t = 30 columns are used in each iteration. Sam Relton (Univ. of Man.) Largest elements of A 12th May / 41

52 Real-life networks - SNAP/ca-AstroPh Parameters used: α = 3, p = 10, t = 30. Time to compute exp(a) and take maximum: secs Time for new algorithm 1.7 secs Speedup 140 Sam Relton (Univ. of Man.) Largest elements of A 12th May / 41

53 Real-life networks - SNAP/ca-CondMat Parameters used: α = 3, p = 10, t = 30. Time to compute exp(a) and take maximum: secs Time for new algorithm 1.2 secs Speedup 170 Sam Relton (Univ. of Man.) Largest elements of A 12th May / 41

54 Real-life networks - MUFC Twitter Parameters used: α = 3, p = 10, t = 30. Time to compute exp(a) and take maximum: secs Time for new algorithm 3.2 secs Speedup 1720 Sam Relton (Univ. of Man.) Largest elements of A 12th May / 41

55 Real-life networks - Summary Time (secs) Matrix Correct Elems Exact Alg New Alg Speedup SNAP/ca-AstroPh SNAP/ca-CondMat MUFC Twitter All the largest elements found correctly Significant speedups Sam Relton (Univ. of Man.) Largest elements of A 12th May / 41

56 Recommendations We learned from our experiments that: Using deflation is always preferable. At least as accurate and reliable (usually more so) Tiny performance hit compared to level-3 BLAS operations To increase accuracy further Try and find more largest elements than required Use deflation Sam Relton (Univ. of Man.) Largest elements of A 12th May / 41

57 Conclusions Finding the largest element of a matrix is an important problem with numerous applications. Our algorithm is a black box applicable to any matrix where Av can be formed (in contrast to other specific algorithms). Algorithm performs well on real-life problems Code available on Github: github.com/sdrelton/matrix-est-maxelts Sam Relton (Univ. of Man.) Largest elements of A 12th May / 41

58 References David W. Boyd, The power method for l p norms. Linear Algebra Appl., 9:95 101, Ernesto Estrada and Naomichi Hatano, Communicability in complex networks. Physical Review E, 77:036111, Leslie V. Foster, The growth factor and efficiency of Gaussian elimination with rook pivoting. J. Comput. Appl. Math, 86: , M. Gu and L. Miranian, Strong rank revealing Cholesky factorization. Electron. Trans. Numer. Anal., 17:76 92, Pham Dinh Tao, Convergence of a subgradient method for computing the bound norm of matrices. Linear Algebra Appl., 62: , Sam Relton (Univ. of Man.) Largest elements of A 12th May / 41