Using R for Iterative and Incremental Processing

Transcription

1 Using R for Iterative and Incremental Processing Shivaram Venkataraman, Indrajit Roy, Alvin AuYoung, Robert Schreiber UC Berkeley and HP Labs UC BERKELEY

2 Big Data, Complex Algorithms PageRank (Dominant eigenvector) Recommendations (Matrix factorization) Anomaly detection (Top-K eigenvalues) User Importance (Vertex Centrality) 6/29/2012 2

3 Big Data, Complex Algorithms PageRank (Dominant eigenvector) Machine learning + Graph algorithms Recommendations (Matrix factorization) Iterative Linear Algebra Operations Anomaly detection (Top-K eigenvalues) User Importance (Vertex Centrality) 6/29/2012 3

4 PageRank Using Matrices N s P 1 P 1 s P 1 P 1 N p M p Z Dominant eigenvector M = modified web graph matrix p = PageRank vector Simplified algorithm: repeat { p = M*p + Z} 6/29/2012 4

5 Breadth-first Search Using Matrices A B C D E X A B C D E A B C D G E G = adjacency matrix X = BFS vector Simplified algorithm: repeat { X = G*X } 6/29/2012 5

6 Breadth-first Search Using Matrices A B D A B D C E C E X A B C D E * * * 0 0 A B C D E A B C D G E G = adjacency matrix X = BFS vector Simplified algorithm: repeat { X = G*X } 6/29/2012 6

7 Breadth-first Search Using Matrices A B D A B D A B D C E C E C E X * * * 0 0 * * * * 0 A B C D E A B C D E A B C D E A B C D G E G = adjacency matrix X = BFS vector Simplified algorithm: repeat { X = G*X } 6/29/2012 7

8 Breadth-first Search Using Matrices A B D A B D A B D A B D C E C E C E C E X * * * 0 0 * * * * 0 * * * * * A B C D E A B C D E A B C D E A B C D E A B C D G E G = adjacency matrix X = BFS vector Simplified algorithm: repeat { X = G*X } 6/29/2012 8

9 Breadth-first Search Using Matrices * * * 0 0 A B C D E A B C D E Matrix operations A B C D * * * * 0 A B C D E * * * * * A B C D E Easy to express Efficient to implement E G = adjacency matrix X = BFS vector Simplified algorithm: repeat { X = G*X } 6/29/2012 9

10 Linear Algebra on Existing Frameworks Matrix Operations: Structured, coarse grained Need global state 6/29/

11 Linear Algebra on Existing Frameworks Matrix Operations: Structured, coarse grained Need global state Data-parallel frameworks MapReduce/Dryad Process each record in parallel Use case: Computing sufficient statistics 6/29/

12 Linear Algebra on Existing Frameworks Matrix Operations: Structured, coarse grained Need global state Data-parallel frameworks MapReduce/Dryad Process each record in parallel Use case: Computing sufficient statistics Graph-centric frameworks Pregel/GraphLab Process each vertex in parallel Use case: Graph models 6/29/

13 Challenge 1 Sparse Matrices 6/29/

14 Challenge 1 Sparse Matrices 6/29/

15 Challenge 1 Sparse Matrices 6/29/

16 Block density (normalized ) Challenge 1 Sparse Matrices LiveJournal Netflix ClueWeb-1B Block ID 6/29/

17 Block density (normalized ) Challenge 1 Sparse Matrices LiveJournal Netflix ClueWeb-1B Block ID 6/29/

18 Block density (normalized ) Challenge 1 Sparse Matrices LiveJournal Netflix ClueWeb-1B x more elements Computation imbalance Block ID 6/29/

19 Challenge 2 Incremental Updates New movie ratings Refine recommendations Better suggestions Incremental computation on consistent view of data 6/29/

20 Presto Framework for large-scale iterative linear algebra Extend R for scalability and incremental updates 6/29/

21 Outline Motivation Programming model Design Applications and Results 6/29/

22 Programming Model One data structure: Distributed Array A darray() 6/29/

23 Programming Model Iteration: foreach 6/29/

24 Programming Model Iteration: foreach Compute Compute Compute Compute 6/29/

25 Programming Model Incremental updates: onchange, update Compute Compute Compute Compute 6/29/

26 Programming Model Incremental updates: onchange, update Data Updated Compute Compute Compute Compute 6/29/

27 Programming Model Incremental updates: onchange, update Data Updated Compute Compute Compute Compute 6/29/

28 PageRank Using Presto N s P 1 P 1 s P 1 P 1 N P M P_old Z M darray(dim=c(n,n),blocks=(s,n)) P darray(dim=c(n,1),blocks=(s,1)) while(..){ foreach(i,1:len, calculate(p=splits(p,i),m=splits(m,i), x=splits(p_old),z=splits(z,i)) { p (m*x)+z } ) } P_old P 6/29/

29 PageRank Using Presto N s P 1 P 1 s P 1 P 1 N P M P_old Z M darray(dim=c(n,n),blocks=(s,n)) P darray(dim=c(n,1),blocks=(s,1)) while(..){ foreach(i,1:len, calculate(p=splits(p,i),m=splits(m,i), Create Distributed Array x=splits(p_old),z=splits(z,i)) { p (m*x)+z } ) } P_old P 6/29/

30 PageRank Using Presto N s P 1 P 1 s P 1 P 1 N P M P_old Z M darray(dim=c(n,n),blocks=(s,n)) P darray(dim=c(n,1),blocks=(s,1)) while(..){ foreach(i,1:len, calculate(p=splits(p,i), m=splits(m,i), x=splits(p_old), z=splits(z,i)) { p (m*x)+z } ) } P_old P 6/29/

31 PageRank Using Presto N s P 1 P 1 s P 1 P 1 N P M P_old Z M darray(dim=c(n,n),blocks=(s,n)) P darray(dim=c(n,1),blocks=(s,1)) while(..){ Execute function in a cluster foreach(i,1:len, calculate(p=splits(p,i), m=splits(m,i), x=splits(p_old), z=splits(z,i)) { p (m*x)+z } ) } P_old P 6/29/

32 PageRank Using Presto N s P 1 P 1 s P 1 P 1 N P M P_old Z M darray(dim=c(n,n),blocks=(s,n)) P darray(dim=c(n,1),blocks=(s,1)) while(..){ Execute function in a cluster foreach(i,1:len, calculate(p=splits(p,i), m=splits(m,i), x=splits(p_old), z=splits(z,i)) { p (m*x)+z } Pass array partitions ) } P_old P 6/29/

33 Incremental PageRank N s P 1 P 1 s P 1 P 1 N P M P_old Z M darray(dim=c(n,n),blocks=(s,n)) P darray(dim=c(n,1),blocks=(s1)) onchange(m) { while(..){ foreach(i,1:len, calculate(p=splits(p,i), m=splits(m,i), x=splits(p_old), z=splits(z,i)) { p (m*x)+z update(p) }) P_old P 6/29/ }}

34 Incremental PageRank N s P 1 P 1 s P 1 P 1 N P M P_old Z M darray(dim=c(n,n),blocks=(s,n)) P darray(dim=c(n,1),blocks=(s1)) onchange(m) { while(..){ foreach(i,1:len, calculate(p=splits(p,i), m=splits(m,i), x=splits(p_old), z=splits(z,i)) { p (m*x)+z update(p) }) P_old P }} Execute when data changes 6/29/

35 Incremental PageRank N s P 1 P 1 s P 1 P 1 N P M P_old Z M darray(dim=c(n,n),blocks=(s,n)) P darray(dim=c(n,1),blocks=(s1)) onchange(m) { while(..){ foreach(i,1:len, calculate(p=splits(p,i), m=splits(m,i), x=splits(p_old), z=splits(z,i)) { p (m*x)+z update(p) }) P_old P }} Execute when data changes Update page rank vector 6/29/

36 Outline Motivation Programming model Design Applications and Results 6/29/

37 Dynamic Partitioning of Matrices 6/29/

38 Dynamic Partitioning of Matrices Profile execution 6/29/

39 Dynamic Partitioning of Matrices Profile execution 6/29/

40 Dynamic Partitioning of Matrices Profile execution Partition 6/29/

41 Dynamic Partitioning of Matrices Profile execution Partition 6/29/

42 Dynamic Partitioning of Matrices Profile execution Partition 6/29/

43 Dynamic Partitioning of Matrices Profile execution Partition Programmers specify size invariants. 6/29/

44 Dynamic Partitioning of Matrices Up to 2x performance improvement 6/29/

45 Incremental Updates Using Consistent Snapshots Web Graph Adjacency Matrix R P Q S /29/

46 Incremental Updates Using Consistent Snapshots Web Graph Adjacency Matrix R P Q S onchange(m 1 ) 6/29/

47 Incremental Updates Using Consistent Snapshots Web Graph Adjacency Matrix Page Rank R P Q S update P 1 6/29/

48 Incremental Updates Using Consistent Snapshots Web Graph Adjacency Matrix Page Rank R P Q S /29/

49 Incremental Updates Using Consistent Snapshots Web Graph Adjacency Matrix Page Rank R P Q S onchange(m 2 ) /29/

50 Incremental Updates Using Consistent Snapshots Web Graph P R S Q Adjacency Matrix update Page Rank /29/

51 Versioned Distributed Arrays Mechanics of versioning update: Increment version number onchange: Bind a version number for the array before executing the handler 6/29/

52 Outline Motivation Programming model Design Applications and Results 6/29/

53 Applications Implemented in Presto Application Algorithm Presto LOC PageRank Eigenvector calculation 41 Triangle counting Top-K eigenvalues 121 Netflix recommendation Matrix factorization 130 Centrality measure Graph algorithm 132 k-path connectivity Graph algorithm 30 k-means Clustering 71 Sequence alignment Smith-Waterman 64 6/29/

54 Applications Implemented in Presto Application Algorithm Presto LOC PageRank Eigenvector calculation 41 Triangle counting Top-K eigenvalues 121 Fewer than 140 lines of code Netflix recommendation Matrix factorization 130 Centrality measure Graph algorithm 132 k-path connectivity Graph algorithm 30 k-means Clustering 71 Sequence alignment Smith-Waterman 64 6/29/

55 Time (sec) Presto is Fast! PageRank per-iteration execution time 1000 Presto Hadoop-InMem Number of workers Data: 100M nodes, 1.2B edges. Setup: 10G network. 12 cores, 96GB RAM. 6/29/

56 Time (sec) Presto is Fast! PageRank per-iteration execution time 1000 Presto Hadoop-InMem 100 More than 20x faster than Hadoop (w/ in-memory storage) Number of workers Data: 100M nodes, 1.2B edges. Setup: 10G network. 12 cores, 96GB RAM. 6/29/

57 More in the Paper Memory management, caching of partitions Scheduling operations Storage driver interface to HBase Fault tolerance 6/29/

58 Conclusion Linear Algebra is a powerful abstraction Easily express machine learning, graph algorithms Challenges: Sparse matrices, Incremental data Presto prototype extends R Open source version soon! 6/29/