Computing approximate PageRank vectors by Basis Pursuit Denoising

Transcription

1 Computing approximate PageRank vectors by Basis Pursuit Denoising Michael Saunders Systems Optimization Laboratory, Stanford University Joint work with Holly Jin, LinkedIn Corp SIAM Annual Meeting San Diego, July 7 11, 2008 SIAM AN08 1/36

2 Outline 1 Introduction 2 PageRank 3 Basis Pursuit Denoising 4 Sparse PageRank (Neumann) 5 Sparse PageRank (LPdual) 6 Preconditioning 7 Summary SIAM AN08 2/36

3 Abstract The PageRank eigenvector problem involves a square system Ax = b in which x is naturally nonnegative and somewhat sparse (depending on b). We seek an approximate x that is nonnegative and extremely sparse. We experiment with an active-set optimization method designed for the dual of the BPDN problem, and find that it tends to extract the important elements of x in a greedy fashion. Supported by the Office of Naval Research SIAM AN08 3/36

4 Introduction Many applications (statistics, signal analysis, imaging) seek sparse solutions to square or rectangular systems Ax b x sparse SIAM AN08 4/36

5 Introduction Many applications (statistics, signal analysis, imaging) seek sparse solutions to square or rectangular systems Ax b x sparse Basis Pursuit Denoising (BPDN) seeks sparsity by solving min 1 2 Ax b 2 + λ x 1 SIAM AN08 4/36

6 Introduction Many applications (statistics, signal analysis, imaging) seek sparse solutions to square or rectangular systems Ax b x sparse Basis Pursuit Denoising (BPDN) seeks sparsity by solving min 1 2 Ax b 2 + λ x 1 Personalized PageRank eigenvectors involve square systems (I αh T )x = v H, v sparse where 0 < α < 1 He e x 0, sparse SIAM AN08 4/36

7 Introduction Many applications (statistics, signal analysis, imaging) seek sparse solutions to square or rectangular systems Ax b x sparse Basis Pursuit Denoising (BPDN) seeks sparsity by solving min 1 2 Ax b 2 + λ x 1 Personalized PageRank eigenvectors involve square systems (I αh T )x = v H, v sparse where 0 < α < 1 He e x 0, sparse Perhaps BPDN can find a very sparse approximate x SIAM AN08 4/36

8 PageRank SIAM AN08 5/36

9 H = 0 1 1/3 1/3 1/3 B 1/2 1/ /2 1/2 PageRank Langville and Meyer 2006 A Hyperlink matrix He e SIAM AN08 6/36

10 H = 0 1 1/3 1/3 1/3 B 1/2 1/ /2 1/2 v = 1 n e or e i PageRank Langville and Meyer 2006 A Hyperlink matrix He e Teleportation vector or personalization vector e T v = 1 SIAM AN08 6/36

11 H = 0 1 1/3 1/3 1/3 B 1/2 1/ /2 1/2 v = 1 n e or e i PageRank Langville and Meyer 2006 A Hyperlink matrix He e Teleportation vector or personalization vector e T v = 1 S = H + av T Stochastic matrix Se = e SIAM AN08 6/36

12 H = 0 1 1/3 1/3 1/3 B 1/2 1/ /2 1/2 v = 1 n e or e i PageRank Langville and Meyer 2006 A Hyperlink matrix He e Teleportation vector or personalization vector e T v = 1 S = H + av T Stochastic matrix Se = e G = αs + (1 α)ev T Google matrix α = 0.85 say Ge = e SIAM AN08 6/36

13 H = 0 1 1/3 1/3 1/3 B 1/2 1/ /2 1/2 v = 1 n e or e i PageRank Langville and Meyer 2006 A Hyperlink matrix He e Teleportation vector or personalization vector e T v = 1 S = H + av T Stochastic matrix Se = e G = αs + (1 α)ev T Google matrix α = 0.85 say Ge = e eigenvector linear system G T x = x (I αh T )x = v SIAM AN08 6/36

14 H = 0 1 1/3 1/3 1/3 B 1/2 1/ /2 1/2 v = 1 n e or e i PageRank Langville and Meyer 2006 A Hyperlink matrix He e Teleportation vector or personalization vector e T v = 1 S = H + av T Stochastic matrix Se = e G = αs + (1 α)ev T Google matrix α = 0.85 say Ge = e eigenvector linear system G T x = x (I αh T )x = v In both cases, x x/e T x (and x 0) SIAM AN08 6/36

15 Web matrices H Name n (pages) nnz (links) i (v = e i ) csstanford cs.stanford.edu Stanford cs.stanford.edu 6753 calendus.stanford.edu Stanford-Berkeley unknown All data collected around 2001 Cleve Moler Sep Kamvar David Gleich (I αh T )x = v SIAM AN08 7/36

16 Power method on G T x = x Stanford, n = , v = e 6753 (calendus.stanford.edu) Time (secs) vs α = 0.1 : 0.01 : α SIAM AN08 8/36

17 Power method Stanford-Berkeley, n = , v = e Time (secs) vs α = 0.1 : 0.01 : α SIAM AN08 9/36

18 Exact x(α) csstanford, n = 9914, v = e 4 (cs.stanford.edu) x j (α), j = 1:n Stanford alpha=0.850 v=e(4) xtrue=p\v α = 0.85 SIAM AN08 10/36

19 Exact x(α) Stanford, n = , v = e (cs.stanford.edu) x j (α), j = 1:n α = 0.85 SIAM AN08 11/36

20 Sparsity of x(α) Stanford, n = , v = e (cs.stanford.edu) x > 250/n x > 500/n nz = nz = 3076 α = 0.1 : 0.01 : 0.99 (90 values) SIAM AN08 12/36

21 n = x > 1000/n x > 2000/n nz = nz = 2305 α = 0.1 : 0.01 : 0.99 (90 values) SIAM AN08 13/36

22 Basis Pursuit Denoising SIAM AN08 14/36

23 Lasso(ν) Tibshirani 1996 min L1 sparse x min 1 2 Ax b 2 st x 1 ν A = SIAM AN08 15/36

24 Lasso(ν) Tibshirani 1996 min L1 sparse x min 1 2 Ax b 2 st x 1 ν A = Basis Pursuit Chen, Donoho & S 1998 min x 1 st Ax = b A = fast operator (Ax, A T y are efficient) SIAM AN08 15/36

25 Lasso(ν) Tibshirani 1996 min L1 sparse x min 1 2 Ax b 2 st x 1 ν A = Basis Pursuit Chen, Donoho & S 1998 min x 1 st Ax = b A = fast operator (Ax, A T y are efficient) BPDN(λ) Same paper min λ x Ax b 2 fast operator, any shape SIAM AN08 15/36

26 BP and BPDN algorithms min λ x Ax b 2 2 OMP Davis, Mallat et al 1997 Greedy BPDN-interior Chen, Donoho & S, 1998, 2001 Interior, CG PDSCO, PDCO Saunders 1997, 2002 Interior, LSQR BCR Sardy, Bruce & Tseng 2000 Orthogonal blocks Homotopy Osborne et al 2000 Active-set, all λ LARS Efron, Hastie, Tibshirani 2004 Active-set, all λ STOMP Donoho, Tsaig, et al 2006 Double greedy l1 ls Kim, Koh, Lustig, Boyd et al 2007 Primal barrier, PCG GPSR Figueiredo, Nowak & Wright 2007 Gradient Projection BCLS Friedlander 2006 Projection, LSQR SPGL1 van den Berg & Friedlander 2007 Spectral GP, all λ BPdual Friedlander & S 2007 Active-set on dual LPdual Friedlander & S 2007 For x 0 SIAM AN08 16/36

27 x 1 when x 0 Suggests regularized LP problems: LPprimal(λ) min x, y e T x λ y 2 st Ax + λy = b, x 0 SIAM AN08 17/36

28 x 1 when x 0 Suggests regularized LP problems: LPprimal(λ) min x, y e T x λ y 2 st Ax + λy = b, x 0 LPdual(λ) min y b T y λ y 2 st A T y e SIAM AN08 17/36

29 x 1 when x 0 Suggests regularized LP problems: LPprimal(λ) min x, y e T x λ y 2 st Ax + λy = b, x 0 LPdual(λ) min y b T y λ y 2 st A T y e Friedlander and S 2007: New active-set Matlab codes SIAM AN08 17/36

30 LPdual solver for Ax b, x 0 min y b T y λ y 2 st A T y e SIAM AN08 18/36

31 LPdual solver for Ax b, x 0 min y b T y λ y 2 st A T y e A few active constraints: B T y = e Initially B is empty, y = 0 Select columns of B in almost greedy manner SIAM AN08 18/36

32 LPdual solver for Ax b, x 0 min y b T y λ y 2 st A T y e A few active constraints: B T y = e Initially B is empty, y = 0 Select columns of B in almost greedy manner Main work per itn: Solve min Bx g Form dy = (g Bx)/λ Form dz = A T dy SIAM AN08 18/36

33 LPdual solver for Ax b, x 0 min y b T y λ y 2 st A T y e A few active constraints: B T y = e Initially B is empty, y = 0 Select columns of B in almost greedy manner Main work per itn: R Solve min Bx g Update QB = 0 Form dy = (g Bx)/λ Form dz = A T dy «without Q SIAM AN08 18/36

34 Sparse PageRank with Neumann Series SIAM AN08 19/36

35 Neumann series (I αh T )x = v (*) x = (I αh T ) 1 v = (I + αh T + (αh T ) 2 + )v A sum of nonnegative terms SIAM AN08 20/36

36 Neumann series (I αh T )x = v (*) x = (I αh T ) 1 v = (I + αh T + (αh T ) 2 + )v A sum of nonnegative terms Equivalent to Jacobi s method on (*) Thanks David! SIAM AN08 20/36

37 Neumann series (I αh T )x = v (*) x = (I αh T ) 1 v = (I + αh T + (αh T ) 2 + )v A sum of nonnegative terms Equivalent to Jacobi s method on (*) x x k decreases monotonically Thanks David! SIAM AN08 20/36

38 Neumann series (I αh T )x = v (*) x = (I αh T ) 1 v = (I + αh T + (αh T ) 2 + )v A sum of nonnegative terms Equivalent to Jacobi s method on (*) Thanks David! x x k decreases monotonically r k x x k 1+α 1 α r k Error bounded by residual SIAM AN08 20/36

39 Neumann series (I αh T )x = v (*) x = (I αh T ) 1 v = (I + αh T + (αh T ) 2 + )v A sum of nonnegative terms Equivalent to Jacobi s method on (*) Thanks David! x x k decreases monotonically r k x x k 1+α 1 α r k Error bounded by residual Eventually small (x k ) j 0, update remainder SIAM AN08 20/36

40 Neumann series (I αh T )x = v (*) x = (I αh T ) 1 v = (I + αh T + (αh T ) 2 + )v A sum of nonnegative terms Equivalent to Jacobi s method on (*) Thanks David! x x k decreases monotonically r k x x k 1+α 1 α r k Error bounded by residual Eventually small (x k ) j 0, update remainder See Gleich and Polito (2006) for sparse Power Method SIAM AN08 20/36

41 Sparse PageRank with BPDN (LPdual) SIAM AN08 21/36

42 Sparse PageRank Regard as (I αh T )x = v v sparse Ax v v = e i say SIAM AN08 22/36

43 Sparse PageRank Regard as (I αh T )x = v v sparse Ax v v = e i say Apply active-set solver LPdual to dual of min x, r λe T x r 2 st Ax + r = v, x 0 SIAM AN08 22/36

44 0.08 Stanford alpha=0.950 v=e(4) x(active) SIAM AN08 23/36

45 Stanford web, n = α = 0.85 v = e (cs.stanford.edu) Direct solve: xtrue = A\v BPDN λ = 2e-3 76 nonzero x j SIAM AN08 24/36

46 Sparsity of xbp(α) Stanford, n = , v = e (cs.stanford.edu) xpower > 1000/n xbp > 1/n nz = 2382 nz = 2305 α = 0.1 : 0.01 : 0.99 (90 values) SIAM AN08 25/36

47 Sparsity of xbp(α) Stanford, n = , v = e (cs.stanford.edu) xpower > 1000/n xbp > 1/n nz = 2305 α = 0.1 : 0.01 : 0.99 (90 values) nz = 2382 xpower > 1/n has λ = 4e rows, nonzeros SIAM AN08 25/36

48 Power method vs BP Stanford, n = , v = e (cs.stanford.edu) Time (secs) for each α BP Direct Power α = 0.1 : 0.01 : 0.99 (90 values) λ = 4e-3 SIAM AN08 26/36

49 Power method vs BP Stanford-Berkeley, n = , v = e 6753 (unknown) Time (secs) for each α BP Direct Power α = 0.1 : 0.01 : 0.99 (90 values) λ = 5e-3 SIAM AN08 27/36

50 Varying α and λ min x, r λ x r 2 st (I αh T )x + r = v H = csstanford web v = e 4 (cs.stanford.edu) α = 0.75, 0.85, 0.95 λ = 1e-3, 2e-3, 4e-3, 6e-3 Plot nonzero x j in the order they are chosen SIAM AN08 28/36

51 csstanford v = e 4 α = 0.75 min λe T x r 2 Stanford alpha=0.750 v=e(4) x(active) Stanford alpha=0.750 v=e(4) x(active) λ = 1e λ = 2e Stanford alpha=0.750 v=e(4) x(active) Stanford alpha=0.750 v=e(4) x(active) λ = 4e λ = 6e SIAM AN08 29/36

52 csstanford v = e 4 α = 0.85 min λe T x r 2 Stanford alpha=0.850 v=e(4) x(active) csstanford alpha=0.850 v=e(4) x(active) λ = 1e-3 λ = 2e Stanford alpha=0.850 v=e(4) x(active) Stanford alpha=0.850 v=e(4) x(active) λ = 4e-3 λ = 6e SIAM AN08 30/36

53 csstanford v = e 4 α = 0.95 min λe T x r 2 Stanford alpha=0.950 v=e(4) x(active) Stanford alpha=0.950 v=e(4) x(active) λ = 1e-3 λ = 2e Stanford alpha=0.950 v=e(4) x(active) 0.08 Stanford alpha=0.950 v=e(4) x(active) λ = 4e-3 λ = 6e SIAM AN08 31/36

54 SPAI: Sparse Approximate Inverse SIAM AN08 32/36

55 SPAI Preconditioning A plausible future application Find sparse X such that AX I SIAM AN08 33/36

56 SPAI Preconditioning A plausible future application Find sparse X such that AX I Find sparse x j such that Ax j e j for j = 1 : n SIAM AN08 33/36

57 SPAI Preconditioning A plausible future application Find sparse X such that AX I Find sparse x j such that Ax j e j for j = 1 : n Embarrassingly parallel use of sparse BP solutions SIAM AN08 33/36

58 Summary SIAM AN08 34/36

59 (I αh T )x v Comments Sparse v sometimes gives sparse x SIAM AN08 35/36

60 (I αh T )x v Comments Sparse v sometimes gives sparse x LPprimal and LPdual work in greedy manner SIAM AN08 35/36

61 (I αh T )x v Comments Sparse v sometimes gives sparse x LPprimal and LPdual work in greedy manner Greedy solvers guarantee sparse x (unlike power method) SIAM AN08 35/36

62 (I αh T )x v Comments Sparse v sometimes gives sparse x LPprimal and LPdual work in greedy manner Greedy solvers guarantee sparse x (unlike power method) Not sensitive to α SIAM AN08 35/36

63 (I αh T )x v Comments Sparse v sometimes gives sparse x LPprimal and LPdual work in greedy manner Greedy solvers guarantee sparse x (unlike power method) Not sensitive to α Sensitive to λ but we can limit iterations SIAM AN08 35/36

64 (I αh T )x v Comments Sparse v sometimes gives sparse x LPprimal and LPdual work in greedy manner Greedy solvers guarantee sparse x (unlike power method) Not sensitive to α Sensitive to λ but we can limit iterations Questions SIAM AN08 35/36

65 (I αh T )x v Comments Sparse v sometimes gives sparse x LPprimal and LPdual work in greedy manner Greedy solvers guarantee sparse x (unlike power method) Not sensitive to α Sensitive to λ but we can limit iterations Questions Run much bigger examples SIAM AN08 35/36

66 (I αh T )x v Comments Sparse v sometimes gives sparse x LPprimal and LPdual work in greedy manner Greedy solvers guarantee sparse x (unlike power method) Not sensitive to α Sensitive to λ but we can limit iterations Questions Run much bigger examples Which properties of I αh T lead to success of greedy SIAM AN08 35/36

67 Thanks Shaobing Chen and David Donoho Michael Friedlander (BP solvers) SIAM AN08 36/36

68 Thanks Shaobing Chen and David Donoho Michael Friedlander (BP solvers) Sou-Cheng Choi and Lek-Heng Lim (ICIAM 07) SIAM AN08 36/36

69 Thanks Shaobing Chen and David Donoho Michael Friedlander (BP solvers) Sou-Cheng Choi and Lek-Heng Lim (ICIAM 07) Amy Langville and Carl Meyer (splendid book) David Gleich SIAM AN08 36/36

70 Thanks Shaobing Chen and David Donoho Michael Friedlander (BP solvers) Sou-Cheng Choi and Lek-Heng Lim (ICIAM 07) Amy Langville and Carl Meyer (splendid book) David Gleich Holly Jin SIAM AN08 36/36