Updating Markov Chains Carl Meyer Amy Langville

Transcription

1 Updating Markov Chains Carl Meyer Amy Langville Department of Mathematics North Carolina State University Raleigh, NC A. A. Markov Anniversary Meeting June 13, 2006

2 Intro Assumptions Very large irreducible chain m = O(10 9 ) number of states Q m m old transition matrix φ T =(φ 1,φ 2,...,φ m ) old stationary distribution

3 Intro Assumptions Very large irreducible chain m = O(10 9 ) number of states Q m m old transition matrix φ T =(φ 1,φ 2,...,φ m ) old stationary distribution Updates Change some transition probabilities Add or delete some states P n n new transition matrix (irreducible) π T =(π 1,π 2,...,π n ) new distribution (unknown)

4 Intro Assumptions Very large irreducible chain m = O(10 9 ) number of states Q m m old transition matrix φ T =(φ 1,φ 2,...,φ m ) old stationary distribution Updates Change some transition probabilities Add or delete some states P n n new transition matrix (irreducible) π T =(π 1,π 2,...,π n ) new distribution (unknown) Aim Use φ T to compute π T

5 Exact (Theoretical) Updating Perturbation Formula P = Q E = π T = φ T ɛ T

6 Exact (Theoretical) Updating Perturbation Formula P = Q E = π T = φ T ɛ T ɛ T = φ T EZ(I + EZ) 1 Z = { Fundamental Matrix (I Q) # (group inverse)

7 Exact (Theoretical) Updating Perturbation Formula P = Q E = π T = φ T ɛ T ɛ T = φ T EZ(I + EZ) 1 { Fundamental Matrix Requires m = n (No states added or deleted) Z = (I Q) # (group inverse)

8 Exact (Theoretical) Updating Perturbation Formula P = Q E = π T = φ T ɛ T ɛ T = φ T EZ(I + EZ) 1 { Fundamental Matrix Requires m = n (No states added or deleted) Z = (I Q) # (group inverse) One Row At A Time (Sherman Morrison) p T i = q T i δ T i = π T = φ T ɛ T

9 Exact (Theoretical) Updating Perturbation Formula P = Q E = π T = φ T ɛ T ɛ T = φ T EZ(I + EZ) 1 Requires m = n (No states added or deleted) Z = { Fundamental Matrix (I Q) # (group inverse) One Row At A Time (Sherman Morrison) p T i = q T i δ T i = π T = φ T ɛ T ɛ T = ( ) φ i 1 + δ T i A # δ T A # A # = (I Q) # i

10 Exact (Theoretical) Updating Perturbation Formula P = Q E = π T = φ T ɛ T ɛ T = φ T EZ(I + EZ) 1 Requires m = n (No states added or deleted) Z = { Fundamental Matrix (I Q) # (group inverse) One Row At A Time (Sherman Morrison) p T i = q T i δ T i = π T = φ T ɛ T ɛ T = ( ) φ i 1 + δ T i A # δ T A # A # = (I Q) # i (I P) # = A # + eɛ T [ A # γi ] A# iɛ T γ = ɛt A # i φ i φ i

11 Exact (Theoretical) Updating Perturbation Formula P = Q E = π T = φ T ɛ T ɛ T = φ T EZ(I + EZ) 1 Requires m = n (No states added or deleted) Z = { Fundamental Matrix (I Q) # (group inverse) One Row At A Time (Sherman Morrison) p T i = q T i δ T i = π T = φ T ɛ T ɛ T = ( ) φ i 1 + δ T i A # δ T A # A # = (I Q) # i (I P) # = A # + eɛ T [ A # γi ] A# iɛ T Not Practical For Large Problems γ = ɛt A # i φ i φ i

12 Restarted Power Method x T j+1 = xt j P with xt 0 = φt (assume aperiodic)

13 Restarted Power Method x T j+1 = xt j P with xt 0 = φt (assume aperiodic) Requires m = n (no states added or deleted)

14 Restarted Power Method x T j+1 = xt j P with xt 0 = φt (assume aperiodic) Requires m = n (no states added or deleted) Requires φ T π T (generally means P Q)

15 Restarted Power Method x T j+1 = xt j P with xt 0 = φt (assume aperiodic) Requires m = n (no states added or deleted) Requires φ T π T (generally means P Q) Asymptotic rate of convergence: R = log 10 λ 2 Need about 1/R iterations to eventually gain one additional significant digit of accuracy

16 Restarted Power Method x T j+1 = xt j P with xt 0 = φt (assume aperiodic) Requires m = n (no states added or deleted) Requires φ T π T (generally means P Q) Asymptotic rate of convergence: R = log 10 λ 2 Example: Need about 1/R iterations to eventually gain one additional significant digit of accuracy Suppose digit 1 (φ T )=digit 1 (π T ) Want 12 digits of accuracy

17 Restarted Power Method x T j+1 = xt j P with xt 0 = φt (assume aperiodic) Requires m = n (no states added or deleted) Requires φ T π T (generally means P Q) Asymptotic rate of convergence: R = log 10 λ 2 Example: Need about 1/R iterations to eventually gain one additional significant digit of accuracy Suppose digit 1 (φ T )=digit 1 (π T ) Want 12 digits of accuracy RPM about 11/R iterations Start from scratch about 12/R iterations

18 Restarted Power Method x T j+1 = xt j P with xt 0 = φt (assume aperiodic) Requires m = n (no states added or deleted) Requires φ T π T (generally means P Q) Asymptotic rate of convergence: R = log 10 λ 2 Example: Need about 1/R iterations to eventually gain one additional significant digit of accuracy Suppose digit 1 (φ T )=digit 1 (π T ) Want 12 digits of accuracy RPM about 11/R iterations Start from scratch about 12/R iterations Only 8.3% reduction

19 Restarted Power Method x T j+1 = xt j P with xt 0 = φt (assume aperiodic) Requires m = n (no states added or deleted) Requires φ T π T (generally means P Q) Asymptotic rate of convergence: R = log 10 λ 2 Example: Need about 1/R iterations to eventually gain one additional significant digit of accuracy Suppose digit 1 (φ T )=digit 1 (π T ) Want 12 digits of accuracy RPM about 11/R iterations Start from scratch about 12/R iterations Only 8.3% reduction A Little Better, But Not Great

20 Censoring Partition (not necessarily NCD) P n n = G 1 G 2... G k G 1 P 11 P P 1k G 2 P 21 P P 2k G k P k1 P k2... P kk

21 Censoring Partition (not necessarily NCD) P n n = G 1 G 2... G k G 1 P 11 P P 1k G 2 P 21 P P 2k G k P k1 P k2... P kk Censored Chains Records state of process only when chain visits states in G i. Visits to states outside of G i are ignored.

22 Censoring Partition (not necessarily NCD) P n n = G 1 G 2... G k G 1 P 11 P P 1k G 2 P 21 P P 2k G k P k1 P k2... P kk Censored Chains Records state of process only when chain visits states in G i. Visits to states outside of G i are ignored. Censored Transition Matrices C i = P ii + P i (I P i ) 1 P i Stochastic Complements

23 Censored Distributions s T i C i = s T i Aggregation

24 Censored Distributions s T i C i = s T i Aggregation Aggregation Matrix s T 1 P 11e... s T 1 P 1ke A k k = e = s T k P k1e... s T k P kke

25 Censored Distributions s T i C i = s T i Aggregation Aggregation Matrix s T 1 P 11e... s T 1 P 1ke A k k = e = s T k P k1e... s T k P kke Aggregated Distribution α T A = α T α T = (α 1,α 2,...,α k )

26 Censored Distributions s T i C i = s T i Aggregation Aggregation Matrix s T 1 P 11e... s T 1 P 1ke A k k = e = s T k P k1e... s T k P kke Aggregated Distribution α T A = α T α T = (α 1,α 2,...,α k ) Aggregation Theorem π T =(π T 1 πt 2... π T k )=(α 1s T 1 α 2s T 2... α 2 s T 2 )

27 Partitioning Intuition Update relatively small number of states in large sparse chain Effects are primarily local Most stationary probabilities not significantly affected.

28 Partitioning Intuition Update relatively small number of states in large sparse chain Effects are primarily local Most stationary probabilities not significantly affected. State Space Partition S = G G G = { states likely to be most affected newly added states

29 Partitioning Intuition Update relatively small number of states in large sparse chain Effects are primarily local Most stationary probabilities not significantly affected. State Space Partition S = G G G = g = G << G { states likely to be most affected newly added states

30 Partitioning Intuition Update relatively small number of states in large sparse chain Effects are primarily local Most stationary probabilities not significantly affected. State Space Partition S = G G G = g = G << G { states likely to be most affected newly added states Induced Matrix Partition P n n = ( G G ) G P 11 P 12 G P 21 P 22 = p p 1g P 1... p g1... p gg P g P 1... P g P 22

31 Specialized Aggregation Censored Transition Matrices C 1 =... = C g =[1] 1 1 C g+1 = P 22 + P 21 (I P 11 ) 1 P 12

32 Specialized Aggregation Censored Transition Matrices C 1 =... = C g =[1] 1 1 C g+1 = P 22 + P 21 (I P 11 ) 1 P 12 Censored Distributions s T 1 =... = s T g = 1 s T g+1 = st g+1 C g+1

33 Specialized Aggregation Censored Transition Matrices C 1 =... = C g =[1] 1 1 C g+1 = P 22 + P 21 (I P 11 ) 1 P 12 Censored Distributions s T 1 =... = s T g = 1 s T g+1 = st g+1 C g+1 Aggregation [ Matrix ] P11 P 12 e A = s T g+1 P 21 1 s T g+1 P 21e (g+1) (g+1)

34 Specialized Aggregation Censored Transition Matrices C 1 =... = C g =[1] 1 1 C g+1 = P 22 + P 21 (I P 11 ) 1 P 12 Censored Distributions s T 1 =... = s T g = 1 s T g+1 = st g+1 C g+1 Aggregation [ Matrix ] P11 P 12 e A = s T g+1 P 21 1 s T g+1 P 21e (g+1) (g+1) Aggregation Theorem π T = ( π 1,...,π g π T) =(α 1,...,α g, α g+1 s T g+1 ) Aggregated Distribution α T = ( ) α 1,...,α g,α g+1

35 Specialized Aggregation Censored Transition Matrices C 1 =... = C g =[1] 1 1 C g+1 = P 22 + P 21 (I P 11 ) 1 P 12 Censored Distributions s T 1 =... = s T g = 1 s T g+1 = st g+1 C g+1 Aggregation [ Matrix ] P11 P 12 e A = s T g+1 P 21 1 s T g+1 P 21e (g+1) (g+1) Aggregation Theorem π T = ( π 1,...,π g π T) =(α 1,...,α g, α g+1 s T g+1 ) Old Distribution φ T =(φ 1,φ 2,... φ T ) (reorded) Aggregated Distribution α T = ( ) α 1,...,α g,α g+1

36 Specialized Aggregation Censored Transition Matrices C 1 =... = C g =[1] 1 1 C g+1 = P 22 + P 21 (I P 11 ) 1 P 12 Censored Distributions s T 1 =... = s T g = 1 s T g+1 = st g+1 C g+1 Aggregation [ Matrix ] P11 P 12 e A = s T g+1 P 21 1 s T g+1 P 21e (g+1) (g+1) Aggregation Theorem π T = ( π 1,...,π g π T) =(α 1,...,α g, α g+1 s T g+1 ) Old Distribution φ T =(φ 1,φ 2,... φ T ) The Assumption φ T π T (reorded) Aggregated Distribution α T = ( ) α 1,...,α g,α g+1

37 Specialized Aggregation Censored Transition Matrices C 1 =... = C g =[1] 1 1 C g+1 = P 22 + P 21 (I P 11 ) 1 P 12 Censored Distributions s T 1 =... = s T g = 1 s T g+1 = st g+1 C g+1 Aggregation [ Matrix ] P11 P 12 e A = s T g+1 P 21 1 s T g+1 P 21e (g+1) (g+1) Aggregation Theorem π T = ( π 1,...,π g π T) =(α 1,...,α g, α g+1 s T g+1 ) Old Distribution φ T =(φ 1,φ 2,... φ T ) The Assumption (reorded) φ T π T = φ T α g+1 s T g+1 Aggregated Distribution α T = ( ) α 1,...,α g,α g+1

38 Specialized Aggregation Censored Transition Matrices C 1 =... = C g =[1] 1 1 C g+1 = P 22 + P 21 (I P 11 ) 1 P 12 Censored Distributions s T 1 =... = s T g = 1 s T g+1 = st g+1 C g+1 Aggregation [ Matrix ] P11 P 12 e A = s T g+1 P 21 1 s T g+1 P 21e (g+1) (g+1) Aggregation Theorem π T = ( π 1,...,π g π T) =(α 1,...,α g, α g+1 s T g+1 ) Old Distribution φ T =(φ 1,φ 2,... φ T ) The Assumption (reorded) Aggregated Distribution α T = ( ) α 1,...,α g,α g+1 φ T π T = φ T α g+1 s T g+1 = s T g+1 φt φ T e

39 Specialized Aggregation Censored Transition Matrices C 1 =... = C g =[1] 1 1 C g+1 = P 22 + P 21 (I P 11 ) 1 P 12 Censored Distributions s T 1 =... = s T g = 1 s T g+1 = st g+1 C g+1 Aggregation [ Matrix ] P11 P 12 e A = s T g+1 P 21 1 s T g+1 P 21e (g+1) (g+1) Aggregation Theorem π T = ( π 1,...,π g π T) =(α 1,...,α g, α g+1 s T g+1 ) Old Distribution φ T =(φ 1,φ 2,... φ T ) The Assumption (reorded) Aggregated Distribution α T = ( ) α 1,...,α g,α g+1 φ T π T = φ T α g+1 s T g+1 = s T g+1 φt φ T e

40 Specialized Aggregation Censored Transition Matrices C 1 =... = C g =[1] 1 1 C g+1 = P 22 + P 21 (I P 11 ) 1 P 12 Censored Distributions s T 1 =... = s T g = 1 s T g+1 = st g+1 C g+1 Aggregation [ Matrix ] P11 P 12 e A = s T g+1 P 21 1 s T g+1 P 21e Aggregated Distribution α T = ( ) α 1,...,α g,α g+1 (g+1) (g+1) Aggregation Theorem π T = ( π 1,...,π g π T) =(α 1,...,α g, α g+1 s T g+1 ) Old Distribution φ T =(φ 1,φ 2,... φ T ) The Assumption (reorded) φ T π T = φ T α g+1 s T g+1 = s T g+1 φt φ T e

41 Specialized Aggregation Censored Transition Matrices C 1 =... = C g =[1] 1 1 C g+1 = P 22 + P 21 (I P 11 ) 1 P 12 Censored Distributions s T 1 =... = s T g = 1 s T g+1 = st g+1 C g+1 Aggregation [ Matrix ] P11 P 12 e A = s T g+1 P 21 1 s T g+1 P 21e Aggregation Theorem Aggregated Distribution α T = ( ) α 1,...,α g,α g+1 (g+1) (g+1) π T =(α 1,...,α g, α g+1 s T g+1 ) Old Distribution (reorded) Updated Distribution φ T =(φ 1,φ 2,... φ T ) The Assumption φ T π T = φ T α g+1 s T g+1 = s T g+1 φt φ T e

42 Summary Reorder & Partition Updated State Space S = G G G = {New + Most affected} G = {Less affected}

43 Summary Reorder & Partition Updated State Space S = G G G = {New + Most affected} G = {Less affected} Use Less Affected Components From Old Distribution φ T components from φ T corresponding to states in G

44 Summary Reorder & Partition Updated State Space S = G G G = {New + Most affected} G = {Less affected} Use Less Affected Components From Old Distribution φ T components from φ T corresponding to states in G Approximate Censored Distribution s T φ T /(φ T e)

45 Summary Reorder & Partition Updated State Space S = G G G = {New + Most affected} G = {Less affected} Use Less Affected Components From Old Distribution φ T components from φ T corresponding to states in G Approximate Censored Distribution s T φ T /(φ T e) Form Approximate [ Aggregation ] Matrix P11 P 12 e A s T P 21 1 s T P 21 e (g+1) (g+1)

46 Summary Reorder & Partition Updated State Space S = G G G = {New + Most affected} G = {Less affected} Use Less Affected Components From Old Distribution φ T components from φ T corresponding to states in G Approximate Censored Distribution s T φ T /(φ T e) Form Approximate [ Aggregation ] Matrix P11 P 12 e A s T P 21 1 s T P 21 e (g+1) (g+1) Compute Approximate Aggregated Distribution α T ( ) α 1,...,α g,α g+1

47 Summary Reorder & Partition Updated State Space S = G G G = {New + Most affected} G = {Less affected} Use Less Affected Components From Old Distribution φ T components from φ T corresponding to states in G Approximate Censored Distribution s T φ T /(φ T e) Form Approximate [ Aggregation ] Matrix P11 P 12 e A s T P 21 1 s T P 21 e (g+1) (g+1) Compute Approximate Aggregated Distribution α T ( ) α 1,...,α g,α g+1 Approximate Updated Distribution π T (α 1,...,α g, α g+1 s T )

48 Summary Reorder & Partition Updated State Space S = G G G = {New + Most affected} G = {Less affected} Use Less Affected Components From Old Distribution φ T components from φ T corresponding to states in G Approximate Censored Distribution s T φ T /(φ T e) Form Approximate [ Aggregation ] Matrix P11 P 12 e A s T P 21 1 s T P 21 e (g+1) (g+1) Compute Approximate Aggregated Distribution α T ( ) α 1,...,α g,α g+1 Approximate Updated Distribution π T (α 1,...,α g, α g+1 s T ) Iterate?

49 Summary Reorder & Partition Updated State Space S = G G G = {New + Most affected} G = {Less affected} Use Less Affected Components From Old Distribution φ T components from φ T corresponding to states in G Approximate Censored Distribution s T φ T /(φ T e) Form Approximate [ Aggregation ] Matrix P11 P 12 e A s T P 21 1 s T P 21 e (g+1) (g+1) Compute Approximate Aggregated Distribution α T ( ) α 1,...,α g,α g+1 Approximate Updated Distribution π T (α 1,...,α g, α g+1 s T ) Iterate? No! At A Fixed Point

50 Iterative Aggregation Move Off Of Fixed Point With Power Step s T φ T /(φ T e) [ ] P11 P 12 e A s T P 21 1 s T P 21 e α T ( ) α 1,...,α g,α g+1 π T (α 1,...,α g, α g+1 s T ) ψ T π T P (g+1) (g+1) If ψ T χ T <τ then quit else s T ψ T /(ψ T e) (Also makes progress toward convergence when aperiodic)

51 Iterative Aggregation Move Off Of Fixed Point With Power Step s T φ T /(φ T e) [ ] P11 P 12 e A s T P 21 1 s T P 21 e α T ( ) α 1,...,α g,α g+1 π T (α 1,...,α g, α g+1 s T ) ψ T π T P (g+1) (g+1) If ψ T χ T <τ then quit else s T ψ T /(ψ T e) (Also makes progress toward convergence when aperiodic) Theorem If C = P 22 + P 21 (I P 11 ) 1 P 12 all partitions S = G G is aperiodic, then convergent for

52 Iterative Aggregation Move Off Of Fixed Point With Power Step s T φ T /(φ T e) [ ] P11 P 12 e A s T P 21 1 s T P 21 e α T ( ) α 1,...,α g,α g+1 π T (α 1,...,α g, α g+1 s T ) ψ T π T P (g+1) (g+1) (Also makes progress toward convergence when aperiodic) If ψ T χ T <τ then quit else s T ψ T /(ψ T e) Theorem If C = P 22 + P 21 (I P 11 ) 1 P 12 is aperiodic, then convergent for all partitions S = G G λ 2 (C) determines rate of convergence

53 Google s PageRank Random Walk On WWW Link Structure { 1/(total # outlinks from page Pi ) if P H ij = i P j, 0 otherwise

54 Google s PageRank Random Walk On WWW Link Structure { 1/(total # outlinks from page Pi ) if P H ij = i P j, 0 otherwise Google Matrix P = α(h + E)+(1 α)f (H + E) &F are stochastic rank (E) = rank (F) = 1 0 <α<1

55 Google s PageRank Random Walk On WWW Link Structure { 1/(total # outlinks from page Pi ) if P H ij = i P j, 0 otherwise Google Matrix P = α(h + E)+(1 α)f (H + E) &F are stochastic rank (E) = rank (F) = 1 0 <α<1 PageRank = π T

56 Google s PageRank Random Walk On WWW Link Structure { 1/(total # outlinks from page Pi ) if P H ij = i P j, 0 otherwise Google Matrix P = α(h + E)+(1 α)f (H + E) &F are stochastic rank (E) = rank (F) = 1 0 <α<1 PageRank = π T Power Law Distribution If ordered by magnitude π(i) αi k for k π(1) π(2)... π(n), then [Donato, Laura, Leonardi, 2002] [Pandurangan, Raghavan,& Upfal, 2004] Relatively few large states (i.e., important sites)

57 L Curves 0.04 movies 0.05 mathworks censorship abortion genetics 0.01 CA

58 Experiments The Updates # Nodes Added = 3 # Nodes Removed = 50 # Links Added = 10 (Different values have little effect on results) # Links Removed = 20 Stopping Criterion 1-norm of residual < 10 10

59 Movies Power Method Iterative Aggregation Iterations Time nodes = 451 links = 713 G Iterations Time

60 Censorship Power Method Iterative Aggregation Iterations Time nodes = 562 links = 736 G Iterations Time

61 MathWorks Power Method Iterative Aggregation Iterations Time nodes = 517 links = 13, 531 G Iterations Time

62 Abortion Power Method Iterative Aggregation Iterations Time nodes = 1, 693 links = 4, 325 G Iterations Time

63 Genetics Power Method Iterative Aggregation Iterations Time nodes = 2, 952 links = 6, 485 G Iterations Time

64 California Power Method Iterative Aggregation Iterations Time nodes = 9, 664 links = 16, 150 G Iterations Time

65 L Curves 0.04 movies 0.05 mathworks censorship abortion genetics 0.01 CA

66 0.04 movies L Curves 0.05 mathworks censorship abortion genetics 0.01 CA

67 Quadratic Extrapolation nodes = 10,000 links = 101,118 [Kamvar, Haveliwala, Manning, Golub, 2003] 0 power method power method with quad. extrap. 2 4 log 10 residual iteration

68 Quadratic Extrapolation nodes = 10,000 links = 101,118 [Kamvar, Haveliwala, Manning, Golub, 2003] 0 power method power method with quad. extrap. i A D 2 4 log 10 residual iteration

69 Quadratic Extrapolation nodes = 10,000 links = 101,118 [Kamvar, Haveliwala, Manning, Golub, 2003] 0 2 power method power method with quad. extrap. i A D i A D with quad. extrap. 4 log 10 residual iteration

70 Timings Iterations Time (sec) G Power Power+Quad IAD IAD+Quad nodes = 10, 000 links = 101, 118

71 Conclusion Iterative aggregation shows promise for updating Markov chains Especially for those having power law distributions

72 Leveling Off Point π(i) αi k dπ(i) di ɛ for some user-defined tolerance ɛ i level ( kα ɛ ) 1/k+1 Perhaps better: ( kα i level f(n) ɛ ) 1/k+1 For WWW: [ 2.109α g opt f(n) ɛ ] 1/3.109