Cheetah: Fast Gaph Kenel Tacking on Dynamic Gaphs Pesente: Liangyue Li Joint wok with Hanghang Tong (ASU), Yanghua Xiao (Fudan), Wei Fan (Baidu) 1 Aizona State Univesity
Gaphs ae Eveywhee Collaboation Netwoks US Powe Gid Bus Netwok 2 Bain Netwoks Patient Netwoks Hospital Netwoks Aizona State Univesity
Application 1: Web Mining Q: How simila ae the two gaphs? A: Gaph Kenel 1. Fo each entity, constuct a neighbohood gaph by beadth-fist seach up to depth k 2. Apply gaph kenel in kenel based leaning methods 3 Lösch, Uta, Stephan Bloehdon, and Achim Rettinge. "Gaph kenels fo RDF data." The Semantic Web: Reseach and Applications. Spinge Belin Heidelbeg, 2012. Aizona State Univesity
Application 2: Compute Vision ea, sun, ( sea, sun,, waves ) sky, waves ) (a) (a) ( cat, foest, ( cat, foest, gass, tige ) gass, tige ) (b) (b) - no caption - no caption (c) (c) (c) 27 7 2 6 6 3 13 1 (f) (f) 5 (e) (d) 5 (d)4 (e) gaphs? 2 Q: How 27 simila ae the two 76 6 3 3 A: Gaph Kenel sea, sun, ( sea, sun,, waves ) 1 waves ) 1 sky, (a)4 (a) ( cat, foest, ( cat, foest, gass, tige ) gass, tige ) 5 5 (b) (b) - no 8caption - no 8caption 9 (c) 9 10 11 9 8 1 2 (d) 13 24 35 4(e) 6 57 6 87 9 11 8 10 9 11 (d) 10 8 10 9(f) 11 10 11 10 11 (e) (f) 1. Fo each image, epesent it as a segmentation gaph i3 methods 2. Apply in kenel based ii1 2 gaph kenel i2 i3 i3 leaning i i 1 3 9 10 11with segmentation 1Hachaoui, 2 3 Zaïd, 4 and5fancis 6 Bach. 7 "Image 8 classification gaph kenels." 10 11 9 1 2 3 4 5 6 7 8 4 Aizona State Univesity CVPR 2007.
Application 3: Neuoscience Q: How simila ae the two gaphs? A: Gaph Kenel 1. Fo each bain image, epesent it as a gaph 2. Apply gaph kenel in kenel based leaning methods 5 L. Shi, H. Tong, X. Mu: BainQuest: Peception-Guided Visual Bain Aizona Compaison, State 2015. Univesity
Random Walk based Gaph Kenel 2 1 5 h b a e i 3 4 c d j 6 7 8 f g Gaph 1 Gaph 2 Intuitions: 1. Compae similaity of evey pai of nodes fom each gaph Eg: (1,2) vs (a, j)! less simila (1,5) vs (a,e)! moe simila 2. Node pai similaity is measued by andom walks 3. Two gaphs ae simila if they shae many simila node pais 6 Aizona State Univesity
Konecke Poduct Gaph Gaph Illustation 2' 2 3' A 1 3 1' 1 X A 2 11' 21' 34' 24' 14' 33' 23' 31' 12' 4' 22' 32' 13' A 1 A 2 A 1 Matix Desciption 2 0 1 3 1 41 0 15 1 1 0 = A 2 = A 1 A 2 = Konecke poduct 2 3 0 1 0 1 61 0 1 0 7 40 1 0 15 1 0 1 0 2 3 0 0 0 0 0 1 0 1 0 1 0 1 0 0 0 0 1 0 1 0 1 0 1 0 0 0 0 0 0 1 0 1 0 1 0 1 0 0 0 0 1 0 1 0 1 0 1 0 0 1 0 1 0 0 0 0 0 1 0 1 1 0 1 0 0 0 0 0 1 0 1 0 0 1 0 1 0 0 0 0 0 1 0 1 1 0 1 0 0 0 0 0 1 0 1 0 0 1 0 1 0 1 0 1 0 0 0 0 61 0 1 0 1 0 1 0 0 0 0 0 7 40 1 0 1 0 1 0 1 0 0 0 05 1 0 1 0 1 0 1 0 0 0 0 0 One Random Walk on + = One Random Walk on One Random Walk on A 1 A 2 A 1 A 2 = A 7 S. V. N. Vishwanathan, Nicol N. Schaudolph, Ime Risi Kondo, and Kasten M. Bogwadt. Gaph Kenels. Jounal of Machine Leaning Reseach, 11:1201 1242, Apil 2010. Aizona State Univesity
RWR Gaph Kenel Fomulation Taking expectations instead of summing Ke(G 1,G 2 ) = P k ck q 0 A k p = q 0 (I ca ) 1 p Computational challenge: A is of size n 2 n 2 O(n 6 ) (Diect computation) o O(n 3 ) (Sylveste equation) Time > 1h, n=3328 8 S. V. N. Vishwanathan, Nicol N. Schaudolph, Ime Risi Kondo, and Kasten M. Bogwadt. Gaph Kenels. Jounal of Machine Leaning Reseach, 11:1201 1242, Apil 2010. Aizona State Univesity
Speed up ARK Idea: pefom low-ank appox on both gaphs Ke(G 1, G 2 )=(q 1 0 q 2 0 )(I ca 1 0 A 2 0 ) 1 (p 1 p 2 ) Step 1: Top- low-ank appox: Step 2: Matix-invese Lemma: 2 2 U 1 1 V 0 1 U 2 2 V 0 2 Ke(G 1, G 2 ) (q 0 1p 1 )(q 0 2p 2 )+c(q 0 1U 1 q 0 2U 2 ) (V 0 1p 1 V 0 2p 2 ) =(( 1 2 ) 1 c(v 0 1 V 0 2)(U 1 U 2 )) 1 Matix of size, easy to invese Oveall complexity: O(n 2 4 + m + 6 ) Time = 7.5s, n=3328 Can be educed to O(n 2 + m + 6 ) 9 U. Kang, Hanghang Tong, Jimeng Sun. Fast Random Walk Gaph Aizona Kenel. State SDM Univesity 2012
Challenges ARK: Good fo static gaphs What if gaphs ae evolving ove time Static,ea, sun, sun, waves ), waves ) ) (a) ( cat, foest, ( cat, foest, gass, tige ) gass, tige ) (b)(b) 11 55 2 2 77 6 6 3 ) (d) 1 ea (e)(e) 1 1 2 2 3 3 4 4 5 5 6 6 7 7 i1 i1 sun sun sky sky ( sea, sun, ( sea, sun, ( cat, foest, ( cat, foest,- no caption - no caption - no caption - no caption ( sea, sun, ( sea, ( cat, foest, - no caption sun, ( cat, foest, - no caption sky, waves ) gass, tige ) sky, waves ) gass, tige ) ( sea, sun, sky, ( cat, foest, - no caption ( sea, waves ) sun, ( cat, foest, - no caption sky, waves ) gass, tige ) gass, tige ) (a) (b) (c) (a) (b) (c) (c) sky, e waves ) gass, tige ) (c) sky, waves ) gass, tige ) m (a) (c) i (a) (b) (c) T(b) (a) (b) (c) (a) (b) (c) 1 1 5 44 1144 11 2 55 5 4 4 2 2 77 2 56 7 7 56 37 2 6 7 6 2 3 6 3 3 (d) 6 3 3 (e) 8 8 9 9 10 10 (f) (f) 8 8 9 9 10 10 11 11 (d) (d) (d) (d) (d) (e) (e) e im T(e) (e) 8 9 9 9 8 8 10 11 11 i3 ii33 i 3 i2 i2 t1 t t 2 t t3 t t 4 t t 5 t t6 t t 7 t t8 2 3 4 5 6 7 8 sea Dynamic 11 11 (e) 10 10 11 (f) (f) (f) 8 8 9 98 10 9 10 10 11 1111 (f)(f) cat waves cat foest foest gass tige gass tige 1 2 3 141 252 363 474 5 5 86 6 79 7 108 811 9 9 10 10 11 11 10 11 1 1 2 2 3 3 41 4 52 5 63 6 74 7 5 8 86 9 97 10 811 9 10 11 i3 i 3 i2 i3 i 3 i i1 i 2 i1 i1 3 i3 i 2 i i1 i 2 i1 i3 i 3 i2 e i3 i3 3 i 3 i1 i2 m i t t t t Tt t t t t1 t2 t3 t1t4 1 t2t5 2 t3t6 3 t4t7 4 t5t8 5 t6 6 t7 7 t8 8 t t t t t t t t t1 1 t2 2 t3 3 t4t1 4 t5t2 5 t6t3 6 t7t4 7 t8t5 8 t6 t7 (g)t8 (g) (g) sea sea sea sun sun sky sky sea waves sun sun cat cat waves sea sun cat waves sky cat waves gass sky foest waves cat tige foest gass tige cat waves sky foest gass tige foest foest gass gass e Tim (f) Q: How to tack gaph kenel efficiently? waves e Tim e Tim tige tige (g) (g) (g) (g) (g) 10 Aizona State Univesity Figue Thee sample gaphimages, two of them annotated; thei egions (d,e,f); and thei e images, two of them annotated; thei egions (d,e,f); and 1. thei sea sun sky foest gass tige
Roadmap Motivation Cheetah-D fo Diected Gaphs Expeimental Results Conclusion 11 Aizona State Univesity
Cheetah-D: gaph kenel tacking Ke(G 1, G 2 )=(q 1 0 q 2 0 )(I ca 1 0 A 2 0 ) 1 (p 1 p 2 ) top- low-ank appox: U 1 1 V 0 1 U 2 2 V 0 2 A 1 A 2 Ideas: Avoid: e-computing low-ank appox [ARK] Goal: tack low-ank stuctue efficiently [Cheetah-D] (SVD in this pape) 12 Aizona State Univesity
Step 0: Low ank appox on A Intuition: A 0 v1 v2 (m =5, = 2) u 1 u 2 0 Details: A A 0 U 0 0 V 0 0 = x 1 A = XY Z 0 =[U 0 X] A = A 0 + A Y z1 apple 0 0 0 Y [V 0 Z] 0 (m 0 =2, 0 = 1) Popety: SVD on SVD on A 0 A takes: takes: O(m + n) O(m 0 0 + n 0 ) O(m + n) [m 0 m, 0 ] 13 Aizona State Univesity
Step 1: Patial QR Decomposition Intuition: Case 1: x 1 2 span(u 1,u 2 ) u 2 x 1 u 1 u 1 u 2 x 1 = u 1 u 2 S Details: [U 0 X]=U 0 S apple 0 0 A =[U 0 X] 0 Y [V 0 Z] 0 apple 0 0 = U 0 S 0 Y T 0 V0 0 O(n 02 ) Popety: Efficiency: takes [ 0 ] Effectiveness: No exta eo 14 Aizona State Univesity
Step 1: Patial QR Decomposition Intuition: Case 2: x 1 /2 span(u 1,u 2 ) x 1 u 1 u 2 x 1 = q u 2 u 1 u 1 u 2 q S Details: [U 0 X]=[U 0 q]s A =[U 0 X] =[U 0 Popety: Efficiency: takes O(n 02 ) [ 0 ] Effectiveness: No exta eo apple 0 0 0 Y [V 0 Z] 0 apple 0 0 Q]S 0 Y T 0 [V 0 Z] 0 Simila Patial QR decomp on Z 15 Aizona State Univesity
Step 2: Full SVD on a Small Matix Intuition: M ( + 0 ) ( + 0 ) = S 0 Y T 0 Details: M = S = L R 0 Updated apple 0 0 0 Y T 0 = L R 0 A =[U 0 apple 0 0 Q]S 0 Y T 0 [V 0 Z] 0 =[U 0 Q]L R 0 T 0 [V 0 Z] 0 Popety: [ 0 ] Efficiency: takes O(( + 0 ) 3 ) Effectiveness: No exta eo 16 Aizona State Univesity
Step 3: Rotate Othonomal Basis Intuition: Rotate u 1 u 2 q by L Rotate v 1 v 2 z by Details: U =[U 0 V =[V 0 R Q]L Z]R Popety: 17 Complexity: O(n 2 ) Oveall SVD Update Complexity: O(n 2 + m 0 0 + n 02 ) Re-compute SVD: O(n 2 + m) Aizona State Univesity
Analysis and Vaiants Time complexity of Cheetah-D: Compaison Example ARK:7.5s Ous:0.4s Vaiants Undiected gaphs Attibuted gaphs O(n 2 + n 02 + 6 ) ARK: O(n 2 4 + m + 6 ) (n = 3328, = 500, 0 = 5) Cheetah-D Algoithm Sketch t = 1, Initialize SVD of A1 and A2 fo t=2,3, Update SVD fo A1 Update SVD fo A2 Update Ke(A1,A2) end O(n( 2 + 02 )) O(n( 2 + 02 )) O(n 2 + 6 ) 18 Aizona State Univesity
Roadmap Motivation Cheetah-D fo Diected Gaphs Expeimental Results Conclusion 19 Aizona State Univesity
Case Study MTA Bus Taffic Gaph constuction Monito taffic volume of 30 bus stops on 3 outes, fom Monday, 03/24/2014 Sunday, 03/30/2014 Repesent each stop as a time seies whee each timestamp is taffic volume within each hou On each day, build a gaph fo the 30 stops using Gange causality test Gaph kenel computation Gaph kenel is computed between two gaphs of two consecutive days 20 Aizona State Univesity
Case Study MTA Bus Taffic 10 10 Nomalized Ke(G1,G2) 9 8 7 6 5 4 Tue Wed Fi Sat (Mon,Tue) (Tue, Wed) (Wed,Thu) (Thu,Fi) (Fi,Sat) (Sat,Sun) Weekdays Schedule Weekends Schedule 21 Aizona State Univesity
Relative Eo Relative Eo =100 =200 =300 =400 =500 0.020% 0.018% 0.016% 0.014% 0.012% 0.010% 0.008% 0.006% 0.004% 0.002% 0.000% 0 5 10 15 20 25 30 All cases, E <0.02% n = 4183, m = 5692 22 Aizona State Univesity
Relative Eo Running Time (Seconds) Avg Eo vs. Rank 0.25% 0.9 0.20% '=5 '=20 0.8 0.7 0.15% '=40 '=60 0.6 0.5 0.10% '=80 '=100 0.4 0.3 0.05% 0.2 0.1 0.00% 0 100 200 300 400 500 Reduced ank 0 0 When >50, E<0.05% n = 4183, m = 5692 Figue 6: di eent ga 23 Aizona State Univesity 0.8
Running Time (Seconds) Running Time (Seconds) Running Time vs. Rank di een 8 7 6 5 4 3 2 1 0 Cheetah-U ARK-U+ 0 100 200 300 400 500 Reduced Rank ARK-U+ 10x OURS 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 24 Figue Aizona State Univesity
Running Time (Seconds) Scalability 0.9 0.8 0.7 0.6 25 0.5 0.4 0.3 0.2 0.1 0 Figue 6: 0 1000 2000 3000 4000 5000 Gaph Size (n) Scale nea linealy =50 =100 =150 =200 =250 Running time of Cheetah-U on AS with Aizona State Univesity
Relative Eo Quality vs. Speed > 10. 1. 0.1 0.01 0.001 0.0001 0.00001 Ous ARK-U+ Cheetah-U Fist-ode Second-ode 0. 0 1 2 3 4 5 6 7 Running Time (Seconds) 26 Aizona State Univesity
Roadmap Motivation Cheetah-D fo Diected Gaphs Expeimental Results Conclusion 27 Aizona State Univesity
Conclusion Goal: tack gaph kenel of dynamic gaphs Ou Solution: Cheetah-D Key idea: tack low-ank appox Results: Complexity: O(n 2 + n 02 + 6 ) In pactice: ~15x faste, E<0.05% Moe in pape: Cheetah-U fo undiected gaphs Eo bound analysis 28 Aizona State Univesity