Sulementary Materal for Sectral Clusterng based on the grah -Lalacan Thomas Bühler and Matthas Hen Saarland Unversty, Saarbrücken, Germany {tb,hen}@csun-sbde May 009 Corrected verson, June 00 Abstract Ths techncal reort s an aendx to the ICML 009 submsson Sectral Clusterng based on the grah -Lalacan [], contanng the roofs whch had to be omtted due to sace restrctons In ths verson, an error occurng n a revous verson has been corrected However, note that the error occurred n an addtonal statement whch was not used n any roof, thus the correctness of the other results s not affected Overvew Ths techncal reort s an aendx to the ICML 009 submsson Sectral Clusterng based on the grah -Lalacan [], contanng the roofs whch had to be omtted due to sace restrctons Our roosed method and some of the results are based on the recent work by Amghbech [] In hs very nterestng artcle, the author rooses the varatonal characterzaton of the second egenvector of the normalzed grah -Lalacan and derves the soermetrc nequalty n the normalzed case Due to the comressed form of the roof gven n [], some mortant lemmas whch lay a role n the roof of the varatonal characterzaton are not exlctely stated n the aer, hence we cover the roofs n more detal n the followng Moreover, we rovde basc roertes of the -Lalacan and related functonals and extend the results of Amghbech to the unnormalzed case However, our man result s to show that the Cheeger cuts obtaned by thresholdng the second egenvector converge to the otmal Cheeger cut as Ths aer s organzed as follows: We start wth some basc roertes of the grah -Lalacan and related functonals n Secton In Secton 3 we rove the varatonal characterzaton of the second egenvalue of the unnormalzed grah -Lalacan Theorem 3 n [], and Secton 4 contans the corresondng characterzatons for the frst and second egenvector n the normalzed case Secton 5 contans the roof of the soermetrc nequalty from Theorem 43 n [], and Secton 6 an outlne of the roof n the normalzed case Fnally,
Secton 7 establshes the connecton between the otmal Cheeger cut and the cut obtaned by thresholdng the second egenvector Theorem 44 n [] As n [], the number of onts s denoted by n = V and the comlement of a set A V s wrtten as A = V \A The degree functon d : V R of the grah s gven as d = n j= w j and the cut of A V and B V, wth A B =, s defned as cuta, B = A, j B w j Moreover, we denote by A the cardnalty of the set A and by vola = A d the volume of A The grah -Lalacan and related functonals The unnormalzed and normalzed grah -Lalacan u defned for any functon f : V R and V as u f = j V w j φ f f j, f = w j φ f f j, d n j V and n are where φ : R R wth φ x = x sgnx As shown n [], one can obtan the egenvalues of the unnormalzed -Lalacan u as local mnma of the functonal F : R V R, F f = Q f f where Q f = f, u f =,j V w j f f j, and each crtcal ont of F corresonds to an egenvector of the -Lalacan To obtan the second egenvalue, we consder the functonal F : R V R, F f = Q f var u f, n analogy to the functonal defned by Amghbech n [] The unnormalzed -varance var u f s defned as var u f = mn f m m R Furthermore, we defne the unnormalzed -mean of f as mean u f = arg mn f m m R In Theorem 3 n Secton 3 we show that the global mnmum of the functonal F s equal to the second egenvalue of the grah -Lalacan u
Analogously, n the case of the normalzed grah -Lalacan, one can defne functonals G : R V R and G : R V R, G f = Q f d f and G f = Q f var n f, wth the normalzed -varance defned as var n f = mn d f m m R, 3 and the normalzed -mean of f as mean n f = arg mn d f m m R Theorems 4 and 4 n Secton 4 establsh the connecton between these functonals and the egenvalues of the normalzed -Lalacan Basc roertes of -Lalacan and related functonals In the followng sectons we restrct ourselves n the roofs to the unnormalzed case Proofs n the normalzed case are smlar Prooston For any functon f : V R, and c R, the unnormalzed -Lalacan u has the followng roertes: u f + c = u f u c f = φ c u f For any functon f : V R, and c R, the normalzed -Lalacan n the followng roertes: has n f + c = n f n c f = φ c n f Proof: These roertes follow drectly from the defnton, as t holds V that u f + c = w j φ f + c f j c = u f j V and u c f = j V w j φ c f f j = j V w j c f f j sgn c f f j = c sgnc j V w j f f j sgn f f j = φ c u f 3
Prooston For any functon f : V R, and c R, the functonal Q f has the followng roertes: Q f + c = Q f Q c f = c Q f Proof: Agan the roertes follow drectly from the defnton Prooston 3 For any functon f : V R, and c R, the unnormalzed -varance var u f has the followng roertes: var u f + c = var u f var u c f = c var u f For any functon f : V R, and c R, the normalzed -varance var n f has the followng roertes: var n f + c = var n f var n c f = c var n f Proof: a Let the -means of f and f + c be gven by m = mean u f and m = mean u f + c Let now m := m + c Then t follows from the defnton of the -varance that var u f + c = mn f + c m m R = f + c m f m = var u f Analogously, for m := m c, we obtan var u var u f = var u f + c f var u f + c and hence b If c = 0, one easly sees that var u c f = 0 = c var u f Let now 4
c 0 Then var u c f = mn c f m m R = c mn f m m R c = c mn f m m R : m = m m R c = c var u f The followng roerty wll later be used to establsh a connecton between the non-constant egenvectors of the unnormalzed and normalzed -Lalacan and the mnmzers of the functonals F res G Prooston 4 Let f R V and m R Then f has unnormalzed -mean value m = mean u f f and only f the followng condton holds: φ f m = 0 Let f R V and m R Then f has normalzed -mean value m = mean n f f and only f the followng condton holds: d φ f m = 0 Proof: We have f m m = f m sgn f m = φ f m, whch mles that a necessary condton for any mnmzer m of the term f m s gven as φ f m = 0 Due to the convexty of the term f m for >, ths s also a suffcent condton Prooston 5 The dervatve of the unnormalzed varance var u f wth resect to f k s gven as var u f = φ 5 f k mean u f
The dervatve of the normalzed varance var n f wth resect to f k s gven as var n f = d k φ f k mean n f Proof: We have var u f = f mean u f = f mean u f sgnf mean u f f mean u f By alyng the defnton of φ and slttng the last term one obtans φ f mean u φ f mean u = φ f k mean u f f f f mean u f mean u f φ f mean u f Due to Pro 4 t holds that φ f mean u f = 0 Thus we obtan var u f = φ f k mean u f The followng rooston rovdes the lnk between the functonals F and F as well as G and G Note that the -mean nsde the -varance s a functon R V R, whch we have to take nto account when takng the dervatve Prooston 6 For any functon f : V R let f denote the unnormalzed -mean of f Then t holds that F F F f l f = F f f f = F f f f k f = f l F f f + F f Ωf k,l, 6
where f l f fk f Ωf k,l = f f f f For any functon f : V R let f denote the normalzed -mean of f Then t holds that where G G G f l f = G f f f = G f f f k f = f l F f f + F f Ωf k,l fl d l d k f fk f Ωf k,l = f f d f f d Proof: The frst statement can be seen drectly by the defntons of F and F and the fact that Q f = Q f f Usng Pro 5 and the defnton of u to f k can be wrtten as Q f var f = var f u, the dervatve of f k Q f var f φ f k f Q f var f wth resect By alyng Pro and Pro as well as the defnton of the -varance, the above exresson can be rewrtten as f f u f f k Q f f f f φ f k f Comarson wth the exresson for F now yelds the second statement For the statement for the second dervatves, one frst shows that f l f = f f l f l = n f and then roceeds analogously to the frst and second statement 3 Varatonal characterzaton of the second egenvalue - Unnormalzed case Theorem 3 The second egenvalue of the unnormalzed grah -Lalacan u s equal to the global mnmum of the functonal F The corresondng 7
egenvector v u of F of u s then gven as v = u c for any global mnmzer n = u c, where c = arg mn c R Furthermore, the functonal F satsfes F tu + c = F u t, c R Lemma 3 Let f be a crtcal ont of the functonal F Then the vector v = f mean u f s an egenfuncton of u wth egenvalue λ = F f Proof: Let f be a crtcal ont of F wth mnmum λ Then F f = 0 By Pro 6 ths mles F f mean u f = 0 as well as λ = F f = F f mean u f It follows that f mean u f s a crtcal ont of F, and by Theorem 3 n [] an egenvector of u wth egenvalue λ Before rovng the other drecton, let us frst derve an mortant roerty of the non-constant egenvectors of the -Lalacan Lemma 3 Let v be a non-constant egenvector of Then φ v = 0 Proof: Let v be a non-constant egenvector of u for all V the equaton v λ φ v = 0 wth egenvalue λ Hence holds As v s not the constant vector, we know that λ 0 and hence V, It follows that φ v = v λ φ v = v λ = w j φ v v j λ j V = w j v v j w j v v j λ λ,j V,v >v j,j V,v <v j = w j v v j w j v j v λ λ,j V,v >v j j,,v j<v = 0, 8
where n the enultmate ste we have erformed a change of the varable names n the second term and n the last ste exloted the fact that w j = w j The above roerty can be seen as a generalzaton of the fact that the larger egenvectors of the unnormalzed standard grah Lalacan are orthogonal to the frst egenvector Lemma 33 Let v be a non-constant egenvector of the -Lalacan u wth egenvalue λ Then there exsts a functon f whch s a crtcal ont of F wth λ = F f and t holds that v = f mean u f Proof: By Theorem 3 n [] we know that v s a crtcal ont of F wth λ = F v Consder now for k R the functon f : V R defned by f = v + k By Lemma 3 t holds that φ v = 0 It follows that k: φ f k = φ f k = φ v = 0 By Pro 4 ths mles that k = mean u f, and hence Pro 6 now mles that F f = F v = f mean u f f mean u f = F v = λ and F f = F f mean u = F v = 0 f Hence t follows that f s a mnmzer of F wth mnmum λ Proof of Theorem 3: Lemma 3 shows the forward drecton of the frst statement of Theorem 3 The reverse drecton follows from Lemma 33 The second statement follows from Pro and 3 9
4 Varatonal characterzaton of the second egenvalue - Normalzed case The followng theorems are the normalzed varants of Theorem 3 and Theorem 3 n [] Theorem 4 The functonal G has a crtcal ont at v R V f and only f v s a -egenfuncton of the normalzed grah -Lalacan n The corresondng egenvalue λ s gven as λ = G v Moreover, we have G αf = G f for all f R V and α R Theorem 4 The second egenvalue of the normalzed grah -Lalacan n s equal to the global mnmum of the functonal G The corresondng egenvector v of n s then gven as v = u c for any global mnmzer u of G n = d u c, where c = arg mn c R Furthermore, the functonal G satsfes G tu + c = G u t, c R The roofs of the above theorems are smlar to the unnormalzed case We just want to sketch the roof of Theorem 4 by gvng the corresondng lemmas wthout roof Lemma 4 Let f be a crtcal ont of the functonal G Then the vector v = f mean n f s an egenfuncton of n wth egenvalue λ = G f Lemma 4 Let v be a non-constant egenvector of n Then d φ v = 0 wth egenvalue λ Then there exsts a functon f whch s a crtcal ont of G Lemma 43 Let v be a non-constant egenvector of the -Lalacan n wth λ = G f and t holds that v = f mean n f 5 Isoermetrc nequalty - Unnormalzed case As shown n [], for > and every artton of V nto C, C there exsts a functon f,c R V such that the functonal F assocated to the unnormalzed -Lalacan satsfes F f,c = cutc, C + 4 C C Exlctely, the functon f,c s gven as f,c = { / C, C, / C, C 5 0
The exresson 4 can be nterreted as a balanced grah cut crteron, and we have the secal cases F f,c = RCutC, C, lm F f,c = RCCC, C It follows that mnmzng the above balanced grah cut crteron s equvalent to mnmzng F wth the restrcton to functons that have the form gven n 5 As the second egenvalue of the -Lalacan s the mnmum of the functonal F taken over all ossble functons wthout the restrcton, the second egenvalue can be seen as a relaxaton of balanced grah cuts The queston s, can we make any statements about the qualty of ths relaxaton? The soermetrc nequalty gves uer and lower bounds on the second egenvalue n terms of the otmal Cheeger cut value defned as h RCC = nf RCCC, C C Theorem 5 Denote by λ the second egenvalue of the unnormalzed grah -Lalacan u For >, hrcc λ h RCC max d Proof of the uer bound n Theorem 5: Let for any > the second smallest egenvalue of the unnormalzed -Lalacan be gven by λ Theorem 3 mles that λ = mn F Q f f R V f = mn, f R V var u f where Q f and var u f are defned as n and Consder now for a artton C, C the functon f,c : V R whch we have defned n 5 Then, usng 4, we have λ F f,c = cutc, C + C C cutc, C { mn C, } C = cutc, C mn { C, C } = RCCC, C As ths nequalty holds for all arttons C, C, t follows that λ nf C RCCC, C = h RCC
For the roof of the lower bound we need to ntroduce some notaton In the followng let us for any functon f : V R denote by f + : V R the functon { f + f, f = 0, 6 0, else Furthermore, we use the notaton Cf t, Ct f V nto the sets for the arttonng of the vertex set C t f = { f > t} and C t f = V Ct f = { f t}, 7 where t R Fnally, for any functon f : V R, we denote by h f,rcc the quantty { } h cutc, C f,rcc = nf C mn { C, C } C = C t f for t 0 8 The value of h f,rcc s the smallest ossble RCC value obtaned by thresholdng f at some t 0 If Cf 0 =, we defne h f,rcc = To rove the lower bound, we roceed n analogy to [] Lemma 5 Suose there exsts a λ 0 such that t holds Cf 0 u f λf Then λ Q f + f + Proof: We have λ f + = λ f + = λ f = λ f f that C 0 f C 0 f Usng the assumton, t follows that λ f + f u f = f + u f C 0 f Alyng the defnton of u, ths can be rewrtten as = f + w j φ f f j j V = w j f + φ f f j + w j f + φ f f j =,j V,j V,j V w j f + f + j φ f f j Let us now have a closer look at the summands n the above sum If both and j are n Cf 0, we have f + f + j φ f f j = f + f + j φ f + f + j = f + f + j
If Cf 0 and j / C0 f, t holds that f + f + j φ f f j = f + f f j sgnf f j = f + f f j f + f, where we have used that f > 0 and f j 0 The last term can be rewrtten as f + = f + f + j Analogously, f / Cf 0 and j C0 f, we have f + f + j φ f f j = f + j f f j sgnf f j = f + j f f j f + j f j = f + j = f + f + j Fnally, n the case that both and j are not n Cf 0, t holds that f + f + j φ f f j = 0 = f + If we combne these results, we obtan that whch comletes our roof,j V,j V f + j w j f + f + j φ f f j w j f + f + j = Q f +, The followng nequalty, whch wll be used n the next lemma, has been shown by Amghbech [] Lemma 5 Amghbech, [] If a, b 0, > and + q =, then b a q b a a + b Lemma 53 For any functon f R V wth 0 < defned n 8 t holds that Q f + f + C 0 f h f,rcc max d V and h f,rcc as Proof: Consder the term w j f + j f + f + j >f + 3
On the one hand we have f + j >f + w j f + j f + = = f + j >f + f + j >f + w j [t ] f + j f + f + j w j f + t dt We can change the order of ntegraton and summaton and obtan f + j >f + f + j w j f + t dt = t 0 f + j >t f + w j dt Note that for t 0, w j = f + j >t f + f j>t f w j = whch leads us to the followng nequalty cutcf t, Cf t = cutct f, Ct f Cf t { nf C j C t f, Ct f C f t = w j = cutc t f, C t f, cutcf t, Ct f { } C mn t f, Cf t C f t } cutc, C mn { C, C } C = C t f for t 0 = h f,rcc C t f, where n the second ste we used the assumton that 0 < ths nequalty holds for all t 0, we obtan t 0 f + j >t f + w j dt 0 t h f,rcc C t f C 0 f dt C t f V As We now use that Cf t = f = >t f + >t for t 0, and change the order of summaton and ntegraton agan, whch leads us to 0 t h f,rcc f + >t dt = h f,rcc = h f,rcc = h f,rcc = h f,rcc f + j >0 f + j 0 [t ] f + j >0 f + j >0 f + j 0 f + j f + t dt 4
So we have just shown the nequalty f + j >f + w j f + j f + h f,rcc On the other hand we have w j f + j f + f + 9 = f + j >f + w j f + j f + + w j f + f + j = f + j >f +,j V w j f + j f +, f + >f + j where agan we exloted the symmetry of the weghts n the second ste Let q be the conjugate of, defned by the equaton + q = The sum can now be decomosed nto,j V = f + j f + f + j f + w j f + j f + / w j f + j f + w j f + j = Q f + / f + j f + f + /q w f + j f + j f + j f + / f + j f + w j f + w j f + q j f + j f + f + j f + q f + j f + where we used Hölder s nequalty n the second ste By alyng Lemma 5 /q, /q 5
we obtan for the second roduct term assumng that /q f + w f + q j j f + f + q w j f + j + f + j f + j f + f + j >f + 4 /q = 4 /q /q f + j f +,j V w j f + + f + j d f + /q max d f + /q / max d = f + Hence we have / f + j f + max d w j Q f + / f + 0 By combnng 9 and 0 we obtan /q /q h f,rcc / f + max d Q f + / f +, whch can be rewrtten as h f,rcc Q f + max d f + Proof of the lower bound n Theorem 5: Let f be the egenfuncton of u corresondng to the second egenvalue λ Let Cf 0 be the set of values where f > 0, as defned n 7 Wthout loss of generalty, we can assume that 0 < C f 0 V, otherwse we just relace f by f We know that u f = λ f on C 0 f, so our condton for Lemma 5 s fulflled Alyng Lemma 5 and 53 yelds λ Q f + h f,rcc f + max d Clearly, we have h f,rcc h RCC, whch comletes the roof 6
6 Isoermetrc nequalty - Normalzed case One can show that also n the normalzed case, for > and every artton of V nto C, C there exsts a functon g,c R V such that the functonal G assocated to the normalzed -Lalacan satsfes G g,c = cutc, C + volc volc Exlctely, the functon g,c s gven as { / volc, C, g,c = / volc, C As n the unnormalzed case, the exresson can be nterreted as a balanced grah cut crteron, and we have the secal cases lm G g,c = NCutC, C, G g,c = NCCC, C It follows that mnmzng the above balanced grah cut crteron s equvalent to mnmzng G wth the restrcton to functons that have the form gven n Wth the same argument as n the unnormalzed case, the second egenvector can be seen as a relaxaton of balanced grah cuts As n the unnormalzed case, the soermetrc nequalty gves uer and lower bounds on the second egenvalue n terms of the otmal Cheeger cut value defned as h NCC = nf NCCC, C C Theorem 6 Amghbech, [] Denote by λ normalzed grah -Lalacan n For >, hncc λ h NCC the second egenvalue of the The roof of the uer bound s smlar to the unnormalzed case For the roof of the lower bound, we use agan the notaton f + for the restrcton of the functon f to ostve values, as well as Cf t, Ct f for a arttonng of the vertex set by thresholdng, as ntroduced n 6 and 7 Furthermore, for any functon f : V R, we denote by h f,ncc the quantty { } h cutc, C f,ncc = nf C mn { volc, volc } C = C t f for t 0 Analogously to the unnormalzed case, the value of h f,ncc s the smallest ossble NCC value obtaned by thresholdng f at some t 0 Agan, we set h f,ncc = n the case C0 f = Lemma 6 Suose there exsts a λ 0 such that t holds Cf 0 n f λf Then Q f + λ f + d that 7
Lemma 6 For any functon f R V wth 0 < volcf 0 volv and h f,ncc as defned above t holds that Q f + h f,ncc f + d Usng the above lemmas the lower bound can now be roven n a smlar way to the unnormalzed case 7 Convergence to the otmal Cheeger cut In -sectral clusterng, a arttonng of the grah s obtaned by thresholdng the real-valued second egenvector v of the grah -Lalacan The otmal threshold s determned by mnmzng the corresondng Cheeger cut, e n the case of the unnormalzed grah -Lalacan u one determnes arg mn C t={ v >t} RCCC t, C t, 3 and smlarly for the second egenvector of the normalzed grah -Lalacan n one comutes arg mn NCCC t, C t 4 C t={ v >t} One can now establsh a connecton between the cut obtaned by thresholdng accordng to the above scheme and the otmal Cheeger cut Theorem 7 Denote by h RCC the rato Cheeger cut value obtaned by tresholdng the second egenvector v of the unnormalzed -Lalacan va 3 Then for >, h RCC h RCC max d h RCC Denote by h NCC the normalzed Cheeger cut value obtaned by tresholdng the second egenvector v of the normalzed -Lalacan va 4 Then for >, h NCC h NCC h NCC Interestngly, the nequaltes become tght for Ths mles that the cut found by thresholdng converges to the otmal Cheeger cut, whch rovdes the man motvaton for -sectral clusterng Proof of Theorem 7: Clearly, the lower bound holds Let now f be the egenfuncton of u corresondng to the second egenvalue λ Let Cf 0 be the set of values where f > 0, as defned n 7 Wthout loss of generalty, we can assume that 0 < C f 0 V, otherwse we just relace f by f We know that u f = λ f on C 0 f, 8
so our condton for Lemma 5 s fulflled Alyng Lemma 5 and 53 yelds λ Q f + f + max d Note that h f,rcc = h RCC, and hence we obtan max d h f,rcc h RCC λ Note that ths bound s tghter than the lower bound from Theorem 5 The above nequalty can be reformulated as h RCC max d λ Usng that λ h RCC, we obtan h RCC max d h RCC As shown by Amghbech [], n the normalzed case one has the nequalty λ hncc Analogously to the unnormalzed case one can show the stronger statement λ h NCC hncc The frst nequalty can be reformulated as h NCC λ, and wth λ h NCC one obtans the result n the normalzed case References [] S Amghbech Egenvalues of the dscrete -Lalacan for grahs Ars Combn, 67:83 30, 003 [] Thomas Bühler and Matthas Hen Sectral Clusterng based on the grah -Lalacan In Proceedngs of the 6th Internatonal Conference on Machne Learnng ICML To aear, 009 9