Machne Leanng -7/5 7/5-78, 78, Spng 8 Spectal Clusteng Ec Xng Lectue 3, pl 4, 8 Readng: Ec Xng Data Clusteng wo dffeent ctea Compactness, e.g., k-means, mxtue models Connectvty, e.g., spectal clusteng Compactness Connectvty Ec Xng
Ec Xng 3 Spectal Clusteng Data Smlates Ec Xng 4
Weghted Gaph Pattonng Some gaph temnology Objects (e.g., pxels, data ponts) W j j I = vetces of gaph G Edges (j) = pxel pas wth W j > Smlaty matx W = [ W j ] Degee d = Σ j G W j d = Σ d degee of G B ssoc(, = Σ Σ j B W j Ec Xng 5 Cuts n a Gaph (edge) cut = set of edges whose emoval makes a gaph dsconnected weght of a cut: cut(, B ) = Σ Σ j B W j =ssoc(, Nomalzed Cut ctea: mnmum cut(,ā) cut(, ) cut(, ) Ncut(, ) = + d d Moe geneally: k Ncut(, K k ) = =, j V \, j V W j = W j = cut(, ) d Ec Xng 6 k 3
Gaph-based Clusteng Data Goupng W j j W j W = f d( x, x )) j ( j G = {V,E} Image sgmentaton ffnty matx: Degee matx: Laplacan matx: W = [ w, j ] D = dag( d ) L = D W (bpatte) patton vecto: x = [ x,...,x = [,,K,,, K ] Ec Xng 7 N ] ffnty Functon W = e, j X X j σ ffntes gow as σ gows How the choce of σ value affects the esults? What would be the optmal choce fo σ? Ec Xng 8 4
Clusteng va Optmzng Nomalzed Cut he nomalzed cut: cut(, cut(, Ncut(, = + d d B Computng an optmal nomalzed cut ove all possble y (.e., patton) s NP had ansfom Ncut equaton to a matx fom (Sh & Malk ): mn x y ( D W) y Ncut( x) = mny y Dy n y {, b} y D = Subject to: Raylegh quotent cut(, cut(, Stll an NP had poblem Ncut(, = + deg() deg( ( + x) ( D S)( + x) ( x) ( D S)( x) > D(, ) x = + ; k = k D ( k) D D(, ) Ec Xng =... 9 Relaxaton mn Ncut( x) = mn x Subject to: y y {, b} y D = y ( D W) y y Dy n Raylegh quotent Instead, elax nto the contnuous doman by solvng genealzed egenvalue system: mn y ( D W) y, s.t. y Dy = y Whch gves: ( D W ) y = λdy Raylegh quotent theoem ( D W ) = Note that so, the fst egenvecto s y = wth egenvalue. he second smallest egenvecto s the eal valued soluton to ths poblem!! Ec Xng 5
lgothm. Defne a smlaty functon between nodes..e.: w = e, j X ( ) X ( j ) σ X. Compute affnty matx (W) and degee matx (D). 3. Solve ( D W ) y = λdy Do sngula value decomposton (SVD) of the gaph Laplacan L V * = ΛV y L = D W 4. Use the egenvecto wth the second smallest egenvalue,, to bpatton the gaph. Fo each theshold k, k ={ y among k lagest element of y*} B k ={ y among n-k smallest element of y*} Compute Ncut( k,b k ) * Output k = ag max Ncut( k, B k ) and k *, B * Ec Xng k y * Ideally y ( D S) y Ncut(, =, wth y {, b}, y D =. y Dy y ( D S) y = λdy y Ec Xng 6
Example (Xng et al, ) Ec Xng 3 Poo featues can lead to poo outcome (Xng et al ) Ec Xng 4 7
Cluste vs. Block matx cut(, cut(, Ncut(, = + d Degee( ) = d B W, j V, j B cut(, cut(, Ncut(, = + d d B B Ec Xng 5 Compae to Mnmum cut Cteon fo patton: mn cut(, = mn Poblem! Weght Weght of of cut cut s s dectly dectly popotonal popotonal to to the the numbe numbe of of edges edges n n the the cut. cut., B W, j B, j B Ideal Cut Cuts wth lesse weght than the deal cut Fst poposed by Wu and Leahy Ec Xng 6 8
Supeo Pefomance? K-means and Gaussan mxtue methods ae based towad convex clustes Ec Xng 7 Ncut s supeo n cetan cases Ec Xng 8 9
Why? Ec Xng 9 Geneal Spectal Clusteng Data Smlates Ec Xng
Repesentaton Patton matx X: [ X ] X =,..., X K segments pxels Pa-wse smlaty matx W: W (, j) = aff (, j) Degee matx D: Laplacan matx L: D w j, j (, ) = L = D W Ec Xng Egenvectos and blocks Block matces have block egenvectos: λ = λ = λ 3 =.7 λ 4 = egensolve.7.7.7 Nea-block matces have nea-block egenvectos: λ =. λ =. λ 3 = -...7 λ 4 = -.. -. egensolve.69.4 -.4.69 -..7 Ec Xng
Spectal Space Can put tems nto blocks by egenvectos: e..7 -..69 -.4..4.69 e -..7 e e Clustes clea egadless of ow odeng: e..7..4.69 -..69 -.4 e -..7 e e Ec Xng 3 Spectal Clusteng lgothms that cluste ponts usng egenvectos of matces deved fom the data Obtan data epesentaton n the low-dmensonal space that can be easly clusteed Vaety of methods that use the egenvectos dffeently (we have seen an example) Empcally vey successful uthos dsagee: Whch egenvectos to use How to deve clustes fom these egenvectos wo geneal methods Ec Xng 4
Method # Patton usng only one egenvecto at a tme Use pocedue ecusvely Example: Image Segmentaton Uses nd (smallest) egenvecto to defne optmal cut Recusvely geneates two clustes wth each cut Ec Xng 5 Method # Use k egenvectos (k chosen by use) Dectly compute k-way pattonng Expementally has been seen to be bette Ec Xng 6 3
Spectal Clusteng lgothm Ng, Jodan, and Wess 3 Gven a set of ponts S={s, s n } Fom the affnty matx w, j = e S S j σ, j, w, = Defne dagonal matx D = Σ κ a k Fom the matx L = D WD / / Stack the k lagest egenvectos of L to fo the columns of the new matx X: X = x x Renomalze each of X s ows to have unt length and get new matx Y. Cluste ows of Y as ponts n R k Ec Xng 7 L x k SC vs Kmeans Ec Xng 8 4
5 Ec Xng 9 Why t woks? K-means n the spectum space! Ec Xng 3 Moe fomally Recall genealzed Ncut Mnmzng ths s equvalent to spectal clusteng = = = = k k V j j V j j k d W W ), cut( ), Ncut(, \, K = = k k d ), cut( ), Ncut( mn K Y WD D Y / / mn I Y Y = s.t. segments pxels Y
oy examples Images fom Matthew Band (R--4) Ec Xng 3 Use s Peogatve Choce of k, the numbe of clustes Choce of scalng facto σ Realstcally, seach ove and pck value that gves the tghtest clustes Choce of clusteng method: k-way o ecusve bpatte Kenel affnty matx w = K( S, S, j j ) Ec Xng 3 6
Conclusons Good news: Smple and poweful methods to segment mages. Flexble and easy to apply to othe clusteng poblems. Bad news: Hgh memoy equements (use spase matces). Vey dependant on the scale facto fo a specfc poblem. W X ( ) X ( j ) σ X (, j) = e Ec Xng 33 7