Context-Aware Clustering

Transcription

1 Context-Awre Clustering Junsong Yun EECS Dept., Northwestern Univ. Evnston, IL, USA Ying Wu EECS Dept., Northwestern Univ. Evnston, IL, USA Abstrct Most existing methods of semi-supervised clustering introduce supervision from outside, e.g., mnully lbel some dt smples or introduce constrins into clustering results. This pper studies n interesting problem: cn the supervision come from inside, i.e., the unsupervised trining dt themselves? If the dt smples re not independent, we cn cpture the contextul informtion reflecting the dependency mong the dt smples, nd use it s supervision to improve the clustering. This is clled context-wre clustering. The investigtion is substntilized on two scenrios of (1) clustering primitive visul fetures (e.g., SIFT fetures) with help of sptil contexts, nd (2) clustering 0 9 hnd written digits with help of contextul ptterns mong different types of fetures. Our context-wre clustering cn be well formulted in closed-form, where the contextul informtion serves s regulriztion term to blnce the dt fidelity in originl feture spce nd the influences of contextul ptterns. A nested-em lgorithm is proposed to obtin n efficient solution, which proves to converge. By exploring the dependent structure of the dt smples, this method is completely unsupervised, s no outside supervision is introduced. 1. Introduction Unsupervised clustering is lrgely settled by the distnce metric tht mesures the dissimilrity or ffinity between two dt points. This cn be regrded s the internl force driving the clustering. Typicl exmples include the k- mens clustering nd spectrl clustering. In prctice, s it is generlly quite difficult to choose the right distnce metric in dvnce, we tend to lern good metric by imposing supervision. Acting s constrint, supervised informtion cn be regrded s the externl force tht blnces or djusts the effect of the internl force. In this wy, we cn sy the distnce metric is tuned or lerned. Supervision is generlly introduced from outside, e.g., mnully lbeling some smples s constrints to perform constrin-bsed clustering [12], or to perform co-trining mong multiple modlities [3], or to perform metric tuning [10]. Then, here is n interesting question: cn the supervision come from inside, i.e., the trining dt themselves? If so, it is still unsupervised, nd cn be clled self-supervised clustering. This is possible when trining dt re not independent. The dependency mong dt is the contextul informtion. Let s tke web-pge grouping s n exmple. The links mong web-pges provide informtion on dependency. We group web-pges not only bsed on if they hve similr contents (fetures) but lso if they shre similr link pges (contexts). Contextul informtion brought by dt dependency provides n importnt clue for dt mining [8]. In computer vision reserch, mny recent work showed tht contextul informtion cn be utilized to resolve the mbiguities nd uncertinties in mny pplictions, including imge serch [6], recognition [9] [1] [2], metric lerning [11], nd imge modeling [16]. If the dt dependency cn be well cptured by the contextul ptterns, which describe the co-occurrences of specific type of dt smples in higher level, it is possible to use it s the supervision to improve clustering. Becuse the contextul informtion is discovered from the unsupervised trining dt themselves, we cll such self-supervised clustering s context-wre clustering. We substntilize it on two cse studies where (1) we cluster primitive visul fetures (e.g., SIFT fetures) for finding locl sptil ptterns in imges, nd (2) we cluster 0 9 hnd written digits with multiple fetures. By feeding bck the contextul ptterns s supervision which chrcterize the cooccurrence sttistics, we cn resolve the mbiguous smples bsed on the hints from their contexts. The novelty of our work lies in two spects. First of ll, we give closed-form formultion of context-wre clustering, where the contextul informtion serves s regulriztion term in trditionl k-mens clustering. Secondly, due to the nice nlyticl properties of the new formultion, we present n efficient nested-em lgorithm for context-wre clustering, which proves to converge. Both simultion nd rel dt vlidte the effectiveness of our method /08/$ IEEE 1

2 2. Context-Awre Clustering 2.1. Motivting exmple: clustering visul primitives We illustrte our context-wre clustering in cse study of clustering visul primitives. Ech visul primitive is denoted s v =(x, y, f), where (x, y) is its sptil loction, nd f denotes the feture vector describing v. In generl f R d cn be ny possible visul fetures to chrcterize locl imge region, like color histogrms or SIFTlike fetures [4] [5]. An imge is collection of visul primitives, nd we denote the visul primitive dtbse s D v = {v i } N. After clustering these visul primitives into words, we cn lbel ech v i D v with l(v i ) Ω, where Ω is the visul word lexicon of size Ω = M. The context of visul primitive is its sptil neighbors in the imge, i.e., those visul primitives tht collocte with it (Fig. 1). For ech visul primitive v i D v, we define its locl sptil neighborhood, e.g. K-nerest neighbors (K- NN) or ɛ-nerest neighbors (ɛ-nn), s its context group G i = {v i,v i1,v i2,,v ik }. The context dtbse is denoted by G = {G i } N. Once the visul primitives re lbeled by Ω, the context dtbse G cn be trnsfered to trnsction dtbse with N records, where ech record t i {0, 1} M is binry vector representtion of G i by indicting which words pper in group G i. This trnsction dtbse is sprse binry mtrix T M N, where ech column is context trnsction t i. The entry t ij =1indictes the j th trnsction contins the i th word nd t ij =0otherwise. In the cse of using sptil K-NN to define context group, we hve M t ij = K, j =1,..., N, becuse ech context group G i contins K visul primitives. An N N sprse binry mtrix Q cn be used to describe the sptil context reltions mong the visul primitives, where q ij =1denotes tht v i belongs to the context group of v j, i.e. v i G j ; nd q ij =0otherwise. Mtrix Q is symmetric when using ɛ-nn to define sptil neighbors, while n symmetric mtrix when using K-NN. The context mtrix Q plys centrl role in our context-wre clustering s it introduces extr reltions mong dt smples other thn in the feture spce. Besides sptil contexts, Q cn present ny other possible contextul informtion mong the N dt smples. In Sec. 3.3, we give nother exmple of pplying contextul informtion from multiple fetures for clustering. Bsed on the word lexicon Ω ( Ω = M), we cn further define phrse lexicon Ψ = {P i } M, where ech phrse P i Ψ is contextul pttern composed of collection of words, i.e. P i Ω. Compred with visul words which lbel visul primitives v, visul phrses lbel trnsctions t. As visul phrses describes the sptil dependencies mong visul words, they cn be more meningful ptterns in higher level [14]. For exmple in Fig. 1, the existence of visul phrse P = {, b} shows tht two words, b Ω Symbol Definition d dimensionlity of the feture vector N number of visul primitives M number of visul words M number of visul phrses t M 1 context trnsction T M N the trnsction dtbse Q N N sptil context reltions of visul primitives u d 1 prototype of visul word ũ M 1 prototype of visul phrse U d M prototypes of M visul words Ũ M M prototypes of M visul phrses R M N lbel mtrix of N primitives with M words R M N lbel mtrix of N groups with M phrses D M N distortion mtrix of N primitives with M words D M N distortion mtrix of N groups with M phrses Tble 1. Nottions of symbols. Bold upper cse letters denote mtrices nd bold lower cse letters denote vectors. b c b c c b b d e c e c b b b e Group Representtion G ={, b, d, e} Trnsction Representtion b c d e t =[ ] T Visul Phrse Representtion u =[ ] T Figure 1. Illustrtion of sptil contexts: context group G, trnsction t nd visul phrse P. The left figure denotes n imge nd ech rectngle denotes visul primitive. We suppose the visul word lexicon contins 5 words: Ω = {, b, c, d, e} nd ech visul primitive is lbeled by word. The circle denotes sptil context group generted by visul primitive. The highlighted visul primitives re instnces of discovered visul phrse P = {, b}. co-occur frequently in locl imge regions nd my form meningful visul pttern. Ech P j Ψ is presented by binry vector ũ j {0, 1} M which describes its word compositions, where ũ j (i) =1indictes tht the i th word is contined in P j. The mtrix Ũ M M is further pplied to represent Ψ, where ech column of Ũ is ũ j. Correspondingly, we use rel mtrix U d M to represent Ω, where ech column is feture vector to represent word prototype u j R d. All of our nottions re listed in tble Problem Formultion We first review the k-mens clustering nd its solution by the EM-lgorithm. By performing trditionl k-mens clustering on collection of visul primitives v i D v,the following men squre distortion is minimized:

3 where N M J 1 = r ij f i u j 2 = tr(r T D), (1) j=1 f i is the d 1 feture vector, nd u j is the center of the cluster (prototype of visul words); denotes the Eucliden distnce nd tr( ) denotes the mtrix trce; D M N denotes the distnce mtrix, where d ij = f j u i 2 denotes the distnce between the j th visul primitives nd the i th visul word prototype; R M N denotes the lbel indictor mtrix of the visul primitives, where r ij =1if the j th visul primitive is lbeled with the i th word; nd r ij =0otherwise. Stndrd EM-lgorithm cn be performed to minimize the distortion in Eq. 1 by itertively updting R (E-step) nd D (M-step). By minimizing the objective function J 1, k- mens clustering tries to mximize the dt likelihood under mixture Gussin distribution nd ssumes ll observtion smples v i D v re independent from one nother: N Pr(D v Ω) = Pr(v i Ω). (2) However, such n independent ssumption does not hold here becuse visul primitives hve sptil dependency with ech other. Thus they re not independent in the feture spce. As result, we need to tke into considertion these sptil contextul informtion nd cnnot cluster visul primitives only bsed on their fetures f i. In order to consider both feture nd contextul informtion for clustering, we propose regulrized objective function bsed on k-mens: N M N M J = r ij f i u j 2 + λ r ijd H (t i, ũ j ) j=1 j=1 = tr(r T D)+λ tr( R T D), (3) where λ>0is positive constnt for regulriztion; r ij is the binry lbel indictor of trnsctions, with r ij =1denoting tht the i th trnsction is lbeled with the j th visul phrse; nd r ij =0otherwise. Similr to R, R N M is mtrix to describe the clustering results of trnsctions t. For deterministic clustering, we hve the following constrints for R nd R: M M r ij =1, r ij =1, i =1,..., N. (4) j=1 j=1 d H (t i, ũ j ) denotes the Hmming distnce between two binry vectors: trnsction t i nd context pttern ũ j, where 1 is the M 1 ll 1 vector: d H (t i, ũ j ) = M [ t T i ũ j +(1 t i ) T (1 ũ j ) ] = t T i 1 + ũ T j 1 2t T i ũ j. (5) Given the objective function in Eq. 3 with M, M nd λ re fixed prmeters, our objectives re two-fold: (1) clustering ll the visul primitives v i into M clsses (word lexicon Ω) nd simultneously (2) clustering ll the context trnsctions t i T into M clsses (phrse lexicon Ψ). The clustering results re presented by R nd R respectively. Since ech visul primitive cn generte sptil context group, we finlly end up with two lbels for every primitive: (1) the word lbel of itself nd (2) the phrse lbel of the sptil group it genertes. Compred with k-mens clustering which ssumes convex (e.g. Gussin) shpe for ech cluster in the feture spce, our regulriztion term cn modify the cluster into n rbitrry shpe by considering the influences from the higher phrse level. Similr to the k-mens clustering, this formultion is lso chicken-ndegg problem where we cnnot estimte D, D, R nd R simultneously Itertive Solution: Nested-EM lgorithm The objective function in Eq. 3 cn be prtitioned into two prts: J = tr(r T T D) + λ tr( R D), }{{}}{{} J 1 J 2 where J 1 = tr(r T D) nd J 2 = λ tr( R T D) correspond to the quntiztion distortions of visul primitives nd context groups respectively. Although it looks we could minimize J by minimizing J 1 nd J 2 seprtely, e.g., through two independent EM-processes, this is ctully infesible becuse J 1 nd J 2 re coupled. By further nlyzing J 1 nd J 2, we find tht lthough visul primitive distortions D only depends on R, the context group distortions D depends on both visul primitive lbels R nd context group lbels R. Thus it is infesible to minimize J 1 nd J 2 seprtely due to their correltion. In the following, we show how to decouple the dependency between J 1 nd J 2 nd propose our nested-em lgorithm. Initiliztion: 1. Clustering ll visul primitives {v i } N into M clsses, e.g. through k-mens clustering, bsed on the Eucliden distnce. 2. Obtining the visul primitives lexicon Ω (represented by U) nd the distortion mtrix D. 3. Clustering ll context groups {G i } N into M clsses bsed on the Hmming distnce, nd obtin the visul phrse lexicon Ψ (represented by Ũ), s well s the distortion mtrix D. E-step: The tsk is to lbel visul primitives v i nd context groups G i with Ω nd Ψ, nmely to updte R nd R given D nd

4 D, where D nd D cn be directly computed from U nd Ũ respectively. Bsed on the nlysis bove, we need to optimize R (corresponding to J 1 ) nd R (corresponding to J 2 ) simultneously to minimize J, becuse J 1 nd J 2 re correlted. According to the Hmming distnce in Eq. 5, we cn derive the mtrix form of context groups distortions: D = 2 Ũ T T + 1 T T + Ũ T 1Ũ, where 1 T is n M M ll 1 mtrix nd 1Ũ is n M N ll 1 mtrix. Moreover, trnsction dtbse T cn be determined by T = RQ, becuse ech trnsction column cn be obtined s t j = N q ij r 1 i, where q ij is binry indictor of whether primitive v i belongs to the context group of v j, nd r i denotes the i th column of R which describe the word lbel of v i. Bsed on the bove, we derive Eq. 3 s follows: J(D, D, R, R) =tr(r T D)+λ tr( R T D) (6) = tr(r T D)+ = tr(r T D)+ = tr(r T D)+ λ tr[ R T ( 2Ũ T T + 1 T T + Ũ T 1Ũ )] (7) λ tr[ R T ( 2Ũ T RQ + 1 T RQ + Ũ T 1Ũ )] λ tr[ R T ( 2(Ũ T T )RQ + Ũ T 1Ũ )] = tr(r T D) 2λ tr[ R T (Ũ T T )RQ]+ λ tr( R T Ũ T 1Ũ ) (8) = tr(r T D) 2λ tr[q T R T (Ũ T T ) T R]+ λ tr( R T Ũ T 1Ũ ) (9) = tr(r T D) 2λ tr[r T (Ũ T T ) T RQ T ]+ λ tr( R T Ũ T 1Ũ ) (10) = tr[r T (D 2λ (Ũ T T ) T RQ T )] + λ tr( R T Ũ T 1Ũ ), (11) where we pply three properties of mtrix trce: for squre mtrix A nd B, wehve(1)tr(a) =tr(a T ) (Eq. 9), (2) 1 Strictly, t j is binry vector only if it contins distinguishble primitives, i.e. ech primitive belongs to different word in t j. However, our solution is generic nd do not need t j to be binry, s long s we pply the distortion mesure s in Eq. 5. tr(ab) =tr(ba) (Eq. 10), nd (3) tr(a) +tr(b) = tr(a + B) (Eq. 11). Bsed on the bove nlysis, we propose n E-step to itertively updte R nd R to decrese J. Recll tht R nd R re lbel indictor mtrices constrined by Eq We first fix R nd updte R. Bsed on Eq. 8, let we hve H = λ ( 2Ũ T RQ + 1 T RQ + Ũ T 1Ũ ), J = tr(r T T D) + tr( R H). (12) }{{}}{{} J 1 J 2 Therefore we only need to minimize J 2 = tr( R T H) s J 1 = tr(r T D) is constnt given R nd U. Becuse ech column of R contins single 1 (Eq. 4), we updte R to minimize J 2 bsed on the following criterion, j =1, 2,...N: { 1 i = rg mink hkj r ij =, (13) 0 otherwise where h kj is the element of H nd r ij is the element of R. H cn be clculted bsed on Q, Ũ nd R which re ll given. 2. Similr to the bove step, now we fix R nd updte R. Bsed on Eq. 11, let H = D 2λ (Ũ T T ) T RQ T We get nother representtion of J: J = tr(r T H) + λ tr( R T Ũ T 1Ũ ), (14) }{{}}{{} J 3 J 4 where J 4 = λ tr( R T Ũ T 1Ũ ) is constnt given R nd Ũ. Therefore, only J 3 cn be minimized. We updte R to minimize J 3 s follows, j =1,...N: { 1 i = rg mink h r ij = kj, (15) 0 otherwise where h kj is the element of H nd r ij is the element of R. The bove E-step itself is n EM-like process becuse we need to updte R nd R itertively until J converges. The objective function J decreses monotoniclly t ech step. M-step: After knowing the lbels of visul primitives nd visul groups (R nd R), we wnt to estimte better visul lexicons Ω nd Ψ. From Eq. 3, D nd D re not interlced nd thus U nd Ũ cn be optimized seprtely. We pply the following two steps to updte U nd Ũ seprtely:

5 1. Reclculte the cluster centroid for ech visul word clss {u i } M like trditionl k-mens lgorithm, with Eucliden distnce. Updte U nd D to decrese J. 2. Reclculte the cluster centroid for ech phrse {ũ i } M, with Hmming distnce (see the Appendix for the updte detils). Updte Ũ nd D to decrese J. Both of the bove steps gurntee tht J is decresing, therefore the whole M-step decreses J monotoniclly. Our method is clled nested-em lgorithm becuse there re two nested EM processes, where the E-step itself is n EM process. We describe the nested-em lgorithm in Alg. 1. Algorithm 1: Nested-EM Algorithm. input : visul primitive dtbse D = {v i}, contextul reltions Q, prmeters: M, M, λ output : visul word nd phrse lexicons: Ω nd Ψ; clustering results R nd R 1 Init: (1) clustering visul primitives to get Ω nd U; 2 (2) bsed on Ω, clustering visul groups to get Ψ nd Ũ; 3 while J is decresing do 4 E-step: fix U nd Ũ, updte R nd R 5 nested-e step: fix R, updte R (Eq. 13) 6 nested-m step: fix R, updte R (Eq. 15) 7 if J is decresing then 8 goto E-setp 9 else 10 Goto M-step M-step: fix R nd R, updte U nd Ũ seprtely. return U, Ũ, R, R. Becuse the solution spces of R nd R re discrete nd finite, ccording to the monotonic decresing of J t ech step of our nested-em lgorithm, we hve theorem 1. Theorem 1 convergence of the nested-em lgorithm The nested-em lgorithm cn converge in finite steps. 3. Experiments 3.1. Simultion results To illustrte the ide of our context-wre clustering, we synthesize sptil dtset for simultion. A concrete exmple of this sptil dtset cn be n imge. All the smples hve two representtions with regrding to (1) feture domin, f R 2 nd (2) sptil domin (x, y) N N s shown in Fig. 3 () nd (b). In our cse, we hve 5 different types of visul primitives lbeled s:,, O,, or. In the sptil domin, {, } is generted together to form co-occurrent contextul pttern, while {O,, } is the other contextul pttern. In the feture domin, ech of the 5 clusters hs different number of smples nd re generted bsed on Gussin distributions of different mens nd vrinces. Bsed on the feture domin only, clustering is chllenging problem becuse some of these Gussin distributions re hevily overlpped, for exmple, clusters, O, nd re hevily overlpped. Our tsks re (1) clustering visul primitives into words, nd (2) recover the visul phrses P 1 = {, } nd P 2 = {O,, }. We compre the performnces of the context-wre clustering with different choices of λ (λ = 0, 400, 0) in Fig. 3 (c),(d) nd (e), where λ = 0 gives the sme results s the k-mens clustering. The mjor differences of the clustering results pper from the cluster. Although hevily overlps with clusters O nd, most of its smples re still correctly lbeled bsed on the help from its sptil context: cluster. For exmple, lthough it is difficult to determine smple v locted in the overlpped regions of nd O in the feture spce, we cn resolve the mbiguity by observing the sptil contexts of v. If is found in its sptil context, then v should be lbeled s becuse discovered visul phrse {, } supports such lbel. Figure 3(f) shows the itertions of nested-em lgorithm with λ = 0. Ech itertion corresponds to n individul E-step or n M-step until converge. We decompose the objective function into J = J 1 + J 2, where J, J 1, J 2 re the red, blck nd pink curves respectively. All these three curves re normlized by J mx = J 0, which is the J vlue t the initiliztion step. Compred to k-mens clustering which minimizes distortions J 1 in feture spce only, our context-wre clustering scrifices J 1 to gin lrger decrese of distortion J 2 in the context spce, which gives smller totl distortion J. The error rte curve (blue) describes the percentge of smples tht re wrongly lbeled t ech step, nd we notice it decreses consistently with our objective function J. In terms of clustering errors, the context-wre clustering (e =0.12 when λ = 0) performs significntly better thn the k-mens method (e =0.25). The prmeter λ blnces the two clustering criteri: (1) clustering bsed on visul fetures f (J 1 ) nd (2) clustering bsed on sptil contexts (J 2 ). The smller the λ, the more fithful the clustering results follow the feture spce, where smples hve similr fetures re grouped together. An extreme cse is λ = 0 when no regulriztion is pplied in Eq. 3 by ignoring the feedbck from contexts. In such cse, our context-wre clustering is equl to k-mens clustering. On the other hnd, lrger λ fvors the clustering results tht support the discovered context ptterns (e.g. visul phrses), thus smples hve similr contexts re more likely to be grouped together Imge texton discovery To vlidte whether the discovered visul phrses cn relly cpture common sptil imge ptterns [13], we test

6 collection of texture imges 2. nd n exmple is presented in Fig. 3.2 Given n imge, we first detect SIFT points [7] nd tret keypoints of scle rnges between 1 nd 2 s visul primitives: D v = {v i }. We pply sptil K-NN groups to build the group dtbses G, with K = 10. Welet λ = τj 0 1/J 0 2, where J 0 1 nd J 0 2 re the initiliztion vlue of J 1 nd J 2 respectively; τ > 0 is the prmeter to blnce the distortions between SIFT fetures (word level) nd contextul ptterns (phrse level). k-mens clustering of visul primitives (k=2). Context-wre clustering: initiliztion of visul phrses. Context-wre clustering: fter the 1 st full EM itertion. Context-wre clustering: finl results (19 full EM iter). Figure 2. The 1 st row shows 2 visul words (red nd purple) formed through k-mens clustering. From the 2 nd to 4 th row, we show 2 visul phrses (red nd purple) discovered through contextwre clustering. There exist two types of ner-regulr textures in the imge. One is the flower pttern locted in the clothes (with deformtions) nd the other is the regulr textures locted in the right bottom. We notice tht k-mens clustering of visul primitives cnnot distinguish from two different textures. Prmeters used re M =25, M =2, τ =0.5. For n imge of size nd contining 0-0 visul primitives, the nested-em lgorithm cn converge within 40 full EM itertions. It is interesting to notice tht the discovered visul phrses re of sptil structures, 2 Imges re from source: Plese see supplementry mterils for more results such s flowers in Fig In Fig. 3.2, we lso show how our nested-em lgorithm corrects the imperfect clustering results itertively, by using the sptil contextul informtion s the feedbck. In comprison, conventionl k-mens clustering cnnot obtin stisfctory results if clustering visul primitives individully Multiple-view clustering Multiple-view clustering is nother typicl ppliction of our context-wre clustering lgorithm. In multiple-view clustering [15], ech dt smple v = {f 1, f 2,..., f c } is represented by different types of fetures f i. Our tsk is to cluster collection of dt smples D v = {v j } N j=1.asimple solution is to conctente ll fetures into long feture vector f = f 1 f 2... f c. Then we cn perform trditionl clustering in the new formed feture spce f. However, becuse the depedent informtion mong different types of fetures re not well utilized, such simple solution cnnot get stisfied results. We select the multiple fetures dt set from the UCI Mchine Lerning Repository for evlution. This multiclss dt set consists of hndwritten numerls ( 0 9 ) extrcted from collection of Dutch utility mps. Ech clss contins dt smples nd the dt set hs 0 digits in totl. Ech digit is represented in terms of the 6 fetures nd we select 3 of them for context-wre clustering: (1) 76 Fourier coefficients of the chrcter shpes (fou); (2) 64 Krhunen-Loeve coefficients (kr) nd (3) 240 pixel verges in 2 x 3 windows (pix). A dt smple v thus genertes 3 primitives f fou, f kr nd f pix in 3 feture spces respectively. As result, we obtin in totl N = 0 3 primitives nd cluster them for word lexicon. The originl dt smple v now corresponds to context group G which chrcterizes the co-occurrent dependency mong different types of fetures f i. Ech v D v genertes to trnsction t of length 3. We further cluster these n = 0 trnsctions into phrses. In the initiliztion step, we build the word lexicon Ω i ( Ω i =10, i = fou,kr,pix)for three types of fetures seprtely nd obtin finl lexicon Ω = Ω fou Ω kr Ω pix ( Ω =30). The phrse lexicon Ψ is then constructed bsed on Ω. By considering both distortions from 3 individul fetures nd their contextul ptterns, the objective function in Eq. 3 now becomes: 3 J = tr(r T i D i )+λ tr( R T D) = tr(r T D)+λ tr( R T D), where R nd D re mtrices contining 3 digonl blocks corresponding to fou, kr nd pix fetures respectively. With stright forwrd djustment of mtrices sizes in Tble 1, we cn still pply the nest-em lgorithm in Alg. 1. Tble 2 compres conventionl k-mens clustering with our context-wre clustering. Specificlly, we try k-mens

7 clustering in 3 fetures individully, nd lso in conctention of 3 fetures. In ech cse, k-mens clustering is repeted times nd the best result with minimum totl distortion is selected for comprison. In context-wre clustering, to blnce between the dt fidelity in feture spce (J 1 ) nd the influence of contextul informtion J 2, we select τ =1, which results in λ = τj 0 1/J 0 2 = From Tble 2, we cn see tht lthough ech individul feture hs limited bility in clustering, the contextul pttern mong them cn help to improve the clustering results significntly. Also our context-wre clustering performs better (with error 13.5%) thn simply conctenting ll fetures for k-mens clustering (with error 17.9%). Tble 2. multiple feture clustering: comprison between trditionl k-mens clustering nd context-wre clustering. #feture #clss error k-mens (fou) 76 k= % k-mens (kr) 64 k= % k-mens (pix) 240 k= % k-mens (ll) k= % context-wre M = 30; M = % 4. Conclusion We present in this pper new formultion of selfsupervised clustering, clled context-wre clustering, nd show how contextul informtion cn feed bck to improve the clustering results. Two kinds of contextul informtion (1) sptil contexts of visul primitives nd (2) contextul ptterns mong different types of fetures re pplied to improve the clustering results in (1) imge texton discovery nd (2) multiple view clustering of hnd written digits, respectively. Compred with trditionl k-mens clustering, our context-wre clustering considers the dt (or feture) dependency in higher level. Thus it not only gets better clustering results, but lso cn revel the hidden structures mong dt smples. The proposed nested-em lgorithm is n efficient itertive solution for the context-wre clustering nd is proved to converge. It provides generl solution to context-wre clustering. Besides sptil contexts nd feture contexts proposed in our experiments, other types of contextul informtion cn lso be incorported. Appendix We discuss how to updte the prototypes of visul phrses (Ũ) in the M-step. Given cluster of M 1 trnsctions X = {t i } n, our trget is to find their centroid ũ {0, 1} M such tht the totl quntiztion distortions re minimized under the Hmming distnce criterion in Eq. 5. The optimiztion problem is formulted s: min ũ {0,1} M n [ M (t T i ũ +(1 t T i )(1 ũ)) ]. Let ũ k denote the k th element of ũ, we minimize the following objective function by using the Lgrngin nd let λ t 0 to obtin the unique mximum solution: f(ũ,λ)= n t=1 By pplying f(ũ,λ) =0, ũ k we obtin nd Finlly, we hve M [ 2t k i ũ k t k i ũ k +1+λ k ũ k (1 ũ k ) ]. f(ũ,λ) λ k =0, λ k 0, k, ũ k = 1 2 (2 n tk i n λ k +1), k, λ k = 2 n t k i n 0. ũ k = sgn(2 n tk i n)+1, 2 where sgn() =1if 0, nd sgn() = 1 if <0. Acknowledgment This work ws supported in prt by Ntionl Science Foundtion Grnts IIS nd IIS References [1] J. Amores, N. Sebe, nd P. Rdev. Context-bsed object-clss recognition nd retrievl by generlized correlogrms. IEEE Trns. on Pttern Anlysis nd Mchine Intelligence, 29(10): , 7. [2] S. Belongie, J. Mlik, nd J. Puzich. Shpe mtching nd object recognition using shpe contexts. IEEE Trns. on Pttern Anlysis nd Mchine Intelligence, 2. [3] A. Blum nd T. Mitchell. Combining lbeled nd unlbeled dt with co-trining. In Proc. of Intl. Conf. on Mchine Lerning, [4] O. Boimn nd M. Irni. Similrity by composition. In Proc. of Neurl Informtion Processing Systems, 6. [5] A. Frome, Y. Singer, nd J. Mlik. Imge retrievl nd clssifiction using locl distnce functions. In Proc. of Neurl Informtion Processing Systems, 6. [6] H. Jegou, H. Hrzllh, nd C. Schmid. A contextul dissimilrity mesure for ccurte nd efficient imge serch. In Proc. IEEE Conf. on Computer Vision nd Pttern Recognition, 7. [7] D. Lowe. Distinctive imge fetures from scle-invrint keypoints. Intl. Journl of Computer Vision, 4. [8] Q. Mei, D. Xin, H. Cheng, J. Hn, nd C. Zhi. Generting semntic nnottions for frequent ptterns with context nlysis. In Proc. ACM SIGKDD, 6. [9] E. Shechtmn nd M. Irni. Mtching locl self-similrities cross imges nd videos. In Proc. IEEE Conf. on Computer Vision nd Pttern Recognition, 7. [10] E. P. Xing, A. Y. Ng, M. I. Jordn, nd S. Russell. Distnce metric lerning, with ppliction to clustering with side-informtion. In Proc. of Neurl Informtion Processing Systems, 2. [11] J. Ye, Z. Zho, nd H. Liu. Adptive distnce metric lerning for clustering. In Proc. IEEE Conf. on Computer Vision nd Pttern Recognition, 7.

8 () Feture Domin (b) Sptil Domin (c) k-mens Clustering (k =5) (d) Context-Awre Clustering (λ = 400) J/Jmx J1/Jmx J2/Jmx error rte itertion (e) Context-Awre Clustering (λ = 0) (f) Performnce (λ = 0) Figure 3. Context-wre clustering on the synthesized sptil dt nd the comprison with the k-mens lgorithm. Prmeter used re M =5, M =2nd ɛ = in serching for ɛ-nn sptil groups. See texts for descriptions. Best seen in color. [12] S. X. Yu nd J. Shi. Segmenttion given prtil grouping constrints. IEEE Trns. on Pttern Anlysis nd Mchine Intelligence, 4. [13] J. Yun nd Y. Wu. Sptil rndom prtition for common visul pttern discovery. In Proc. IEEE Intl. Conf. on Computer Vision, 7. [14] J. Yun, Y. Wu, nd M. Yng. Discovery of colloction ptterns: from visul words to visul phrses. In Proc. IEEE Conf. on Computer Vision nd Pttern Recognition, 7. [15] D. Zhou nd C. J. Burges. Spectrl clustering nd trnsductive lerning with multiple views. In Proc. Intl. Conf. on Mchine Lerning, 7. [16] S.-C. Zhu, C. en Guo, Y. Wng, nd Z. Xu. Wht re textons? Intl. Journl of Computer Vision, 5.