CLUSTER ANALYSIS. SUKANTA DASH M.Sc. (Agricultural Statistics), Roll No I.A.S.R.I., Library Avenue, New Delhi Chairperson: Sh. S.D.

Transcription

1 CLUSTER ANALYSIS SUKANTA DASH M.Sc. (Agrcultural Statstcs), Roll No I.A.S.R.I., Lbrary Avenue, New Delh-002 Charperson: Sh. S.D. Wah Abstract: Cluster analyss s a technque for groupng ndvdual or objects nto unknown groups. It dffers from other methods of classfcaton n vew that the number and characterstcs of the groups are to be derved from the data and are not usually known pror to the analyss. Here, the commonly used methods of clusterng lke Herarchcal clusterng and Non herarchcal clusterng wll be dscussed n detal. Lnkage methods are sutable for clusterng tems, as well as varables, lke Sngle lnkage, Complete lnkage and Average lnkage. Ward s herarchcal clusterng procedure s based on mnmzng the loss of nformaton from jonng two groups. Ths procedure s usually mplemented wth loss of nformaton taken to be as an ncrease n an error sum of squares. Dendrogram s also called herarchcal tree dagram or plot, and shows the relatve sze of the proxmty coeffcents at whch cases were combned. Proxmty measures are used to represent the nearest of two objects. There are two types of measure lke smlarty measure and dssmlarty measure.here, dfferent clusterng methods and dstance (smlarty) measures wll be explaned n detal. Besdes, an llustraton of clusterng procedure on lve data followed by use of software (SAS) wll be dscussed. Keywords: Cluster Analyss, Herarchcal Clusterng, Non Herarchcal Clusterng (Kmeans clusterng), Dendrogram, Proxmty Measures, Smlarty and Dssmlarty Measures.. Introducton Cluster analyss s usually done n an attempt to combne cases nto groups when the group membershp s not known pror to the analyss. Cluster analyss s a technque for groupng ndvdual or objects nto unknown groups. It dffers from other methods of classfcaton such as Dscrmnant analyss, n that n cluster analyss the number and characterstcs of the groups are to be derved from the data and are not usually known pror to the analyss. In bology, cluster analyss has been used for decades n the area of taxonomy, where lvng thngs are classfed nto arbtrary groups on the bass of ther characterstcs group. The classfcaton proceeds from the most general to the most specfc n steps. The most general classfcaton s kngdom followed by phylum, subphylum, and class etc. Cluster analyss has been used n medcne to assgn patent to specfc dagnostc categores on the bass of ther presentng symptoms and sgns. Cluster analyss s also an mportant tool for nvestgaton n data mnng. For example consumers can be clustered on the bass of ther purchases n marketng research. Here the emphass may be on the methods that can be used for large data sets. In short t s possble to fnd applcaton of cluster analyss n vrtually any feld of research. It s also possble to cluster the varables rather than the cases. Clusterng of varables s sometmes used n analyzng the tems n a scale to determne whch tems tends to be close together n terms of ndvdual response to them.

2 2. Clusterng Methods (Johnson and Wchern, 2006) The commonly used methods of clusterng fall nto two general categores. () Herarchcal and () Non herarchcal. Herarchcal clusterng technques proceed by ether a seres of mergers or a seres of successve dvsons. Agglomeratve herarchcal method starts wth the ndvdual objects, thus there are as many clusters as objects. The most smlar objects are frst grouped and these ntal groups are merged accordng to ther smlartes. Eventually, as the smlarty decreases, all subgroups are fused nto a sngle cluster. Dvsve herarchcal methods work n the opposte drecton. An ntal sngle group of objects s dvded nto two sub groups such that the objects n one sub group are far from the objects n the others. These subgroups are then further dvded nto dssmlar subgroups. The process contnues untl there are as many subgroups as objects.e., untl each object form a group. The results of both agglomeratve and dvsve method may be dsplayed n the form of a two dmensonal dagram known as Dendrogram. It can be seen that the Dendrogram llustrate the mergers or dvsons that have been made at successve levels. Lnkage methods are sutable for clusterng tems, as well as varables. Ths s not true for all herarchcal agglomeratve procedure. The followng types of lnkage are now dscussed: () Sngle lnkage (mnmum dstance or nearest neghbour), () Complete lnkage (maxmum dstance or farthest neghbour) and () Average lnkage (average dstances). The mergng of cluster under the three lnkage crtera s llustrated schematcally n the fgure gven below. cluster dstance d d 5..3 d 3 +d 4 +d 5 +d 23 +d 24 +d

3 From the above fgure, we see that Sngle lnkage results when groups are fused accordng to the dstance between ther nearest members. Complete lnkage occurs when groups are fused accordng to the dstance between there farthest members. For Average lnkage, groups are fused accordng to the average dstance between par of members n the respectve sets. The followng are the steps n the agglomeratve herarchcal clusterng algorthm for groups of N objects (tems or varables).. Start wth N clusters, each contanng a sngle entty and an N N symmetrc matrx of dstance (or smlartes) D = {d k }.. Search the dstance matrx for the nearest (most smlar) par of clusters. Let the dstance between most smlar clusters U and V be d uv.. Merge clusters U and V. Label the newly formed cluster (UV). Update the entres n the dstance matrx by (a) deletng the rows and columns correspondng to clusters U and V and (b) addng a row and column gvng the dstances between cluster (UV) and the remanng clusters. v. Repeat steps () and () a total of N- tmes (All objects wll be n a sngle cluster after the algorthm termnates). Record the dentty of clusters that are merged and the levels (dstances or smlartes) at whch the mergers take place. The basc deas behnd the cluster analyss are now shown by presentng the algorthm components of lnkage methods. 2. Sngle Lnkage The nputs to a sngle lnkage algorthm can be dstances or smlartes between par of objects. Groups are formed from the ndvdual enttes by mergng nearest neghbors,.e. smallest dstance or largest smlartes. Intally, we must fnd the smallest dstance n D = {d k } and merge the correspondng objects, say, U and V, to get cluster (UV). For step 3 of general algorthm the dstance between (UV) and any other cluster W are computed by d (u,v),w = mn {d uw, d vw } The results of sngle lnkage clusterng can be graphcally dsplayed n the form of Dendrogram or tree dagram. The branches n the tree represent clusters. The branches come together (merge) at nodes whose postons along a dstance (or smlarty) axs ndcate the level at whch the fuson occurs. 2.2 Complete Lnkage Here at each stage, the dstance (smlarty) between clusters s determned by the dstance (smlarty) between the two elements. One from each cluster that s most dstant. Thus complete lnkage ensures that all tems n a cluster are wth n some maxmum dstance (or mnmum smlarty) of each other. The general agglomeratve algorthm agan starts by fndng the mnmum entry n D = {d k } and mergng the correspondng objects, such as U and V, to get cluster (UV). For step () of general algorthm, the dstance between (UV) and any other cluster W s D (uv)w = max {d uw, d vw } 3

4 Here d uw and d vw are the dstances between the most dstant members of clusters U and W and clusters V and W. 2.3 Average Lnkage Average lnkage treats the dstances between two clusters as the average dstance between all pars of tems where one member of par belongs to each cluster. Agan the nput to average lnkage algorthm may be dstances or smlartes and the method can be used to group objects or varables. The average lnkage algorthm proceeds n the manner of the general algorthm, we begn by searchng the dstance matrx D = {d k } to fnd the nearest (most smlar) objects for example U and V. These objects are merged to form the cluster (UV). For step 3 of general agglomeratve algorthm the dstance between (UV) and other cluster W are determned by d (uv)w = ( d k ) / (N (uv) * N w ), k where d k s the dstance between object n the cluster (UV) and object k n the cluster W, and N uv and N w are the member of tems n clusters (UV) and W respectvely. 2.4 Centrod Ths method assgns each tem to the cluster havng nearest centrod (means). The process has three steps, Partton the tems nto k ntal clusters. Proceed through the lst of tems assgnng an tem to the cluster whose centrod (mean) s nearest. Recalculate the centrod (mean) for the cluster recevng the new tem and the cluster losng the tem. Repeat step () untl no more assgnments take place. 2.5 Ward s Herarchcal Clusterng Methods Ward consdered herarchcal clusterng procedure based on mnmzng the loss of nformaton from jonng two groups. Ths method s usually mplemented wth loss of nformaton taken to be an ncrease n an error sum of squares crteron, ESS. Frst for a gven cluster k, let ESS k be the sum of the square devaton of every tem of the cluster from the cluster mean (centrod). If there are currently K clusters, defne ESS as the sum of the ESS k or ESS = ESS + ESS ESS k. At each step n the analyss the unon of every possble par of cluster s consdered and the two clusters whose combnaton results n the smallest ncrease n ESS (mnmum loss of nformaton) are joned. Intally each cluster consst of a sngle tem, and f there are N tems, ESS k = 0, k =, 2,, N so ESS = 0 at the other extreme, when all the clusters are combned n a sngle group of N tems, the value of ESS s N ESS = (X X) (X X), j= j j where X j s the multvarate measurement assocated wth the j th tem and X s the mean of all the tems. The results of Ward s method can be dsplayed by a Dendrogram. The vertcal axs gves the value of ESS at whch the mergers occur. 4

5 2.6 Non Herarchcal Clusterng Method Non herarchcal clusterng technques are desgned to group tems, rather than varables, nto a collecton of K clusters. The number of clusters, K, may ether be specfed n advance or determned as part of the clusterng procedure. Because a matrx of dstance does not have to be determned and the basc data do not have to be stored durng the computer run. Non herarchcal methods can be appled to much larger data sets than can herarchcal technques. Non herarchcal methods start from ether () an ntal partton of tems nto groups or (2) an ntal set of seed ponts whch wll form nucle of the cluster. 2.7 K means Clusterng (Aff, Clark and Marg, 2004) The K means clusterng s a popular non herarchcal clusterng technque. For a specfed number of clusters K the basc algorthm proceeds n the followng steps. Dvde the data nto K ntal cluster. The number of these clusters may be specfed by the user or may be selected by the program accordng to an arbtrary procedure. Calculate the means or centrod of the K clusters. For a gven case, calculate ts dstance to each centrod. If the case s closest to the centrod of ts own cluster, leave t n that cluster; otherwse, reassgn t to the cluster whose centrod s closest to t. v Repeat step () for each case. v Repeat steps (), (), and (v) untl no cases are reassgned. The frst step consders all the data as one cluster. For the hypothetcal data set ths step s llustrated as n the fgure below. The algorthm then searches for the varable, wth the hghest varance n ths case X. The orgnal cluster s now splt nto two clusters usng the md range of X as the dvdng pont as shown n plot (b) of fgure drawn below. If the data are standardzed, then each varable has a varance of one. In that case the varable wth the smallest range s selected to make the splt. The algorthm n general proceeds n ths manner by further splttng the clusters untl the specfed member K s acheved. That s, t successvely fnds that partcular varable and the cluster producng the largest varance and splts that cluster accordngly untl K clusters are obtaned. At ths stage, step () of the basc algorthm s completed and t proceeds wth the other steps (a) Starts wth all ponts n one cluster. 5

6 (b) Cluster s splt nto 2 clusters at md range of X (varable wth largest var.) (c) Pont 3 s closure to centrod of cluster (, 2, 3) and stays assgned to (, 2, 3) (d) Every pont s now closest to centrod of ts own cluster. 3. Dendrogram Dendrogram s also called herarchcal tree dagram or plot, and shows the relatve sze of the proxmty coeffcents at whch cases are combned. The bgger the dstance coeffcent or the smaller the smlarty coeffcent, the more clusterng nvolved combnng unlke enttes, whch may be undesrable. Trees are usually depcted horzontally, not vertcally, wth each row representng a case on the Y axs, whle the X axs s a rescaled verson of the proxmty coeffcents. Cases wth low dstance/hgh smlarty are close together. Cases showng low dstance are close, wth a lne lnkng them a short dstance from the left of the Dendrogram, ndcatng that they are agglomerated nto a cluster at a low dstance coeffcent, ndcatng alkeness. When, on the other hand, the lnkng lne s to the rght of the Dendrogram the lnkage occurs at a hgh dstance coeffcent, ndcatng the cases/clusters were agglomerated even though much less alke. If a smlarty measure s used rather than a dstance measure, the rescalng of the X axs stll produces a dagram wth lnkages nvolvng hgh alkeness to the left and low alkeness to the rght. 6

7 4. Proxmty Measures (Tmm, 2002) Proxmty measures are used to represent the nearest of two objects. If a proxmty measure represents smlarty, the value of the measure ncreases as two objects become more smlar. Alternatvely f the proxmty measure represents dssmlartes the value of the measure decreases n value as two objects become more alke. Let X and Y represents two objects n a p-varate space then an example of dssmlarty measures s the Eucldan dstance between X and Y. For measure of smlarty, we may use the proporton of the elements n the two vectors that match. 4. Dssmlarty Measures Gven two objects X and Y n a p dmensonal space, a dssmlarty measure satsfes the followng condtons:. d (X,Y) 0 for all objects X and Y. 2. d (X,Y) = 0 ff X = Y. 3. d (X,Y) = d (Y,X). Condton (3) mples that the measure s symmetrc so that the dssmlarty measure that compares X and Y s same as the comparson for object Y verses X. Condton (2) requres the measures to be zero, when ever object X equals to object Y. The objects are dentcal f d(x, Y) = 0. Fnally, Condton () mples that the measure s never negatve. Some dssmlarty measures are as follows. 4.. Eucldan Dstance Ths s probably the most commonly chosen type of dstance. It smply s the geometrc dstance n the multdmensonal space. It s computed as, 2 / 2 } d(x,y) = { (X Y ) n matrx form p = or d (X,Y)= ( X Y) ( X Y) where X' = (X,X 2,, X p ), Y' = (Y, Y 2,, Y p ) The statstcal dstance between the same two observatons s of the form d (X,Y) = ( X Y) A( X Y), where A = S - and S contans the sample varances and covarances. Eucldan and square Eucldan dstances are usually computed from raw data and not from standardzed data Square Eucldean Dstance Square the standard Eucldean dstance n order to place progressvely greater weght on objects that are further apart. Ths dstance s computed as: d²(x,y) = (X or n matrx form p = 2 Y ) 7

8 d²(x,y) = (X - Y) (X - Y) 4..3 Mnkowsk Metrc When there s no dea about pror knowledge of the dstance group then one goes for mnkowsk metrc. Ths can be computed as gven below: n } d(x,y) = { X Y p = n For m =, d(x, Y) measures the cty block dstance between two ponts n p dmensons. For m = 2, d(x, Y) becomes the Eucldean dstance. In general, varyng m changes the weght gven to larger and smaller dfferences Cty-Block (Manhattan) Dstance Ths dstance s smply the average dfference across dmensons. In most cases, ths dstance measure yelds result smlar to the smple Eucldean dstance. Ths can be computed as: p d(x,y) = X = Y 4..5 Chebychev Dstance Ths dstance measure may be approprate n case when we want to defne the objects as dfferent f they are dfferent on any one of the dmensons. The chebychev dstance s computed as: d(x,y) = maxmum X Y. Two addtonal popular measures of dstance or dssmlarty are gven by the Canberra metrc and the Czekanowsk coeffcent. Both of these measures are defned for non negatve varables only. We have p X Canberra Metrc: d(x, Y) = = (X Y + Y ) 2 mn (X,Y ) = Czekanowsk Coeffcent = p (X Y ) p = 4.2 Smlarty Measure Gven two objects X and Y n a p-dmensonal space, a smlarty measure satsfes the followng condtons:. 0 S(X,Y) for all objects X and Y 2. S(X,Y) = ff X = Y 3. S(X,Y) = S(Y, X) Here S(X,Y) = d(x,y) S(X,Y) = smlarty measure, D(X,Y) = dssmlarty measure 8

9 Let the frequency of matches and mx matches for objects X and Y be arranged n the form of a contgency table as follows: Table 4. Object (X) 0 Totals Object(Y) a b a + b 0 c d c + d Totals a+c b+d P = a + b + c + d a represents the frequency of - matches b represents the frequency of -0 matches c represents the frequency of 0- matches d represents the frequency of 0-0 matches Followng s the lst of common smlarty coeffcents defned n terms of the frequency n the table. Table 4.2 Coeffcent Ratonale. (a+d)/p Equal weghts for - matches and 0-0 matches. 2. 2(a+d)/(2(a+d)+b+c) Double weght for - matches and 0-0 matches. 3. (a+d)/(a+d+2(b+c)) Double weght for unmatched pars. 4. a/p No 0-0 matches n numerator. 5. a/(a+b+c) No 0-0 matches n numerator or denomnator. 6. 2a/(2a+b+c) No 0-0 matches n numerator and denomnator. Double weght for - matches 7. a/(a+2(b+c)) No 0-0 matches n numerator or denomnator. Double weght for unmatched pars 8. a/(b+c) Rato of matches to msmatches wth 0-0 Matches excluded. Coeffcent of, 2, and 3 n the table are monotoncally related. Suppose coeffcent- s calculated for two contngency table. If [(a + d )/p] [(a + d )/p], then we also have [2(a +d )/(2(a +d )+b +c )] [2(a +d )/(2(a +d )+b +c )] and coeffcent 3 wll be at least as large for Table 4. as t s for Table 4.2. Here a, b, c, d are from Table 4. and a, b,c, d are from Table Illustraton (Chatfeld and Collns, 990) Gven below s food nutrent data on calores, proten, fat, calcum and ron. The objectve of the study s to dentfy sutable clusters of food nutrent data based on the fve varables. Food Items Calores Proten Fat Calcum Iron

10 Output from SAS Centrod Herarchcal Cluster Analyss Egenvalues of the Covarance Matrx Egenvalue Dfference Proporton Cumulatve Root-Mean-Square Total-Sample Standard Devaton =

11 CLUSTER= Obs food cal pro fat calc ron CLUSTER= Obs food cal pro fat calc ron

12 CLUSTER= Obs food cal pro fat calc ron CLUSTER= Obs food cal pro fat calc ron CLUSTER= Obs food cal pro fat calc ron Dendrogram for above data D s t a n c e 300 B e t w e e n C l u s t e r C e n t r o d s f ood 2

13 Interpretaton The man objectve of our analyss s to groupng the food tems on the bass of ther nutrent content based on the fve varables such that food tems wth n the groups are homogeneous and between the groups are heterogeneous. Number of Groups Two groups Three groups Four groups Fve groups Fve groups Sx groups Food Items Group- (,,2,,27) Group-2 (25) Group- (,,,0) Group-2 (5,5,,27) Group-3 (27) Group- (,,,0) Group-2 (5,5,,9) Group-3 (7,8,,27) Group-4 (25) Group- (,,,0) Group-2 (5,5,,9) Group-3 (7,8) Group-4 (22,24,27) Group-5 (25) Group- (,,,3) Group-2 (2,9,0) Group-3 (5,5,,9) Group-4 (7,8) Group-5 (22,24,27) Group-6 (25) 6. Examples of Clusterng Applcaton Marketng: Help marketers dscover dstnct groups n ther customer bases, and then use ths knowledge to develop targeted marketng programs. Land Use: Identfcaton of areas of smlar land use n earth observaton database. Insurance: Identfes groups motor nsurance polcy holders wth a hgh average clam cost. Cty Plannng: Identfcaton of group of houses accordng to ther house type, value and geographcal locaton. Earthquake Studes: Observed earthquake epcenters should be clustered along contnent faults. Feld of medcne: Clusterng of dseases, cure for dsease of symptoms of dsease can lead to very useful taxonomes. Feld of psychatry: The correct dagnoss of clusters of symptoms such as Paranoa, Schzophrena etc. s essental for successful therapy. In Archeology: Researches have attempt to establsh taxonomes of stone tools, funerals object etc by applyng cluster analytc technques. Feld of plant and anmal ecology: Clusterng s used to descrbe and to make spatal and temporal comparson of communtes of organsm n heterogeneous envronment. Feld of Bonformatcs: In transcrptomcs, clusterng s used to buld groups of genes wth related represents patterns and also n sequence analyss, t s used to group homologous sequence nto gene famles. 3

14 Socal network analyss: In the study of socal network, clusterng may be used to recognze communty wth large group of people. In general, when ever we need to classfy a mountan of nformaton nto manageable meanngful ples, cluster analyss s of great utlty. It s also used n data mnng. 7. Conclusons Here, dfferent ssues related to cluster analyss have been dscussed. Unlke other methods of classfcaton, cluster analyss however, has not yet ganed a standard methodology. Nonetheless, a number of technques are developed for dvdng multvarate sample on a composton whch s not known n advance nto several groups. Cluster analyss s a heurstc technque for classfyng cases nto groups when knowledge of the actual group membershp s unknown. There are numerous method for performng the analyss, wthout good gudelnes for choosng among them. Unless there s consderable separaton among the nherent group, t s not realstc to expect very clear results wth cluster analyss. In partcular f the observatons are dstrbuted n a nonlnear manner, t may be dffcult to acheve dstnct groups. Cluster analyss s qute senstve to outlers. In fact t s sometmes used to fnd outler. The data should be carefully screened before runnng cluster programs. Many statstcal package programs are also beng used for the purpose of cluster analyss. References Aff, A., Clark, V. A. and Marg, S. (2004). Computer aded multvarate analyss. USA, Chapman & Hall. Chatfeld, C. and Collns, A. J. (990). Introducton to multvarate analyss. Chapman and Hall Publcatons. Har, J. F., Anderson R. E., Tatham, R. L. and Black, W. C. (2006). Multvarate data analyss. 5 th Edn., Pearson Educaton Inc. Johnson, R. A. and Wchern, D. W. (2006). Appled multvarate statstcal analyss. 5 th Edn., London, Inc. Pearson Prentce Hall. Tmm, N. H. (2002). Appled multvarate ANALYSIS. 2 nd Edn., New York, Sprnger- Verlag. 4