Grouped data clustering using a fast mixture-model-based algorithm

Transcription

1 Poceedings of the 29 IEEE Intenational Confeence on Systems, Man, and Cybenetics San Antonio, TX, USA - Octobe 29 Gouped data clusteing using a fast mixtue-model-based algoithm Allou SAMÉ Laboatoie des Technologies Nouvelles Institut National de Recheche su les Tanspots et leu Sécuité Noisy-le-Gand, Fance same@inets.f Abstact Mixtue-model-based clusteing has become a popula appoach in many data analysis poblems fo its statistical popeties and the implementation simplicity of the EM algoithm. Howeve the computation time of the EM algoithm and its vaiants inceases significantly with the sample size. Fo lage data sets, pefoming clusteing on gouped data constitutes an efficient altenative to speed up the algoithms execution time. A apid and effective algoithm dedicated to gouped data clusteing is then poposed in this pape. Inspied by the Classification EM algoithm (), the poposed appoach estimates the missing sample at each iteation. An expeimental study using simulated data and eal acoustic emission data in the context of a flaw detection application on gas tans eveals good pefomances of the poposed appoach in tems of patitioning pecision and computing time. Index Tems Clusteing, mixtue models, EM algoithm, Classification EM algoithm, gouped Data, histogam I. INTRODUCTION Clusteing is an exploatoy data analysis technique which consists in identifying homogeneous goups in data and thus constitutes a cental poblematic in many applications including web data mining, bio-infomatics and image segmentation. Mixtue-model-based clusteing [1][6] continues to eceive inceasing attention fo its statistical popeties and its implementation simplicity. Its two commonly used appoaches [6] fo multidimensional data ae the maximum lielihood (ML) appoach and the classification maximum lielihood (CML) appoach. The ML appoach consists in estimating the model paamete by maximizing the log-lielihood by the Expectation Maximization (EM) algoithm and then to estimate a patition by the maximum a posteioi (MAP) ule. In the CML appoach, the model paamete and the patition ae simultaneously estimated by maximizing the classification loglielihood using the Classification Expectation Maximization () algoithm []. With the impovement of measuement devices and compute, data sets with a lage numbe of obsevations become fequent and the challenge is to develop clusteing methods whose execution time does not depend on the numbe of obsevations. Howeve the computation time of the EM and algoithms inceases significantly with the sample size. To speed up the EM algoithm, vaious attempts have been poposed: the incemental and spae veions poposed by Neal and Hinton [1], the scalable veion of the EM algoithm poposed by Badley et al. [2] to handle lage databases with a limited memoy buffe, the Mooe s appoach that summaizes data using the so-called multiesolution d-tee [9], the Ng and McLachlan appoach [11] that combines the incemental EM and the multiesolution d-tee appoach. Anothe geneic way of impoving the clusteing algoithms execution time is to opeate on gouped data o histogams instead of oiginal obsevations. This data stuctue also efeed as binned data [4] simply esults fom the conveion of oiginal obsevations into counts in disjoined subspaces of the oveall obsevations space. Gouped data can also natually occu when the measuement devices have finite pecision. Figue 1 shows example of gouped data esulting fom the discetization of eal acoustic emission data. McLachlan and Jones [8], and Cadez et al. [4] have theoetically fomulated the poblem of estimating a mixtue model fom gouped data as a missing data poblem, and have developed a dedicated EM algoithm. Howeve, the multivaiate fomulation of this algoithm equies the numeical evaluation of multiple integals which can be computationally expensive. Fig. 1. Gouped data esulting fom the discetization of eal acoustic emission data A faste mixtue-model-based algoithm fo gouped data clusteing is pesented in this pape. The poposed algoithm is inspied by the classification maximization lielihood appoach and incopoates a sample estimation at each iteation. The section 2 intoduces the essential concepts of gouped /9/$. c 29 IEEE /9/$. 29 IEEE 23

2 data in the famewo of mixtue models, and biefly ecalls the Cadez et al. appoach [4] fo paamete estimation via the EM algoithm. Ou clusteing appoach adapted to binned data is descibed in the thid section and an expeimental study using simulated data and eal acoustic emission data is summaized in Section 4. II. ESTIMATING A MIXTURE MODEL FROM GROUPED DATA The oiginal data ae epesented by an independent sample x (x 1,...,x n ) esulting fom a mixtue density of K components, defined on R p by K f(x i ; Φ) π f (x i ; θ ), (1) 1 whee Φ (π 1,...,π K, θ 1,...,θ K ), π 1,...,π K denotes the popotions of the mixtue and (θ 1,...,θ K ) the paamete of each component density. We denote by z (z 1,...,z n ) the set of classes associated to the sample x, wheez i {1,...,K}. In addition, we conside a patition (H 1,...,H v ) of R p into v hypeectangula bins and we assume that the only infomation we have petaining to the sample is the set of fequencies n of the obsevations x i belonging to the bins H, summaized by a vecto n (n 1,...,n v ) of counts, with v 1 n n. The obsevations of bin H can then be ewitten as (x ) 1 s n and thei associated classes labels by ( ) 1 s n. The estimation of the mixtue model paamete fom gouped data was fit intoduced by Dempste et al. [7] as a missing data poblem in the famewo of the EM algoithm, and then developed in the one-dimensional case by McLachlan and Jones [8]. It can be noticed that in this new situation, the missing data ae the oiginal obsevations and thei cluste membehip. The genealization to the multidimensional case can be attibuted to Cadez et al. [4]. To maximize the obseved log-lielihood defined by v L(Φ) logp(n; Φ) n log f(x; Φ)dx c, (2) 1 H c being a contant tem which does not depend on Φ, the EM algoithm stats fom an initial paamete Φ () and then altenates the two following steps until convegence : E-step (Expectation) Computation of the expectation of the complete data log-lielihood conditionally on the available data and the cuent paamete vecto Φ (q) : Q(Φ, Φ (q) )E[L c (Φ, x, z) n, Φ (q) ], whee the complete data log-lielihood is defined by L c (Φ, x, z) logp(n, x, z; Φ) K v n log π f (x ; θ ). 1 1 s1 with 1if, andothewise. M-step (Maximization) Computation of the paamete vecto Φ (q1) maximizing Q(Φ, Φ (q) ). Given the paamete vecto estimated pa the EM algoithm, a patition of the bins can easily be deived by assigning each bin H to the cluste which maximizes the posteio pobability that an obsevation x i of the bin H aises fom the th component of the mixtue. The main difficulty when implementing the EM algoithm applied to gouped data, is the numeical evaluation of multiples integals at each iteation, which can be computationally expensive. The following section intoduces a faste classification algoithm which does not equie integals calculations. III. THE PROPOSED CLASSIFICATION APPROACH The clusteing method intoduced hee, in the context of gouped data, is inspied by the Classification EM algoithm () [] applied to oiginal obsevations. The standad algoithm is an iteative clusteing algoithm yielding simultaneously the paamete and the classification. It iteatively maximizes, with espect to the component membehip vecto z (z 1,...,z n ) and the paamete vecto Φ, the classification log-lielihood citeion. The algoithm also can be intepeted as a classification veion of the EM algoithm []. In the famewo of gouped data, we popose an adaptation of the algoithm which consists in estimating simultaneously the missing data (x, z) and the paamete vecto Φ by maximizing the complete data log-lielihood which can be ewitten as v n L c (Φ, x, z) log π z f z (x ; θ z ). (3) 1 s1 The poposed algoithm stats with an initial paamete Φ () and then altenates, at the qth iteation, the two following steps until convegence: Step 1 Computation of (x (q1), z (q1) ) ag max (x,z) L c(φ (q), x, z) (4) Step 2 Computation of Φ (q1) agmax L c(φ, x (q1), z (q1) ) () Φ As in the classical veion of applied to oiginal obsevations, it can easily be poved that each iteation of this algoithms inceases the complete data log-lielihood citeion and that the convegence is obseved afte a finite numbe of iteations. The following sections detail the two steps of this algoithm unde the assumption of a Gaussian mixtue with diagonal covaiance matices which is a ealistic hypothesis fo many eal applications and moe paticulaly fo the application that will be intoduced in section 4. A. Step 1: maximization of the complete data log-lielihood with espect to the sample and the patition Unde the Gaussian mixtue assumption, the maximization of L c with espect to (x, z) is equivalent to the minimization 21

3 of h 1 (x, z) v n log Σ (q) 2logπ z (q) 1 s1 (x μ (q) ) T Σ 1 (x μ (q) ), (6) whee the π, μ and Σ ae espectively the popotions, mean vecto and covaiance matices of the mixtue. Minimizing h 1 is equivalent to minimize, fo all and s, h 2 (x, ) log Σ (q) 2logπ z (q) (x μ (q) ) T Σ 1 (x μ z (q) ) (7) unde the constaints x H and {1,...,K}. Since the minimization of h 2 depends only on the bin label, letus denote by (x (q1),z (q1) ) its solution. This solution can be obtained using the two following steps: (a) Fo 1...K, compute x (q1) (b) Set (x (q1) agmin(x μ (q) x H agmin,z (q1) )(x (q1) (x (q1) )T Σ (q) 1 (q) (x μ );, ),whee log Σ (q) 2logπ(q) μ (q) )T Σ (q) 1 (x (q1) μ (q1) ). The poblem (a) is a convex optimization poblem with linea constaints. It can be seen that x (q1) emains in the bounday of the bin H if μ (q1) is outside the bin H.Fo example, unde the assumption of a 2-dimensional Gaussian mixtue distibution with diagonal covaiance matices, if we set H 2 l1 [al ; bl [ and μ(q) (μ (q) 1,μ(q) 2 )T, it can be shown that x (q1) (x 1,x 2 )T with x 1 { a if μ (q) 1 <a μ (q) 1 b if a μ (q) 1 if μ (q) b x 2 1 >b { c if μ (q) 2 <c μ (q) 2 d if c μ (q) 2 d. if μ (q) 2 >d Figue 2 shows examples of locations of x in elation to the cuent value of μ. B. Step 2: maximization of the complete data log-lielihood with espect to the paamete vecto This step is simila to the M step of the standad algoithm [] applied to the sample (x (q1) 1,...,x v (q1) ) weighted by the fequencies (n 1,...,n v ). The popotions, mean vecto and covaiance matices which maximizes L c (Φ, x (q1), z (q1) ) ae given by π (q1) v 1 nz(q1) μ (q1) Σ (q1) diag n v 1 nz(q1) x (q1) v1 ( n z (q1) v 1 nz(q1) (x (q1) μ (q1) v 1 nz(q1) ) )(x (q1) μ (q1) ) T whee z 1if z, andothewise,anddiag(a) is the diagonal covaiance matix whose diagonal elements ae those of matix A. (a) (b) (c) Fig. 2. Exemples of localisations of x : fo cases (a) and (b) the Gaussian mean is outside the bin H and fo case (c) the Gaussian mean is inside the bin IV. EXPERIMENTAL STUDY This section evaluates the poposed algoithm, that we shall call Bin-, in tems of pecision and computing time using simulated data and eal wold data. A. Expeiments using simulated data The potocol of the simulations is as follows: n obsevations ae geneated accoding to a mixtue of 2 bivaiate Gaussian densities; these obsevations ae used to compute an histogam containing v bins (gouped data). Two diffeent mixtue models with % of theoetical Bayes eo ate have been consideed: mixtue model A: π 1 π 2 1/2, μ 1 ( 2, ), μ 2 (, ), Σ 1 Σ 2 diag(1, 1); mixtue model B: π 1 π 2 1/2, μ 1 (1.6, ), μ 2 (, ), Σ 1 diag(1, 1/8), Σ 2 diag(1/8, 1). Figue 3 shows, examples of simulations of gouped data accoding to mixtues A and B. Thee algoithm ae compaed in all these simulations: the standad algoithm applied to the oiginal data (individual obsevations), the Bin- and Bin-EM (EM applied to gouped data) algoithms applied to gouped data. The 22

4 bin Fig. 3. Example of a simulation of the gouped data accoding to model A (left) and B (ight) at bins bin quality of a patition povided by each algoithm is measued by computing its misclassification ate with espect to the simulated patition. Fo each simulated oiginal data set o gouped data set, the misclassification ates and CPU times ae aveaged ove diffeent samples (Monte-Calo simulation scheme). Figue 4 epots the misclassification ate obtained fo mixtues A and B in elation to the numbe of bins pe dimension, when n. We obseve a apid decease until the numbe of bins pe dimension eaches. T Theeafte, the misclassification ate is almost constant and close to that of : the thee algoithms become simila Fig.. CPU time in second fo the (no ma), bin-em (cicle) and bin- (squae) algoithms in elation to the sample size at bins fo mixtue B bin Model A bin Model B Fig. 4. Misclassification ate (%) in elation to the numbe of bins pe dimension, obtained fo n with (no ma), bin-em (cicle) and bin- (squae) fo mixtues A (top) and B (bottom) Figue epesents CPU times in second fo the thee algoithms with espect to the sample size fo a fixed numbe of bins pe dimension ( bins pe dimension). The execution time of Bin- and Bin-EM algoithms does not vay while that of the algoithm gows consideably with the sample size. Bin- outpefoms the and bin-em algoithms. B. Expeiments using eal acoustic emission data A motivation of this wo was to develop a compute-aided decision pocedue to assist the detection, in eal time, of damaged zones on the suface of a gas tan, though the use of acoustic emissions. Expets ae in ageement that spatial concentations of acoustic emissions, identified using spatial coodinates, ae of pimay impotance in the detection of damaged zones. Othe featues of acoustic emissions ae useful in distinguishing between majo and mino flaws. Ou method, theefoe, consists of two steps : Identification of zones whee acoustic emissions ae concentated (clusteing step) which is the object of the pesent study; Sepaation of the identified cluste into diffeent categoies accoding to the seveity of the impefection: these categoies ae temed mino, active and citical. Thisstep is done using standad discimination methods such as Bayesian discimination with Gaussian densities. Accoding to specialists in the field, the method we have descibed poduces satisfactoy esults using the algoithm fo the fit step, so long as the numbe of acoustic emissions does not exceed 1. When thee ae moe than 1 emissions the clusteing step becomes too slow (moe than a few seconds delay) fo a eal-time application. The assumption of Gaussian mixtue with diagonal covaiance matices can be attibuted to the fact that the damage zones ae usually to be found hoizontally and vetically along welding lines. To evaluate ou new stategy, we pefomed a compaison of and Bin- on a eal data sets of acoustic emissions. The numbe of cluste is selected by maximizing the integated classification lielihood citeion (ICL) [3] 23

5 which is the complete lielihood penalized by the numbe of fee paamete of the mixtue model. Both stategies, using and bin-, select a model with 11 cluste. Table I shows the misclassification ate between and Bin- in elation to the numbe of bins. Figue 6 shows the esulting patitions given by and Bin- at bins pe dimension and 8 bins pe dimension. TABLE I MISCLASSIFICATION RATES BETWEEN AND BIN- FOR REAL ACOUSTIC EMISSION DATA Numbe of bins Vs. Bin-.7% 6 6.8% % % % This shows that esults obtained with Bin- ae close to those of the algoithm applied to the oiginal sample. In this application, we have theefoe achieved ou aim of obtaining patitions simila to patitions but in less time. V. CONCLUSION A new clusteing algoithm fo gouped data has been poposed in this pape. The main objective was to obtain a apid and effective algoithm fo clusteing low-dimensional massive data sets. The poposed Bin- clusteing appoach poduces patitions fom gouped data, which ae close to those computed by the algoithm applied to the oiginal obsevations. Almost no diffeences in esults ae obseved between and Bin- patitions when the numbe of bins is sufficiently lage (geate than in the simulations we have pesented). The execution time of the poposed algoithm is almost constant, since it depends only on the numbe of bins, while the execution time fo inceases significantly as the numbe of available obsevations inceases. The application of the poposed algoithm on eal acoustic emission data has also eveal good pefomances of the poposed algoithm which theefoe epesents an efficient altenative fo clusteing lage data sets. REFERENCES [1] J.D. Banfield and A.E. Raftey, Model-based Gaussian and non Gaussian clusteing. Biometics, 49, , 23. [2] P.S. Badley, M.U. Fayyad and C.A. Reina, Scaling EM (Expectation Maximization) clusteing to lage databases. Technical Reppot MSR- TR-98-, Micosoft Reseach, [3] C. Bienaci and G. Celeux and G. Govaet, Assessing a mixtue model fo clusteing with the integated completed lielihood. IEEE Tansactions on Patten Analysis and Machine Intelligence. 22, 7, 719-7, 2. [4] I.V. Cadez and P. Smyth and G.J. McLachlan and C.E. McLaen, Maximum lielihood estimation of mixtue densities fo binned and tuncated multivaiate data. Machine Leaning, 47, 7-34, 2. [] G. Celeux and G. Govaet, A classification EM algoithm fo clusteing and two stochastic veions. Computational Statistics and Data Analysis, 14, 3-332, [6] G. Celeux and G. Govaet, Gaussian paimonious clusteing models. Patten Recognition, 28(), , 199. [7] A.P. Dempste and N.M. Laid and D.B. Rubin, Maximum lielihood fom incomplete data via the EM algoithm. Jounal of the Royal Statistical Society, Seies B, 39(1), 1-38, (oiginal sample) bins bins Fig. 6. Results obtained with (top) and Bin- (middle and bottom) on eal acoustic emission data [8] G.J. McLachlan and P.N. Jones, Fitting mixtue models to gouped and tuncated data via the EM algoithm. Biometics, 44(2), 71-78, [9] A. W. Mooe, Vey fast EM based mixtue model clusteing using multiesolution d-tees. In Solla M.S. Keans and D.A Cohn, edito, Advances in Neual Infomation Pocessing Systems, 11, 43-49, MIT Pess, [1] R.M. Neal and Hinton, A view of the EM algoithm that justifies incemental, spae and othe vaiants. In M.I. Jodan, edito, Leaning in Gaphical Models, -368, Kluwe, Dodecht, 1998 [11] S.-K. Ng and G. J. McLachlan, On some vaiants of the EM algoithm fo the finite mixtue models. Autian Jounal of Statistics, 32(1-2), ,