An EM Algorithm for the Student-t Cluster-Weighted Modeling

Transcription

1 An EM Alorithm for the Student-t luster-weihted Modelin Salvatore Inrassia, Simona. Minotti, and Giuseppe Incarbone Abstract luster-weihted Modelin is a flexible statistical framework for modelin local relationships in heteroeneous populations on the basis of weihted combinations of local models. Besides the traditional approach based on Gaussian assumptions, here we consider luster Weihted Modelin based on Student-t distributions. In this paper we present an EM alorithm for parameter estimation in luster-weihted models accordin to the maximum likelihood approach. 1 Introduction The functional dependence between some input vector X and output variable Y based on data comin from a heteroeneous population, supposed to be constituted by G homoeneous subpopulations 1 ;:::;, is often estimated usin methods able to model local behavior like Finite Mixtures of Reressions (FMR) and Finite Mixtures of Reressions with oncomitant variables (FMR), see e.. Frühwirth-Schnatter (005). As a matter of fact, the purpose of these models is to identify roups by takin into account the local relationships between some response variable Y and some d-dimensional explanatory variables X D.X 1 ;:::;X d /. Here we focus on a different approach called luster- Weihted Modelin (WM) proposed first in the context of media technoloy in S. Inrassia G. Incarbone Dipartimento di Impresa, ulture e Società, Università di atania orso Italia 55, atania, Italy s.inrassia@unict.it; incarbo@diit.unict.it S.. Minotti ( ) Dipartimento di Statistica, Università di Milano-Bicocca Via Bicocca deli Arcimboldi Milano, Italy simona.minotti@unimib.it W. Gaul et al. (eds.), hallenes at the Interface of Data Analysis, omputer Science, and Optimization, Studies in lassification, Data Analysis, and Knowlede Oranization, DOI / , Spriner-Verla Berlin Heidelber 01 13

2 14 S. Inrassia et al. order to recreate a diital violin with traditional inputs and realistic sound, see Gershenfeld et al. (1999). From a statistical point of view, WM can be rearded as a flexible statistical framework for capturin local behavior in heteroeneous populations. In particular, while FMR considers the conditional probability density p.yjx/, the WM approach models the joint probability density p.x;y/ which is factorized as a weihted sum over G clusters. More specifically, in WM each cluster contains an input distribution p.xj / (i.e. a local model for the input variable X) and an output distribution p.yjx; / (i.e. a local model for the dependence between X and Y ). Some statistical properties of the luster-weihted (W) models have been established by Inrassia et al. (010); in particular it is shown that, under suitable hypotheses, W models eneralize FMR and FMR. In the literature, W models have been developed under Gaussian assumptions; moreover, a wider settin based on Student-t distributions has been introduced and will be referred to as the Student-t WM. In this paper, we focus on the problem of parameter estimation in W models accordin to the likelihood approach; in particular to this end we present an EM alorithm. The rest of the paper is oranized as follows: the luster-weihted Modelin is introduced in Sect. ; the EM alorithm for the estimation of the WM is described in Sect. 3; in Sect. 4 we provide conclusions and ideas for further research. The luster-weihted Modelin Let.X;Y/ be a pair of a random vector X and a random variable Y defined on with joint probability distribution p.x;y/,wherex is the d-dimensional input vector with values in some space X R d and Y is a response variable havin values in Y R. Thus.x;y/ R d1. Assume that can be partitioned into G disjoint roups, say 1 ;:::; G,thatis D 1 [ [ G. luster-weihted Modelin (WM) decomposes the joint probability of.x;y/as: p.x;yi / D p.yjx; /p.xj / ; (1) D1 where D p. / is the mixin weiht of roup, p.xj / is the probability density of x iven and p.yjx; / is the conditional density of the response variable Y iven the predictor vector x and the roup, D 1;:::;G,see Gershenfeld et al. (1999). Vector denotes the set of all parameters of the model. Hence, the joint density of.x;y/ can be viewed as a mixture of local models p.yjx; / weihted (in a broader sense) on both the local densities p.xj / and the mixin weihts. Throuhout this paper we assume that the input-output relation can be written as Y D.xI ˇ/ ", where.xi ˇ/ is a iven function of x (dependin also on some parameters ˇ) and" is a random variable with zero mean and finite variance.

3 An EM Alorithm for the Student-t luster-weihted Modelin 15 Usually, in the literature about WM, the local densities p.xj / are assumed to be multivariate Gaussians with parameters. ; /,thatisxj N d. ; /, D 1;:::;G; moreover, also the conditional densities p.yjx; / are often modeled by Gaussian distributions with variance "; around some deterministic function of x, say.xi ˇ/, D 1;:::;G. Thus p.xj / D d.xi ; /, where d denotes the probability density of a d-dimensional multivariate Gaussian and p.yjx; / D.yI.xI ˇ/; "; /,where denotes the probability density of a uni-dimensional Gaussian. Such model will be referred to as the Gaussian WM. In the simplest case, the conditional densities are based on linear mappins.xi ˇ/ D b 0 x b 0, with ˇ D.b 0 ;b 0/ 0, b R d and b 0 R, yieldin the linear Gaussian WM: p.x;yi / D.yIb 0 x b 0;"; / d.xi ; / : () D1 Under suitable assumptions, one can show that model () leads to the same posterior probability of Finite Mixtures of Gaussians (FMG), Finite Mixtures of Reressions (FMR) and Finite Mixtures of Reressions with oncomitant variables (FMR). In this sense, we shall say that WM contains FMG, FMR and FMR, as summarized in the followin table, see Inrassia et al. (010) for details: p.xj / p.yjx; / assumption FMG Gaussian Gaussian linear relationship between X and Y FMR none Gaussian. ; / D.; /,=1,...,G FMR none Gaussian D and D,=1,...,G In order to provide more realistic tails for real-world data with respect to the Gaussian modelsand rely on a more robustestimation of parameters, the WM with Student-t components has been introduced by Inrassia et al. (010). Let us assume that in model (1) both p.xj / and p.yjx; / are Student-t densities. In particular, we assume that Xj has a multivariate t distribution with location parameter, inner product matrix and derees of freedom, that is Xj t d. ; ; /, and Y jx; has a t distribution with location parameter.xi ˇ/, scale parameter and derees of freedom, that is Y jx; t..xi ˇ/; ; /,sothat p.x;yi / D t.yi.xi ˇ/; ; /t d.xi ; ; / : (3) D1 This implies that, for D 1;:::;G, Xj ; ; ;U N d ; ; u

4 16 S. Inrassia et al. Y j.x; ˇ/; ; ;W N.x; ˇ/; ; w where U and W are independent random variables such that U j ; ; ; and W j.x; ˇ/; ; ; : (4) Model (3) will be referred to as the Student-t WM; the special case in which.xi ˇ/ is some linear mappin will be called the linear t-wm. In particular, we remark that the linear t-wm defines a wide family of densities which includes mixtures of multivariate t-distributions and mixtures of reressions with Student-t errors. Obviously, the two models have a different behavior. An example is illustrated usin the NO dataset, which relates the concentration of nitric oxide in enine exhaustto the equivalenceratio, see Hurn et al. (003). Data have been fitted usin both Gaussian and Student-t WM. The two classifications differ by four units, which are indicated by circles around them (two units classified in either roup 1 or roup ; two units classified in either roup or roup 4), see Fi. 1. In particular, there are two units that the Gaussian WM classifies in roup 4 but which are a little bit far from the other points of the same roup; instead, such units are classified in roup by means of the Student-t WM. If we consider the followin index of weihted model fittin (IWF) E defined as: = B E 1 4y n I ˇ/p. jx n ;y n / A5 A ; (5) N D1 Enine Mix Enine Mix Nitric Oxide Gaussian WM Nitric Oxide Student-t WM Fi. 1 Gaussian and Student-t W models for the NO dataset

5 An EM Alorithm for the Student-t luster-weihted Modelin 17 we et E D 0:108 in the Gaussian case and E D 0:086 in the Student WM. Thus the latter model showed a slihtly better fit than the Gaussian WM. 3 WM Parameter Estimation Via the EM Alorithm In this section we present the main steps of the EM alorithm in order to estimate the parameters of WM accordin to the likelihood approach. We illustrate here the Student-t distribution case; similar ideas can be easily developed in the Gaussian case. Let.x 1 ;y 1 /;:::;.x N ;y N / beasampleofn independent observation pairs drawn from (3) andsetx D.x 1 ;:::;x N /, Y D.y 1 ;:::;y N /. Then, the likelihood function of the Student-t WM is iven by L 0. I X ; Y/ D NY p.x n ;y n I / D 3 ny 4 p.y n jx n I ˇ; /p d.x n I ; /f.wi /f.ui / 5 ; D1 where f.wi / and f.ui / denote the density functions of W and U respectively iven in (4). Maximization of L 0. I X; Y/ with respect to, for iven data (X; Y), yields the estimate of. If we consider fully cateorized data f.x n ;y n ; z n ; u n ; w n / W n D 1;:::;N, then the complete-data likelihood function can be written in the form L c. I X ; Y/ D Y n; p.y n jx n I ˇ; / z n p d.x n I ; / z n f.wi / z n f.ui / z n z n ; (6) where z n D 1 if.x n;y n / comes from the -th population and z n D 0 elsewhere. Let us take the loarithm, then after some alebra we et: L c. I X ; Y/ D ln L c. I X ; Y/ D L 1c.ˇ/ L c./ L 3c./ L 4c./ L 5c./ where L 1c.ˇ/ D D1 " 1 z n ln ln w n ln ; w n.y n b 0 x # n b 0 / ;

6 18 S. Inrassia et al. L c./ D L 3c./ D L 4c./ D L 5c./ D D1 D1 D1 D1 1 h i z n p ln p ln u n ln j j u n.x n / 0 1.x n / z n ln z n h ln z n Œln : ln ln.ln w n w n / ln w n.ln u n u n / ln u n i The E-step on the.k 1/-th iteration of the EM alorithm requires the calculation of the conditional expectation of the complete-data lolikelihood function ln L c. I X; Y/, sayq. I /, evaluated usin the current fit for.to this end, the quantities E Zn jx n ;y n, E fu njx n ; z n, E fln U njx n ; z n, E fw njy n ; z n and E fln W njy n ; z n have to be evaluated, for n D 1;:::;N and D 1;:::;G. It follows that E fz n jx n ;y n D n D P G j D1 j p.y n jx n I ˇ p.y n jx n I ˇ j ; /p d.x n I ; j ; / /p d.x n I j ; j / is the posterior probability that the n-th observation.x n ;y n / belons to the -th component of the mixture. Further, we have that E fu njx n ; z n D u n D x n ( E fln U njx n ; z n D ln u n d d 0 1 ln x n d ) E fw njy n ; z n D w n D 1 y n b 0 x n b 0 ; ( E fln W njy n ; z n D ln w n 1 ln 1 ) ; where.s/ Df@.s/=@s=.s/. In the M-step, onthe.k 1/-th iteration of the EM alorithm, we maximize the conditional expectation of the complete-data lolikelihood Q, with respect to

7 An EM Alorithm for the Student-t luster-weihted Modelin 19. The solutions for the posterior probabilities.k1/ and the parameters.k1/, of the local input densities p d.x n j ; /,for D 1;:::;G,existina.k1/ closed form and coincide with the case of mixtures of multivariate t distributions (Peel and McLachlan 000), that is:.k1/.k1/.k1/ D 1 N n D n u n x n n u n D n u n.x n.k1/ /.x n.k1/ n u n / 0 The updates b.k1/ ;b.k1/ 0 and ;.k1/ of the parameters for the local output densities p.y n jx n I ˇ; /,for D 1;:::;G, are obtained by the solution of the fl c. jx n ;y n / D 0 fl c. jx n ;y n / D fl c. jx n ;y n / D ; and it can be proved that they are iven by b 0.k1/ D n w n y n x 0 n n w P n y N n n w P N n n w n n w n x n x 0 n 1 n w n x 0 n n w P A N n n w n : n w n x 0 n n w n 1 b.k1/ 0.k1/ ; D D n w n y n n w n n w b.k1/ n w n x 0 n n w n n Œy n.b.k1/ x 0 n b.k1/ n w n 0 / :

8 0 S. Inrassia et al. The updates.k1/ and.k1/ for the derees of freedom and need to be computed iteratively and are iven by the solutions of the followin equations, respectively: and ln ln 1 1 P N n d ln d D P N n 1 ln 1 D 0 n ln u n n ln w n u n w n where.s/ Df@.s/=@s=.s/. 4 oncludin Remarks WM is a flexible approach for clusterin and modelin functional relationships based on data comin from a heteroeneous population. Besides traditional modelin based on Gaussian assumptions, in order to provide more realistic tails for real-world data and rely on a more robust estimation of parameters, we considered also WM based on Student-t distributions. Here we focused on a specific issue concernin parameter estimates accordin to the likelihood approach. For this aim, in this paper we presented an EM alorithm which has been implemented in Matlab and R lanuae for both the Gaussian and the Student-t case. Our simulations showed that the initialization of the alorithms is quite critical, in particular the initial uess has been chosen accordin to both a preliminary clusterin of data usin a k-means alorithm and a random roupin of data, but our numerical studies pointed out that there is no overwhelmin stratey. Similar results have been obtained by Faria and Soromenho (010) in the area of mixtures of reressions, where the performance of the alorithm depends essentially on the confiuration of the true reression lines and the initialization of the alorithms. Finally we remark that, in order to reduce such critical aspects, suitable constraints on the eienvalues of the covariance matrices could be implemented. This provides ideas for future work.

9 An EM Alorithm for the Student-t luster-weihted Modelin 1 References Faria S, Soromenho G (010) Fittin mixtures of linear reressions. J Stat omput Simulat 80:01 5 Frühwirth-Schnatter S (005) Finite mixture and markov switchin models. Spriner, Heidelber Gershenfeld N, Schöner B, Metois E (1999) luster-weihted modelin for time-series analysis. Nature 397:39 33 Hurn M, Justel A, Robert P (003) Estimatin mixtures of reressions. J omput Graph Stat 1:55 79 Inrassia S, Minotti S, Vittadini G (010) Local statistical modelin via the cluster-weihted approach with elliptical distributions. ArXiv: v Peel D, McLachlan GJ (000) Robust mixture modellin usin the t distribution. Stat omput 10:

10