A proposed nonparametric mixture density estimation using B-spline functions

Transcription

1 A proposed nonparametric mixture density estimation using B-spine functions Atizez Hadrich a,b, Mourad Zribi a, Afif Masmoudi b a Laboratoire d Informatique Signa et Image de a Côte d Opae (LISIC-EA 4491), ULCO, 50 Rue Ferdinand Buisson BP 719, Caais Cedex, France. b Laboratory of Probabiity and Statistics, Facuty of Sciences of Sfax, Sfax University. B. P Sfax. Tunisia Abstract In this paper, we suppose that a density of probabiity f is expressed as a finite inear combination of second order B-spine functions. Then, we obtain a finite mixture of B-spine. We extend the Expectation Maximization (EM) agorithm in order to estimate the new mixture density. The experiments show that the proposed estimator using B-spine functions can produce a satisfactory estimation of mixture density than gaussian cassica theory. Key words: function. B-spine functions, EM agorithm, Estimator, Mixture density, Probabiity density 1. Introduction In statistics, concerning the density estimation in the construction, there are severa competitive casses of nonparametric estimators, the two most popuar being the Kerne estimators introduced by Rosenbatt [8] and the Orthogona series estimators [8]. Usuay, a Kerne estimator has a greater efficiency than an estimator based on Orthogona series, athough it may be more compex to cacuate and update. In case the density has its support confined to the positive haf ine and may not be continuous at the origin, the Kerne method wi not be so attractive since the estimator may perform poory in the neighborhood of the origin. In such a situation, an estimator based on Orthogona series may be more appropriate. Athough scarcey used, the Orthogona series density estimator bears a striking resembance to the B-spine density estimator [1]. However, the Orthogona series estimator may sometimes ead to a function that is not a pdf. Again, the ony reason for incuding this estimator is that the B-spine estimator actuay has the same form, except that the basis functions are not Orthogona. In this paper, we propose a nonparametric method to estimate a mixture density using the second order B-spine and the EM agorithm [3]. The combination of the B-spine functions with the EM agorithm aows us to define a new agorithm which we denote as EM Generaized B-spine (EMGB) agorithm. The paper is organized as foows: In section 2, we describe the nonparametric density estimation methods as we as the B-spine functions. Convergence properties of the iterative Maximum Likeihood Estimation (MLE) are given in section 3. The performance comparison based on the Mean Squared Errors (MSE) is presented in section 4 in order to show the accuracy of the proposed estimator. Finay, the concusion appears in section 5. Emai addresses: hadrich atizez@isic.univ-ittora.fr (Atizez Hadrich a,b ), Mourad.Zribi@isic.univ-ittora.fr (Mourad Zribi a ), Afif.Masmoudi@fss.rnu.tn (Afif Masmoudi b ) URL: (Atizez Hadrich a,b ), (Atizez Hadrich a,b ) Preprint submitted to Esevier Juy 13, 2016

2 2. Proposed estimation methods by using B-spine functions. In the mathematica subfied of numerica anaysis, a B-spine is a spine function that has a minimum support with respect to a given degree, smoothness and domain partition. It s we known that every spine density function can be represented as a finite inear combination of B-spine [1]. The term B-spine stands for basis spines according to Isaac Jacob Schoenberg [2]. A B-spine nonparametric density estimator with uniformy spaced knots convenient for arge data sets was discussed by Gehringer [4]. Curry and Schoenberg (1966) [2] have proved that every spine function S of degree d (d 1, 2,...) with m knots (m 1, 2,...) has a unique expansion S(x) m+d b h B d (x), for a < x < b and h B d (x)dx, (2.1) where a, b R and b s are unknown parameters that need to be estimated. Note that b 0 and m+d b 1 is a specia requirement when using B-spines in order to estimate probabiity density functions. The B-spine of d degree are defined recursivey by B d (x) x x B d 1 (x) + x +d+1 x B d 1 +1 (x) (2.2) x +d x x +d+1 x +1 where B 0 (x) { 1, if x [x, x +1 ) 0, esewhere. For genera spines, this specia requirement is not true. The case d 1 corresponds to a piecewise inear approximation which is attractivey simpe but produces a visibe roughness, uness the knots are cose to each other. In more technica terms, a spine function S of degree d with m interior knots, a x 1 < x 2 <... < x b is a (d 1) continuousy differentiabe function, such that S P d [2], (P d is the cass of poynomias of a maximum degree d in each of the intervas (a, x 2 ), (x 2, x 3 ),..., (x m+1, b)). When we consider the approximation by the B-spines of d degree, the density function f is supposed to be (d 1) continuousy differentiabe in each of the intervas (a, x 2 ), (x 2, x 3 ),..., (x m+1, b). Hence, we have chosen the quadratic B-spines functions which are ony continuousy differentiabe, that is to say, having a minima requirement, as compared with the B-spines of degree d > 2. In our work, we have used the second order B-spines functions (B 2)... defined by (x x ) 2 (x +1 x )(x +2 x ), x [x, x +1 ) 1 B 2 [ (x x )(x +2 x) + (x x +1)(x +3 x) ], x [x (x) +1, x +2 ) x +2 x +1 x +2 x x +3 x +1 (x +3 x) 2 (x +3 x +1 )(x +3 x +2 ), x [x +2, x +3 ) 0, esewhere, where is an integer as usua. We notice that 0 B 2 (x) 1 is aways verified. The support of each spine covers three intervas. It is a quadratic poynomia on each support interva. Note that the peak vaue of B 2(x) is ess than 1. If a the B2 spines coud be potted in any interva, there woud be contributions from the three spines. Note that any pdf f can be approximated by the foowing mixture (2.3) of B-spines: f(x) 2 b B 2,h (x), (2.3)

3 + x+3 with b 0, b 1, B,h 2 (x) B2 (x) and h B 2 (x)dx B 2 (x)dx. The estimation h x of the density f reduces the estimation of the finite-dimensiona parameters (b 1,..., b ) that characterize f Estimation of b by the EM agorithm Assuming independence between the observations X 1, X 2,..., X n, we define the Log-ikeihood function by (X 1,..., X n b) Log( b B,h 2 (X i )). To obtain the MLE of (b 1, b 2,..., b ), we appy the EM agorithm [3]. We initiaize the B-spine coefficients by b (0) 1 B,h 2 n (X j ). E-Step:,j b(p) B,h 2 (X j ) b (p) s Bs,h 2 (X j ) s1 ; 1,..., m + 2, j 1,..., n where,j is the conditiona probabiity at the pth iteration. M-Step: The coefficients of B-spine at (p + 1)th iteration are given by b (p+1) 1 n,j ; 1,..., m + 2 After some iterations of the EM agorithm, we obtain the mixture B-spine estimator of f as f(x) b B,h 2 (x) where b is the maximum ikeihood of b. In what foows, we suggest to extend the second order B-spine estimator to the probem of the mixture density estimation [1]. The combination of the B-spine functions with the EM agorithm aows us to define a new agorithm denoted as EM Generaized B-spine (EMGB) agorithm Description of the EMGB agorithm Let s consider the foowing mixture density of mixture of B-spine f(x) π k f(x b.,k ) π k b,k B 2,h (x), where π k 0 and π k 1. Θ { θ (π 1,..., π K, b.,1,..., b.,k ); π k 0, and f(x b.,k ) π k 1; k 1,..., K; b.,k [0, 1] ; b,k 1 } b,k B 2,h (x). Let X 1, X 2,..., X n be n random variabes with an unknown common pdf f. For each X i, we consider a random variabe Z i (Z 1,i,..., Z K,i ) such that Z i foows a 3

4 mutinomia distribution Mu(1, π 1,..., π K ). Let X (X 1,..., X n ) and Z (Z 1,..., Z n ), the og maximum ikeihood is given by (X, Z θ) [ Z Log(π k f(x i b.,k ))]. We define the mean of the og ikeihood Q(θ, θ (p) ) E((X, Z θ) θ (p) ) [ E(Z X, θ (p) )Log(π k f(x i b.,k ))]. If E(Z X, θ (p) ), then Q Q(θ, θ (p) ) [ Log(π kf(x i b.,k ))] K [ Log(π kf(x i b.,k )) + 1,i Log((1 π k )f(x i b.,1 ))]. k2 k2 The EMGB agorithm seeks to find the MLE by appying iterativey the foowing two steps: E-Step: We estimate a posterior probabiity beonging to the cass k at the pth iteration: E(Z X i, θ (p) ) π (p) k f(x i b (p).,k ). π (p) k f(x i b (p).,k ) M-Step: we cacuate the parameter θ (p+1) that maximizes θ (p+1) arg max θ Θ On the one hand, Q π k which impies π k k2 k2 ( π k n 1,i 1 (1 1,i k2 k2 π k ) 0. This means that πk π k ). Therefore, π (p+1) k Q(θ, θ (p) ). (1 1,i 1 n π k ) Q On the second hand, (B,h 2 (X i ) B1,h 2 (X i )) 0; 2,..., m + 2 b,k f(x i b.,k ) and 2 Q n b 2 (B,h 2 (X i ) B1,h 2 (X i )) 2; 2,..., m + 2.,k f(x i b.,k ) In order to sove Q 0, we use the Newton Raphson method [5] or the steepest descent method, b,k k2. 4

5 we obtain b (p+1),k b (p),k + (B2,h (X i ) B1,h 2 (X i ) f(x i b (p).,k ) ) (B2,h (X i ) B1,h 2 (X i ) f(x i b (p).,k ) ) 2 for 2,..., m + 2 andk 1,..., K. (2.4) The cacuation cyces are made from one step to another unti we achieve the convergence. As starting vaues for the EMGB agorithm, we take the initia vaues of π k and b,k which have to be equa respectivey to n k n and 1 n k B,h 2 n (X j ); where n k is the tota number of observations in the k kth cass. 3. Convergence of the proposed estimator In this section, we wish to show first that the EMGB iterations converge to a vaue ( π k, b,k ), second that this vaue ( π k, b,k ) indeed satisfies the ikeihood equations. Proposition 3.1. Let θ (π 1,..., π K, b.,1,..., b.,k ) Θ. If θ (p+1) converges to θ, as p +, then θ satisfies the ikeihood equations. Proof. On the one hand, for a fixed k, the derivative of the Log-ikeihood function with respect to b,k is given by (X 1,..., X n θ) (B,h 2 π (X j ) B1,h 2 (X j )) k ; 2,..., m + 2. where b,k f(x j ) f(x j ) and f k (X j ) b,k B,h 2 (X j ). By etting p + in (2.3), by using the fact that b (p),k b,k and k,j τ k,j π f k k (X j ), one has τ k,j (B 2,h (X j ) B 2 1,h (X j )) f k (X j ) 0; 2,...,. Therefore 0. And, b,k satisfies the derivation of the ikeihood equations. π k (B 2,h (X j ) B 2 1,h (X j )) f(x j ) 5

6 On the other hand, π k 1 n 0 n Let f k (X j ). This impies that π 1 f1 (X j ) + π 2 f2 (X j ) π K 1 fk 1 (X j ) + (1 π 1 π 2... π K 1 ) f K (X j ) f k (X j ) π 1 ( f 1 (X j ) f K (X j )) π K 1 ( f K 1 (X j ) f K (X j )) + f K (X j ) f k (X j ). H (3.5) π 1 ( f 1 (X j ) f K (X j )) π K 1 ( f K 1 (X j ) f K (X j )). (3.6) Then, mutipying (3.4) by π k for k 1,..., K 1 and by summing the resuts of the (K 1) successive mutipications, we obtain ( π 1 + π π K 1 )H H. Since π k 1, then (1 π K )H H and π K H 0. Note that we chose π K arbitrariy in (3.5). We coud have chosen any one of the π k, k 1, 2,..., K. So, we can say that π k H 0, k 1,..., K. Necessariy, H 0. From (3.4), this impies (X 1,..., X n θ) π k f k (X j ) f K (X j ) 0 for k 1,..., K 1. Therefore, π 1, π 2,..., π K of the EMGB satisfy the ikeihood equations. 4. Performance comparison In our study we simuate n 1000 observations according to different distributions (Norma (N), Beta (B) and Gamma (G)) with given true parameters. We cacuate the density estimator by using Histogram, Kerne, Orthogona and B-spine methods. Then, we compute the Mean Squared Errors MSE 1 (f(x j ) n f(x j )) 2 for each method. Among the different methods, Tabe 1 shows that the B-spine method is the best one by giving the owest MSE. This finding is vaid for a unique density as we as a mixture density. In Tabe 2, we have first computed the MSE between the empirica distribution and the estimated mixture density by using the Gaussian EM agorithm. Second, we have computed the MSE between the empirica distribution and the estimated mixture density by using the EMBG agorithm. This work was done for two images (Lena and Boat). We notice that the EMGB method gives a much ower MSE for both images.in Figure 1, in the case of mixture densities, the B-spine method remains the best one in terms of being cose to the rea density. 6

7 Tabe 1: MSE of different estimations methods. True mode Methods Histogram Kerne Orthogona B-spine N(0, 1) B(3, 5) G(2, 3) Mixed of N(-2, 2), G(3, 1) and B(11, 5) Tabe 2: MSE of mixture distribution obtained by EM and EMGB. Methods Images MSE of Lena MSE of Boat Gaussian EM EMGB Figure 1: Estimation of mixture density using both cassica EM agorithm and EMGB agorithm. 5. Concusion In this paper, we have introduced a new nonparametric B-spine estimator for mixture distributions. Many resuts presented show that the estimation of pdf density by using the proposed estimator is better than those of other methods. This comparison is obtained by the computation of the Mean Squared Errors between the estimated density and the empirica distribution. References [1] Atigan, Bozdogan, (1992). Convergence properties of MLE s and asymptotic simutaneous confidence intervas in fitting cardina B-spine for density estimation. Statistics and Probabiity Letters, Vo. 13, pp , North-Hoand. [2] Curry H.B, Schoenberg I.J., (1966). On poy freqency functions IV: The fundamenta spine functions and their imits. J.Ana. Math, Vo. 17, pp [3] Dempster A.P., Laird N.M., Rubin D.B, (1977). Maximum Likeihood from Incompete Data via the E.M. Agorithm, Journa of the Roya Statistica Society, Ser. B, Vo. 39, pp [4] Gehringer, K.R. and Redner, R.A., (1992), Nonparametric density estimation using tensor product spines. Communications in Statistics-Simuation and Computation, Vo. 21, pp

8 [5] Hazewinke, Michie, ed. (2001), Newton method, Encycopedia of Mathematics, Springer, ISBN [6] Kim J. and a., (2005). A Nonparametric Statistica Method for image segmentation using information theory and curve evoution. IEEE transactions on image processing, Vo. 14, pp [7] Peter Ha, (1983). Orthogona series distribution function estimation, with appication. J, R, Statist, Vo 48, pp [8] Rosenbatt M., (1956). Remarks on some nonparametric estimates of a density function. Ann. Math.Statist, Vo. 27, pp