FITTING A RECTANGULAR FUNCTION BY GAUSSIANS AND APPLICATION TO THE MULTIVARIATE NORMAL INTEGRALS

Transcription

1 Appl. Comput. Math., V.4, N.2, 25, pp FITTING A RECTANGULAR FUNCTION BY GAUSSIANS AND APPLICATION TO THE MULTIVARIATE NORMAL INTEGRALS HATEM A. FAYED, AMIR F. ATIYA 2, ASHRAF H. BADAWI 3 Abstract. Ths artcle ntroduces a new scheme to express a rectangular functon as a lnear combnaton of Gaussan functons. The man dea of ths scheme s based on fttng samples of the rectangular functon by adaptng the well-known clusterng algorthm, Gaussan mxture models (GMM). Ths method has several advantages compared to other exstng fttng algorthms. Frst, t ncorporates an effcent algorthm that can ft more Gaussan functons. Second, weghts of the lnear combnaton are already constraned n the algorthm to le n the nterval [,], whch avods large/small values that cause numercal nstablty. Thrd, almost the entre ftted Gaussan functons le wthn the nterval of the rectangular functon, whch can be utlzed effcently to approxmate dffcult defnte ntegrals such as the multvarate normal ntegral. Experments show that t s effcent when low accuracy s requred (error of order of 4 ) especally for small values of the correlaton coeffcents. Keywords: Functon Approxmaton, Gaussan Functons, Gaussan Mxture Models, Multvarate Normal Integrals. AMS Subject Classfcaton: 4Axx, 65D5, 65D3.. Introducton The problem of functon approxmaton s wdely used n many areas of scence and engneerng where numercal technques are ncorporated. Common approaches nvolve Taylor seres, orthogonal polynomals (Chebyshev, Hermte, Legendre, etc.), Gaussan functons, and Fourer seres. An mportant applcaton of functon approxmaton s the numercal approxmaton of ntegrals that do not have closed forms. The multvarate normal complementary ntegral s one of the sgnfcant ntegrals that appear n many engneerng and statstcs computatons. A lot of research was nvestgated to approxmate t, however no unque approxmaton s used for all dmensons and a predefned accuracy. The ntegral s defned by: L(h, Σ) = (2π) d Σ h h d exp } 2 xt Σ x dx, where x = (x, x 2, x d ), and Σ s an d d symmetrc postve defnte covarance matrx. The problem has receved consderable attenton n lterature [3, 4, 6]. For d = 2, there are some seres expressons [8, 8, 2] and effcent numercal technques [4 6, ]. For the multvarate case, d > 2, there exst several powerful numercal methods based on multvarate ntegraton technques that rely on ordnary Monte-Carlo methods, along wth some common varance reducton technques [2, 3]. Another group of algorthms s based on computng upper Department of Engneerng Mathematcs and Physcs, Caro Unversty, 263, Caro, Egypt & Unversty of Scence and Technology, Zewal Cty of Scence and Technology, 2588, Caro, Egypt e-mal: h fayed@eng.cu.edu.eg, hfayed@zewalcty.edu.eg 2 Department of Computer Engneerng, Caro Unversty, 263, Caro Unversty, 263, Caro, Egypt e-mal: amr@alumn.caltech.edu 3 Unversty of Scence and Technology, Zewal Cty of Scence and Technology, 2588, Caro, Egypt e-mal: abadaw@zewalcty.edu.eg Manuscrpt receved 2 November

2 H. FAYED et al.: FITTING A RECTANGULAR FUNCTION BY GAUSSIANS and lower bounds on the probablty (see [, 2] for a survey of these methods). Recently, Mwa [7] developed an algorthm that evaluates the multple ntegral by transformng t nto recursve evaluaton of a one-dmensonal ntegraton over a fne grd of ponts. Ths method s consdered among the most effcent methods for d. Fayed and Atya [9] derved a seres expanson based on Fourer seres that s more effcent up to d = 7. In most studes, the case when the components of x are equcorrelated, that s ρ j = ρ for all jand ρ =, s often used as a benchmark. Ths case can be evaluated as ( [23], p. 92): L(h, Σ) = 2π exp( t2 2 ) d = ( h + t ) ρ Φ dt. ρ 2. Gaussan mxture models as an approxmaton One of the approaches to approxmate a functon by a lnear combnaton of Gaussan s to sample the functon, and then use the radal bass functon networks (RBF networks) where the radal bass functons are chosen to be Gaussan functons [5]. However, to acheve a good accuracy n approxmatng a rectangular functon, a large network s often requred. Another approach s to use the method proposed n [], however ths method faled to obtan good accuracy for the rectangular functon due to the abrupt change at the begnnng and end of the functon. Nonlnear regresson s also another alternatve, but t becomes consderably slow f the number of components s moderately large, and t requres to bounhts to avod very small/large values that often deterorate the approxmaton n the multvarate case. So we proposed the combnaton weg a smple method that crcumvents the above problems leadng to a good approxmaton to a rectangular functon. Let us consder the one-dmensonal problem for approxmatng the followng rectangular functon: R(x, T ) = x T otherwse. The nterval [, T ] s sampled unformly to generate M data ponts. These data ponts are used to ft the rectangular functon by the followng Gaussan mxture model: R(x, T ) = T ω p (x λ ), = where K s the number of mxture components, λ = µ, σ 2 }, p (x λ ) N(µ, σ 2 ) s the normal dstrbuton wth mean µ and varance σ 2, and ω s the component weght n the mxture. So, for a predetermned number of mxtures (K), the mxture s parameters can be estmated teratvely usng the expectaton-maxmzaton (EM) algorthm as follows [7, 22]. Let X = x m R; m =,, M} be the sample sequence, and θ = ω, µ, σ 2 }, =,, K denotes, respectvely, the component weght, the mean and varance of the th normal component. To fnd the optmum values of the normal components usng the EM algorthm, maxmzaton of the followng lkelhood functon s performed: f (Θ) = ln p (x m, m Θ)} = m= ln p (x m m, Θ) P m }, where Θ = θ,, θ K }, m,, K} denotes that x m was generated from component. At each teraton j of EM algorthm, two steps are performed: the expectaton step (E-step), and the maxmzaton step (M-step) as descrbed below. m=

3 96 APPL. COMPUT. MATH., V.4, N.2, 25 E-step: Takng the expectaton of f (Θ) based on the current estmate Θ j, Q (Θ, Θ j ) } = E ln p (x m m, Θ j ) P m } = m= = M P ( m x m, Θ j ) ln p (x m m, Θ j ) P m }, m= m = where Q (Θ, Θ j ) s a functon of Θ, assumng that Θ j s fxed. The notaton can now be smplfed by droppng the ndex m from m. Ths s because, for each m, we sum up over all possble values of m, and these are the same for all m. Note also that P becomes smply the component weght ω. However, for GMM we have: } p (x m, Θ j ) = (2πσ ) exp (x m µ ) 2 2 d 2σ 2. So Q (Θ, Θ j ) = m= = P ( x m, Θ j ) d 2 ln ( 2πσ 2 ) 2σ 2 (x m µ ) 2 + ln ω }. M-step: By maxmzng the Q (Θ, Θ j ) wth respect to ω, µ, σ 2, we get: ω (j) = M P ( x m, Θ j ), m= (j) σ 2 = µ (j) = m= P ( x m, Θ j ) x m, P ( x m, Θ j ) m= m= ( P ( x m, Θ j ) x m µ (j) m= P ( x m, Θ j ) = P ( x m, Θ j ) k= ) 2 ω(j ) P (x m, Θ j ) ω (j ) k P (x m k, Θ j ) Fg. shows the results of fttng R(x, 4) usng a sample step of., and dfferent values of K. The shown results are the best of runs wth dfferent random ntalzaton that led to the mnmum mean absolute error:,. MAE = M R(x, T ). m= A smple form can also be obtaned by constranng the Gaussan functons to have the same varance σ 2 and equally spaced means;.e. µ = µ + ( ) δ. Thereby, the tradtonal EM

4 H. FAYED et al.: FITTING A RECTANGULAR FUNCTION BY GAUSSIANS algorthm s modfed to obtan µ and δ from the followng equatons: M ( ) P ( x m, Θ j ) ( ) = m= µ ( ) P ( x m, Θ j ) ( ) 2 = δ P ( x m, Θ j ) = m= = m= x m = m= ( ) P ( x m, Θ j ) x m = m= and the varance formula s reduced to: (j) σ 2 = Md = m= ( P ( x m, Θ j ) x m µ (j) Fg. 2 shows the results of fttng R(x, 4) usng a sample step of. and dfferent values of K usng these constrants. ) 2. or 3. The normal ntegrals Suppose that t s requred to approxmate the normal ntegral: I(h) = ) exp ( x2 dx. 2π 2 It can be approxmated usng the rectangular functon R (x h, T ) as: I(h) = 2π I(h) = T K 2π whch can smply be obtaned as: I(h) = = ω h R (x h, T ) exp T K 2π = N ( µ + h, σ 2 ω + σ 2 exp where T s chosen such that the Gaussan functon 2π exp ) ( x2 dx, 2 ) ) exp ( x2 dx, 2 } (µ + h) 2 2 ( ) + σ 2, ( T 2 2 to choose T = 4). For the multvarate case, suppose that t s requred to approxmate: L(h, Σ) = (2π) d Σ h h d exp } 2 xt Σ x dx. ) = (for h, t s reasonable One way to approxmate ths multple ntegral s to sample data n d-dmensonal space, and apply EM for the sampled data as before. However ths approach led to poor results as the effcency of EM degrades as both the number of samples and the number of Gaussan functons ncrease. Alternatvely, t was approxmated along each dmenson separately as:

5 98 APPL. COMPUT. MATH., V.4, N.2, (a) K = 5, MAE =.3 (b) K = 6, MAE = (c) K =, MAE =.49 (d) K = 2, MAE = (e) K = 3, MAE =.24 (f) K = 5, MAE =.9 Fgure : Results of fttng R(x, 4) (sold lne) usng the general form of GMM (dashed lne).

6 H. FAYED et al.: FITTING A RECTANGULAR FUNCTION BY GAUSSIANS (g) K = 5, MAE =.28 (h) K = 6, MAE = () K =, MAE =.85 (j) K = 2, MAE = (k) K = 3, MAE =.45 (l) K = 5, MAE =.33 Fgure 2: Results of fttng R(x, 4) (sold lne) usng the smple form of GMM (dashed lne).

7 2 APPL. COMPUT. MATH., V.4, N.2, 25 L(h, Σ) = whch can be evaluated as [2]: T d (2π) d Σ = exp 2 xt Σ x } dx d = ω ω d N(µ + h, Σ ) where L(h, Σ) = T d (2π) d = d = ω ω d exp (α ) Σ + Σ, α = 2 (µ + h) T Σ h h =. h d ( Σ + Σ, µ = µ. µ d ) Σ, Σ = (µ + h) 2 (µ + h) T Σ (µ + h), σ 2 σ σ 2 d Ths form, lke all other exstng algorthms used n approxmatng the normal ntegral, suffers from the curse of dmensonalty. However, to attan a smple expresson that can speed t up sgnfcantly, we used the smple form descrbed above; that s, the Gaussan functons are constraned to have the same varance σ 2 and µ l = µ + ( l ) δ, l d. Hence the ntegral can be approxmated by: L(h, Σ) = T d (2π) d Σ + σ 2 I = d = } ω ω d exp 2σ 2 C (µ + h) 2 2, where C s an upper trangular matrx obtaned from Cholesky decomposton of the matrx [ I Σ ( Σ + σ 2 I ) ]. In ths way, the computatonal complexty can be reduced to be of order O(d 2 K d ) flops. 4. Expermental results The orthant probabltes, L(, Σ) s evaluated usng the proposed method (GMM) for 7 d, and s compared wth Mwa s algorthm (avalable at: As a benchmark, we used the Gauss-Kronrod (7, 5) par quadrature ntegraton method for the equcorrelated case [23]. The probabltes are evaluated for ρ.,.2,.9}. For GMM, T = 4 s used to sample the rectangular functon R(x, T ), and K = 5 and K = 6 are nvestgated. For Mwa s algorthm, grd szes examned are G = 8 and G = 6 (whch have comparable runnng tmes wth the proposed approach). We used C language n our mplementaton on Wndows 7 runnng on Pentum 2.4GHz PC wth 3GB RAM. The absolute error and the elapsed tme are reported n Table to Table 4. It can be notced that GMM s comparable to Mwa s algorthm n accuracy, especally for ρ.7, whle consderably faster for d 8. Moreover, as the dmenson ncreases, processng of Mwa s algorthm becomes much slower than GMM. Thus, for 7 d, when low accuracy s requred, GMM becomes a reasonable choce, however, f hgh accuracy s needed, Mwa s algorthm s recommended.

8 H. FAYED et al.: FITTING A RECTANGULAR FUNCTION BY GAUSSIANS Conclusons In ths paper, an approxmaton of a rectangular functon s obtaned by adaptng the wellknown Gaussan mxture models to express t as a lnear combnaton of Gaussan functons. The proposed approxmaton s used to derve an approxmate expresson for the multvarate normal ntegral. The obtaned expresson s found to be fast f an error of order of 4 s plausble, especally for d 7 and ρ.7. Moreover, for d, t s consderably faster and thus more approprate and feasble than Mwa s algorthm. In future work, explorng other optmzaton strateges than the EM algorthm may be nvestgated, as t may yeld further mprovements [9]. Table. Results of the orthant probabltes for d = 7 Elapsed Tme Error ρ Mwa GMM Mwa GMM G = 8 G = 6 K = 5 K = 6 G = 8 G = 6 K = 5 K = E-4 3.6E-5.76E-4.3E E E-5.77E-4.34E E-4 2.5E E-5.5E E E-5.7E-4.52E E-4 2.E E-4 2.7E E-4.84E-5.47E-3 7.6E E-4.69E-5 2.9E-3.35E E-4.54E E-3.69E E-4.2E E E-4 Table 2. Results of the orthant probabltes for d = 8 Elapsed Tme Error ρ Mwa GMM Mwa GMM G = 8 G = 6 K = 5 K = 6 G = 8 G = 6 K = 5 K = E E-5.23E-4 8.4E E-4 6.E-5.25E-4.6E E E E-5 8.9E E E E E E E E-4 3.6E E E-5.67E-3 7.6E E-3 8.5E-5 3.3E-3.33E E E E-3.64E E-3 9.5E E E-4 Table 3. Results of the orthant probabltes for d = 9 Elapsed Tme Error ρ Mwa GMM Mwa GMM G = 8 G = 6 K = 5 K = 6 G = 8 G = 6 K = 5 K = E-3 8.6E E E E-3.8E-4 9.E E E-3.3E E-6 7.7E E-3.53E E E E-3.76E-4 8.8E-4 3.2E E-3 2.E-4.79E E E E-4 3.3E-3.25E E E-4 6.5E-3.4E E-3 2.9E-4.E-2 3.5E-5

9 22 APPL. COMPUT. MATH., V.4, N.2, 25 Table 4. Results of the orthant probabltes for d = Elapsed Tme Error ρ Mwa GMM Mwa GMM G = 8 G = 6 K = 5 K = 6 G = 8 G = 6 K = 5 K = E-3.5E E-5 4.4E E-3.67E E-5 7.5E E E-4.83E E E E E E E-3 3.5E E E E-3 4.2E-4.79E E E E-4 3.3E-3.7E E-3 5.9E E-3.E E-2 7.2E-4.7E E-5 References [] Calcaterra, C. Lnear combnaton of Gaussans wth a sngle varance are dense n L2, Proceedngs of the World Congress on Engneerng WCE, V.2, 28. [2] Deák, I. Three dgt accurate multple normal probabltes, Numer. Math., V.35, 98, pp [3] Deák, I. Random Number Generators and Smulaton, Akadéma Kadó, 99. [4] Dvg, D.R. Calculaton of unvarate and bvarate normal probablty functons, Ann. Stat., V.7, N.4, 979, pp [5] Drezner, Z. Computaton of the bvarate normal ntegral, Math. Comput., V.32, 978, pp [6] Drezner, Z., Wesolowsky, G.O. The computaton of the bvarate normal ntegral, J. Stat. Comput. Smul., V. 35, 99, pp.-7. [7] Duda, R.O., Hart, P.E., Stork, D.G. Pattern Classfcaton, 2nd ed., Wley, New York, 2. [8] Fayed, H.A., Atya, A.F. An evaluaton of the ntegral of the product of the error functon and the normal probablty densty, wth applcaton to the bvarate normal ntegral, Math. Comput., V.83, N.285, 24, pp [9] Fayed, H.A., Atya, A.F. A novel seres expanson for the multvarate normal probablty ntegrals based on fourer seres, Math. Comput., V.83, N.289, 24, pp [] Gassmann, H. Multvarate normal probabltes: Implementng an old dea of Plackett s, J. Comp. Graph. Stat., V.2, N.3, 23, pp [] Genz, A. Numercal computaton of rectangular bvarate and trvarate normal and t probabltes, Stat. Comput., V.4, N.3, 24, pp [2] Genz, A. Comparson of methods for the computaton of multvarate normal probabltes, Comp. Sc. Stat., V.25, 993, pp [3] Gupta, S.S. Probablty ntegrals of multvarate normal and multvarate t, Ann. Math. Statst., V.34, 963, pp [4] Harrs, B., Soms,A.P. The use of the tetrachorc seres for evaluatng multvarate normal probabltes, J. Multvarate Anal., V., 98, pp [5] Haykn, S. Neural Networks: a Comprehensve Foundaton, 2nd ed., Prentce-Hall, 999. [6] Kendall, M.G. Proof of relatons connected wth the tetrachorc seres and ts generalzatons, Bometrka, V.32, 94, pp [7] Mwa, T., Hayter, A.J., Kurk, S. The evaluaton of general non-centred orthant probabltes, J. R. Statst. Soc. B, V.65, 23, pp [8] Owen, D.B. Tables for computng bvarate normal probabltes, Ann. Math. Stat., V.27, N.4, 956, pp [9] Pardalos, M.P. and Chnchuluun, A. Some recent developments n determnstc global optmzaton (survey), Appl. Comput. Math., V.5, N., 26, pp [2] Pearson, K. Mathematcal contrbutons to the theory of evoluton. VII. on the correlaton of characters not quanttatvely, Phlos. Trans. R. Soc. S-A, V.96, 9, pp.-47. [2] Petersen, K.B. and Pedersen, M.S. The matrx cookbook, Nov 22, verson 225. [Onlne]. Avalable: detals.php?d=3274 [22] Theodords, S., Koutroumbas, K. Pattern Recognton, 2nd ed., Elsever, New York, 23. [23] Tong, Y.L. The Multvarate Normal Dstrbuton, Sprnger Seres n Statstcs, Sprnger-Verlag, New York, 99.

10 H. FAYED et al.: FITTING A RECTANGULAR FUNCTION BY GAUSSIANS Hatem A. Fayed - s an Assocate Professor at the Engneerng Mathematcs and Physcs Department, Caro Unversty, and currently a seconded Assocate Professor at Zewal Cty of Scence and Technology. He receved hs Ph.D. from the Engneerng Mathematcs and Physcs Department, Caro Unversty, 25. Hs research nterests are n the areas of machne learnng, tme seres forecastng, neural networks, optmzaton technques, and mage ptrocessng. Amr F. Atya - receved hs B.S. and M.S. degrees from Caro Unversty, and hs M.S. and Ph.D. degrees from Caltech, Pasadena, CA, all n electrcal engneerng. Dr. Atya s currently a Professor at the Department of Computer Engneerng, Caro Unversty. Hs research nterests are n the areas of machne learnng, theory of forecastng, computatonal fnance, dynamc prcng, and Monte Carlo Methods. Ashraf H. Badaw - s a Dean of Student Affars, Assstant Professor at the Center of Nanotechnology at Zewal Cty. He s also the Drector for the Learnng Center of Learnng Technologes. Pror to jonng SMART Ashraf was the lead WMAX Solutons Specalsts for Intel n the Mddle East and Afrca. He was an assstant professor, Caro Unversty, Engneerng Math and Physcs department from 22 tll 29. He graduated from the Systems and Bomedcal Engneerng Department n 99 n Caro, where he started pursung hs M.Sc. n Engneerng Physcs. He then traveled to Wnnpeg, Canada to pursue hs PhD n Electrcal Engneerng from the Unversty of Mantoba.