Gaussian Processes and Polynomial Chaos Expansion for Regression Problem: Linkage via the RKHS and Comparison via the KL Divergence

Transcription

1 entropy Artce Gaussan Processes and Poynoma Chaos Expanson for Regresson Probem: Lnkage va the RKHS and Comparson va the KL Dvergence Lang Yan * ID, Xaojun Duan, Bowen Lu and Jn Xu Coege of Lbera Arts and Scences, Natona Unversty of Defense Technoogy, Changsha 47, Chna; xjduan@nudt.edu.cn (X.D.); bowen_u@6.com (B.L.); xujn_nudt@6.com (J.X.) * Correspondence: yanang@nudt.edu.cn; Te.: Receved: January 8; Accepted: March 8; Pubshed: March 8 Abstract: In ths paper, we examne two wdey-used approaches, the poynoma chaos expanson (PCE) and Gaussan process (GP) regresson, for the deveopment of surrogate modes. The theoretca dfferences between the PCE and GP approxmatons are dscussed. A state-of-the-art PCE approach s constructed based on hgh precson quadrature ponts; however, the need for truncaton may resut n potenta precson oss; the GP approach performs we on sma datasets and aows a fne and precse trade-off between fttng the data and smoothng, but ts overa performance depends argey on the tranng dataset. The reproducng kerne Hbert space (RKHS) and Mercer s theorem are ntroduced to form a nkage between the two methods. The theorem has proven that the two surrogates can be embedded n two somorphc RKHS, by whch we propose a nove method named Gaussan process on poynoma chaos bass (GPCB) that ncorporates the PCE and GP. A theoretca comparson s made between the PCE and GPCB wth the hep of the Kuback Leber dvergence. We present that the GPCB s as stabe and accurate as the PCE method. Furthermore, the GPCB s a one-step Bayesan method that chooses the best subset of RKHS n whch the true functon shoud e, whe the PCE method requres an adaptve procedure. Smuatons of D and D benchmark functons show that GPCB outperforms both the PCE and cassca GP methods. In order to sove hgh dmensona probems, a random sampe scheme wth a constructve desgn (.e., tensor product of quadrature ponts) s proposed to generate a vad tranng dataset for the GPCB method. Ths approach utzes the nature of the hgh numerca accuracy underyng the quadrature ponts whe ensurng the computatona feasbty. Fnay, the expermenta resuts show that our sampe strategy has a hgher accuracy than cassca expermenta desgns; meanwhe, t s sutabe for sovng hgh dmensona probems. Keywords: Gaussan process; poynoma chaos expanson; reproducng kerne Hbert space; Kuback Leber dvergence; expermenta desgn. Introducton Computer smuatons are wdey used n earnng tasks, where a snge smuaton s an nstance of the system [,]. A smpe approach to the earnng task s to randomy sampe nput varabes and run the smuatons for each nput to obtan the features of the systems. Smar approaches are utzed n Monte Caro technques []. However, even a snge smuaton can be computatonay costy due to ts hgh compexty, and so, obtanng a trustworthy resut va suffcent smuatons becomes ntractabe. Mathematca methods and statstca theorems are ntroduced to generate surrogate modes to repace the smuatons, especay when deang wth compex systems wth many parameters [4,5]. Athough the man drawback of surrogate modes s that ony approxmatons can be obtaned, they are computatonay effcent whst mantanng the essenta nformaton of the systems, hence Entropy 8,, 9; do:.9/e9

2 Entropy 8,, 9 of anayzng the propertes of the system. Attemptng to construct surrogate modes wth an acceptabe number of smuatons necesstates the deveopment of robust technques to determne ther reabty and vadty [6 8]. Penty of researchers are workng on mprovng sampng strateges to decrease the number of smuatons, whch makes the task more sgnfcant [9]. Wth an ncreasng number of surrogate modes beng deveoped, there needs to be a comprehensve understandng of the uncertantes ntroduced by those modes. The man purpose of uncertanty quantfcaton (UQ) s to estabsh a reatonshp between nput and output,.e., the propagaton of nput uncertantes, and then to quantfy the dfference between surrogate modes and orgna smuatons. UQ can provde a measure of the surrogate mode s accuracy and an ndcaton of how to update the mode at the same tme [ ]. Denote f as a functon (or smuator) of the orgna system, then gven expermenta desgn X, the output Y = f (X) s produced, where capton notaton s used because the nput and output are usuay vectors (or matrces) n smuatons. From a statstca perspectve, the nput uncertantes are ntroduced by ther randomness, so we represent the nput wth a random varabe x, whose pror probabty densty functon (PDF) s p(x), such as the mutvarate Gaussan dstrbuton; as for the output uncertantes, a common technque s to ntegrate the system uncertanty and the approxmaton error as a nose term ɛ. In fact, the output y s aso a random varabe y = f (x) + ɛ determned by f, x and ɛ. Now, suppose a surrogate f (x) s constructed to approxmate f (x), then UQ s used to dentfy the dstrbuton and statstca features (for exampe, Kuback Leber dvergence) of y, whch are essenta to the vadaton and verfcaton of surrogates. Bascay, there are two precondtons that need to be satsfed: frsty, the surrogate modes are we defned,.e., any f s a measurabe functon wth respect to (w.r.t) correspondng probabty space p(x); secondy, technques are needed that earn from the pror nformaton to obtan the best guess of the true functon. There s a number of studes proposng dfferent surrogates for specfc appcatons n the terature, such as mutvarate adaptve regresson spnes (MARS) [], support vector regresson (SVR) [4], artfca neura network (ANN) [5] for reabty and senstvty anayses and krgng [6] for structura reabty anayss. We many focus on two popuar methods that have been extensvey studed recenty. One popuar method that s extensvey studed n the terature s the poynoma chaos expanson (PCE), aso known as a spectra approach [7]. PCE ams to represent an arbtrary random varabe of nterest as a spectra expanson functon of other random varabes wth pror PDF. Xu et a. [8 ] have generazed the PCE n terms of the Askey scheme of poynomas, so the surrogates can be expressed by a seres of orthogona poynomas w.r.t the dstrbutons of the nput varabes. These poynomas can be extended as a bass of a poynoma space. In genera, methods used to sove PCE probems are categorzed as two types: ntrusve and non-ntrusve. The man dea behnd the ntrusve methods s the substtuton of the nput x and output f (x) wth the truncated PCE and cacuatng the coeffcents wth the hep of Gaerkn projecton []. However, the expct formaton of f s requred to compose the Gaerkn system, and a specfc agorthm or program s needed to sove a partcuar probem. It s for these reasons that the ntrusve modes are not wdey used; non-ntrusve methods have been deveoped to avod these mtatons [,]. There are two man aspects of the non-ntrusve methods: one s the choce of sampng strateges, for exampe Monte Caro technques; the other one s computatona approaches. These two aspects are not ndependent of each other: for exampe, f x N(, ), then the Gaussan quadrature method s ntroduced to sove the numerca ntegraton and X s the set of correspondng quadrature ponts. Another one of the more common methods n constructng surrogate modes s the Gaussan process (GP), whch s actuay a Bayesan approach. Instead of attemptng to dentfy a specfc rea mode of the system, the GP method provdes a posteror dstrbuton over the mode n order to make robust predctons about the system. As descrbed n the hghy nfuenta works [ 6], the GP can be treated as a dstrbuton over functons wth propertes controed by a kerne. For the two prerequstes dscussed n the prevous paragraph, the GP generates a surrogate mode that es n a space spanned

3 Entropy 8,, 9 of by kernes; meanwhe, Bayesan near regresson or cassfcaton methods are ntroduced to utze the pror nformaton. Both the PCE and GP methods bud surrogates, but there are some dfferences between them. The PCE method buds surrogates of a random varabe y as a functon of another pror random varabe x rather than the dstrbuton densty functon tsef. The PCE surrogates are based on the orthogona poynoma bass correspondng to the p(x), so t s smpe to obtan the mean and standard devaton of y. In contrast, the GP utzes the covarance nformaton so that t performs better n capturng the oca features. Athough both the PCE and GP approaches are feasbe methods to compute the mean and standard devaton of y, the PCE performs more effcenty than the GP method. As mentoned above, both the PCE and GP methods have ther own trade-offs to consder when budng surrogates, and there exsts a connecton to be expored. Accordng to Pau Constantne s work [7], ordnary krgng (.e., GP n geostatstcs) nterpoaton can be vewed as a transformed verson of the east squares (LS) probem, and the PCE can be vewed as the east squares wth seected bass and weghts. However, the GP reverts to nterpoaton when the nose term s zero. When takng the nose term nto consderaton, the Gaussan process wth the kerne (.e., covarance matrx) X T X can be vewed as a rdge regresson probem [8] wth a reguarzaton term. Furthermore, dfferent numerca methods can affect the precson of the PCE method, as we. For exampe, Xu [] anayzed the aasng error w.r.t the projecton method and nterpoaton method. Thus, the nherent connecton of the two modes cannot be smpy summarzed as an LS souton, and how to output a mode wth hgh precson remans an nterestng queston. There are connectons between the PCE and GP methods that have been expored by R. Schob, etc. They ntroduced a new meta-modeng method namng PC-krgng [9] (poynoma-chaos-based krgng) to sove the probems ke rare event estmaton [], structura reabty anayss [], quante estmaton [], etc. In ther papers, the PCE modes can be vewed as a speca form of GP where a Drac functon s ntroduced as the kerne. They aso proposed the dea that the PCE modes have better performance n capturng the goba features and that the GP modes approxmate the oca characterstcs. We woud ke to descrbe the PC-krgng method as a GP mode wth a PCE-form trend functon aong wth a nose term. The goba features are domnated by the PCE trend, and oca structures (resduas) are approxmated by the ordnary GP process. The PC-krgng mode thus ntroduces the coeffcents as parameters to be optmzed, and the souton can be derved by Bayesan near regresson wth the bass consstng of the PCE poynomas. They aso use the LARSagorthms to cabrate the mode and to seect a sparse desgn. They construct a rgd framework to optmze the parameters, vadate and cabrate the mode and evauate the mode accuracy. Unke the PC-krgng, whch takes the PCE as a trend, ths paper focuses on the constructon of the kerne n the GP to sove the regresson probems, through whch we can combne the two methods nto a unfed framework, unfyng postve aspects from both and n so dong refnng the surrogates. In other words, we wsh to fnd the connecton between the GP and the PCE by anayzng the attrbuton of ther soutons, and we want to propose a new approach to acheve hgh-precson predctons. The man dea of ths paper s descrbed as foows. Frsty, the PCE surrogate s embedded n a Hbert space whose bases are the orthonorma poynomas themseves, then a sutabe nner product and a Mercer kerne [] are defned to bud a reproducng kerne Hbert space (RKHS) []. Secondy, on the other hand, the kerne of the GP can be de-composted as the product of egenfunctons, and we can defne an nner product to generate a RKHS, as we. We have expcty eaborated the two procedures respectvey and proven that the two RKHS are sometrcay somorphc. Hence, a connecton between these two approaches has been estabshed va RKHS. Furthermore, we can obtan a souton of the PCE mode by sovng a GP mode wth the Mercer kerne w.r.t the PCE poynoma bass. We name ths approach Gaussan processes on poynoma chaos bass (GPCB). In order to ustrate the capabty of the GPCB method, we use the Kuback Leber dvergence [4] to expcty compare the PDFs of the posteror predcton of the GPCB and PCE method. Provded

4 Entropy 8,, 9 4 of that the true functon can be approxmated by a fnte number of PCE bases, t can be concuded that the GPCB can converge to the optma subset of the RKHS wheren the true functon es. The expermenta desgn from the PCE mode,.e., the fu tensor product of quadratures n each dmenson, s used n the GPCB. We have overcome two concerns about the PCE and GP, respectvey. Frsty, the PCE s based on a truncated poynoma bass, whe the GPCB keeps a poynomas, whch can be regarded as mantanng nformaton n every feature. Secondy, the GP s behavor depends on the expermenta desgn; however, t often acheves the optma resut n oca sma datasets. The quadrature ponts derved from the PCE mode are dstrbuted eveny n the nput space, and those ponts have hgh numerca precson w.r.t the poynoma bass; hence, they can work we wth the GPCB. However, we must admt that the GPCB s st a GP approach, so when the dmenson of nput varabes grows, the computatona burden s on the tabe. In order to cope wth the hgh dmensona probems, sampng strateges to ower the number of expermenta desgns are put on the tabe. The AK-MCSmethod [5] s a usefu too that adaptvey seects new expermenta desgns; however, the expermenta desgn tends to vadate the seected surrogate mode. We propose a new method that s mode-free and that makes fu use of the quadrature ponts. We randomy choose a sparse subset from the quadrature ponts to form a new expermenta desgn whe mantanng the accuracy. Severa cassca sampng strateges ke MC, Haton and LHS are ntroduced to compare ther capabtes. Our sampe scheme has superor performance under the condtons n ths paper. The GPCB s a nove method to bud surrogate modes, and t can be used for varous physca probems such as reabty anayss and rsk assessment. Ths paper s dvded nto two parts. In Part, we dscuss the mathematca rgor of the method: a bref summary of PCE and GP s presented n Secton ; the reproducng kerne Hbert space (RKHS) s ntroduced to connect these two methods n Secton ; the GPCB method s proposed based on the dscusson n Secton 4; meanwhe, a theoretca Kuback Leber dvergence between the GPCB and PCE method s demonstrated. In Part, an expct Meher kerne s presented wth the Hermte poynoma bass n the ast part of Secton 4; severa tests of the GPCB wth some benchmark functons are presented n Secton 5, aong wth the random constructve sampng method for hgh dmensona probems.. Bref Revew of PCE and GP Frsty, we want to have a cear dea of how the PCE and GP work under the crcumstances that, e.g., ony sampes of nput X and output Y are obtaned. Dfferent assumptons are made to cope wth PCE and GP, respectvey, and the processng procedures are presented n the foowng subsectons... Poynoma Chaos Expanson Just as dscussed n Secton., the output s assumed to be represented by a mode y = f (x) + ɛ, where f (x) : x Ω R s the rea functon underneath and ɛ N (, σɛ ). Here, we defne x = (x, x,..., x d ) T as a d-dmensona vector of ndependent random varabes n a bounded doman Ω R d. Suppose {x, =,..., d} are ndependent and dentcay dstrbuted; the jont PDF has the form p(x) = = d p(x ). In the context of PCE, we am to seek a surrogate of the mode f (x) as an expanson of a seres of orthonorma poynomas φ α (x): f (x) = α N d β α φ α (x), α = {α,..., α d } () where α s the mut-ndex, φ α (x) = d = φ() α (x ), Ω φ () m (x )φ () n (x )p (x )dx = δ mn wth Ω the margna doman of Ω and δ mn the Kronecker deta. Xu et a. [] have summarzed varous correspondences between the dstrbuton and poynoma bass to form generazed poynoma chaos.

5 Entropy 8,, 9 5 of It s proven that the orgna mode f (x) can be approxmated to any degree of accuracy n a strong sense [], e.g., mean-square norm f (x) α N D β α φ α (x) n an L norm defned on Ω, athough f s not necessary the span of orthonorma poynoma bases. Snce we are unabe to cacuate an nfnte seres, the truncaton scheme correspondng to mut-ndex α s ntroduced such that we can rearrange the poynomas. For smpcty, we can rewrte Equaton () n the foowng form: f (x) M β φ (x) f P, () = We can smpy sove the above system va the ordnary east squares method or the non-ntrusve method. Specfcay, we focus on the non-ntrusve projecton method, whereby we can drecty obtan the coeffcents by takng the expectaton vaue of Equaton () mutped by φ (x): β = f (x)φ (x)p(x)dx N ω f (X )φ (X ), =,..., M () = where the second equaton s derved by the numerca ntegraton technques, such as the Gaussan quadrature rue, and {X, =,..., N} and {ω, =,..., N} are the correspondng nodes and weghts. The ntegraton s exact when f (x) s of poynoma compexty. Together wth Equatons () and (), f P (x) has the form: f P (x) [ M N ] ω f (X )φ (X ) φ (x) = = M β φ (x). (4) = { f (X ), =,..., N} reman unknown to us, and usuay, they are substtuted by {Y, =,..., N}. Note Y = f (X ) + ɛ, so such a substtuton w ntroduce nose nto the surrogate; hence, the approxmaton error s negected as a source of uncertanty... Gaussan Process Regresson The anayss of the Gaussan process regresson mode [6] s revewed n ths secton. A Gaussan pror s paced over functon f (x),.e., f (x) GP(m(x), k(x, x)), where m s the mean functon and k s the kerne functon, whch s postve sem-defnte bounded. More specfcay, et X = {X, =,..., N} Ω N be the nput data, and et Y = {Y, =,..., N} R N be the output data, then we have Y = f (X) + ɛ wth f (X) N (m(x), k(x, X)) and ɛ N (, σ ɛ I). Bear n mnd that the mathematca expresson of f (x) s mpct, so f (x) s approxmated to acheve the best guess predcton f G n the statstca sense. Wth the hep of Bayes theorem, predcton and correspondng varance at a new pont x can be obtaned by the foowng equatons [6]: p G ( f (x) Y, X, x, θ) = N ( f G (x), cov( f G (x))), f G (x) E[ f (x) Y, X, x, θ] = K T x [K + σ ɛ I] Y, cov( f G (x)) = K xx K T x [K + σ ɛ I] K x. (5) where K = k(x, X) R N N as the covarance matrx wth K j = k(x, X j ) and K x = k(x, x) R N, K xx = k(x, x) R are defned smary. Note that Equaton (5) shows that the mean vaue of the posteror dstrbuton can be expressed as a near combnaton of N kerne functons as foows: f G (x) = N α k(x, x), α = (K + σɛ I) Y (6) =

6 Entropy 8,, 9 6 of. Lnks between the PCE and GP The basc concepts of PCE and GP are dscussed n Secton. GP generates a surrogate based on Bayes theorem and the Gaussan hypothess; however, t s controed by the kerne functon and the expermenta desgn and usuay does not utze pror dstrbuton nformaton; PCE substtutes the mode wth orthonorma poynomas, whch s more computatona effcent, but performs bady when facng nosy or bg data. Ths secton ams to bud a connecton between PCE and GP; hence, they can be studed n the same structure and be combned to mprove the performance of the surrogates. The reproducng kerne Hbert space w be of great hep to bud such a brdge, and we are gong to present t as foows... Generate an RKHS from a Mercer Kerne Constructed by the PCE Bass We have obtaned a compete orthonorma bass {φ } of Hbert space H := span{φ (x)} wth nner product < f (x), g(x) >= f (x)g(x)p(x)dx n Secton.. We can see that the PCE method generates surrogates, whch are actuay a near combnaton of {φ }, so there exsts a unque expanson f = f φ H. Accordng to Mercer s theorem [], we am to defne a kerne havng the foowng form: k(x, x ) = λ φ (x)φ (x ) s.t. k(x, x) < f or x Ω. (7) If we have postve weghts λ that satsfy λ φ (x) <, then for any x Ω, together wth the Cauchy Schwarz nequaty, we have: f (x) < f (x), φ (x) > λ λ φ (x). (8) f (x) s pont-wse bounded because f (x) H for any x Ω. By checkng the rght sde of the above nequaty, the second term s ensured n advance, then f (x) es n a subspace of H such that: H P = { f H < f (x), φ < f, f > HP = (x) > λ } <, λ φ (x) <. (9) Proposton. H P defned n Equaton (9) s an RKHS wth Mercer kerne defned n Equaton (7)... Generate an RKHS from the Reproducng Kerne Map Constructon We am to compose a space of functons n whch a the GP surrogates are embedded. Gven Equaton (6) and an arbtrary expermenta desgn X, defne a space of functons as foows: H G = { f (x) = N = f k(x, X ) N N, X Ω N, x Ω, f R, N = N j= f f j k(x, X j ) < + }. () Proposton. H G s a pre-hbert space wth the nner product <, > H G Now that H G s a pre-hbert space and gven the norm f (x) H G = < f (x), f (x) > H G, we can defne a cosure of H G as H G derved by the cassca Hbert space theory. Ths s an abstract space where the norm of H G extends to the cosure H G. Thus, we have a Hbert space H G. Proposton. H G defned above s the unque RKHS of the kerne k(, ).

7 Entropy 8,, 9 7 of.. Reproducng Kerne Hbert Spaces as a Lnkage H P and H G are RKHS wth the Mercer kerne and GP kerne, respectvey. We are gong to nvestgate the reatonshp between the two RKHS, by whch we can dscuss the two approaches n a unfed structure. Let X be a sampe set and GP kerne k(, ) be a rea postve sem-defnte kerne, then accordng to Mercer s theorem, k(x, X j ) has an egenfuncton expanson: k(x, X j ) = λ φ (X )φ (X j ), () where the egenfunctons {φ } are orthonorma,.e., < φ, φ >= φ (x)φ (x)p(x)dx = δ and {λ, φ } satsfes λ φ (x) <. Let f X(x) H G wth expermenta desgn X, then we can rewrte t accordng to Equatons () and (): f X (x) = N = f [ ] N λ φ (X )φ (x) = λ f φ (X ) = φ (x) c (X)φ (x). () where c (X) s dentcay determned by and X, and t has a smar form as a functon es n H P. Actuay, gven f X (x), g X (x) H G, we have: < f X (x), g X (x) > HG = N = = N j= {[ ] f g j [ λ φ (X )λ φ (X j )/λ N λ f φ (X ) = ] [λ N ]} g j φ (X j ) /λ j= = c (X)c (X )/λ = < f X (x), φ >< g X (x), φ > /λ =< f X (x), g X (x) > HP. The above equaton gves us the nformaton that ther nner product stands n H P, as we, so we can concude that f X es n H P. It aso shows us that the two nner products are equvaent. Next, we are gong to propose a rgd theorem to prove that the two spaces are sometrcay somorphc. Theorem. The reproducng kerne Hbert space H G of a gven kerne k s sometrcay somorphc to the space H P. Accordng to the proof of Theorem n Appendx A.4, t s reasonabe to ntroduce a weghted space /λ because t s dffcut to fnd a drect near map between H G (where c vares accordng to X and k) and H P (where c vares accordng to the dstrbuton of x). Fgure shows two fowcharts used to descrbe dfferent processes n generatng the RKHS. Furthermore, the GP predcton s a combnaton of the kerne functons, whch consst of nfnte egenfunctons, whe the PCE predcton s aways a combnaton of fnte poynoma bases. The Kuback Leber dvergence (KL dvergence) s a usefu crteron to ndcate the performance of dfferent surrogate modes. We are gong to present the comparson of the GPCB and PCE methods wth the hep of KL dvergence n the next secton. ()

8 Entropy 8,, 9 8 of Gaussan Process Dstrbuton of x We defned kerne functon k(, ) Sampng X Known Bass {φ(x)} Parameter λ Space H G defned n Equaton () Egenfuncton expanson n Equaton () Space H P defned n Equaton (9) Mercer Kerne defned n Equaton (7) RKHS wth k(, ) Isomorphc to /λ Isomorphc to /λ RKHS wth Mercer kerne Fgure. Left: Generate the reproducng kerne Hbert space (RKHS) wth the reproducng kerne map; rght: generate the reproducng Mercer kernes wth the poynoma chaos expanson (PCE) bass. 4. Gaussan Process on Poynoma Chaos Bass H G and H P are somorphc as dscussed n prevous Secton, so t s natura to come up wth the dea that GP can be conducted wth k(, ) as the Mercer Kerne generated by poynoma bass n the PCE, and the new mode s caed Gaussan process on poynoma chaos bass (GPCB). In fact, the GPCB generates a PCE-ke mode, but wth a dfferent phosophy. Note that the posteror dstrbuton of the predctons regardng expermenta desgn {X, Y} can be cacuated anaytcay, so we are abe to compute the KL dvergence as we, whch are presented as foows. 4.. Comparson of the PCE and GPCB wth the Kuback Leber Dvergence The true dstrbuton of the system s aways mpct n practce. Wthout oss of generaty, the underyng true system s assumed to be f P (x) = M = β φ (x) f f P (x) C ( Ω) such that t can be approxmated by the poynomas to any degree of accuracy []. Frsty, we presume that M M,.e., β n Equaton (4) s an unbased estmator of β. Hence, the PCE approxmaton can be consdered as a precse approxmate of the true functon. We compare the performance of the GPCB and the PCE method by comparng ther dfference n the posteror dstrbuton of the predcton. It s known that gven expermenta desgn {X, Y} and kerne functon k(x, x ) = = λ φ (x)φ (x ), the dstrbuton of the predcton of the GPCB reads: p G ( f G (x)) = N (Kx T K Y Y, K xx Kx T K Y K x) N (µ, Σ ), (4) where condtons X, Y, x are dropped n p G ( f G (x) X, Y, x) for smpcty and K Y = K + σ I. Smary, the predcton of the PCE wth the projecton method s f P (x) = φφ T WY, whch s derved from the estmaton β = Φ T WY. Here, φ = φ(x) R (P+), Φ = φ(x) R N (P+), W = dag{ω,..., ω N } s a dagona matrx. The correspondng predcton varance s cov( f P (x)) = φcov(β)φ T = σ φφ T W Φφ T. Droppng the condtons n p P ( f P (x) X, Y, x) as we, the prevous resuts ndcate: p P ( f P (x)) = N (φφ T WY, σ φφ T W Φφ T ) N (µ, Σ ). (5) We can evauate the dscrepancy between p G ( f G (x)) and p P ( f P (x)), hence comparng ther performance. The KL dvergence can be cacuated anaytcay: D KL (p P, p G ) = ( ) + Σ + og Σ + (µ µ ) Σ Σ Σ ) ( + b og b + a b, (6) Σ

9 Entropy 8,, 9 9 of where a, b are smpfed notatons for the correspondng parts n Equaton (6). In fact, b s the pont-wse rato between the posteror varances of the predctons, and a represents the dfference between the posteror mean of the predcton. We dscuss the propertes of D KL startng from a speca case to the genera condtons hereafter. Let k be a truncated kerne wth the M bass of PCE,.e., k(x, x ) = M = λ φ (x)φ (x ). Actuay, t can be seen as the assgnment of {λ, > M} wth the vaue of zero, whch can be acheved by optmzng the vaue λ wth a specfc procedure. b can be smpfed as: b = = σ φφ T W Φφ ( T ) φ Λ = ΛΦ T K Y ΦΛ φ T φφtwφφt φφ T WK Y KWΦφT [ smax + σɛ φφ T W Φφ T φφ T WU(S + σ I) U T USU T WΦφ T s max, s mn + σ ɛ s mn ], (7) where Λ = dag{λ } s a dagona matrx and K = USU T s the egenvaue decomposton. Let s max be the maxmum egenvaue and s mn be the mnma one; the above nterva hods because K s a postve defnte matrx. Note that b s an nvarant wth fxed x. On the other hand, the dstrbuton of a s as foows: ( ) a = φ Φ T WY ΛΦ T K Y Y = σɛ φφ T WK Y Y ( N σ ɛ φφ T WK Y Ŷ, σ6 ɛ φφ T WK Y K Y WΦφT). Here, Ŷ denotes the mean vaue of the observatons,.e., the true response. It s necessary to state that the randomness of a s brought by the random varabe ɛ n observaton Y. In fact, we have the expectaton of D KL (p P, p G ) as foows: E ɛ [D KL (p P, p G )] = ( + b og b + E ɛa ) b Σ = ) ( + b og b + (var(c) + (E ɛ a) )b/σ ( + b og b + σ ɛ + Ŷ ( σ ) ɛ b) σɛ s mn + σɛ. Presume that the observaton Y s normazed, as we as Ŷ; hence, Ŷ can be estmated as O(). The man dfference s affected by the kerne k(, ) (or Λ) and the term σ ɛ. More specfcay, The GPCB can acheve a smaer varance than the PCE method n a pont-wse manner because b >, and the dfferences between the predctons of the two methods s of the order of σ ɛ. Furthermore, f σ ɛ s suffcent sma,.e., σ ɛ s mn, we have b, thus E ɛ [D KL (p P, p G )]. If σ ɛ =,.e., we nvestgate the nose-free modes, then b = and a =, whch enforces Σ = Σ and µ = µ, respectvey. Ths means that p G and p P are dentca dstrbutons,.e., D KL (p P, p G ) =. We can concude that the expected vaue of D KL (p P, p G ) s bounded by a certan constant, whch many depends on the σ ɛ and Λ. In other words, snce σ ɛ s gven n the pror, and the Λ are optmzed; hence, the D KL (p P, p G ) s constraned, whch means that the GPCB s as stabe as the PCE method. Nonetheess, f f P has reached a desred predcton precson, then f G wth the kerne constructed wth the same bass can have a desrabe precson, as we as smaer varance. Secondy, we consder that M > M, where the β of the PCE method s not an unbased estmate of β any onger. Let p P denote the PCE approxmaton wth the M bass; under the crcumstance, k(x, x ) = M k= λ φ (x)φ (x ) s acheved by tunng the vaue of Λ va a certan earnng method; hence, (8) (9)

10 Entropy 8,, 9 of D KL (p P, p G) s aso bounded by a constant accordng to Equaton (9),.e., the GPCB can converge to the precse PCE predcton p P, as we. However, the KL dvergence D KL(p P, p P) s gven as: D KL (p P, p P) = ( + Σ og Σ + ( µ ) µ ) Σ Σ Σ b = φφt W Φφ T + φ r Φr T W Φ r φr T + φaφr T φφ T W Φφ T, ( ) ā = φ r Φr T WY N φ r Φr T WŶ, σɛ φ r Φr T W Φ r φr T. ( + b og b + ā We denote φ r as the bass that beongs to the mode f P, but f P. It s shown that the based PCE f P has smaer varance, however wth a bas whose mean vaue depends on φ r. We notce that φ r represents the hgh-order poynomas; hence, the bas can be consderaby arge for genera cases, and so s the D KL (p P, p P). We can concude that even though the based PCE has smaer varance, the reatvey arge bas can ead to a fase predcton. In fact, f the underyng system s smooth enough to be modeed by a poynoma approxmaton, then we can adaptvey ncrease the number of poynoma bases (and the expermenta desgns f necessary) to reach a precse approxmaton. However, on the other hand, we can drecty use the GPCB method, whch s a one-step Bayesan approxmaton, that converges to the hypothetca true system f P. Roughy speakng, the GPCB fnds M automatcay by tunng the parameters Λ nstead of adaptvey changng the vaue of P n the PCE method. It ndeed provdes more convenence for computaton. The key probem s the evauaton of Λ. Specfcay, we ntroduce the Meher kerne [7], whch s an anaytc expresson of the Mercer kerne constructed by Hermte poynomas, and dscuss the earnng procedure of Λ. 4.. Constructon of the Kerne wth Hermte Poynomas Reca that we have {φ α } n PCE as an orthonorma bass, then we regard them as egenfunctons of a kerne k(, ). Snce p(x) foows the standard Gaussan dstrbuton, then φ () (x ) = He (x )/!, where x s the -th varabe of x and He (x ) s the Hermte poynoma of degree : Σ ), () He (x ) = ( ) e x d dx e x. () Here, we denote the rea -th mut-ndex of the -th poynoma n Equaton () as α () = (α (),..., α() d ), α() = α () α () d and α ()! = α ()!... α() d!, then accordng to the prevous anayss, we can get φ α ()(x) = He α ()(x)/ α ()!. We have Meher kerne Me(X, X ) [7] wth orthonorma Hermte poynomas as egenfunctons: Me(x, x ) = exp { D xd T x + D x D T x } [ exp d = ρ x x ] = ρ α() φ α ()(x)φ α ()(x ), () = where the egenvaue s λ α () = ρ α() = = d ρα() for parameter ρ, D x s the symbo representng the row gradent operator,.e., D x = ( / x,..., / x d ), and D x s defned smary. Specfcay, n the one-dmensona case: ( Me(x, x ) = exp ρ (x + x ) ρxx ) ρ ( ρ = ) ρ φ (x)φ (x ), () = where the egenvaue λ = ρ >. The truncated kerne Me M (x, x ) = M = ρ φ (x)φ (x ), and ts attrbuton can be nvestgated by varyng ρ and M. Fgure a ustrates the truncated kerne Me M (x, x ), whch shows that Me M (x, x ) tends to converge to Me(x, x ) as M grows. Fgure b

11 Entropy 8,, 9 of shows the vaues of Me(x,.8) wth dfferent ρ. It presents to us that the nfuence of egenvaue λ s greater on the Meher kerne (a) (b) Fgure. Comparson of the effect of M and ρ on the D Meher kerne wth one fxed pont {.8}. (a) Kerne vaue of Me M (x,.8) wth ρ =.5; (b) kerne vaue of Me(x,.8). 4.. Learnng the Hyper-Parameter ρ of Me(x, x ) It s cear that λ = ρ has a great mpact on the kerne vaues, hence affectng the convergence of the f G (x). We start wth a smpe exampe, where f (x) = 5 + x + exp(x), x N (, ) s the true underyng functon, and the nose term s gnored n the observaton. Consderng the Tayor expanson of exp(x), f (x) can be approxmated by the PCE wth suffcenty arge M. In fact, we can cacuate the projecton < f (x), φ (x) > to seek the vaue of M. When > M, < f (x), φ (x) >. On the other hand, as dscussed n Secton 4., the GPCB can fnd M automatcay by tunng the hyper-parameter ρ. Let the expermenta desgn X be the zeros of He (x) of Equaton (),.e., quadrature ponts correspondng to degree ; we compare the performance of the GPCB wth ρ equa to.,.45,.7, respectvey. The resuts are dspayed n Fgure. Note that the projecton vaue s the absoute vaue of the true vaue n the fgure for better ustraton. It shows that we are abe to approxmate f (x) wth poynomas up to degree 4. If ρ =.45 for the Meher kerne, the GPCB amost converges to exact f (x), whereas ρ =. eads to a fast convergence rate and ρ =.7 resuts n a sow convergence rate Fgure. Projectons on the frst poynomas of f (x) and the Gaussan process on poynoma chaos bass (GPCB) wth ρ =.,.45,.7.

12 Entropy 8,, 9 of Fgure shows that the ρ has a cruca mpact on the performance of the GPCB method, so a tractabe method to optmze ρ s needed. A natura crteron s the KL dvergence, whch can be mnmzed by fndng the optma hyper-parameters ρ. We dscussed the KL dvergence of the GPCB and the PCE surrogates under the assumpton that the PCE surrogate mode can approxmate the true system to any degree of accuracy. However, the dstrbuton of the rea system s usuay unknown, whch makes the cacuaton of KL dvergence ntractabe. In fact, t can be easy deduced that the mnmzaton of KL dvergence s equvaent to mnmzng the negatve og margna kehood, whch (actuay s ) reads: ] = Y T K Y [(π) Y + og N K Y It s mportant to optmze ρ and σ ɛ to obtan a sutabe kerne to get an accurate approxmaton. Cassca methods ke gradent-based technques can be used to search for the optma ρ; however, t may perform poory because t s ocay optmzed. As we can see n Equaton (), t s ndcated that ρ shoud take a vaue between zero and one, so we can propose a goba method to sove our optmzaton probem. The agorthm for generatng a GPCB approxmaton s gven n Agorthm : (4) Agorthm : Genera procedure of the GPCB method wth the Meher kerne. Input: Smuator y(x), pror dstrbuton p(x) Output: A GPCB approxmaton f G (x) Data: =, =, ρ =., ρ =.999, eps = e 6 Intaze poynoma bass φ (x) w.r.t p(x); Construct correspondng Meher kerne accordng to Equaton (); Sampe an expermenta desgn {X, Y} wth Y = y(x); 4 whe eps do 5 Dvde [ρ, ρ ] nto ntervas to get ρ () = mn{ρ, ρ }, ρ (),..., ρ () = max{ρ, ρ }; 6 Cacuate accordng to Equaton (4) w.r.t ρ () to get () ; 7 Seect the frst two mnma () to get ( ), ( ) ; 8 Assgn = ( ), = ( ) ; 9 Assgn ρ = ρ ( ), ρ = ρ ( ) ; end Seect ρ wth mnma, together wth Equaton () and Equaton (5), to obtan f G (x). 5. Numerca Investgaton In ths secton, we nvestgate the GPCB method for varous benchmark functons. Frsty, we nvestgate the same exampe n Fgure ; however, the nose term s consdered,.e., y = f (x) + ɛ = 5 + x + exp(x) + ɛ s the observaton, where x N (, ), ɛ N (,. ). Three methods,.e., GP wth the RBF kerne, PCE and GPCB, are mpemented. It s necessary to note that the Monte Caro (MC) sampng strategy s used n the norma GP approaches, and Gaussan quadrature ponts are ntroduced n the GPCB approach. The man reason s that the quadrature ponts are too sparse for the wdey-used kernes to capture the oca features. For exampe, n ths case, P = n PCE, then the maxmum quadrature pont s.76, whch s beyond σ x. In the frst set of experments, et P = n PCE, whch means sampe ponts are used n the experments; furthermore,, sampes are ntroduced as test dataset to output the ECDF (emprca cumuatve dstrbuton functon) and RMSE (root-mean-squared error). The GP agorthm s mpemented by the gpm toobox [8] wrtten n MATLAB wth four dfferent kernes,.e., near, quadratc, Gaussan and Matérn-/ kernes. The comparsons of the resuts are dspayed n Fgure 4.

13 Entropy 8,, 9 of (a) (b) (c) Fgure 4. Comparsons among the GP, poynoma chaos expanson (PCE) and GPCB surrogates for the D exampe. (a) Comparson of the KL dvergence, P = ; (b) comparson of the ECDF of predcton, P = ; (c) comparson of the RMSE wth dfferent degrees P n the D exampe. Fgure 4a ustrates the pont-wse KL dvergence between the true vaue of f (x) and the predctons on the nterva [ 4, 4] based on Equaton (6). It s cear that the dstrbuton of GPCB predcton s statstcay cosest to the true response, athough the GP methods wth quadratc, Gaussan and Matérn-/ kernes outperform the GPCB at some ponts. Fgure 4b compares the ECDF of y based on the test dataset. It shows that both the PCE and GPCB have a smar ECDF wth the true vaue. Upon coser nspecton, whch s shown n the magnfed subregon, t s obvous that the ECDF of the GPCB s amost exacty the same as the rea ECDF, whch shows that GPCB has actuay captured the feature of f (x) wth hgh precson. At the same tme, the RMSEs of the PCE, near kerne, quadratc kerne, Gaussan kerne, Matérn-/ kerne and GPCB are.57, 7.56,.68, 5.644, ,.48, respectvey. We have mpemented another experment, whch uses the degree P (.e., the number of expermenta desgn) as the second set of experments, whch are ustrated n Fgure 4c. Fgure 4c shows that GPCB generay outperforms the ordnary GP wth the RBF kerne, whch ndcates that GPCB performs better wth a few (or sparse) tranng ponts. It s notabe that the PCE and GPCB perform wth amost the same precson when the degree s greater than 6. It echoes the dea that PCE and GPCB are statstca equvaent, as we present n Secton 4.. Smar experments are conducted wth a two-dmensona functon, whch s expressed as f (x) = exp(x )/ exp(x ). Let x, x N (, ), y = f (x) + ɛ be the rea mode where ɛ s an ndependent nose term wth a norma dstrbuton N(,. I ). Unke the frst test functon, ths test exampe s a mt state functon. Let the maxmum degree for each dmenson p t be seven for the PCE method, whch makes 64 tranng ponts n tota. Another dataset of, ndependent sampes s ntroduced as the test set to cacuate the ECDF and RMSE, as we. Smary, we have the pont-wse KL dvergence n the regon [, ;, ] as shown n Fgure 5a. It s cear that the GPCB s gobay coser to the true dstrbuton than other methods. The GP wth quadratc, Gaussan and Matérn-/ kernes can approxmate the center part we, whe the PCE does not seem to perform as we. Fgure 5b shows that the sx methods except the near kerne are abe to reconstruct the dstrbuton of the predcton, and upon coser observaton, we fnd that the ECDF of the GPCB and Matérn-/ kerne are the best approxmatons among the sx methods. We aso consder another set of experments focusng on the number of expermenta desgns, whch equas (p t + ) for the D functon. The RMSEs of the three methods wth respect to dfferent p t are dspayed n Fgure 5c. It aso shows that the PCE and GPCB generay outperform the norma GP approaches, and the GPCB has the best performance.

14 Entropy 8,, 9 4 of (a) (b) (c) Fgure 5. Comparsons among the GP, PCE and GPCB surrogates for the D exampe. (a) Comparson of the KL dvergence wth p t = 7; (b) comparson of the ECDF of predcton, p t = 7 ; (c) comparson of the RMSE wth dfferent degrees p t n the D exampe. To summarze, the GPCB generates a surrogate of nfnte seres, whe the PCE can ony generate a surrogate wth up to P + poynomas, and they tend to behave wth smar precson when P s arge enough. A set of sparse quadrature ponts samped n the PCE, whch are derved from the Gaussan quadrature rue, s a good desgn for the GPCB. The GPCB wth those tranng ponts generay performs better than the norma GP methods and PCE. However, the sze of such a tranng set grows dramatcay wth the dmenson (N = (p t + ) d n tota), so t s not practca n rea-fe appcatons. We am to present a strategy of sampng from those quadrature ponts, namey canddate ponts n the next secton, and anayze the performance of our agorthm on the seected ponts. 5.. A Random Constructve Desgn n Hgh Dmensona Probems As the dmenson of a system grows, so do the number of desgn ponts of PCE due to a tensor product of quadrature ponts n each dmenson. It s possbe that PCE coud dea wth thousands of ponts of tranng data wth acceptabe computatona tme; however, t becomes expensve for

15 Entropy 8,, 9 5 of GP approaches, ncudng our GPCB approach. Monte Caro sampng technques can substtute quadrature desgn; however, these are not aways stabe. Other sampng strateges ke Haton sampng and Latn hypercube sampng [9] are wdey used. In ths work, we want to utze the hgh accuracy of quadrature ponts and aso want to reduce the massve number of ponts. Let x R d, and p t s the maxmum degree n each dmenson, so p t + quadrature ponts are needed n each dmenson, whch makes the tota number of tensor products of quadrature ponts be #{X c } = (p t + ) d. We seek to fnd a subset of the canddate desgn X c. Furthermore, we wsh to obtan a subset havng a good coverage rate n the space. Therefore, we proposed the random defnte desgn n our paper. Note that the LHS desgn can be extended to a arger nterva (, (p t + ) D ) and can produce ponts at mdponts (endponts), so we use the LHS desgn to sampe N ndces from the nterva. More specfcay, we presume that the ponts n X c are equay mportant, so we arrange those ponts wth a certan order to get ther ndces. Then, we sampe from the ndces wth the LHS desgn, and each ndex s reated to a certan quadrature pont. It can be easy mpemented by the MATLAB but-n functon hsdesgn. The correspondng N ponts are what we need. Take a three-dmensona nput space as an exampe, where x N (, ), =,,. Set p t = 6, then #{X c } = 4. The canddate desgn and ts subset of 5 ponts X are ustrated n Fgure 6. We can see from the fgure that our sampng s sparse n the whoe set of canddate ponts, and t behaves unformy n dmenson one as ustrated n Fgure 6b, wth smar concusons n the other two dmensons. When projectng our sampng from dmenson three to get Fgure 6c, we can see that the seected ponts amost cover every pont of X c, whch means t has a features n dmensons one and two,.e., quadrature pont vaues of the two dmensons. It shows that such a method can generate a sparse subset meanwhe guaranteeng the coverage rate n the whoe canddate desgn. We name t the random constructve desgn (a) (b) (c) Fgure 6. Left: X c and X n D vew; the bue dots represent the quadrature ponts, whe the red ponts represent our sampngs; mdde: ths shows the sparsty of our sampng n X c ; rght: ths shows that our sampng actuay covered amost every feature of X c. (a) X c and X n D vew; (b) one sce of X c ; (c) projecton of X on X c n dmensons one and two. Now, we want to fnd out whether these sampes retan ther capabty of accuracy. Frsty, we use the PCE method to test those sampes. We w ook nto the benchmark Ishgam functon [4]: f (x) = sn(x ) + 7 sn (x ) +.x 4 sn(x ), where four dfferent sampng strateges are compared here. Set p t = 5, and the canddate desgn X c has a sze of 496. The error term ɛ s emnated n ths smuaton for the accuracy test. The RMSE are computed on, ndependenty-samped data, and the resuts are presented beow n Fgure 7. It can be seen that our sampe aways performs better than other sampes. When the number of sampes surpasses 9, the RMSE becomes.65 5, whch equas the RMSE wth the whoe canddate ponts. Therefore, we ony seect % of X c and get the same precson. Furthermore, f we set our precson to be, ony 4 ponts are needed.

16 Entropy 8,, 9 6 of Ths shows that the quadrature ponts have hgh precson n numerca cacuaton. In other words, the ponts n the canddate set are good ponts Fgure 7. Comparson of the RMSE between four sampng strateges wth the PCE method. Then, the random constructve desgn s used wth the PCE, GP and GPCB methods for the Ishgam functon, wth the nose term added n the observatons. We take the RMSE as a crteron to compare ther performance, and the resuts are ustrated n Fgure 8. Fgure 8a shows that the GPCB s aways better than the PCE method, and they tend to behave the same. However, as the number of sampng ponts grows, the GP wth the quadratc kerne, Gaussan kerne and Matérn-/ kerne generay outperform the other methods. We can see that the Ishgam functon s a bounded functon; therefore, t s key to f the whoe observaton space as the number of sampes ncreases, hence mprovng the accuracy of the GP method. We pot the ECDF wth respect to the three methods when N = n Fgure 8b. It s cear that the GP wth the quadratc kerne, Gaussan kerne and Matérn-/ kerne can amost recover the true dstrbuton of the response, whch s beyond the capabty of the PCE and GPCB (a) (b) Fgure 8. Comparsons among the GP, PCE and GPCB surrogates for the Ishgam functon. (a) Comparson of the RMSE; (b) comparson of the ECDF; N =. A sx-dmensona probem s beng tested wth the G-functon [4], whch s not ke the Ishgam functon and s unbounded n the doman [, ] 6 : f (x) = 6 4x + a, where a + a =, =,..., 6 (5) =

17 Entropy 8,, 9 7 of The experment s performed wth the same approaches, and the resuts are shown beow n Fgure 9. Fgure 9a shows that the GPCB outperforms the PCE and GP wth the Gaussan Kerne, and t has smar precson wth the quadratc and Matérn-/ kernes. We notce that the GPCB s more stabe than the PCE method, whch behaves bady especay when N = 5,. Fgure 9b shows that none of these three methods can reconstruct the probabty of y very we; however, we can note that the GPCB s st comparatvey the cosest (a) (b) Fgure 9. Comparsons among the GP, PCE and GPCB surrogates for the G-functon. (a) Comparson of the RMSE; (b) comparson of the ECDF; N =. Fnay, we are gong to present a more compcated mode wth 5 dmensona functons wth the foowng form: f (x) = a T x + at sn(x) + at cos(x) + xt Mx. (6) The dstrbuton of x s the product of 5 ndependent dstrbutons,.e., x N (, ), =,..., 5. Ths functon s ntroduced by the work of O Hagan, where a, a, a, M are defned n [4]. We can see that ths functon s domnated by the near and quadratc term, so t may be we approxmated by the ow-order PCE mode. Let p t = n the PCE mode; we can see from Fgure a that the GP wth the quadratc kerne performs best among the sx methods, whe the PCE performs better than the GPCB and other GP methods. On the other hand, the GPCB s aways generay better than the GP method except wth the quadratc kerne for ths functon. When N =, the PCE can generate y, whch foows the rea dstrbuton accordng to the ECDF n Fgure b (a) (b) Fgure. Comparsons among the GP, PCE and GPCB surrogates for Equaton (6). (a) Comparson of the RMSE; (b) comparson of the ECDF; N =.

18 Entropy 8,, 9 8 of 6. Concusons Ths paper has examned two dfferent surrogates of computatona modes,.e., poynoma chaos expanson and Gaussan process regresson. Frst, we present a bref revew of these two approaches. Next, we dscuss the reatonshp between PCE and GP and fnd that PCE and GP surrogates are embedded n two somorphc RKHS. Mercer s theorem s ntroduced to generate a kerne based on a PCE bass, by whch a new approach s proposed, whch we name GPCB. An exampe shows that wth the same expermenta desgn, GPCB tends to retan usefu nformaton n a sutabe subspace of the RKHS by changng the hyper-parameters, whereas PCE smpy sets the nformaton of the resdua to zero. We further nvestgate the approxmaton performance on two test functons n D and D, respectvey, and ther approxmaton propertes are ustrated. In order to dea wth the hgh dmensona scenaro, a random constructve desgn from the quadrature ponts s used to generate an expermenta desgn. The resuts gve us severa drectons for choosng modes: bascay, the GPCB outperforms the PCE, but when the orgna mode can be we approxmated by ow-order PCE (Fgure ), t seems cumbersome to ntroduce the GPCB and GP; when the response functon s bounded (Fgure 8), f we have enough tranng resources, the GP can be a better choce; when the objectve functon s unbounded (Fgures 4 and 9) or cannot be approxmated by fnte poynomas (Fgure 5), we shoud probaby choose the GPCB. Future work can extend the famy of the Mercer kerne or equvaent kerne (other than the Meher kerne presented n ths paper) beyond a cassca approxmaton method. We can aso anayze the expermenta desgn for GP regresson n many ways. Athough we can see that our sampng method behaves far enough n the experments, there s aso the opportunty to dscover further sutabe expermenta desgn schemes to ft dfferent computatona purposes, whch woud be of great nterest. The stabty of our method w be nvestgated n future work,.e., how many ponts are needed to tran a good surrogate and whether our method aways produces a sutabe desgn. Furthermore, we can estabsh coser connectons between numerca anayss and statstcs va such combnatons. Acknowedgments: Ths work s supported by the program for New Century Exceent Taents n Unversty, State Educaton Mnstry n Chna (No. NCET -89) and the Natona Scence Foundaton of Chna (No. 7745, No ). Author Contrbutons: Lang Yan proposed the orgna dea, mpemented the experments n the work and wrote the paper. Xaojun Duan contrbuted to the theoretca anayss and smuaton desgns. Bowen Lu partay undertook the wrtng and smuaton work. A authors read and approved the manuscrpt. Confcts of Interest: The authors decare no confct of nterest. Appendx A. Proofs of Proposton, Proposton, Proposton and Theorem Here, we use the same notatons as n Secton. Appendx A.. Proof of Proposton Proof. Defne the nner product n the above subspace H P as foows: < f (x), g(x) > HP = f g /λ. (A) Frsty, t s obvous that H P s a Hbert space n H. Secondy, for any x Ω, k(x, ) beongs to H P because: < k(x, ), k(x, ) > HP = It aso has the reproducng property for: < k(x, ), φ ( ) > λ = λ φ (x) <. (A)

19 Entropy 8,, 9 9 of < f ( ), k(x, ) > HP = f λ φ (x)/λ = f (x) f or x Ω, (A) We have the concuson that H P s an RKHS derved from the Mercer kerne. Appendx A.. Proof of Proposton Proof. In fact, H G s a space of a fnte near combnatons of functons k(x, ) : Ω R, so we can denote H G := span{k(x, ) x Ω}, and the eements n H G have the genera form of f (x) = = N f k(x, X ). Therefore, dfferent N and a expermenta desgns X are aowed, whch enabes that f (x) = = N f k(x, X ), g(x) = = N f k(x, X ) H G. The nearty of H G s gven by the foowng expanaton. Let t, t R be scaars, then we can rewrte t f () (x) + t f () (x) as a functon f (x) such that: f (x) = N t f () k(x, X () = ) + N j= t f () j k(x, X () j ) N f k(x, X ), = (A4) where N = N + N, { f, =,..., N} = {t f (), =,..., N } {t f () j, j =,..., N }, X = X () X (). Addtonay, we have: N N = m= f f m k(x, X m ) = N = + N = N j= < +, t f () f () k(x (), X () ) N j = t f () j f () j k(x () j, X () j ) + N = N j= t t f () f () j k(x (), X () j ) (A5) whch means that t f () (x) + t f () (x) aso beongs to H G. Then, we woud ke to show that H G s an nner product space. Let the kerne functon be postve sem-defnte, and we defne the nner product of H G as foows: < f (x), g(x) > H G = N N = j= f g j k(x, X j ). (A6) <, > H G s a we-defned nner product by checkng the foowng condtons:. Symmetry: < f, g > H G =,j f g j k(x, X j ) = j, g j f k(x j, X ) =< g, f > H G ;. B-nearty: < t f () + t f (), g > H G = =a N N = j= N = N j= f g j k(x, X j ) N = j= f () g j k(x (), X j ) + b N =a < f (), g > H G +b < f (), g > H G ; f () g j k(x (), X j ). Postve-defnteness: It s obvous that < f, f > H G = f T K f wth the equaty ff f =.

20 Entropy 8,, 9 of Appendx A.. Proofs of Proposton Proof. Frsty, we can prove that the reproducng formua hods for the space H G. For any X, k(x, x) s a functon of x and beongs to H G. Furthermore, we have: < f ( ), k(x, ) > H G = N f k(x, X ) = f (x). = (A7) The above reproducng property s vad for any f H G ; thus, t s st vad for the cosure H G n the sense of generazng the above equaton as < f ( ), k(x, ) > HG = f (x). Then, we need to prove that H G s unque. Suppose that we have another Hbert space H G, whch s possby an RKHS of the kerne k(, ), then, for a specfc X, we can get: < k(x, X), k(x, X) > H G = k(x, x ) =< k(x, X), k(x, X) > HG. (A8) Ths proves that the two nner products are the same on H G ; then, H G must contan H G because t s the cosure of H G. H G must be equvaent to H G, otherwse we can fnd a nonzero eement f H G H G such that t s orthogona to H G. However, we can aways get that f =< f, k(, X) > HG for a partcuar X, whch s a contradcton. Appendx A.4. Proofs of Theorem Proof. Defne a weghted space such that: } λ {h = < h, h > = λ λ h <. (A9) It s cear that {c (X)} /λ for c (X) defned n Equaton (). /λ s the competon of the span of a {c (X)}, then H G s sometrcay somorphc to /λ, because frsty, c (X) s dentcay determned by k(, ) and X, secondy, accordng to Equaton (), < f X (x), g X (x) > HG = c (X)c (X )/λ. On the other hand, there exsts a near map such that: T : /λ H P, T(c) = c φ. (A) Ths s a surjectve map for every f H P, { f } /λ. Now, we need to prove that the projecton T s njectve. Assume there exst c and c such that T(c) = T(c ), then: = T(c) T(c ) H P =< T(c c ), T(c c ) > HP = (c c ) /λ <, (A) whch proves c = c. Meanwhe, < f (x), g(x) > HP = f g /λ, { f }, {g } /λ. (A) The nner products reman equa, so t s aso cear that H P s sometrcay somorphc to /λ. To summarze, we can say that H G and H P are somorphc by estabshng a Hbert space /λ to connect them; or t can be sad that PCE and GP wth the Mercer kerne generate surrogates n the same Hbert space.

21 Entropy 8,, 9 of References. Schwefe, H.P.P. Evouton and Optmum Seekng: The Sxth Generaton; John Wey & Sons, Inc.: Hoboken, NJ, USA, 99.. Santner, T.J.; Wams, B.J.; Notz, W.I. The Desgn and Anayss of Computer Experments; Sprnger Scence & Busness Meda: Bern, Germany,.. Hurtado, J.; Barbat, A.H. Monte Caro technques n computatona stochastc mechancs. Arch. Comput. Methods Eng. 998, 5, Cont, S.; O Hagan, A. Bayesan emuaton of compex mut-output and dynamc computer modes. J. Stat. Pan. Inference, 4, Hgdon, D.; Gattker, J.; Wams, B.; Rghtey, M. Computer mode cabraton usng hgh-dmensona output. J. Am. Stat. Assoc. 8,, Bac, O. Verfcaton, vadaton, and certfcaton of modeng and smuaton appcatons. In Proceedngs of the 5th Conference on Wnter Smuaton: Drvng Innovaton, New Oreans, LA, USA, 7 December. 7. Rubno, G.; Tuffn, B. Rare Event Smuaton Usng Monte Caro Methods; John Wey & Sons, Inc.: Hoboken, NJ, USA, Sundar, V.; Sheds, M.D. Surrogate-enhanced stochastc search agorthms to dentfy mpcty defned functons for reabty anayss. Struct. Saf. 6, 6,. 9. Shan, S.; Wang, G.G. Survey of modeng and optmzaton strateges to sove hgh-dmensona desgn probems wth computatonay-expensve back-box functons. Struct. Mutdscp. Optm., 4, Fadae, T.D.; Nenarokomov, A.V.; Emery, A.F. Uncertantes n parameter estmaton: The nverse probem. Int. J. Heat Mass Transf. 995, 8, Lang, B.; Mahadevan, S. Error and uncertanty quantfcaton and senstvty anayss n mechancs computatona modes. Int. J. Uncertan. Quantf.,, De Curs, E.S.; Sampao, R. Uncertanty Quantfcaton and Stochastc Modeng wth Matab; Esever: Amsterdam, The Netherands, 5.. Fredman, J.H. Mutvarate adaptve regresson spnes. An. Stat. 99, 9, Drucker, H.; Burges, C.J.; Kaufman, L.; Smoa, A.J.; Vapnk, V. Support vector regresson machnes. Adv. Neura Inf. Process. Syst. 997, 9, Oparaj, U.; Sheu, R.J.; Bankhead, M.; Austn, J.; Pate, E. Robust artfca neura network for reabty and senstvty anayses of compex non-near systems. Neura Netw. 7, 96, Sun, Z.; Wang, J.; L, R.; Tong, C. LIF: A new krgng based earnng functon and ts appcaton to structura reabty anayss. Reab. Eng. Syst. Saf. 7, 57, Ghanem, R.; Spanos, P.D. Stochastc Fnte Eements: A Spectra Approach; Sprnger: Bern, Germany, Xu, D.; Karnadaks, G.E. The Wener Askey poynoma chaos for stochastc dfferenta equatons. SIAM J. Sc. Comput., 4, Xu, D.; Hesthaven, J.S. Hgh-order coocaton methods for dfferenta equatons wth random nputs. SIAM J. Sc. Comput. 5, 7, Xu, D. Numerca Methods for Stochastc Computatons: A Spectra Method Approach; Prnceton Unversty Press: Prnceton, NJ, USA,.. Le Maître, O.P.; Reagan, M.T.; Najm, H.N.; Ghanem, R.G.; Kno, O.M. A stochastc projecton method for fud fow: II. Random process. J. Comput. Phys., 8, Ghoce, D.M.; Ghanem, R.G. Stochastc fnte-eement anayss of sesmc so-structure nteracton. J. Eng. Mech., 8, Kennedy, M.C.; O Hagan, A. Bayesan cabraton of computer modes. J. R. Stat. Soc. Ser. B (Stat. Methodo.), 6, Cresse, N. Statstcs for spata data: Wey seres n probabty and statstcs. Wey-Intersc. N. Y. 99, 5, MacKay, D.J. Introducton to Gaussan processes. NATO ASI Ser. F Comput. Syst. Sc. 998, 68, Rasmussen, C.E. Gaussan processes n machne earnng. In Advanced Lectures on Machne Learnng; Sprnger: Bern, Germany, 4; pp Constantne, P.G.; Wang, Q. Resdua mnmzng mode nterpoaton for parameterzed nonnear dynamca systems. SIAM J. Sc. Comput., 4, A8 A44.

22 Entropy 8,, 9 of 8. Quñonero-Candea, J.; Rasmussen, C.E. A unfyng vew of sparse approxmate Gaussan process regresson. J. Mach. Learn. Res. 5, 6, Schob, R.; Sudret, B.; Wart, J. Poynoma-chaos-based krgng. Int. J. Uncertan. Quantf. 5, 5, Schöb, R.; Sudret, B.; Mare, S. Rare Event Estmaton Usng Poynoma-Chaos krgng. ASCE-ASME J. Rsk Uncertan. Eng. Syst. Part A Cv. Eng. 7,, D46.. Schöb, R.; Sudret, B. Combnng poynoma chaos expansons and krgng for sovng structura reabty probems. In Proceedngs of the 7th Internatona Conference on Computatona Stochastc Mechancs (CSM7), Santorn, Greece, 5 8 June 4.. Schöb, R.; Sudret, B. PC-krgng: A new meta-modeng method and ts appcatons to quante estmaton. In Proceedngs of the 7th IFIP Workng Group 7.5 Conference on Reabty and Optmzaton of Structura Systems, Huangshan, Chna, 7 Juy 4.. Aronszajn, N. Theory of reproducng kernes. Trans. Am. Math. Soc. 95, 68, Kuback, S. Informaton Theory and Statstcs; Courer Corporaton: North Chemsford, MA, USA, Echard, B.; Gayton, N.; Lemare, M. AK-MCS: An actve earnng reabty method combnng krgng and Monte Caro smuaton. Struct. Saf.,, Dubourg, V. Adaptve Surrogate Modes for Reabty Anayss and Reabty-Based Desgn Optmzaton. Ph.D. Thess, Unversté Base Pasca-Cermont-Ferrand II, Aubère, France,. 7. Kbbe, W. An extenson of a theorem of Meher s on Hermte poynomas. Math. Proc. Camb. Phos. Soc. 945, 4, Rasmussen, C.E.; Ncksch, H. Gaussan processes for machne earnng (GPML) toobox. J. Mach. Learn. Res.,, Nederreter, H. QuasMonte Caro Methods; John Wey & Sons, Inc.: Hoboken, NJ, USA,. 4. Ishgam, T.; Homma, T. An mportance quantfcaton technque n uncertanty anayss for computer modes. In Proceedngs of the Frst Internatona Symposum on Uncertanty Modeng and Anayss, Coege Park, MD, USA, 5 December 99; pp Marre, A.; Iooss, B.; Van Dorpe, F.; Vokova, E. An effcent methodoogy for modeng compex computer codes wth Gaussan processes. Comput. Stat. Data Ana. 8, 5, Oakey, J.E.; O Hagan, A. Probabstc senstvty anayss of compex modes: A Bayesan approach. J. R. Stat. Soc. Ser. B (Stat. Methodo.) 4, 66, c 8 by the authors. Lcensee MDPI, Base, Swtzerand. Ths artce s an open access artce dstrbuted under the terms and condtons of the Creatve Commons Attrbuton (CC BY) cense (