Gaussian Process Vine Copulas for Multivariate Dependence

Transcription

1 Gaussian Process Vine Copulas for Multivariate Dependence José Miguel Hernández Lobato 1,2, David López Paz 3,2 and Zoubin Ghahramani 1 June 27, University of Cambridge 2 Equal Contributor 3 Ma-Planck-Institute for Intelligent Systems 1 / 23

2 Copulas and Multivariate Probabilistic Models Copulas link univariate marginal distributions into joint multivariate probabilistic models. Marginal Densities Copula Joint Density y Copulas specify dependencies between the random variables. 2 / 23

3 Separating Marginals and Dependencies Using Copulas p( 1,..., d ) = c(u 1,..., u d ) }{{} unique copula d p i ( i ), where u i := P i ( i ) and P i denotes the cumulative distribution function (cdf) for the variable i. We can model each of the d marginal distributions and the copula function independently. There eists a broad catalog of two-dimensional copula models i=1... However, for the high-dimensional regime, the number of copula models available and their epressiveness is more limited. 3 / 23

4 More than Two Dimensions: Vine Factorizations G G 1 /T 2 /T 2 1 e 4 = 1, , 3 2, 3 e1 = 1, 3 e 2 = 2, 3 e5 = 1, 4 3 G 3 /T 3 e 6 = 2, 4 1, e 3 = 3, 4 3, 4 1, 2 3 1, 4 3 c 1234 = c 13 c 12 3 }{{}}{{}}{{}}{{}}{{} c 23 c 34 e 1 e 2 e 3 } {{ } T 1 c 14 3 c e 4 e 5 } {{ } T 2 }{{} e 6 }{{} T 3 Parameters: bivariate copula families, tree construction method. 4 / 23

5 Eample of Vine Factorizations: Three Variables We can factorize c(u 1, u 2, u 3 ) using the product rule of probability as c(u 1, u 2, u 3 ) = p(u 3 u 1, u 2 )p(u 2 u 1 ) and we can epress each factor in terms of bivariate copula functions p(u 3 u 1, u 2 ) = p(u 3, u 1 u 2 )[p(u 1 u 2 )] 1 = c 31 2 (u 3 {u 2 }, u 1 {u 2 })p(u 3 u 2 ), p(u 3 u 2 ) = p(u 3, u 2 ) = c 32 (u 3, u 2 ), p(u 2 u 1 ) = p(u 2, u 1 ) = c 21 (u 2, u 1 ), where u i {u j } is the conditional cdf of u i given u j. Finally, we obtain c(u 1, u 2, u 3 ) = c 31 2 (u 3 {u 2 }, u 1 {u 2 })c 32 (u 3, u 2 )c 21 (u 2, u 1 ). 5 / 23

6 Eample of Vine Factorizations: Four Variables Using vines, we can factorize p( 1, 2, 3, 4 ) as p( 1, 2, 3, 4 ) = p 1 ( 1 )p 2 ( 2 )p 3 ( 3 )p 4 ( 4 ) c 13 (u 1, u 3 )c 23 (u 2, u 3 )c 34 (u 3, u 4 ) c 12 3 (u 1 {u 3 }, u 2 {u 3 })c 14 3 (u 1 {u 3 }, u 4 {u 3 }) c (u 2 {u 1, u 3 }, u 4 {u 1, u 3 }), We need to use conditional copulas and conditional cdfs! The conditional cdfs can be recursively computed using u i v = C ij v j (u i v j, u j v j ) u j v j. The Vine Simplifying Assumption: conditional copulas are independent from their conditioning variables. Can we do better? 6 / 23

7 A Semi-parametric Model for Conditional Copulas Assumption: the conditional copula is parametric and the conditioning variables have a direct effect on the copula parameter, specified in terms of Kendall s tau 4 τ [ 1, 1]: Eample: The Gaussian copula. c(u 1, u 2 τ) = φ 2[Φ 1 (u 1 ), Φ 1 (u 2 ) ρ = sin(τπ0.5)] φ(φ 1 (u 1 ))φ(φ 1 (u 2 )) φ and Φ are the standard Gaussian pdf and cdf. φ 2 is a 2D Gaussian pdf with zero mean, correlation ρ and variances 1. Let z be a vector of conditioning variables. Then we assume the conditional dependence τ = σ[f(z)], where f is a real function and σ() = 2Φ() 1 is a sigmoid function., 4 Many one-parameter copulas can be specified in terms of Kendall s τ rank correlation coefficient. 7 / 23

8 Estimation of Conditional Copulas with GP+EP Given the data D u,v = {(u i, v i )} n i=1 and D z = {z i } n i=1, we place a Gaussian process prior on f and do Bayesian inference. The posterior for f = (f 1,..., f n ) T, where f i = f(z i ), is prior likelihood p(f D u,v, D z ) = p(f D z )p(d u,v f), }{{} evidence posterior where p(f D z ) = N (f m, K) }{{} GP prior and p(d u,v f) = n c(u i, v i τ = σ(f i )). i=1 } {{ } copula likelihood model 8 / 23

9 Ingredient 1: The Gaussian Process Prior n p(f D u,v, D z ) N (f m, K) c(u i, v i τ = σ(f i )). The prior distribution of any set of values of f is jointly Gaussian. A covariance function specifies the value of K R n n : K ij = k(z i, z j ) = s ep( (z i z j ) T diag(λ)(z i z j )) + s 0. (1) This imposes smoothness properties on the possible prior values of f. i=1 In the GP prior m is given by a constant mean function. 9 / 23

10 Ingredient 2: The Copula Likelihood n p(f D u,v, D z ) N (f 0, K) c(u i, v i τ = σ(f i )). If several copula families are available as candidates, build one posterior for each one and perform Bayesian model selection. i=1 10 / 23

11 Ingredient 3: Bayesian Inference To make predictions at z n+1 for u n+1 and v n+1 average over all possible functions f, weighting each one by its posterior probability: p(u n+1, v n+1 z n+1, D UV, D z ) = c(u n+1, v n+1 τ = σ[f n+1 ]) p(f n+1 f, z n+1, D z ) p(f D UV, D z ) dfdf n+1. }{{}}{{}}{{} new likelihood conditional prior old posterior }{{} new posterior for f n+1 Problem 1: The likelihood is not Gaussian. We cannot perform eact inference (solve the integral analytically). Problem 2: Working with GPs is epensive. It often requires O(n 3 ) operations (compute Cholesky decompositions of n n matrices). 11 / 23

12 Ingredient 4: Epectation Propagation Solution for 1: Approimate likelihood factors with Gaussians. EP tunes ˆm i and ˆv i by minimizing KL[q i (f i )Q(f)[ˆq i (f i )] 1 Q(f)]. After convergence, the new (approimate) posterior is Gaussian. An approimation of the evidence and its gradient is returned. 12 / 23

13 Ingredient 5: FITC Approimation for GPs Solution for 2: Use a covariance matri K with a simple form: K K = Q + diag(k Q), Q = K nn0 K 1 n 0 n 0 K n0 n, where a subset of n 0 n pseudo-inputs is used. Q = K nn0 K 1 n0n0 K n0n The n 0 pseudo-inputs are tuned by maimizing the evidence returned by EP. New cost of O(nn 2 0 ). 13 / 23

14 All Together: The GPCC Method Input: Data {(u i, v i, z i )} n i=1. Parametric copula density function c(u, v τ). Number of pseudo-inputs n 0. Algorithm: Until Convergence: Use EP to get approimate Gaussian posterior, evidence and evidence gradient. Follow evidence gradient to tune pseudo-inputs and the parameters of the covariance function. Prediction of (u, v ) at z : sample f = f(z ) from the EP-posterior and average over copulas with τ = σ(f ). Output: Conditional copula model. 14 / 23

15 Eperimental Results: Synthetic Test Z is uniform in [ 6, 6] (U, V ) are Gaussian with correlation 3/4 sin(z) We sample the data {( ˆP U (u i ), ˆP V (v i ), ˆP Z (z i ))} 50 i=1. τ U,V Z GPVINE MLLVINE TRUE P Z (Z) MLLVINE is a nearest-neighbor algorithm proposed by Acar (2011). 15 / 23

16 Eperimental Results: Real-World Data Real-world data: UCI datasets, meteorological data, mineral concentrations and financial data Average test log likelihood when limited to two trees in the vine: Results are averages on 50 random train/test splits. SVINE are Vines that use the simplifying assumption. 16 / 23

17 Eperimental Results: Using More Trees GPVINE SVINE 17 / 23

18 Eperimental Results: Weather in Barcelona τ(atmospheric pressure, cloud percentage {latitude, longitude}). 18 / 23

19 GP Conditional Copulas and Financial Time Series We learn time-changing dependencies in financial returns. Three parametric copulas: Gaussian Copula Student's t Copula Symmetrized Joe Clayton Copula The marginals are described using AR-GARCH processes: t = φ 0 + φ 1 t 1 + σ t ɛ t, σ t = ω + ασ t 1 ( ɛ t 1 γɛ t 1 ) + βσ t / 23

20 Eperimental Results: Conditioning on time: NASDAQ HD AXP CNW ED HPQ BARC RBS RBS Method HON BA CSX EIX IBM HSBC BARC HSBC GPCC-G GPCC-T GPCC-SJC HMM TVC DSJCC CONST-G CONST-T CONST-SJC Data is preprocessed using AR(1)-GARCH(1,1) models. GPCC-G: GPs on τ(t) for Gaussian Copula. GPCC-T: GPs on ν(t) and τ(t) for tcopula. GPCC-SJC: GPs on τ L (t) and τ U (t) for Sym. Joe Clayton. HMM: High/Low Correlated tcopula (Jondeau 2006). TVC: Dynamic tcopula (Tse 2002). DSJCC: Dynamic SJC Copula (Patton 2006). 20 / 23

21 Eperimental Results: Conditioning on time: FOREX EUR CHF GPCC-T, EUR CHF GPCC-T, EUR CHF GPCC-SJC, Mean GPCC T Mean GPCC T Mean GPCC-SJC Mean GPCC-SJC Oct 06 Mar 07 Aug 07 Jan 08 Jun 08 Nov 08 Apr 09 Oct 09 Mar 10 Aug 10 Oct 06 Mar 07 Aug 07 Jan 08 Jun 08 Nov 08 Apr 09 Oct 09 Mar 10 Aug 10 Oct 06 Mar 07 Aug 07 Jan 08 Jun 08 Nov 08 Apr 09 Oct 09 Mar 10 Aug 10 Method AUD CAD JPY NOK SEK EUR GBP NZD GPCC-G GPCC-T GPCC-SJC HMM TVC DSJCC CONST-G CONST-T CONST-SJC Data against USD from 02/01/1990 to 15/01/2013 (6011 points). Decorrelations at Fall 2008 (onset of global recession) and Summer 2010 (worsening European sovereign debt crisis) 21 / 23

22 Summary and Conclusions Vines are fleible dependence models. They factorize a copula density into a product of conditional bivariate copulas. In practical implementations of vines, some of the conditional dependencies in the bivariate copulas are usually ignored. To avoid this, we have proposed a method for the estimation of fully conditional vines using Gaussian processes (GPVINE). GPVINE outperforms a baseline that ignores conditional dependencies (SVINE) and other alternatives (MLLVINE). The building block of GPVINE are GP Conditional Copulas (GPCC). GPCC obtain state-of-the-art performance in the task of predicting time-changing dependencies in financial data. 22 / 23

23 Thanks! Lopez-Paz D., Hernandez-Lobato J. M. and Ghahramani Z. Gaussian Process Vine Copulas for Multivariate Dependence International Conference on Machine Learning (ICML 2013). Acar, E. F., Craiu, R. V., and Yao, F. Dependence calibration in conditional copulas: A nonparametric approach. Biometrics, 67(2): , Bedford, T. and Cooke, R. M. Vines-a new graphical model for dependent random variables. The Annals of Statistics, 30(4): , 2002 Minka, T. P. Epectation Propagation for approimate Bayesian inference. Proceedings of the 17th Conference in Uncertainty in Artificial Intelligence, pp , Naish-Guzman, A. and Holden, S. B. The generalized FITC approimation. In Advances in Neural Information Processing Systems 20, Patton, A. J. Modelling asymmetric echange rate dependence. International Economic Review, 47(2): , / 23