AUTOREGRESSIVE IMPLICIT QUANTILE NETWORKS FOR TIME SERIES GENERATION MAXIMILIEN BAUDRY *

Transcription

1 AUTOREGRESSIVE IMPLICIT QUANTILE NETWORKS FOR TIME SERIES GENERATION MAXIMILIEN BAUDRY * *DAMI Chair & Université Lyon 1 - ISFA

2 AKNOWLEDGEMENTS 2 ACKNOWLEDGEMENTS TO Christian Y. ROBERT for the many discussions, the DataLab of BNP Paribas Cardif

3 WHAT WE DO 3 TIME SERIES GENERATION Let xxxxxxx (X t ) t2z be a stochastic process in Rd. Objective: Generating a sample x xxxxxxxxxxxxxx =(x 1,...,x n ) such that x has the same distribution as (X xxxxxxxxxxx t ) t2{1,...,n} without fixing any prior time dependency structure on (X xxxxxxx. t ) t2z R d

4 MOTIVATIONS 4 MOTIVATIONS Insurance industry: Real-World Economic Scenario Generator (Real-World ESG) which, associated with a Stochastic Discount Factor, allows the computation of the Best Estimate Liabilities, Traffic: generate many traffic scenarios in order to prevent traffic jams, Video: open the way to generate movies, or part of movies given a situation.

5 GENERATING IS NOT FORECASTING 5 GENERATING VS FORECASTING Forecasting: computing the conditional expectation of its future values: Equation E[X t+h ici X t,x t 1,...], 8h >0, Generating: estimating the multivariate conditional distributions of (X xxxxxxxxxxxxxx t+1,...,x t+h ) by considering the product of its univariate conditional distributions: hy Equation p(x t+h,...,x ici t+1 X t,x t 1,...)= p(x t+j X t+j 1,...), j=1

6 STRUCTURAL MODELS 6 STRUCTURAL MODELS Traditional generative methods for time series generation consist of fixing a prior structure on (X xxxxxxx t ) t2z such as an AR, MA, ARMA, VAR, ARCH/GARCH, DCC-GARCH model, and estimate their parameters. ARM A(p, q) : GARCH(p, q) : X t = px ' i X t i + i=1 2 t = 0 + qx i " t i=1 qx i Xt 2 i + i=1 i + " t Here, we do NOT make any assumption on the structure of xxxxxxx. (X t ) t2z However, we expect our model to be able to learn such structures. px i=1 i 2 t i

7 STATE-OF-THE-ART DEEP GENERATIVE MODELS 7 DEEP GENERATIVE MODELS Variational Auto-Encoders (VAE) (Kingma & Welling, 2013), Generative Adversarial Networks (GAN) (Goodfellow et al., 2014), PixelCNN (Oord et al., 2016). Baseline of our architecture Such models are able to learn & generate samples along any multivariate, high dimensional distribution, They are mainly used in image processing. Two of the above images are real, the other two are fake

8 8 STATE-OF-THE-ART DEEP GENERATIVE MODELS GENERATION STEP PixelCNN allows to generate conditional images (scenarios) Partial image generated images True image

9 QUANTILE REGRESSION 9 QUANTILE REGRESSION Quantile loss: where u is the difference between the observation and the estimated quantile. For a scalar distribution Zx Z with c.d.f. Fzz and a quantile x, we have: F 1 Z ( ) = arg min Implicit Quantile Networks (IQN) add an extra input xxxxxxxxxxxxx, which is reparameterized by the network to directly learn xxxxxxx. Huber quantile loss (1964): q F Z E z Z [ (z q)] U([0, 1]) F 1 Z ( ) 8

10 QUANTILE REGRESSION 10 REPARAMETERIZATION

11 CAUSAL CONVOLUTIONS 11 MASKED CONVOLUTIONS To ensure that the model is causal, we perform masked convolutions. Hence, our model belongs to the class of the Autoregressive Implicit Quantile Networks (AIQN). Masked convolutions consist of zeroing the weights which corresponds to future, yet unknown, values of Xt. (X t ) t2z

12 OUR ARCHITECTURE 12 OUR MODEL

13 IMPORTANCE SAMPLING 13 BETA PRIOR Generative models on images usually provide a probability on a range of integer color levels (from 0 to 255) for each pixel. Here, (X xxxxxxxxxxx, t ) t2z 2 R d which may raise issues when estimating the distribution tails. We overcome this by using a Beta distribution xxxxxxx B(, ) for u. This gives an importance sampling for some u which may be difficult to estimate.

14 EXPERIMENTS (1) 14 SIMULATIONS We compared our model to the state-of-the-art model, the Recurrent GAN (RGAN) (Esteban et al., 2017) on the following structural models: AR(1) (with different coefficients), ARMA(1,1), GARCH(1,1), 3-dimensional DCC-GARCH.

15 EXPERIMENTS (2) 15 REAL DATA We also compare the AIQN and RGAN models on two realworld datasets: Financial data, containing the end-of-day increments of prices of 9 equities of a given portfolio. Challenge: weak white noise, with also very weak time dependencies on squared data, GPS data, containing the GPS coordinates for each second on several car trajectories, for each driver (~1700 drivers). Challenge: very strong time dependencies.

16 AR MODEL 16 RESULTS Gaussian AR(1) with coefficient.9 Real AIQN RGAN

17 AR MODEL 17 RESULTS Gaussian AR(1) with coefficient -.5 Real AIQN RGAN

18 GARCH MODEL 18 RESULTS GARCH(1,1) Real AIQN RGAN

19 3-DIMENSIONAL DCC-GARCH MODEL (1) 19 RESULTS DCC-GARCH Real AIQN RGAN

20 3-DIMENSIONAL DCC-GARCH MODEL (2) 20 RESULTS DCC-GARCH AIQN RGAN

21 FINANCIAL DATA RESULTS 21 RESULTS Financial data 9 equities from the financial portfolio AIQN RGAN

22 GPS DATA RESULTS (1) 22 RESULTS GPS data joint plots of speed (x-axis) and acceleration (y-axis) Real AIQN RGAN

23 GPS DATA RESULTS (2) 23 RESULTS GPS data Speed Acceleration

24 MMD STATISTIC 24 MMD SCORES The table below shows the means and standard deviations (in brackets) of the MMD values computed along 50 independent simulations. Results with the symbol (*) are computed on realigned quantiles.

25 BETA PRIOR 25 INFLUENCE OF THE BETA PRIOR B(1/2, 1/2) U([0, 1])

26 DISCUSSION 26 FUTURE WORK Propose a meta-architecture suited for high dimensional data, Adapt the loss function in order to penalise the marginal distribution

27 QUESTIONS 27 THANK YOU

28 APPENDICES 28 APPENDICES

29 EVALUATION 29 EVALUATION MMD statistic (Gretton et al., 2008) is a non parametric tool which measures the distance of the distributions of two high dimensional samples. Let x xxxxxxxxxxxxxx =(x 1,...,x n ) and y xxxxxxxxxxxxxx =(y 1,...,y m ) be two samples drawn from distributions p and q respectively. We define the MMD by the following formula: MMD 2 (F; x, y) =sup f2f E X p [f(x)] E Y q [f(y )] We estimate it by the following unbiased estimator: MMD 2 (x, y) = 1 n(n 1) X k(x i,x j ) i6=j 2 mn X k(x i,y j )+ i,j 1 m(m 1) X i6=j k(y i,y j )