Regression with correlation for the Sales Data

Transcription

1 Regression with correlation for the Sales Data

2 Scatter with Loess Curve Time Series Plot Sales Sales Week Week

3 Sales Data What is our goal with the sales data? 1. We need to predict (i.e. forecast ) Why is this goal important? Helps to set expectations for upcoming year.

4 Sales Data What are the challenges with the sales data? 1. Maybe non-linear relationship with time. 2. Correlated in time Can we just ignore the correlation? Yes if we don t care about uncertainty the typical regression estimates are unbiased when data is correlated but all your confidence and prediction intervals will be wrong.

5 Sales Data Big Picture Overview of What We are Going to Learn: Y N X, 2 I ) Y N (X, ) This is called generalized least squares if you don t assume a distribution: min NX (y i x 0 i ) 2 =min(y X ) 0 (Y X ) ) min(y X ) 0 1 (Y X ) i=1

6 Important Definitions for Correlated Data Mean Function: µ t = E(x t ) Covariance Function: Correlation function: (t 1,t 2 )=Cov(x t1,x t2 ) (t 1,t 2 ) = Corr(x t1,x t2 ) Note: For a finite set of time periods, correlation functions give rise to correlation matrices. ij th element of R = (t i,t j )

7 Examples of Correlation Functions Global Temperatures ACF Lag NYSE ACF Lag Southern Oscillation Index ACF Lag

8 Regression for Time Series Big Idea: Treat time (t) as a covariate in a regression model. Linear Trend Series my.lm$residuals Global Temperature Deviations Global Temperature Deviations Time Natural Spline Fit ACF ACF Lag Series my.lm$residuals Just putting time as a covariate isn t enough because you have residual correlation Time Lag

9 Regression w/correlated Errors Correlated Regression Model: The model for observed data: Y = X + N (0, ) Y N N 1 or X, N P P 1 N N We need to also consider the distribution of future data points, how they will correlated with the observed data, and what their distribution is.

10 Gaussian Process Gaussian Process: Any finite collection of random variables follow a multivariate normal distribution (e.g. the joint distribution of observed and future data is MVN). More technical definition: For any set of t 1,...,t N 2 T vector Y =(Y (t 1 ),...,Y(t N )) 0 N (µ, Y )., the Defining a GP: To define Gaussian distribution, we only need to define what and are. µ X

11 Gaussian Process The Mean of a GP: µ = X where X contains any trends in time (e.g. linear) and any other covariates (splines etc.). The Covariance of a GP: 1. We need a correlation function, say ( ), so that observations Y (t i ) and Y (t j ) are strongly correlated when t i t j is small. 2. The correlation function, ( ), also needs to be positive definite so that is a valid covariance matrix.

12 Covariance Structures Lag-1 Autoregressive Process AR(1): Written as a conditional distribution: t t 1 N ( t 1, 2 ) or t = t 1 +! t,! t iid N (0, 2 ) Which equates to the joint distribution = B A N T 0 0 0, T T CC..... AA T } R

13 Covariance Structures Lag-1 Autoregressive Process AR(1): Written as a conditional distribution: t t 1 N ( t 1, 2 ) or t = t 1 +! t,! t iid N (0, 2 ) Notes about AR(1) processes: 1. Corr( t1, t2 )= t 1 t 2 2. is a correlation so 1 < < 1 (Lagrange) 3. Can be extended to AR(p) processes (but no closed form correlation function): t t 1,..., 1 N px `=1 ` t `, 2!

14 Autoregressive Processes AR(1) φ = x Time AR(1) φ = 0.9 x Time

15 Covariance Structures Moving Average Process MA(1): Written as a conditional distribution: t =! t 1 +! t,! t iid N (0, Which gives the correlation function 2 ) Corr( t1, t2 )= ( 1+ 2 if t 1 t 2 =1 0 otherwise = 2 R

16 Covariance Structures Moving Average Process MA(1): Written as a conditional distribution: t =! t 1 +! t,! t iid N (0, A few notes about MA(1) processes: 1. Correlation function is tapered in that correlation only between successive time points. 2. Can be extended to MA(q) processes t = qx `! t `=1 2 ) ` +! t,! t iid N (0, 2 )

17 Moving Average Processes MA(1) θ = x Time MA(1) θ = 0.9 x Time

18 Covariance Structures You can combine AR and MA processes to get an ARMA process: px qx ARMA(p, q) : t = l 1 t l + j! t j +! t l=1 j=1 Big Issue with AR, MA and ARMA processes: These models are only defined for equally spaced time (which, admittedly, is common but not the rule).

19 Covariance Structures Exponential Correlation Function: t1 t 2 Corr( t1, t2 )=exp Properties: 1. : range parameter. As increases, correlation (at a fixed distance) increases. 2. =1/ : decay parameter 3. E. Range : distance where correlation decays to Time does not have to be equally spaced 5. Exponential is a special case of a broader class of correlation functions called the Matern correlation.

20 Covariance Structures Exponential correlation without a nugget: t1 t 2 Corr( t1, t2 )=exp = 2 R Exponential correlation with a nugget: = 2 ((1!)R +!I) What s up with the nugget? The nugget (ω) gives added variance when t 1 t 2 =0 and allows sampling variability. Should I include a nugget? Yes if you expect to see sampling variability under a frozen time scenario. Also, nugget s help stabilize estimation so I almost always do.

21 Coding for Regression with Correlated Errors library(nlme) #Best library that I have found but still not great ## GLS with AR(1) Errors ar1.gls <- gls(y~x,correlation=corar1(form=~time),data=mydata,method="ml") summary(ar1.gls) intervals(ar1.gls) ## GLS with MA(1) Errors ma1.gls <- gls(y~x,correlation=corarma(form=~time,p=0,q=1),data=mydata,method="ml") ## GLS of Exponential Errors w/nugget exp.gls <- gls(y~x,correlation=corexp(form=~time,nugget=true),data=mydata,method="ml")

22 Prediction with Gaussian Processes Prediction for Gaussian Process Regression: Suppose we want to predict y at times T +1,...,T + K. By the Gaussian process assumption, 0 1 Y Y (T + 1) C Y Y? = Y (T + K) C A N X X? So, the predictions for Y? are the expected values of the conditional distribution for Y? Y, E(Y? Y) =X? ˆ + 2 R Y?,Y 2 1 R Y Y Xˆ, 2 RY R Y,Y? R Y?,Y R Y?

23 Gaussian Process Regression Prediction Intervals using Gaussian Process Regression: Due to the joint normal assumption, the 95% prediction interval is the and quantiles of the conditional distribution for Y? Y, Y? Y N µ Y? Y, Y? Y

24 Gaussian Process Regression Assumptions of a Gaussian process: 1. Data is multivariate normal. In other words, Y N (X, ) =N X, 2 R But we can t verify this assumption because we only have 1 observation. 2. Constant variance but, again, can t verify because we only have one observation 3. (maybe) Linearity depends on how you construct your X matrix 4. Independence No, we are assuming dependence.

25 Prediction Coding for Gaussian Processes library(nlme) #Doesn t do the prediction right so you have to do this by hand ## Set up the R matrix for observations AND predictions N <- nrow(mydata) #Number of observed time periods K <- 12 #number of time periods forward R <- diag(k+n) R <- phi^(abs(row(r)-col(r))) ## AR(1) correlation matrix R <- (theta/(1+theta^2))^(abs(row(r)-col(r))) R <- R*(abs(row(R)-col(R))<=1) ## MA(1) correlation matrix R <- exp(-as.matrix(dist(1:(k+n)))/phi) ##Exp correlation with no nugget R <- (1-nug)*R+nug*diag(N+K) ##Exp correlation with nugget pred.mn <- Xstar%*%bhat + R[N+(1:K), (1:N)]%*%solve(R[(1:N),(1:N)])%*% (Y-X%*%bhat) #conditional mean of MVN pred.var <- sig2*(r[n+(1:k),n+(1:k)]-r[n+(1:k), (1:N)]%*%solve(R[(1:N),(1:N)]) %*%R[(1:N),N+(1:K)]) # conditional variance of MVN

26 Gaussian Processes in Spatial Statistics Issue in Spatial Statistics: y(s) is correlated as a function of spatial location s (i.e. closer together in space means more correlated)

27 Gaussian Processes in Spatial Statistics Spatial Statistics: We observe y(s 1 ),...,y(s N ) and the covariates x(s 1 ),...,x(s N ) at N distinct spatial locations s 1,...,s N in some spatial region D. Spatial Statistics Model: Use the Gaussian process so 0 1 Y (s 1 ) B Y =. A N X, 2 ((1!)R +!I) Y (s N ) where R captures the spatial correlation w/nugget via ksi s j k R ij =exp Euclidean distance

28 Gaussian Processes in Spatial Statistics A few points about spatial statistics: Spatial statistics (almost) always uses a nugget variance. AR & MA correlation structures don t make sense in the spatial context because locations are not evenly spaced. A more common correlation function is the Matern correlation.

29 Gaussian Processes in Spatial Statistics Spatial Kriging (Prediction): Calculate conditional expectation at new locations given observations. Spatial Uncertainty Quantification: Variance at new location conditional on observations. (see previous slides on prediction)

30 Good and Bad of GPs The Good: 1. Have great predictive behavior prediction just reverts to the X outside the range of the data. The Bad: 1. Computation if you have a large dataset then fitting GP models is very slow.

31 Expectations for Sales Case Study 1. Give an overview of the problem. 2. Give a bird s eye view of regression with correlated errors and why we would need to use it here. 3. Write out a correlated regression model for the sales data. - Be sure to justify the correlation function you use (e.g. exponential, AR or MA) - Be sure to define/explain any parameters (including correlation parameters). 4. Show a plot of your predictions over the next year w/ uncertainty. 5. Give some measure of predictive accuracy (coverage and width). Be careful how you split test and training data (hint: preserve the time series).