Probabilistic prediction of real-world time series: A local regression approach

Transcription

1 Click Here for Full Article GEOPHYSICAL RESEARCH LETTERS, VOL. 34, L03403, doi: /2006gl028776, 2007 Probabilistic prediction of real-world time series: A local regression approach Francesco Laio, 1 Luca Ridolfi, 1 and Stefania Tamea 1 Received 16 November 2006; accepted 15 December 2006; published 9 February [1] We propose a probabilistic prediction method, based on local polynomial regressions, which complements the point forecasts with robust estimates of the corresponding forecast uncertainty. The reliability, practicability and generality of the method is demonstrated by applying it to astronomical, physiological, economic, and geophysical time series. Citation: Laio, F., L. Ridolfi, and S. Tamea (2007), Probabilistic prediction of real-world time series: A local regression approach, Geophys. Res. Lett., 34, L03403, doi: /2006gl Dipartimento di Idraulica, Trasporti ed Infrastrutture Civili, and Water and Environment Research Center, Politecnico di Torino, Torino, Italy. Copyright 2007 by the American Geophysical Union /07/2006GL Introduction [2] The challenge of predicting the future by looking at the past has led to a wide variety of methods that provide point estimates of future values of time series [Makridakis et al., 1998]. Point (or deterministic) predictions are however strongly arguable, since they do not supply an assessment of the uncertainty associated with the forecast. In contrast, a probabilistic (or density) forecast gives an explicit representation of the uncertainty through the probability distribution of the predicted value [Abramson and Clemen, 1995; Tay and Wallis, 2000]. Probabilistic forecasts enable rational decision making and offer substantial economic benefits [Krzysztofowicz, 2001], which can be especially relevant when geophysical variables (e.g., river discharge, rainfall, earthquake intensity,...) are the object of investigation; however, a general, practical, and reliable method is not available yet. Here we propose a novel probabilistic prediction method, extending and improving a family of deterministic forecast approaches based on local polynomial regression techniques [Fan and Gijbels, 1996; Cleveland and Loader, 1996]. Known with different names in different contexts (nonlinear prediction method [Farmer and Sidorowich, 1987; Porporato and Ridolfi, 2001; Sivakumar et al., 2002], lazy learning [Atkeson et al., 1997], cluster weighted modelling [Gershenfeld et al., 1999], nearestneighbors method or method of analogues [Viboud et al., 2003], just in time prediction [Stenman et al., 1996]) these methods have proved to be reliable and effective in the deterministic setting. Our probabilistic extension maintains these qualities, with the added value of providing a complete probabilistic representation of the forecast uncertainty. The prediction method is applied to six time series and the reliability of the probabilistic forecasts is rigorously evaluated, demonstrating the excellent qualities of the proposed approach. 2. Prediction Method [3] A short review of the deterministic prediction method is hereinafter presented, together with a novel parameter calibration technique, based on an ensemble approach, and a detailed description of the probabilistic extension of the local polynomial regression method Deterministic Prediction [4] Suppose a time series of measurements of the variable to forecast, x(t), is available, along with n 1 additional time series of the variables x 2 (t),..., x n (t), in the case of a multivariate analysis. The series are sampled at regular intervals Dt, have the same length N, and are preliminarily standardized to have zero mean and unit variance. The point prediction aims at obtaining an estimate of a future value of x(t) as a function of the available information at a time t 0, ^xðt 0 þ TÞ ¼ Fxt ½ ð 0 ÞŠ; ð1þ where the lead time (or prediction horizon) of the forecast, T, is a multiple of Dt. The available information can be represented as a vector of endogenous and exogenous regressors [Sauer et al., 1991], xðt 0 Þ ¼ ½xt ð 0 Þ; xt ð 0 t 1 Þ;...; xt ð 0 ðm 1 1Þt 1 Þ;...; x 2 ðt 0 Þ; x 2 ðt 0 t 2 Þ;...; x 2 ðt 0 ðm 2 1Þt 2 Þ;...; x n ðt 0 Þ; x n ðt 0 t n Þ;...; x n ðt 0 ðm n 1Þt n ÞŠ; ð2þ where t 1,..., t n are multiples of Dt, and m 1,..., m n are integer numbers (m m n = m). The prediction function F: R m! R in equation (1) converts the information in the query point x(t 0 ) into the predicted future value. In local methods the map F changes with x(t 0 ); in particular, in local polynomial regression methods the k nearest neighbors, x(t j )(j =1,..., k; t j + T < t 0 ), of the query point are collected, and a local predictor is constructed by fitting a polynomial of order r to the pairs (x(t j ), x(t j + T)) [Fan and Gijbels, 1996; Cleveland and Loader, 1996; Farmer and Sidorowich, 1987]. The nearest neighbors x(t j ) are the points closest to the query point according to the unweighted Euclidean distance. [5] The local polynomial regression takes the matrix form z ¼ Y a þ ; where z =[x(t 1 + T), x(t 2 + T),..., x(t k + T)] 0 (the apex denotes the transpose operator); Y is a (k p) matrix (p = (m+r)!/(m!r!))whosejthrowisy(t j )=[1,x(t j ),x 2 (t j ),...,x r (t j )] ð3þ L of5

2 x r (t j )] and x r (t j ) is a row vector with components of the form x q u x r q w (x u is the uth element of x(t j ); u,w =1,...,m; q =0,...,r); a is the vector of unknown coefficients; and is a vector of mutually independent errors (residuals), with common distribution P E (). Since the model is linear in the unknown coefficients, the least squares estimator is available [Kendall and Stuart, 1977], ^a =(Y 0 Y) 1 Y 0 z, where the suffix ( 1) denotes the inverse matrix. The forecast reads ^xðt 0 þ TÞ ¼ yðt 0 Þ^a; ð4þ which can be interpreted as the Taylor expansion of the prediction function F in equation (1) around the query point x(t 0 ), truncated to the order r. Zero order (r = 0), linear (r = 1) and quadratic (r = 2) approximations are adequate in most cases [Cleveland and Loader, 1996]. The local polynomial method is highly adaptable to the underlying dynamical behavior of the time series: high values of k and low values of r are suitable for modelling and predicting linear stochastic time series, low values of k and high values of r for non-linear chaotic systems [Farmer and Sidorowich, 1987; Kugiumtzis, 2002; Casdagli, 1992], while large m and t values can be useful to represent strongly persistent processes Parameter Calibration [6] The vector of parameters q =[m 1,..., m n, t 1,..., t n, r, k] is determined through cross calibration [Cleveland and Loader, 1996; Atkeson et al., 1997; Regonda et al., 2005]. The available data are preliminarily split into three independent non-overlapping portions: a fitting (or training, or learning) set S F, where the neighbors are searched for; a calibration set S C used for cross calibration purposes; and a validation (or testing) set S V, used as the ersatz of the future data for providing an a posteriori evaluation of the quality of the predictions. Starting from different trial parameter sets q s, several predictions are then carried out for all the points belonging to S C. A score function is defined to assess the quality of the predictions: we use the mean absolute error z s = E[jx ^x s j] (hereinafter the dependence on time is omitted), where the expectation E[] is taken over all points in S C. Because of the relatively low number of parameters in q, it is feasible to perform a full analysis of their influence. Reasonable trial parameter vectors q s can be constructed by taking values of m in the range {1, 20}, t 2 {Dt, 5Dt}, r 2 {0, 2} and k The parameter set associated to the lowest score function is selected and used for forecasting. [7] A problem with the cross calibration approach arises when the calibration set is not fully representative of the actual dynamics of the future data, which typically happens when dealing with short and highly variable geophysical time series. If this is the case, a single set of parameter values chosen by minimizing the score function in S C can produce suboptimal predictions. For this reason, instead of selecting only one set of parameter values, we use the N q sets (with, say, N q = 100) that give the smallest z s over the calibration set and calculate an ensemble of N q predictands. The ensemble average (or the median) is taken as the actual point forecast, as in classical ensemble techniques [Krogh and Sollich, 1997; Roulston and Smith, 2003]. This averaging procedure provides robustness to the calibration and improves the quality of the predictions, consistently with Makridakis et al. [1998], Regonda et al. [2005] and Clemen [1989] Probabilistic Forecast [8] We now turn to the probabilistic aspects of the method. Compared to other forecasting methods, it is here relatively simpler to perform a full probabilistic prediction, i.e. to determine the probability distribution of the predicted values, P ^X (^x). In fact, the prediction model locally reduces to a linear regression, whose residuals can be used as first approximations of the global errors affecting ^x. These residuals (insample errors) can be opportunely inflated to revert them into out-of-sample errors [Chatfield, 2001; Stine, 1985], and then summed up to the point prediction to produce a sample of predictands. The procedure is detailed hereinafter. [9] The point prediction in equation (4) is affected by a global forecast error x = x ^x, which can be quantified by taking x = y a + from equation (3), x ¼ y ða ^a Þþ: ð5þ The first term on the right hand side of equation (5) represents the uncertainty associated with the regression coefficient estimation; its distribution is approximatively N(0, ^s 2 ), i.e., normal (even when the residuals are non- Gaussian, see Makridakis et al. [1998] and Stine [1985]) with zero mean and variance [Makridakis et al., 1998; Kendall and Stuart, 1977] ^s 2 ¼ ^ 0 ^ k p y ð Y0 YÞ 1 y 0 ; ð6þ where ^ =(z Y ^a) is the vector of estimated residuals (in-sample errors). The term in equation (5) accounts for the propagation of uncertainty in prediction [Kendall and Stuart, 1977], and has distribution P E (), which can be estimated from the rescaled empirical distribution function, S k (^), of the residuals. This is defined as sffiffiffiffiffiffiffiffiffiffiffi S k ðþ¼ ^ k k p i k ; ^ i ^ < ^ iþ1 ði ¼ 1;...; kþ; ð7þ where ^ i are residuals arranged in ascending order, and the factor under square root guarantees that the variance of S k (^) is (^ 0 ^)/(k p), as expected from the least squares theory [Kendall and Stuart, 1977]. [10] Since the two terms in equation (5) are independent [Kendall and Stuart, 1977; Stine, 1985], the convolution of N(0, ^s 2 ) and S k (^) provides an estimate of the distribution of the global error; in practice this is obtained by sampling with replacement from N(0, ^s 2 ) and S k (^), and summing up the results. A sample of ~ k global errors is obtained, which is used to dress the point forecast ^x, obtaining a sample of predictands of the form ^x j = ^x + x j ( j =1,..., ~ k). Here we take ~ k = 100. The model structural uncertainty is accounted for by repeating this procedure for several parameter values (model averaging approach): a sample of predictands ^x j is obtained from each of the N q ensemble members, and the resulting cumulative probability distribution of the predictands, P ^X (^x), is estimated as the empirical distribution function of the large sample of N q ~ k elements ^x s,j = ^x s + x s,j, (s =1,..., N q, j = 1,.., ~ k). The median of this distribution is taken as the point forecast, and the prediction interval 2of5

3 Table 1. Characteristics of the Six Time Series a Series m s Sk AC 1 AC 10 N N C N V I II III IV V VI a m, s and Sk are the mean value, the standard deviation and the skewness coefficient of the time series, respectively; AC 1 is the autocorrelation coefficient at lag Dt and AC 10 at lag 10Dt. N is the total number of points in the time series, N C and N V are the length of the calibration and validation sets, respectively. corresponding to a significance level a is obtained by taking the (a/2) and (1 a/2) quantiles of P ^X (^x). The probability distributions are different for each point to be predicted and they change with the lead time, T, of the prediction. [11] The proposed approach is novel and general, since it does not require any limiting assumption on the distribution of the residuals. Parameter uncertainty, model structural uncertainty, and input uncertainty are all quantified through the processing of the model residuals. However, to guarantee the correctness of the proposed method, it is essential to have data which are representative of the future dynamics, also in terms of their associated uncertainty: for example, in the case the input data become less reliable during application, an additional uncertainty is induced that cannot be adequately quantified with the proposed procedure. 3. Application and Discussion [12] The proposed probabilistic approach has been applied to the following six time series, with very different statistical and dynamical characteristics (see Table 1). [13] Series I is a chaotic Henon map [Henon, 1976] of 2 equation y i = 1 1.4y i y i 2 corrupted by an observational Gaussian noise (i.e., x i = y i + h i ). The noise term h i is added after the time series generation, sampling from a normal distribution N(0, s 2 ) with s equal to the 10% of the map s standard deviation. The Henon map is a benchmark series, often used to assess the validity of nonlinear prediction methods [Farmer and Sidorowich, 1987; Regonda et al., 2005]. [14] Series II is the same Henon map [Henon, 1976] corrupted by an additive dynamical Gaussian 2 noise: x i =1 1.4x i x i 2 + h i. The noise term h i plays as an external forcing of the dynamical system and it is sampled from a normal distribution N(0, s 2 ) with s equal to the 1% of the map s standard deviation. [15] Series III is the E time series of the Santa Fe forecasting competition [Weigend and Gershenfeld, 1992], i.e., an astronomical recording of the light variation of a white dwarf star, measured at Dt =10s. [16] Series IV is the B time series of the Santa Fe forecasting competition [Weigend and Gershenfeld, 1992]: a physiological recording of the chest volume of a patient suffering from sleep apnea, taken at Dt =0.5s. A multivariate analysis is carried out, using also the related heart rate and blood oxygen concentration time series. [17] Series V is the time series of the first differences of the monthly exchange rate between U.S. Dollar and G.B. Pound from January 1971 to August 2005 [see Kugiumtzis, 2002]. [18] Series VI is the hourly discharge time series of the Tanaro River (Italy) at the Farigliano gauging station; the mean rainfall depth over the basin (extension of 1522 km 2 ; mean elevation of 938 meters a.m.s.l. [see Tamea et al., 2005; Laio et al., 2003]) is also available, which represents the main forcing of the runoff dynamical system. [19] Before investigating the performances of the probabilistic model, we mention that the point prediction method, evaluated on the validation set S V (see Table 2), demonstrates high reliability, except that for series V, which is very noisy [Kugiumtzis, 2002]. Overall, the quality of the point predictions in Table 2 is good and compares favorably to that of other point prediction methods [see Kugiumtzis, 2002; Weigend and Gershenfeld, 1992; Laio et al., 2003] applied to the same time series. Some examples of predictions obtained with the probabilistic method (Figure 1) confirm the good quality of the results. Even with a lead time as high as 10Dt, the system behavior is predicted with reasonable accuracy, especially when a strong deterministic component is present (cases I and II) or a multivariate prediction is carried out (cases IV and VI). The prediction method produces forecast errors that are scarcely autocorrelated at lags greater than the lead time T (the autocorrelation coefficient is always below 0.13, see also Figures 1d 1f), meeting a basic requirement for prediction methods [Makridakis et al., 1998; Diebold et al., 1998]. One of the major advantages of our probabilistic approach lies on its capability of providing prediction bands (Figure 1). The prediction bands are smooth and their amplitude naturally increases where the forecast is more difficult (see for example Figures 1e and 1f). Only in cases where the noise component is predominant, the prediction method tends to a global regression model and the bands have an almost constant amplitude (as in Figure 1c). [20] The reliability of the probabilistic forecast can be evaluated by considering the transformation h = P ^X (x) [Tay and Wallis, 2000; Diebold et al., 1998; Laio and Tamea, 2006], which attaches to the real future value x a probability h from the distribution of the predicted values, P^X (). The h values are calculated for each point belonging to the validation set, obtaining a sample of N V values h i = P^X (x i) (note that P^X () is different from point to point). If T-stepahead forecasts are considered, the sample is split into T subseries {h 1, h 1+T, h 1+2T,...}, {h 2, h 2+T, h 2+2T,...},..., {h T, h 2T, h 3T,...}[Diebold et al., 1998; Laio and Tamea, 2006]. When the probabilistic forecast is correct (i.e., P^X () Table 2. Evaluation of the Probabilistic Predictions at Lags Dt and 10Dt for the Six Time Series a T = Dt T =10Dt Series z/z V U. Test I. Test z/z V U. Test I. Test I II * * III * 2.786* IV * 2.771* V VI * * a The point prediction quality is verified through the mean absolute error of the forecast z, normalized by the average absolute deviation of the validation set, z V. The quality of the probabilistic predictions is determined using the Kolmogorov uniformity test (U. test) and the Kendall s t test of independence (I. test), applied to the probability integral transform of the prediction, h i. Marked values(*) are significant at a 5% level. 3of5

4 Figure 1. (a c) Probabilistic predictions and (d f) forecast errors x(t)=x(t) ^x(t) for the six time series. The black points are the real data, the marked continuous line is the point prediction, and the shaded areas represent the 50% (dark grey) and 90% (light grey) prediction bands; the predictions are at T = Dt in Figures 1a, 1b, 1e, and 1f; at T =10Dtin Figures 1c and 1d. Figure 2. Evaluation of the reliability of the probabilistic predictions. Probability plots for the six time series, with h i ¼ P ^X ðx iþ on the horizontal axis, and the corresponding empirical distribution function, P H (h i ) on the vertical axis. The probabilistic prediction is reliable if the points are close to the bisector; the Kolmogorov limits with 5% significance are reported as dashed lines. Predictions are at (a c) T = Dt and (d f) T =10Dt; in all cases the evaluation of the ensemble forecasts (grey marked lines) indicates that ensembles do not represent reliable probabilistic predictions (see text for details). 4of5

5 is the true distribution of the predictands), these subseries of h i values constitute T samples of independent values from a uniform distribution [Tay and Wallis, 2000; Diebold et al., 1998]. The mutual independence is verified applying the Kendall s tau independence test to the samples of h i values [Laio and Tamea, 2006]. The resulting values for the six time series are reported in Table 2, demonstrating that the temporal dependence of the h i values is negligible for predictions with T = 10Dt, while there is a statistically significant dependence for some of the Dt-lag predictions. However, we note that even for these series the autocorrelation coefficient remains below [21] The uniformity of the T samples of h i values is verified by plotting the h i s against their (empirical) cumulative distribution function [Clements and Smith, 2000], as shown in Figure 2: under the hypothesis of uniformity, the resulting points should plot as a straight line corresponding to the bisector of the diagram [Tay and Wallis, 2000; Diebold et al., 1998; Laio and Tamea, 2006]. The distance of the points from the bisector can be evaluated adopting a Kolmogorov goodness-of-fit criterion [Diebold et al., 1998], which is graphically represented by two other lines parallel to the bisector (dashed lines in Figure 2): when the points are inside these lines, the Kolmogorov test of uniformity at the 5% level is passed [Clements and Smith, 2000]. The results of the Kolmogorov uniformity test for all series and lags are also reported in Table 2. [22] Figure 2 demonstrates the reliability of the probabilistic forecasts. Good results are obtained for all time series; even when the hypothesis that P ^X () is the true distribution should be rejected (see Table 2), the uncertainty description remains adequate for practical applications. Possible reasons for occasional failures of the uniformity and independence tests are the lack of representativeness of the fitting set for the future data, or the inadequacy of some of the assumptions behind equation (3) (e.g., independence of the residuals, additive nature of the noise term, etc...). We report in Figure 2 also the points that would be obtained if one estimated P^X (x) from the sample of ^x s (s = 1,..., N q ) ensemble members [Regonda et al., 2005], neglecting the uncertainty associated to each ^x s. The uncertainty representation in this case is clearly inappropriate, demonstrating that the ensemble technique should not be intended as a stand-alone method to provide probabilistic predictions [Krzysztofowicz, 2001; Roulston and Smith, 2003]. [23] A correct evaluation of the uncertainty of the predicted values is one of the key requirements for a good forecast method. The proposed probabilistic method satisfies this condition, being able to describe the whole probability distribution of the future values. Moreover, the robust and non-subjective calibration and the reasonable computational demand make this method suitable for real time forecasting applications (the run of a probabilistic forecast requires about 1.6 seconds, with a processor with speed 1.86 GHz; the Matlab source files are available at the website Finally and most importantly, the method is very flexible and adaptable to different problems, as demonstrated by the positive results obtained from its application to six highly heterogeneous time series. References Abramson, B., and C. Clemen (1995), Probability forecasting, Int. J. Forecasting, 11, 1 4. Atkeson, C. G., A. W. Moore, and S. Schaal (1997), Locally weighted learning, Artif. Intel. Rev., 11, Casdagli, M. (1992), Chaos and deterministic versus stochastic nonlinear modelling, J. R. Stat. Soc., Ser. B, 54(2), Chatfield, C. (2001), Prediction intervals for time series forecasting, in Principles of Forecasting: a Handbook for Researchers and Practitioners, edited by J. Armstrong, pp , Springer, New York. Clemen, R. T. (1989), Combining forecasts: A review and annotated bibliography, Int. J. Forecasting, 5, Clements, M. T., and J. Smith (2000), Evaluating the forecast densities of linear and non-linear models: Application to output growth and uneployment, J. Forecasting, 19, Cleveland, W. S., and C. L. Loader (1996), Smoothing by local regression: Principles and methods, in Statistical Theory and Computational Aspects of Smoothing, edited by W. Haerdle and M. G. Schimek, pp , Springer, New York. Diebold, F. X., T. A. Gunther, and A. S. Tay (1998), Evaluating density forecasts with applications to financial risk management, Int. Econ. Rev., 39(4), Fan, J., and I. Gijbels (1996), Local Polynomial Modelling and its Applications, CRC Press, Boca Raton, Fla. Farmer, J. D., and J. J. Sidorowich (1987), Predicting chaotic time series, Phys. Rev. Lett., 59(8), Gershenfeld, N., B. Schoner, and E. Metois (1999), Cluster-weighted modelling for time-series analysis, Nature, 397(6717), , doi: / Henon, M. (1976), A two-dimensional mapping with a strange attractor, Commun. Math. Phys., 50, Kendall, M. G., and A. Stuart (1977), The Advanced Theory of Statistics, 2nd ed., Griffin Press, London. Krogh, A., and P. Sollich (1997), Statistical mechanics of ensemble learning, Phys. Rev. E, 55(1), Krzysztofowicz, R. (2001), The case for probabilistic forecasting in hydrology, J. Hydrol., 249, 2 9. Kugiumtzis, D. (2002), State space local linear prediction, in Nonlinear Deterministic Modelling and Forecasting of Economic and Financial Time Series, edited by A. Soofi and L. Cao, pp , Springer, New York. Laio, F., and S. Tamea (2006), Verification tools for probabilistic forecasts of continuous hydrological variables, Hydrol. Earth Syst. Sci. Discuss., 3, Laio, F., A. Porporato, R. Revelli, and L. Ridolfi (2003), A comparison of nonlinear flood forecasting methods, Water Resour. Res., 39(5), 1129, doi: /2002wr Makridakis, S., S. C. Wheelwright, and R. J. Hyndman (1998), Forecasting: Methods and Applications, 3rd ed., John Wiley, Hoboken, N. J. Porporato, A., and L. Ridolfi (2001), Multivariate nonlinear prediction of river flows, J. Hydrol., 248, Regonda, S., B. Rajagopalan, U. Lall, M. Clark, and Y. Moon (2005), Local polynomial method for ensemble forecast of time series, Nonlinear Process. Geophys., 12, Roulston, M. S., and L. A. Smith (2003), Combining dynamical and statistical ensembles, Tellus, Ser. A, 55, Sauer, T., J. A. Yorke, and M. Casdagli (1991), Embedology, J. Stat. Phys., 65, Sivakumar, B., A. W. Jayawardena, and T. M. K. G. Fernando (2002), River flow forecasting: use of phase-space reconstruction and artificial neural networks approaches, J. Hydrol., 265, Stenman, A., F. Gustafsson, and L. Ljung (1996), Just in time models for dynamical systems, Proc. IEEE, 1, Stine, R. A. (1985), Bootstrap prediction intervals for regression, J. Am. Stat. Assoc., 80(392), Tamea, S., F. Laio, and L. Ridolfi (2005), Probabilistic nonlinear prediction of river flows, Water Resour. Res., 41, W09421, doi: / 2005WR Tay, A. S., and K. F. Wallis (2000), Density forecasting: A survey, J. Forecasting, 19, Viboud, C., P. Boelle, F. Carrat, A. Valleron, and A. Flahault (2003), Prediction of the spread of influenza epidemics by the method of analogues, Am. J. Epidemiol., 158(10), Weigend, A. S., and N. A. Gershenfeld (Eds.) (1992), Time Series Prediction: Forecasting the Future and Understanding the Past, Addison-Wesley, Boston, Mass. F. Laio, L. Ridolfi, and S. Tamea, Dipartimento di Idraulica, Trasporti ed Infrastrutture Civili, Politecnico di Torino, Corso Duca degli Abruzzi 24, I Torino, Italy. (francesco.laio@polito.it) 5of5