PITFALLS IN BACKTESTING HISTORICAL SIMULATION MODELS

Transcription

1 PITFALLS IN BACKTESTING HISTORICAL SIMULATION MODELS Juan Carlos Escanciano Indiana University Pei, Pei Indiana University February 14, 2011 Abstract Historical Simulation (HS) and its variant, the Filtered Historial Simulation (FHS), are the most widely used Value-at-Risk forecast methods at commercial banks. These forecast methods are traditionally evaluated by means of the unconditional backtest. This paper formally shows that the unconditional backtest is inconsistent for testing HS and FHS models, with a power function that can be even smaller than the nominal level. Our findings have fundamental implications in the determination of market risk capital requirements, and explain empirical findings in previous studies. To overcome the lack of power of the unconditional backtest, we propose a data-driven weighted backtest with good power properties to evaluate HS and FHS forecasts. Finally, our theoretical findings are confirmed through a Monte Carlo simulation study and an empirical exercise is implemented using actual data from five international commercial banks. The empirical application reveals the practical importance of our theoretical findings in the determination of multiplication factors and market risk capital requirement. Keywords and Phrases: Backtesting; Basel Accord; Risk management; Value at Risk; Conditional Quantile. JEL Classficiation: C52; C32; G21; G32 We thank Christophe Pérignon and Daniel R. Smith for providing the data we use in our application. Department of Economics, 105 Wylie Hall, 100 S. Woodlawn, Bloomington, IN 47405, U.S.A.; address: jescanci@indiana.edu. This research is funded by Spanish Plan Nacional de I+D+I grant number SEJ Corresponding Address: Department of Economics, Indiana University, 105 Wylie Hall, 100 S. Woodlawn, Bloomington, IN ppei@indiana.edu. 1

2 1 Introduction Value-at-Risk (VaR) has become the standard tool for measuring market risk used by financial institutions and their regulators. VaR is defined as the maximum expected loss on an investment over a specified horizon at a particular confidence level, see Jorion (2006). VaR summarizes market risk into a single number which, in statistical terms, is a quantile of the conditional distribution of portfolio returns given investor s information set. It not only offers a measure of market risk but also forms the basis for the determination of market risk capital, since the 1996 Amendment of the Basel Accord (see Basel Committee on Banking Supervision, 1996a and 1996b). Obviously, the foremost concern of VaR users is to forecast VaRs accurately in order to guide their decisions. According to a recent international survey by Perignon and Smith (2010), the Historical Simulation (HS) forecast model and its variant, the Filtered HS (FHS), are the most currently used methods at commercial banks 1. These methods use unconditional quantiles of raw data and standardized innovations, respectively, to forecast the conditional quantiles. For alternative forecast models and comprehensive finite-sample comparisons see e.g. McNeil et al. (2005) or Kuester et al. (2006). Within the framework of the Basel Accord, besides the VaR forecasts, the other two crucial ingredients in determining market risk capital requirements are the forecast evaluation or backtesting procedure and the multiplication factors based on the outcomes of the backtesting procedure. Therefore, backtesting, as a forecast evaluation method of a VaR model, is also of vital importance for financial institutions and their regulators. There are several backtesting methods available. See, for example, the unconditional coverage test of Kupiec (1995, henceforth K-test), the conditional backtest of Christoffersen (1998, henceforth C-test), the CaViar test of Engle and Manganelli (2004), the duration-based approach of Christoffersen and Pelletier (2004) and the density forecast evaluation approach of Berkowitz (2001). For a review of backtesting methods see Berkowitz, Christoffersen and Pelletier (2009). Among existing backtests, the K-test and C-test are the most widely employed backtesting procedures, given their simplicity. The K-test simply computes the frequency of the ex post trading results that are less than or equal to the ex ante VaR forecasts, 1 The authors find that 73% of the firms that disclose their VaR methodology in their 2005 annual reports use HS or related techniques. 2

3 and compares this frequency with the theoretical VaR level, e.g. 1%. Large differences between the frequency and the VaR level indicates that, on average, the forecasts are not optimal. 2 This paper draws on the observation that the most widely used VaR models, i.e. HS or FHS models, are commonly evaluated with the K-test, and it formally shows that this combination is highly misleading. We prove that the K-test is always inconsistent in detecting non-optimal HS or FHS forecasts. That is, the rejection probability of the K-test does not converge to one as the sample size increases under the alternative. This theoretical finding explains the low power associated with the K-test in simulations carried out in the literature, see Kupiec (1995), Pritsker (2006) and Pérignon and Smith (2008), among many others. Prisker (2006), who particularly studies the poor performance of HS models in estimating conditional quantiles, does find the low power of the K-test in detecting non-optimal forecasts of HS models in its simulation exercise where data were only generated from GARCH(1,1) processes, but does not find out it is the combination of the K-test and HS models that results in the low power. As far as we are concerned this is the first paper points out the pitfalls in backtesting HS models. In order to solve the deficiencies of the K-test in detecting non-optimality of HS or FHS forecasts, this paper proposes a new data-driven weighted backtest with better power properties than the K- test and alternative tests. The new test statistic is a consistent test in testing HS and FHS models and combines a nonparametric weight with a parametric one such that reasonably good power performances can be obtained. In Prisker (2006), the author suggests a parametric bootstrapping approach, but the approach is subjective to model misspecifications. We show through simulations that the new test has an excellent finite-sample performance and overcomes the limitations of the K-test for the evaluation of HS or FHS methods. The remainder of the paper is organized as follows. Section 2 describes the forecast evaluation problem from a decision point of view and shows the inconsistency of the K-test for HS and FHS models. In Section 3 we propose a large class of backtesting methods for HS models. The asymptotic distribution theory for the new class of weighted tests is established. The finite-sample performances of the K-test, C-test and the new test for HS and FHS models are investigated using 2 For a formal definition of optimal VaR forecast, see next section. 3

4 Monte Carlo methods in Section 4. Section 5 analyzes, in an application to daily actual trading results at five major commercial banks, the testing results for the optimality of FHS forecasts and the correponding implications on the determination of market risk capital requirements. Finally, section 6 concludes. Mathematical proofs of our results are gathered in the Appendix. 2 Backtesting Historical Simulation Models In this section, we provide a formal introduction to the forecast evaluation problem and the HS and FHS models, and we prove the inconsistency of the K-test in testing optimality of HS forecasts. Denote the real-valued time series of portfolio returns or Profit and Losses (P&L) account by Y t, and assume that at time t 1 the agent s information set is given by I t 1, which contains past values of Y t and other relevant economic and financial variables. Let F t 1 be the σ field generated by I t 1. We assume that VaR users face the following decision problem: with the information available at time t 1, they aim to find q (I t 1 ) minimizing the conditional expected asymmetric or tick loss function, that is q (I t 1 ) := arg min L t,α (q) := E [L α (Y t, q(i t 1 )) I t 1 ], (1) q(i t 1) F t 1 where L α (Y t, q(i t 1 )) := {α 1(Y t q(i t 1 ))} (Y t q(i t 1 )) and 1(A) is the indicator function of the event A, that is, 1(A) = 1 if A occurs and 0 otherwise. Henceforth, an optimal forecast according to this loss function is simply called optimal (see e.g. Christoffersen, 1998). It is well-known that, provided the conditional distribution of Y t given I t 1 is continuous, the optimal forecast q (I t 1 ) is the α-th conditional quantile (i.e. VaR) of Y t given I t 1, q α (I t 1 ), satisfying the conditional moment restriction P (Y t q α (I t 1 ) I t 1 ) = α, almost surely (a.s.), α (0, 1), t Z. (2) A common approach in empirical work is to assume a parametric VaR model M = {m α (I t 1, θ) : θ Θ R p } and to proceed to make VaR forecasts using the model M. In parametric models 4

5 the nuisance parameter θ 0 belongs to Θ, a compact set in R p. The literature on parametric VaR modelling has been advancing rapidly; see e.g. Engle and Manganelli (2004), Koenker and Xiao (2006) and Gourieroux and Jasiak (2006), among many others 3. Semiparametric and nonparametric specifications for q α ( ) have been also entertained, see e.g. Fan and Gu (2003) and Cai and Wang (2008), where θ 0 belongs to an infinite-dimensional space of functions. The preferred model by financial institutions is the HS model or its variant, the FHS, see Barone- Adesi et al. (1998, 1999) and Barone-Adesi and Giannopoulos (2002). The HS model specifies a time-constant VaR model m α (I t 1, θ 0 ) = θ 0 F 1 Y (α), (3) where, henceforth, for a generic strictly stationary sequence {X t } t Z, F 1 (α) denotes the unconditional quantile function of X t, evaluated at α, i.e. X the generalized inverse of the cumulative distribution function (cdf) of X t, denoted by F X ( ). In contrast, the FHS assumes an underlying location-scale structure, and hence specifies a time-varying VaR model m α (I t 1, θ 0 ) = µ(i t 1, β 0 ) + σ(i t 1, β 0 )F 1 ε (α), (4) where µ t 1 µ(i t 1, β 0 ) and σ t 1 σ(i t 1, β 0 ) are the conditional mean and standard deviation of Y t given I t 1, respectively, and the standardized innovations {ε t = (Y t µ t 1 )/σ t 1 } t Z are assumed to be independent and identically distributed (i.i.d.) and independent of I t 1. Among the most common models for µ t 1 and σ t 1 are the autoregressive moving average (ARMA) and the generalized autoregressive conditional heteroscedasticity (GARCH) models, respectively. Once a forecast model has been specified, a natural testing problem is to evaluate the forecasts. This is the so-called backtesting procedure for VaR models. That is, we aim to test for the null hypothesis H 0 : m α (I t 1, θ 0 ) is an optimal forecast, 3 The reader should not be confused with a fully parametric approach in which the whole conditional distribution of Y t given I t 1 is fully specified. We do not need such fully specification since our concern is just the conditional quantile function at a particular quantile level α. In this sense and in the statistical jargon our specified model is semiparametric. 5

6 against the alternative hypothesis, H A, that m α (I t 1, θ 0 ) is not optimal. From our previous discussion, we know that the null entails that m α (I t 1, θ 0 ) is a correctly specified VaR model, m α (I t 1, θ 0 ) = q α (I t 1 ) a.s.. In this paper we are concerned with the tests for H 0 when m α (I t 1, θ 0 ) is given by a HS or FHS model in (3) or (4), respectively. Taking expectations of (2), we conclude that a necessary condition for an optimal forecast is that it is unbiased, i.e. E[h t,α (θ 0 )] = α, where h t,α (θ 0 ) := 1(Y t m α (I t 1, θ 0 )) are the so-called hits or exceedances. This implication suggests a test for H 0 based on rejecting for large absolute values of K P K(P, R) := 1 P n [h t,α (θ 0 ) α], where P and R are the out-of-sample and in-sample sample sizes, respectively, n = P + R, and θ 0 is assumed to be known. This is the so-called unconditional backtest (Kupiec, 1995), in short the K-test. Escanciano and Olmo (2009) has extended the K-test to the more realistic situation of unknown parameter θ 0. The K-test is the reference backtest in the banking industry and the academia. Under the current regulatory framework, the multiplication factors used for computing capital reserves are also functions of the K-test. Our main result formally shows that the K-test and its generalization to estimated parameters are inconsistent in detecting non-optimal forecasts from HS or FHS models. Intuitively, K P will remain small under lack of forecast optimality for HS or FHS models, as the corresponding hit sequence has a mean approximately equal to α under the alternative. This intuition is formalized below in our Theorem 1. This theoretical result has fundamental implications for risk management practices at commercial banks and other financial institutions. To derive our theoretical results, we first need to introduce some notations and assumptions. 6

7 Assume that I t 1 = (X t 1, X t 2,...), where X t includes Y t, and define the α-mixing coefficients as α(m) = sup sup P (A B) P (A)P (B), m 1 B F n,a P n+m n Z, where the σ-fields F n and P n are F n := σ(x t, ε t, t n) and P n := σ(x t, ε t, t n), respectively. Under appropriate regularity conditions, K P / α(1 α) converges into a standard normal random variable, see Kupiec (1995). The power function of Kupiec s two sided test at τ% nominal level is then defined as ( ) Π(τ, α) := lim Pr K P > z P 1 τ/2 H A, α(1 α) where z 1 τ/2 is the corresponding level-τ critical value from a standard normal random variable. That is, Π(τ, α) is the limit probability of rejecting non-optimality of HS or FHS forecasts when in fact they are not optimal. It will be shown that the power function depends on the long-run variance σk 2 = α(1 α) + 2 Cov (h t,α (θ 0 ), h t j,α (θ 0 )) j=1 = : α(1 α) + 2σ hk, which is well defined under the following assumption. Assumption A1: {X t, ε t } t Z is strictly stationary and strong mixing process with mixing coefficients satisfying j=1 α(j) <. F (y) and F ε(y) are strictly increasing and continuous in y R. The conditional variance σ 2 (I t 1, β 0 ) > 0 a.s., and σ 2 K > 0. The following Theorem shows the inconsistency of the K-test. Theorem 1: Assume that m α (I t 1, θ 0 ) is given by a HS or FHS model, and let Assumption A1 hold. Then, under the alternative hypothesis H A, Π(τ, α) = 2Φ ( z τ/2 α(1 α) σ K ), 7

8 where Φ is the cdf of the standard normal random variable. A consequence of Theorem 1 is that K-test is inconsistent, that is, Π(τ, α) < 1. The magnitude of the power function depends on the relative size of the long-run variance, σk 2, and the null asymptotic variance, α(1 α), of the K-test. If σ 2 K < α(1 α), or equivalently σ hk < 0, the power function is smaller than the nominal level τ, otherwise it is larger than τ. Empirically, σ hk > 0 is more consistent with actual financial data due to the well-known property of clustering of extremes. Therefore, our theoretical results suggest that the K-test will have low power, but still larger than the nominal level for large sample sizes. This is confirmed in our simulations below. Theorem 1 sheds some light on simulations and applications carried out in the literature. We remark on some conclusions that the literature has drawn from simulations. First, our Theorem 1 shows that the low power of the K-test cannot be in general improved by considering small VaR confidence levels 1 α, large sample sizes P or different approximations to its finite-sample distributions, such as Monte Carlo or bootstrap 4, when applied to HS or FHS models. Second, the lack of power affects both HS and FHS models. To solve this deficiency of the K-test we introduce a general class of weighted backtesting procedures in the next section. 3 A Class of Weighted Backtesting Procedures In this section we propose a class of backtesting procedures based on the test statistic K P,w := 1 P n w(i t 1 )(h t,α ( θ t 1 ) α), (5) where θ t 1 is an estimator for θ 0 and w(i t 1 ) is a given univariate weight function, whose choice will be based on power considerations of the resulting test. See our discussion after Theorem 3. We 4 For instance, Jorion (2006) suggests small VaR confidence level and large sample sizes to increase the power of the tests in general. Prisker (2006) mentions that more than 2-1/2 years of daily data are required to reject the null of correct unconditional coverage in a one-sided 95% test. Christoffersen and Pelletier (2004) introduces the use of Dufour (2000) Monte Carlo testing technique. However, our theorem implies that only the relative magnitude of the long-run variance and the null asymptotic variance of the K-test affects the size of the power function, so to employ more data or to choose small VaR confidence level or some Monte Carlo technique would not improve the power performances. 8

9 will develop the asymptotic theory for K P,w. Large values of K P,w will indicate non-optimality of the forecasts. Define the conditional distributions F x (y) := P (Y t y I t 1 = x). (6) For the sake of completeness we discuss the asymptotic distribution theory under three different forecasting schemes, namely, the recursive, rolling and fixed forecasting schemes. They differ in how the parameter θ 0 of the HS or FHS model is estimated. In the recursive scheme, the estimator θ t is computed with all the sample available up to time t. In the rolling scheme, only the last R values of the series are used to estimate θ t, that is, θ t is constructed from the sample s = t R + 1,..., t. Finally, in the fixed scheme, the parameter is not updated when new observations become available, i.e., θ t = θ R, for all t, R t n. Examples of estimates θ t for HS and FHS models are as follows. In the HS model, the natural estimate of θ 0 is the empirical quantile estimator θ t = F 1 1,t 1,Y (α), θ t = F 1 t R,t 1,Y (α) and θ t = F 1 1 1,R,Y (α), for the recursive, rolling and fixed forecasting schemes, respectively, where Fs,t,Y (α) is the empirical quantile function of {Y r } t r=s, s < t. In the FHS model, the parameter β 0 is usually estimated by the Quasi-Maximum Likelihood Estimator (QMLE), say β t, in a first stage, and estimators for Fε 1 (α) are constructed as for the HS but based on residuals { ε r } t r=s, s < t, where ε t = Y t µ(i t 1, β t ) σ(i t 1, β, t = 1,..., n. t ) We introduce some assumptions that are necessary for the development of the asymptotic theory for K P,w in (5) for the HS model. Assumption A2: The family of distributions functions {F x, x R } has Lebesgue densities {f x, x R } that are uniformly bounded sup x R,y R f x (y) C, and equicontinuous: for every ɛ > 0 there exists a δ > 0 such that sup x R, y z δ f x (y) f x (z) ɛ. Assumption A3: The parameter space Θ is compact in R p. The true parameter θ 0 belongs to the 9

10 interior of Θ. The estimator θ t satisfies the asymptotic expansion θ t θ 0 = H(t) + o P (1), where H(t) is a p 1 vector such that H(t) = t 1 t s=1 l(y s, I s 1, θ 0 ), R 1 t s=t R+1 l(y s, I s 1, θ 0 ) and R 1 R s=1 l(y s, I s 1, θ 0 ) for the recursive, rolling and fixed schemes, respectively. We assume that, under H 0, E[l(Y t, I t 1, θ 0 ) I t 1 ] = 0 a.s. and V := E[l(Y t, I t 1, θ 0 )l (Y t, I t 1, θ 0 )] exists and is positive definite. Moreover, l(y t, I t 1, θ) is continuous (a.s.) in θ in Θ 0 and E sup θ Θ0 l(y t, I t 1, θ) 2] [ C, where Θ 0 is a small neighborhood around θ 0. Assumption A4: R, P as n, and lim n P/R = π, 0 π <. Assumptions A2 to A4 are required in Escanciano and Olmo (2009) and they are explained in detail there. Assumption A3 has been established in the literature for the empirical quantile estimator under a variety of regularity conditions, see e.g. Wu (2005) and references therein. For the empirical quantile estimator in the HS model, the influence function is l(y t, I t 1, θ 0 ) = h t,α(θ 0 ) α, f Y (θ 0 ) where f Y (θ 0 ) is the density function of Y t evaluated at θ 0, which is assumed to satisfy f Y (θ 0 ) > 0. For HS, θ t is consistent for θ 0 both under the null and the alternative. For FHS, with some abuse of notation we use θ 0 to denote the probabilistic limit of θ t, bearing in mind that, in general, this probabilistic limit depends on whether we are under the null or the alternative. We now establish the asymptotic distribution of K P,w for the HS model. Define B w := E [ w 2 (I t 1 ) ], A HS,w := E [ w(i t 1 )f It 1 (θ 0 ) ], ρ w := E[w(I t 1 ) (h t,α (θ 0 ) α) l(y t, I t 1, θ 0 )] and the constants λ hl and λ ll depending on the forecasting scheme Scheme λ hl λ ll Recursive 1 π 1 ln(1 + π) 2 [ 1 π 1 ln(1 + π) ] Rolling, π 1 π/2 π π 2 /3 (7) Rolling, 1 < π < 1 (2π) 1 1 (3π) 1 Fixed 0 π. 10

11 Theorem 2: Let Assumptions A1-A4 hold for the HS model. Then, (i) Under the null hypothesis H 0, K P,w d N(0, σ 2 HS,w), where σ 2 HS,w = α(1 α)b w + 2λ hl A HS,w ρ w + λ ll A 2 HS,w V. (ii) Under the alternative hypothesis H A, P 1/2 K P,w P E[w(It 1 )(h t,α (θ 0 ) α)]. In order to investigate the asymptotic theory of K P,w for the FHS model, we need to define A F HS,w := f ε (F 1 ε (α))e [w(i t 1 )a 1,t (β 0 )] + f ε (Fε 1 (α))fε 1 (α)e [w(i t 1 )a 2,t (β 0 )] (8) with a 1,t (β) = µ t (β)/σ(i t 1, β) and a 2,t (β) = σ t (β)/σ(i t 1, β), where µ t (β) = µ(i t 1, β)/ β, σ t (β) = σ(i t 1, β)/ β and f ε is the density function associated with the quantile function Fε 1. For the existence of these derivatives we need the following assumption: Assumption A5: Assume µ(i t 1, β) and σ(i t 1, β) are continuously differentiable in β (a.s.) such [ that E sup β B0 µ t (β) 2] [ < C and E sup β B0 σ t (β) 2] < C for a neighborhood B 0 of β 0. Then, we have the analogue to Theorem 2 for the FHS model. Theorem 3: Let Assumptions A1-A5 hold for the FHS model. (i) Under the null hypothesis H 0, K P,w d N(0, σ 2 F HS,w), 11

12 where σ 2 F HS,w = α(1 α)b w + 2λ hl A F HS,w ρ w + λ ll A F HS,w V A F HS,w. (ii) Under the alternative hypothesis H A, P 1/2 K P,w P E[w(It 1 )(h t,α (θ 0 ) α)]. Theorem 2 and 3 establish the asymptotic null distribution of K P,w and also show the probability limit of P 1/2 K P,w under the alternative hypothesis H A, for the HS and FHS model, respecitively. Notice that under H A, if E[w(I t 1 )(h t,α (θ 0 ) α)] 0, K P,w will diverge to ±, and the test based on K P,w will be consistent. On the contrary, if E[w(I t 1 )(h t,α (θ 0 ) α)] = 0, then the test based on K P,w will be inconsistent, and an analogous result to Theorem 1 will hold. Notice that estimating the asymptotic variances σ 2 HS,w and σ2 F HS,w in Theorems 2 and 3 is a nontrivial task. There is, however, a situation where these quantities are substantially simplified. If R is arbitrarily large relative to P, i.e. π = 0, that is, if there is infinite information contained in θ t 1 about θ 0 with respect to K P,w, then the asymptotic variances become α(1 α)b w, which can be easily estimated. We can estimate B w by ˆB w := P 1 n w2 (I t 1 ), and then construct a test based on K 2 P,w / ˆB w (α(1 α)). In what follows we investigate how to choose the weight function w. The choice of w can be based on computational conveniences and/or power considerations of the resulting test. It is interesting to notice that the class of weighted backtests we propose includes traditional tests such as the test of independence of Christoffersen (1998) or extensions to higher order autocorrelations in Berkowitz, Christoffersen and Pelletier (2009). To see this, take w(i t 1 ) := w aj (I t 1 ) = h t j,α (θ 0 ), j 1, and note that by definition, E[h t,α (θ 0 )] = α, 5 hence, E[h t j,α (θ 0 )(h t,α (θ 0 ) α)] = Cov(h t j,α (θ 0 ), h t,α (θ 0 )), j 1, 5 See (ii) in the proof of Theorem 1 in Appendix 12

13 which is the j th order autocovariance of the hit sequence. Tests based on autocorrelations are computationally convienent. And their power properties depend on whether E[h t j,α (θ 0 )(h t,α (θ 0 ) α)] is zero or not under the alternative. Given the well-known clustering effect of financial data we expect tests based on autocorrelations to be powerful in detecting non-optimal forecasts of the HS model. However, for FHS, E[h t j,α (θ 0 )(h t,α (θ 0 ) α)] = Cov(1(ε t j Fε 1 (α)), 1(ε t Fε 1 (α)), j 1, (9) so the consistency of tests based on autocorrelations turns out to depend on the dependence in the pseudo-innovations (here ε t = p lim n ε t ( β t )) at the left tail. Our simulations suggest that (9) is small, so tests based on autocorrelations such as those of Christoffersen (1998) or Berkowitz, Christoffersen and Pelletier (2009), among others, have low power. Therefore, the above theoretical results suggest that the choice of w has an influential effect on the power properties of the resulting test. The natural question arises: does there exist any choice of w that leads to a test with good power properties? We address this question in the next section. 3.1 A Consistent Data-Driven Weighted Backtest In order to deliver a consistent test, our theoretical results suggest to choose w such that E[w(I t 1 )(h t,α (θ 0 ) α)] 0 (see Theorem 2 and 3). The following Lemma shows that there exists a choice of w that guarantees the consistency of K P,w. Lemma 1: Under the assumptions of Theorem 2 or 3. If we choose w(i t 1 ) w (I t 1 ) := F It 1 (m α (I t 1, θ 0 )), which is assumed to be non-zero, then the resulting test K P,w is consistent. Lemma 1 shows that our class of weighted backtest includes consistent tests. A practical limitation of Lemma 1 is that w (I t 1 ) = F It 1 (m α (I t 1, θ 0 )) is in general unknown, and its nonparametric estimation is generally complicated by having a possibly infinite dimensional information set. In situations where the researcher is confident about a parametric model for F It 1 (y), we suggest the choice of w as the weight in K P,w. For instance, for testing optimality of HS forecasts, if the 13

14 researcher thinks that a Gaussian AR(1)-GARCH(1,1) model fits the data well, then Lemma 1 suggests to use w (I t 1 ) = Φ ((θ 0 µ t )/σ t ), where θ 0 is the unconditional quantile, and µ t and σ t are the conditional mean and standard deviation specification of the AR(1)-GARCH(1,1) model, respectively. If a nonparametric approach is entertained, we suggest to reduce the dimensionality of the problem and consider the choice of w1(y t 1 ) = F Yt 1 (m α (I t 1, θ 0 )), where F Yt 1 ( ) denotes the conditional distribution of Y t given Y t 1. A similar result to Lemma 1 holds for w1 provided that the class of alternatives is restricted to F Yt 1 (m α (I t 1, θ 0 )) α. This represents a large class of alternatives, while the so-called curse of dimensionality problem of nonparametric estimation is avoided. To obtain the nonparametric estimation of w1, note that F Yt 1=y(m α (I t 1, θ 0 )) = E[h t,α (θ 0 ) Y t 1 = y] = E[h t,α (θ 0 ) F Y (Y t 1 ) = F Y (y)] = E[h t,α (θ 0 ) U t 1 = u], where the conditioning variable Y t 1 is normalized by taking a quantile transform, provided that F Y is strictly increasing. That is, we define U n,t 1 := F n (Y t 1 ) := 1 n 1{Y s 1 Y t 1 } and U t := F Y (Y t ), (10) n s=1 where the empirical cdf F n,t is implicitly defined. Note that U t is U[0, 1] distributed. Hence, w1 is a regression function. There are many nonparametric estimators available for a regression function, see Li and Racine (2007). For our particular situation, one appealing estimator for w1 is a global estimator obtained by series methods. We introduce the series estimator for w1 as follows. First, we introduce a vector of series base functions: p S (u) := (p 0 (u), p 1 (u),, p S 1 (u)), u [0, 1], (11) 14

15 where p 0 (u) 1. Using these base functions, we approximate g(u) := E[h t,α (θ 0 ) U t 1 = u] by p S (u) γ with an appropriate vector γ. And γ can be estimated by the ordinary least squares (OLS) [ regression of H n,α := h 1,α ( θ n ),..., h n,α ( θ n )] on Pn := [p S (F n (Y 0 )),..., p S (F n (Y n 1 ))]. We define the series estimator for g(u) as ĝ(u) := p S (u) γ n, (12) where γ n = [P np n ] 1 P nh n,α is the OLS estimator for γ. Then, the final estimator for w 1(Y t 1 ) = F Yt 1 (m α (I t 1, θ 0 )) is ŵ 1(Y t 1 ) := ĝ(f n (Y t 1 )), 1 t n. (13) Note that Ŵ 1 = (ŵ 1(Y 0 ),..., ŵ 1(Y n 1 )) can be simply computed as the fitted values of the OLS regression, that is, Ŵ 1 = P n [P np n ] 1 P nh n,α. Then, we construct the weighted test statistic using Ŵ 1 as K P,ŵ := 1 P n ŵ 1(Y t 1 )(h t,α ( θ t 1 ) α). (14) Note that K P,ŵ will depend on the number of components S in the series estimator Ŵ 1, so we make this dependence explicit and write K P,ŵ (S). Next, we address the challenging and important practical issue of bandwidth choice S for testing as follows. Let S max denote an upper bound for S, and denote by γ n = (γ n1,..., γ nsmax ) the corresponding vector of the OLS estimates. Note that under the null F Yt 1 (m α (I t 1, θ 0 )) = α, hence the coefficients γ n1,..., γ nsmax are expected to be close to zero, but significantly different from zero under the alternative if S max is sufficiently large. Then, we construct an aggregate measure of significance as and choose S such that Q S = n S j=1 γ 2 nj, S = min{s : 1 S S max ; L S L h, h = 1, 2,..., S max } where L S = Q S S log n. The above procedure is motivated by the fact that γ n = (γ n1,..., γ nsmax ) 15

16 behaves as the sample autocorrelations and thus the data-driven selection of the order of an autoregression can be applied. As the evidence points to the autocorrelations appear to be small under the null, we employ the Bayesian Information Criterion (BIC), see Schwarz (1978), which imposes a large penalty and results in the chosen values for S are small. Tests using the BIC criterion can control the type I error properly and are more powerful when the serial correlation exists in the first order autocorrelation. See Escanciano and Lobato (2007) for a review of the approaches to selecting autocorrelation orders. Our final recommended test combines a nonparametric weight with a parametric weight as D n = (K P,ŵ(S )) 2 ˆB wα(1 α) + (K P,w a1 )2 α 2 (1 α) 2, where ˆB w := P 1 n [ŵ 1(Y t 1 )] 2. We call the new test D n the D-test, as it is fully datadriven. The test combines the nonparametric optimal weight using Y t 1 as conditioning variable (the first term) with the parametric optimal weight, the lagged hit h t 1,α (θ 0 ) (the second term). This is a feasible test when the in-sample size R is large enough relative to the out-of-sample size P. In our simulations, we show the D-test has good finite-sample size and power properties when the ratio of P to R is as small as 0.1. To investigate the consistency properties of D n, we note that each alternative for which F Yt 1 (m α (I t 1, θ 0 )) α can be identified with a sequence of coordinates {γ j } j=1 in the expansion F Yt 1=y(m α (I t 1, θ 0 )) = γ j p j (F Y (y)), j=0 where γ 0 = α and p 0 1. Theorem 4: Let Assumptions 1-5 hold. Then, for the HS and FHS model: (i) Under the null hypothesis H 0, D n d χ 2 2, where χ 2 2 is the chi-squared random variable with two degrees of freedom. 16

17 (ii) Under the alternative hypothesis H A, D n +, provided γ j 0 for some j S max. Hence, the proposed data-driven test rejects H 0, if D n > χ 2 2,τ, where χ 2 2,τ denotes the 1 τ quantile of the chi-squared distribution with two degrees of freedom. The assumption for consistency is quite general, as there is always such j when S max is sufficiently large, as long as F Yt 1 (m α (I t 1, θ 0 )) α. Allowing S max with the sample size will lead to a consistent test for any alternative for which F Yt 1 (m α (I t 1, θ 0 )) α. In finite samples, S max is finite, so for all practical matters a theory with finite S max suffices. We expect the role of S max to be minor as long as is moderate relative to the sample size P, and we confirmed this in our simulations. A rule of thumb that we chose was S max = P 2/5, which seems to work well with the data generating processes considered in the simulations On the multiplication factors As mentioned earlier, one of the important ingredients of the determination of market risk capital requirements is the computation of the multiplication factors based on the outcomes of the backtesting procedure. According to the supervisory interpretation of backtesting results, see Basle Committee on Banking Supervision (1996b), the multiplication factors are determined by classifying the number of VaR violations in the previous 250 days, say N, into three zones. For instance, for a 5% VaR level, the multiplication factors mf t are computed as mf t = 3.0, (N 17), 4.0, if N 17, the green zone if 18 N 27, the yellow zone if N 28, the red zone, where the yellow zone begins at the point such that the probability of obtaining that number or fewer violations equals or exceeds 95%, and the red zone begins at the point such that the probability of obtaining that number or fewer violations equals or exceeds 99.99%. The currently 17

18 used factors are based on the unconditional backtest as n N := h t,α ( θ t 1 ) = P K P + P α. Here we propose an alternative computation based on the more powerful test K P,ŵ. Define N D := α 1 n P K P,ŵ + P α = α 1 ŵ1(y t 1 )(h t,α ( θ t 1 ) α) + P α. The key insight is to notice that under the optimality of HS or FHS models, N D behaves asymptotically as the number of violations N. That is, under H 0 N D = N + o P (1), n : = h t,α ( θ t 1 ) + o P (1). This is so, because w1(y t 1 ) = α under H 0. Hence, we consider the same definition of multiplication factor mf t but with N D replaced by N. 4 Monte Carlo Experiment In this section, the traditional unconditional backtest of Kupiec (K-test), the conditional backtest of Christoffersen (C-test) and the proposed new data-driven test (D-test) are compared in a set of Monte Carlo experiments. As Theorem 1 proves that the K-test is not a consistent test for evaluating optimality of the HS or FHS model forecasts, it is expected to have poor power performances in the finite samples. However, the C-test and D-test are both consistent tests for testing HS and FHS models as shown previously in the paper, and thus they are expected to have reasonably good power properties in the finite samples. In simulations, we also compare the finite sample performances of the proposed D-test with data-driven selection of S and the one with the chose value of S = 5. To implement our test, we employ a power series estimator using the Legendre polynomials (after taking the quantile transform of Y t 1 ) to estimate the nonparametric weights w1(y t 1 ). The upper 18

19 bound of the number of the base functions included in the series estimator is set to be S max = 9. Under the null hypothesis, the data generating process (DGP) is taken as iid for the HS model, so we define the null model as Y t = ε t, where ε t i.i.d.n(0, 1). For the FHS model, we define the null model as a GARCH(1,1) model: Y t = σ t η t, where η t are i.i.d. standardized student-t disturbances with degrees of freedom ν = 5 and σ t follows a GARCH(1,1) model, σ 2 t = ω + αy 2 t 1 + βσ 2 t 1, with parameters (ω, α, β) = (0.05, 0.1, 0.85). GARCH(1,1) is the simplest and most widely used specification for capturing daily financial data, see Andersen et al. (2006). Additionally, the parameter values we choose represent typical parameter values in empirical applications. To perform a power comparison, we choose the following alternatives: ALT1: RiskMetrics model: Y t = σ t ε t, σ 2 t = 0.06Y 2 t σ 2 t 1. ALT2: AR(1)-GARCH(1,1) model: Y t = 0.3Y t 1 + u t, u t = σ t ε t, σ 2 t = u 2 t σ 2 t 1. ALT3: EGARCH(1,1) model: Y t = h t ε t, ln h 2 t = ln h 2 t ( ε t 1 (2/π) 1/2 ) 0.8ε t 1. ALT4: TAR model: Y t = a t Y t 1 + ε t, a t = 0.7 1(ε t 1 < 0.5) 0.7 1(ε t 1 > 0.5). ALT5: Bilinear model (BIL): Y t = 0.7Y t 1 ε t 1 + ε t. ALT6: Exponential Autoregressive model (EXP): Y t = 0.6Y t 1 exp( 0.5Y 2 t 1) + ε t. ALT7: AR(1)-GARCH(1,1) model with Hansen s (1994) skewed t innovations denoted by ɛ t GT ( η t, λ t ) (AR-GARCH-GT): Y t = 0.5Y t 1 + v t, v t = σ t ɛ t, (15) σ 2 t = σ 2 t 1ɛ 2 t σ 2 t 1, ɛ t GT ( η t, λ t ), η t = g (2.1,30) ( η t ), λ t = g ( 0.9,0.9) ( λ t ), η t = 1 0.5Y t 1 0.1Y 2 t 1, λt = Y t 1 0.1Y 2 t 1, 19

20 where g is the logistic function, i.e. g (L,U) (x) = L + (U L)/(1 + exp( x)). In models ALT1-ALT6, ε t i.i.d.n(0, 1). Model ALT1 is the JP Morgan s (1996) RiskMetric model. Model ALT3-ALT5 are studied in Escanciano and Olmo (2009). Model ALT6 is used in Hong (2000) and model ALT7 is from Hansen (1994). For all the tests implemented, the nominal level is fixed at 5%, and the rejection rates are calculated over 1000 Monte Carlo trials. For simplicity, we employ the fixed forecast scheme for both the HS and FHS models. We use the asymptotic critical values in the simulations. In order to alleviate the impact of estimation risk on backtesting results such that our theoretical findings could be illustrated exclusively, we choose the ratio of the out-of-sample to in-sample size, P/R, as small as 0.1. The reason is that the estimation effect on backtesting results will get smaller and smaller as the ratio of P to R approaches to zero, see Escanciano and Olmo (2009). In addition, the D-test becomes feasible when R is large enough relative to P. Two different combinations of the in-sample and out-of-sample size are considered. They are (R, P ) = (2500, 250) and (R, P ) = (5000, 500), through which we will also be able to see the power performances as the sample size increases. And two VaR levels, α = 5% and 1%, are considered, respectively. Table 1 and 2 contain all the simulation results for HS and FHS models, respectively. In each table, the upper and lower panel report the results for (R, P ) = (2500, 250) and (R, P ) = (5000, 500), respectively. The empirical sizes are reported in the first row of each panel labeled as A0 HS or A0 F HS and the empirical powers for alternative models in the following rows labeled as ALT1 ALT7, respectively. The heading of each column such as 5% VaR and 1% VaR represents the level of VaRs being backtested. The main conclusions from Table 1 and Table 2 are the following: First, we see that all the tests have good size properties in the finite samples. Second, consistent with our Theorem 1, the K-test performs poorly in terms of power in finite samples. We see that the powers for the K-test are all lower than 42.1% but still higher than 5%, the nominal level, except for ALT1, the RiskMetrics model. We find that the results under the RiskMetrics model are very abnormal. In Table 1, we even observe one power is as high as 1 when R=5000 and P=500. This extreme result is because the time-varing volatilities generated from the RiskMetrics model decay very fast to zero at the 20

21 Table 1: Simulation Results for Backtesting HS Models at 5% nominal level K-test D-test D5-test C-Test 5% VaR 1% VaR 5% VaR 1% VaR 5% VaR 1% VaR 5% VaR 1% VaR R=2500, P=250 A0 HS ALT ALT ALT ALT ALT ALT ALT R=5000, P=500 A0 HS ALT ALT ALT ALT ALT ALT ALT Notes: This table compares the size and power properties of Kupiec s test, the proposed test with data-driven selection of S, the test with chosen value of S = 5 and Christoffersen s likelihood ratio test for HS models, denoted as K-test, D-test, D5-test and C-test, respectively. The nominal level for all the tests is fixed at 5% Monte Carlo replications. The upper and lower panel report the results for (R, P ) = (2500, 250) and (R, P ) = (5000, 500), respectively. In each panel, the first row displays the empirical size, labeled as A0 HS, and the other rows report the empirical powers under each alternative model, labeled as ALT1 - ALT7, respectively. Each test is implemented for two levels of VaRs, labeled as 5% VaR and 1% VaR, respectively. 21

22 Table 2: Simulation Results for Backtesting FHS Models at 5% nominal level K-test D-test D5-test C-Test 5% VaR 1% VaR 5% VaR 1% VaR 5% VaR 1% VaR 5% VaR 1% VaR R=2500, P=250 A0 F HS ALT ALT ALT ALT ALT ALT ALT R=5000, P=500 A0 F HS ALT ALT ALT ALT ALT ALT ALT Notes: This table compares the size and power properties of Kupiec s test, the proposed test with data-driven selection of S, the test with chosen value of S = 5 and Christoffersen s likelihood ratio test for FHS models, denoted as K-test, D-test, D5-test and C-test, respectively. The nominal level for all the tests is fixed at 5% Monte Carlo replications. The upper and lower panel report the results for (R, P ) = (2500, 250) and (R, P ) = (5000, 500), respectively. In each panel, the first row displays the empirical size, labeled as A0 F HS, and the other rows report the empirical powers under each alternative model, labeled as ALT1 - ALT7, respectively. Each test is implemented for two levels of VaRs, labeled as 5% VaR and 1% VaR, respectively. 22

23 beginning of the in-sample, and the use of the fixed forecast scheme as well as the large in-sample size, R=5000, result in no VaR violations in the out-of-sample. Referring to Theorem 1, σ hk = 0 for this case, so we get the power of 1. Similar reasoning would apply to the other abnormal results under the RiskMetrics model. Third, as expected, both the D-test and C-test have reasonably good power performances. And the D-test basically outperforms the C-test. Fourth, the proposed D-test with data-driven selection of S generally performs better than the one with the chosen value of S = 5, as we observe higher powers for the D-test. This confirms that our data-driven test, the D-test, is a better and natural choice. There are several other important conclusions from the tables. First, the performances of each test are generally getting better as the sample size increases. We observe that the powers in the lower panel are generally higher than those in the upper panel in each table. Second, the higher the VaR level, the better the performances by the asymptotic theory of the finite sample. We see that the results for 5% VaR are much better than those for 1% VaR, and the C-test has poor power performances in backtesting 1% VaRs, especially for FHS models. This is probably because less valid observations are involved for backtesting 1% VaR. To conclude, the K-test is not the appropriate test for detecting non-optimality of the HS or FHS forecasts due to its poor power performances in the finite sample. Comparing with the finite-sample performances of the C-test and D5-test, the new proposed data-driven test (the D-test), seems to be a better and natural backtesting method given its consistency and simplicity. Therefore, we would recommend the use of the D-test in practice, but we need to employ a relatively larger in-sample size to the out-of-sample size, for example, to choose R=2500 and P=250 as our simulation exercise does. 5 Application In the application, we aim to illustrate the practical importance of our main findings especially in the determinations of multiplication factors and market risk capital requirements. The dataset we use contains the actual daily trading revenues from five large international commercial banks. They 23

24 are data for Bank of America, Credit Suisse First Boston, Deutsche Bank, Royal Bank of Canada and data for Société Générale, denoted as BOA, CSFB, DB, RBC and SG, respectively. The data was collected and previously used by Perigon and Smith (2008, 2010). 6 In order to get a preliminary view of the data, we also present some summary statistics in Table 3 and the graph of each bank s daily trading revenues in Figure 1. Overall, trading revenues are highly volatile, right skewed, leptokurtic, moderately autocorrelated and exhibit volatility clustering. Table 3: Summary statistics for the daily trading revenues at five large commercial banks BOA CSFB DB RBC SG Mean Variance Skewness Kurtosis Bera-Jarque Test Autocorrelation ADF LB ARCH(12) Minimum Maximum Notes: This table is from Perignon and Smith (2010), which presents some summary statistics for the daily trading revenues at five large commercial banks. The summary statistics include the first four moments, minimum, and maximum of each variable, the Bera-Jarque normality test, the first-order autocorrelation coefficient, the Augmented Dickey-Fuller test (ADF), the Ljung-Box autocorrelation test using 12 lags, and the ARCH-12 test, which is a LB test applied to the squared demeaned returns. represents significance at the 5% confidence level. With the application to the same dataset, we compare the following four application results for each bank on a daily basis in the last year of the data (250 observations): (1) the p-values of the K-test, D-test and C-test; (2) the number of violations N from the K-test and its equivalent number N D from the D-test; (3) the multiplication factors determined by the backtesting results from the K-test and D-test and (4) the market risk capital requirements that only depends on VaR forecasts 7. In order to get the above results on each day in the last year of the data, we need to 6 We thank the authors for sharing their data with us. In their paper, the data was obtained by a method of extracting daily trading revenues from the graphs disclosed by the banks. For more details, see Perigon and Smith (2008, 2010). 7 Ever since Basel Committee on Banking Supervision (2009), stressed VaR is incorporated into the calculation of the capital requirement. The capital requirement that each bank must meet on a daily basis is the sum of two parts, the part that depend on VaR and the part that depends on stressed VaR. This paper only focuses on the former 24

25 Figure 1: Daily Trading Revenues Notes: This figure displays the daily trading revenues of Bank of America, Credit Suisse First Boston, Deutsche Bank, Royal Bank of Canada and Société Générale between January 1, 2001 and December 31, All values are in million and expressed in local currencies. 25

26 get daily one-day ahead VaR forecasts in the last two years and implement backtesting procedures each day in the last year. We would like to choose R=2500 and P=250 as in our simulations such that the effect of estimation risk is avoided and the proposed data-driven test becomes valid, but the available data are not enough. Therefore, we take the third year s original data from the end as the auxiliary in-sample, based on which we use the technique of block bootstrap to generate 2500 artificial observations to form a new in-sample. In this way, we get enough data that we need and also keep the dependence in the original data. The block length is chosen following Politis and White (2003). In fact, in real practice there are enough data available, so we do not have to do the above procedure to generate more data. Finally we only use the last three years data (750 observations) for each bank. We forecast 5% and 1% VaRs by the FHS method with the rolling forecast scheme following the conventional practice, and an AR(1)-GARCH(1,1) model is used to filter the data. Consistent with our simulations, we choose the nominal level at 5%. If the p-value is lower than 5%, the FHS model is rejected, otherwise, it is accepted. The results for each bank are presented in Figure 2-6, respectively. In each figure, there are four panels, which plot the p-values of all three tests, the number of violations, the multiplication factors and the market risk capital requirements that are updated on a daily basis in the last year, respectively. The conclusions from the application results are the following: Based on p-values of each test, the three tests always lead to the consistent conclusions. They always reject or accept the null at the same time. Unfortunately, this is opposite to what we expect from the application that we would get more rejections from the C-test or D-test. In fact, we do not get as many rejections from the C-test and D-test as expected. Each time we get rejection from the C-test or D-test, we also get one from the K-test. And the p-values obtained from the K-test are always lower than those from the C-test and D-test. For example, with 1% VaR level, the null that the FHS model is optimal is accepted for BOA, CSFB, DB, and SG altogether by the three tests, when p-values are higher than 20%, but rejected for RBC, when p-values are less than 5%. Only from day 1 to day 50 and from day 105 to day 124 for RBC, we get different results from the K-test and D-test, but those are rejections from the K-test and acceptances from the D-test, which part, which is expressed as the higher of its previous day s VaR and an average of the daily VaR measures on each of the preceding sixty business days, multiplied by a multiplication factor. 26