Robust Backtesting Tests for Value-at-Risk Models

Transcription

1 Robust Backtesting Tests for Value-at-Risk Models Jose Olmo City University London (joint work with Juan Carlos Escanciano, Indiana University) Far East and South Asia Meeting of the Econometric Society

2 Outline Differences between a specification test for the quantile process and backtesting. Model risk in unconditional and independence Backtesting Exercises A Subsampling solution Monte Carlo analysis (Asymptotic theory vs Block bootstrap and Subsampling) Application to risk management in financial data Conclusions Far East and South Asia Meeting of the Econometric Society

3 Differences between Specification Tests and Backtesting Value-at-Risk: We say that m α (W t 1, θ) is a correctly specified α-th conditional VaR of Y t given W t 1 = {Y s, Z s} t 1 s=t h, h <, if and only if P (Y t m α (W t 1, θ 0 ) F t 1 ) = α, almost surely (a.s.), α (0, 1), t Z, (1) for some unknown parameter θ 0 Θ, with Θ a compact set in an Euclidean space R p and F t 1 denoting the set of available information. This condition is equivalent to the following in terms of conditional expectations: E[I t,α (θ 0 ) F t 1 ] = α, almost surely (a.s.), α (0, 1), t Z, (2) with I t,α (θ) := 1(Y t m α (W t 1, θ)) the so called hits or exceedances associated to the model m α (W t 1, θ 0 ). Far East and South Asia Meeting of the Econometric Society

4 This condition implies the so-called joint hypothesis (cf. Christoffersen (1998)), {I t,α (θ 0 )} are iid Ber(α) random variables (r.v.) for some θ 0 Θ, (3) where Ber(α) stands for a Bernoulli r.v. with parameter α. In the risk management literature Backtesting is identified with this joint hypothesis: Unconditional Backtesting test: E[I t,α (θ 0 )] = α, (4) and Independence Backtesting test: {I t,α (θ 0 )} being iid. (5) Motivation: The joint test E[I t,α (θ 0 ) F t 1 ] = α and the two tests above are not equivalent. In fact, the latter two tests do not imply necessarily the former. Far East and South Asia Meeting of the Econometric Society

5 Example 1. (Unconditional coverage test): Suppose that the true model for conditional VaR is with F Wt 1 (x) := P {Y t x I t 1 }. m α (W t 1, θ 0 ) = F 1 W t 1 (α), (6) The researcher proposes, instead, an unconditional quantile: m α (W t 1, θ 0 ) = F 1 (α), for all t, (7) with F 1 ( ) the inverse function of the unconditional distribution function F. Note from these equations that E[1(Y t m α (W t 1, θ))] = F (F 1 (α)) = α. Far East and South Asia Meeting of the Econometric Society

6 However, the unconditional quantile is misspecified to assess the conditional quantile unless F 1 W t 1 (α) = F 1 (α) a.s. Note: The popular Historical simulation (HS) VaR falls in this group of unconditional risk models. Note that in the HS case the unconditional VaR quantile is estimated nonparametrically from the empirical distribution function F n : m α (W t 1, θ 0 ) = F 1 n (α), for all t, (8) and therefore implies that this method exhibits model risk effects. Message: This method to quantify risk based on the unconditional α quantile can be successful in passing the backtesting unconditional coverage test, being however a wrong specification of the true conditional quantile process. Far East and South Asia Meeting of the Econometric Society

7 Example 2. (Independence test): Suppose that {Y t } is an autoregressive model given by Y t = ρ 1 Y t 1 + ε t, (9) with ρ 1 < 1, and ε t an error term serially independent that follows a distribution function F ε ( ). Then m α (W t 1, θ 0 ) = ρ 1 Y t 1 + Fε 1 (α), with Fε 1 (α) the α quantile of F ε. Suppose now that the researcher assumes incorrectly an alternative autoregressive process where the error term follows a Gaussian distribution function (Φ( )). The conditional (misspecified) quantile process is with Φ 1 ε (α) Fε 1 (α). m α (W t 1, θ 0 ) = ρ 1 Y t 1 + Φ 1 ε (α), Far East and South Asia Meeting of the Econometric Society

8 Note that although the risk model is wrong: m α (W t 1, θ 0 ) m α (W t 1, θ 0 ), the sequence of hits associated to m α (W t 1, θ 0 ) are iid since P {ε t Φ 1 ε (α), ε t 1 Φ 1 ε by independence of the error term. (α)} = P {ε t Φ 1 ε (α)}p {ε t 1 Φ 1 ε (α)}, Far East and South Asia Meeting of the Econometric Society

9 Implementation of Backtesting The test statistic introduced by Kupiec (1995) for the unconditional coverage hypothesis is K P K(P, R) := 1 P n t=r+1 (I t,α (θ 0 ) α), with R being the in-sample size and P the corresponding out-of-sample testing period. In practice, however, the test statistic implemented is S P S(P, R) := 1 P n t=r+1 (I t,α ( θ t 1 ) α), (10) with θ t 1 an estimator of θ 0 satisfying certain regularity conditions (cf. A4 below). Far East and South Asia Meeting of the Econometric Society

10 Estimation and Model Risk I: Assumptions Assumption A1: {Y t, Z t } t Z is strictly stationary and strong mixing process with mixing coefficients satisfying j=1 (α(j))1 2/d <, with d > 2 as in A4. Assumption A2: The family( of distributions functions {F x,) x R dw } has Lebesgue densities {f x, x R dw } that are uniformly bounded sup x R dw,y R f x(y) C and equicontinuous: for every ɛ > 0 there exists a δ > 0 such that sup x R dw, y z δ f x(y) f x (z) ɛ. Assumption A3: The model [ m α (W t 1, θ) is continuously differentiable in θ (a.s.) g α (W t 1, θ) such that E sup θ Θ0 g α (W t 1, θ) 2] < C, for a neighborhood Θ 0 of θ 0. with derivative Assumption A4: The parameter space Θ is compact in R p. The true parameter θ 0 belongs to the interior of Θ. The estimator θ t satisfies the asymptotic expansion θ t θ 0 = H(t) + o P (1), where H(t) is a p 1 vector such that H(t) = t 1 t s=1 l(y s, W s 1, θ 0 ), R 1 t s=t R+1 l(y s, W s 1, θ 0 ) and R 1 R s=1 l(y s, W s 1, θ 0 ) for the recursive, [ rolling and fixed schemes, respectively. Moreover, l(y t, W t 1, θ) is continuous (a.s.) in θ in Θ 0 and E l(y t, W t 1, θ 0 ) d] < for some d > 2. Assumption A5: R, P as n, and lim n P/R = π, 0 π <. Far East and South Asia Meeting of the Econometric Society

11 Estimation and Model Risk II: Decomposition Under the above assumptions Escanciano and Olmo (2008) showed that S P = 1 P n t=r+1 [ It,α (θ 0 ) F Wt 1 (m α (W t 1, θ 0 )) ] (11) +E [ g α(w t 1, θ 0 )f Wt 1 (m α (W t 1, θ 0 )) ] 1 H(t 1) P t=r+1 }{{} n Estimation Risk + 1 [ FWt 1 (m α (W t 1, θ 0 )) α ] P t=r+1 }{{} Model Risk n + o P (1), where g α (W t 1, θ) is the derivative of m α (W t 1, θ) with respect to θ. Far East and South Asia Meeting of the Econometric Society

12 Estimation and Model Risk III: Theorem Theorem 1: Under Assumptions A1-A5 and E[I t,α (θ 0 )] = α, S P d N(0, σ 2 a), with σa 2 = j= E[a ta t j ] + λ al (A j= E[a tl t j ] + j= E[a tl t j ] A ) + λ ll A j= E[l tl t j ]A, where Scheme λ al Recursive 1 π 1 ln(1 + π) λ ll 2 [ 1 π 1 ln(1 + π) ] Rolling, π 1 π/2 π π 2 /3 Rolling, 1 < π < 1 (2π) 1 1 (3π) 1 Fixed 0 π (12) and A = E [ g α(w t 1, θ 0 )f Wt 1 (m α (W t 1, θ 0 )) ] 1 P n t=r+1 H(t 1) and l(y s, W s 1, θ 0 ) is the score function. Far East and South Asia Meeting of the Econometric Society

13 Independence Tests I: test statistics To complement backtesting exercises one can be interested in testing the following {I t,α (θ 0 )} n t=r+1 are iid. (13) Since for Bernoulli random variables serial independence is equivalent to serial uncorrelation, it is natural to build a test for (13) based on the autocovariances γ j = Cov(I t,α (θ 0 ), I t j,α (θ 0 )) j 1. (14) They can be consistently estimated (under E[I t,α (θ 0 )] = α) by either of these tests: (Joint Test) γ P,j = 1 P j n t=r+j+1 (I t,α (θ 0 ) α)(i t j,α (θ 0 ) α) j 1. Far East and South Asia Meeting of the Econometric Society

14 (Marginal Test) ζ P,j = 1 P j n t=r+j+1 where, for θ Θ, E n [I t,α (θ)] = 1 P j (I t,α (θ 0 ) E n [I t,α (θ 0 )])(I t j,α (θ 0 ) E n [I t j,α (θ 0 )]), { n t=r+j+1 I t,α (θ) In practice, however, joint or marginal tests for (13) need to be based on estimates of the relevant parameters, such as }. or γ P,j = 1 P j n t=r+j+1 (I t,α ( θ t 1 ) α)(i t j,α ( θ t j 1 ) α) ζ P,j = 1 P j n t=r+j+1 (I t,α ( θ t 1 ) E n [I t,α ( θ t 1 )])(I t j,α ( θ t j 1 ) E n [I t j,α ( θ t j 1 )]). Far East and South Asia Meeting of the Econometric Society

15 Independence Tests II: Theorem Theorem 2: Under Assumptions A1-A5 and the joint test, for any j 1, γ P,j d N(0, σ 2 b ), where σ 2 b = E[b2 t] + λ al (B j= E[b tl t j ] + j= E[b tl t j ] B ) + λ ll B j= E[l tl t j ]B, and B = E [ g α(w t 1, θ 0 )f Wt 1 (m α (W t 1, θ 0 )){I t j,α (θ 0 ) + α} ] and b t = (I t,α (θ) α)(i t j,α (θ) α). If instead of the joint hypothesis, only (13) holds, then ζ P,j d N(0, σ 2 c), where σ 2 c = E[c 2 t] + λ al (C j= E[c tl t j ] + j= E[c tl t j ] C ) + λ ll C j= E[l tl t j ]C, with C = E [ g α(w t 1, θ 0 )f Wt 1 (m α (W t 1, θ 0 )){I t j,α (θ 0 ) + E[I t,α (θ 0 )]} ] and c t = (I t,α (θ) E[I t,α (θ)])(i t j,α (θ) E[I t j,α (θ)]). Far East and South Asia Meeting of the Econometric Society

16 Subsampling approximation: Known parameters Let T P (θ 0 ) = T P (X R+1,..., X n ; θ 0 ), be the relevant test statistic and G P (w) the corresponding cumulative distribution function: G P (w) = P (T P (θ 0 ) w). Let T b,i (θ 0 ) = T b (X i,..., X i+b 1 ; θ 0 ), i = 1,..., n b+1, be the test statistic computed with the subsample (X i,..., X i+b 1 ) of size b. Subsampling: One can approximate the sampling distribution G P (w) using the distribution of the values of T b,i (θ 0 ) computed over the n b+1 different consecutive subsamples of size b. This subsampling distribution is defined by G P,b (w) = 1 n b + 1 n b+1 i=1 1(T b,i (θ 0 ) w), w [0, ). and let c P,1 τ,b be the (1 τ)-th sample quantile of G P,b (w), i.e., c P,1 τ,b = inf{w : G P,b (w) 1 τ}. Far East and South Asia Meeting of the Econometric Society

17 Subsampling approximation: Theorem Theorem 3: Assume A1-A5 and that b/p 0 with b as P. Then, for T P (θ 0 ) any of the tests statistics K P, γ P,j or ζ P,j, we have that (i) under the joint null hypothesis, c P,1 τ,b P c1 τ and P (T P (θ 0 ) > c P,1 τ,b ) τ; (ii) under the marginal null hypothesis of unconditional coverage probability, c P,1 τ,b P c1 τ and (iii) under any fixed alternative hypothesis, P (T P (θ 0 ) > c P,1 τ,b ) τ; P (T P (θ 0 ) > c P,1 τ,b ) 1. Far East and South Asia Meeting of the Econometric Society

18 Subsampling approximation: Unknown parameters The subsampling distribution of the relevant test statistics, now S P, γ P,j or ζ P,j, is G P,b(w) = 1 n b + 1 n b+1 i=1 1( T b,i w), w [0, ), with T b,i the subsample version of the test statistic, computed with data X i,..., X i+b 1. Under the presence of estimation risk we divide each subsample X i,..., X i+b 1 in insample and out-of-sample observations according to the ratio π used for computing the original test: R b observations are used for in-sample and P b observations for out-of-sample, such that R b + P b = b and lim b P b /R b = π. The computation of T b,i is done as with the original test, say T P, but replacing the original sample by X i,..., X i+b 1 and R and P by R b and P b. Far East and South Asia Meeting of the Econometric Society

19 Theorem 4: Assume A1-A5 and that b/p 0 with b as P. Moreover, assume that lim b P b /R b = lim n P/R = π. Then, for T P any of the tests statistics S P, γ P,j or ζ P,j, and, we have that (i) under the joint null hypothesis, c P,1 τ,b P c 1 τ and P ( T P > c P,1 τ,b ) τ; (ii) under the marginal null hypothesis of unconditional coverage probability, c P,1 τ,b P c 1 τ and (iii) under any fixed alternative hypothesis, P ( T P > c P,1 τ,b ) τ; P ( T P > c P,1 τ,b ) 1. Far East and South Asia Meeting of the Econometric Society

20 Monte Carlo Experiment: Unconditional Backtesting test Data generating process: Y t = σ t ε t, with ε t N(0, 1) Φ( ), and where β 0 = 0.05, β 1 = 0.10 and β 2 = σ 2 t = β 0 + β 1 Y 2 t 1 + β 2 σ 2 t 1, (15) Model 1: True conditional VaR process (Correct specification). m α,1 (W t 1, β) = σ t Φ 1 (α), (16) with Φ 1 (α) the α quantile of the standard Gaussian distribution function. Comments: The asymptotic distribution of the unconditional coverage test is N(0, α(1 α)). Subsampling and block bootstrap methods are in this case valid alternatives to approximate the finite-sample distribution of the test statistic. Far East and South Asia Meeting of the Econometric Society

21 Model 2 : Estimated version of the true VaR process. m α,2 (W t 1, β) = σ t Φ 1 (α), (17) with β = ( β 0, β 1, β 2 ), and σ 2 t = β 0 + β 1 Y 2 t 1 + β 2 σ 2 t 1 the estimated parameters obtained from the in-sample size (R=1000 in-sample size to estimate the process, and P=500 out-of-sample evaluation period). Problem: Estimation effects. The asymptotic distribution is Gaussian but the variance is different from α(1 α). It depends on a nonlinear manner on the asymptotic ratio P/R and the values of the parameter vector β. Solutions: Subsampling and block bootstrap versions of the test account for estimation effects. Far East and South Asia Meeting of the Econometric Society

22 Model 3: Historical Simulation method. m α,3 (W t 1, β) = 1 F R (α), (18) with F R the empirical distribution function of the R = 1000 in-sample data. Problem: This method can potentially exhibit very strong estimation and model risk effects. Therefore, the asymptotic critical values of the unconditional backtesting test is not N(0, α(1 α)). The size of the test using these critical values will be very distorted under general situations. Comment: simplicity. This method is widely used by practitioners in the industry due to its Solutions: Subsampling and block bootstrap versions of the test account for estimation and model risk effects and are, in theory, able to approximate the appropriate nominal sizes. Far East and South Asia Meeting of the Econometric Society

23 Simulations of Size Φ( ) α = 0.10 α = 0.05 α = 0.01 P = 500/size K asy,1 P K bb,1 P K ss,1 P P = 500/size S asy,2 P S bb,2 P S ss,2 P P = 500/size S asy,3 P S bb,3 P S ss,3 P Table 1. Unconditional backtesting test for models (16), (17) and (18). R = 1000 and P = 500. Coverage probability α = 0.10, 0.05, b = kp 2/5 with k = Monte-Carlo replications. Far East and South Asia Meeting of the Econometric Society

24 Monte Carlo Experiment: Independence test Data generating process: Y t = σ t ε t, with ε t Φ( ), and σ 2 t = β 0 + β 1 Y 2 t 1 + β 2 σ 2 t 1, where β 0 = 0.05, β 1 = 0.10 and β 2 = Note that γ P,j is a joint test (unconditional and independence). If E[I t,α (θ 0 )] = α this test does not exhibit model neither estimation risk, its asymptotic distribution is N(0, α 2 (1 α) 2 ). ξ P,j, does not exhibit model risk, its asymptotic distribution is N(0, α 2 (1 α) 2 ). The estimated version of these test statistics, γ P,j and ξ P,j, however exhibit estimation risk. Solution: subsampling and block bootstrap methods. In the simulations below we only study γ P,1 and ξ P,1 in terms of size. Far East and South Asia Meeting of the Econometric Society

25 Simulations of Size Φ( ) α = 0.10 α = 0.05 α = 0.01 P = 500/size γ asy,1 P, γ bb,1 γ ss,1 P, P, P = 500/size ξ asy,2 P, ξ bb,2 P, ξ ss,2 P, Table 2. Empirical size for independence backtesting test for models (16) and (17). R = 1000 and P = 500. Coverage probability α = 0.10, 0.05, b = kp 2/5 with k = Monte-Carlo replications. Far East and South Asia Meeting of the Econometric Society

26 Application: Backtesting performance of VaR forecast models The models under study are Hybrid method: m α,1 (W t 1, θ t 1 ) = σ t F 1 R, ε (α), (19) with F R, ε the empirical distribution function constructed from an in-sample size of R observations, and σ t 2 = β 0 + β 1 Yt β 2 σ t 1, 2 the estimated GARCH process, where ( β 0, β 1, β 2 ) is the vector of estimates of the parameter vector (β 0, β 1, β 2 ) and { ε t } is the residual sequence. Far East and South Asia Meeting of the Econometric Society

27 Gaussian GARCH model: m α,2 (W t 1, θ t 1 ) = σ t Φ 1 (α). (20) Student-t GARCH model: m α,3 (W t 1, θ t 1 ) = σ t t 1 ν (α). (21) Riskmetrics model: m α,4 (W t 1, θ t 1 ) = σ t Rm Φ 1 (α), (22) with ( σ t Rm ) 2 = β 1 Yt (1 β 1 )( σ t 1) Rm 2. Far East and South Asia Meeting of the Econometric Society

28 Design of Backtesting Exercise A backtesting exercise for daily returns on Dow Jones Industrial Average Index over the period 02/01/1998 until 23/05/2008 containing 2500 observations. Use a fixed forecasting scheme for estimating the relevant empirical distribution functions F R and the GARCH parameters, and where the in-sample size R = 1000 is considerably greater than the out-of-sample size P = 500. The choice of this sample size is a compromise between absence of estimation risk effects (P/R < 1) and meaningful results of the subsampling and asymptotic tests (P sufficiently large). We repeat this experiment using all the information of the time series available considering rolling windows of 250 observations and giving a total of five different periods where computing backtesting tests. The rejection regions at 5% considered for the unconditional coverage and independence tests are those determined by the corresponding asymptotic normal distributions and the alternative subsampling approximations. Far East and South Asia Meeting of the Econometric Society

29 Unconditional test: Hybrid Method Backtesting test Sn at 5% for HS GARCH. k=8. Sn Subsampling Asymptotic out of sample rolling period Far East and South Asia Meeting of the Econometric Society

30 Unconditional test: Gaussian GARCH(1,1) Method Backtesting test Sn at 5% for Gaussian GARCH. k=8. Sn Subsampling Asymptotic out of sample rolling period Far East and South Asia Meeting of the Econometric Society

31 Unconditional test: Student-t GARCH(1,1) Method Backtesting test Sn at 5% for Student GARCH. k=8. Sn Subsampling Asymptotic out of sample rolling period Far East and South Asia Meeting of the Econometric Society

32 Unconditional test: Riskmetrics Method Backtesting test Sn at 5% for Rismetrics. k=8. Sn Subsampling Asymptotic out of sample rolling period Far East and South Asia Meeting of the Econometric Society

33 Comments on the plots We observe the differences between the subsampling rejection regions and the asymptotic ones for the first three methods based on GARCH estimates, and the similarities of the Riskmetrics approach with the asymptotic intervals. Therefore, whereas we reject the VaR forecasting methods obtained from using GARCH models we do not do it if Riskmetrics is employed. The choice of the asymptotic critical values is misleading most of the times since one can accept VaR models that are wrong if no attention is paid to the misspecification effects. We observe an increase in the uncertainty in all of the unconditional backtesting tests that use subsampling. Far East and South Asia Meeting of the Econometric Society

34 Independence test: Hybrid Method Dependence backtesting test \hat{\xi}_n at 5% for HS GARCH. k=8. Sn Subsampling Asymptotic out of sample rolling period Far East and South Asia Meeting of the Econometric Society

35 Independence test: Gaussian GARCH(1,1) Method Dependence backtesting test \hat{\xi}_n at 5% for Gaussian GARCH. k=8. Sn Subsampling Asymptotic out of sample rolling period Far East and South Asia Meeting of the Econometric Society

36 Independence test: Student-t GARCH(1,1) Method Dependence backtesting test \hat{\xi}_n at 5% for Student GARCH. k=8. Sn Subsampling Asymptotic out of sample rolling period Far East and South Asia Meeting of the Econometric Society

37 Independence test: Riskmetrics Method Dependence backtesting test \hat{\xi}_n at 5% for Riskmetrics. k=8. Sn Subsampling Asymptotic out of sample rolling period Far East and South Asia Meeting of the Econometric Society

38 Conclusions Backtesting techniques are statistical tests designed, specially, to uncover an excessive risk-taking from financial institutions and measured by the number of exceedances of the VaR model under scrutiny. Econometric methods that are well specified in-sample for describing the dynamics of extreme quantiles are not necessarily those that best forecast their future dynamics. Therefore, financial institutions can choose risk models for forecasting conditional VaR that although badly specified they succeed to satisfy unconditional and independence backtesting requirements. In order to implement correctly the standard backtesting procedures one needs to incorporate in the asymptotic theory certain components accounting for possible misspecifications of the risk model. A feasible alternative to the complicated asymptotic theory is resampling methods: block bootstrap and subsampling. Far East and South Asia Meeting of the Econometric Society

39 Our simulations indicate that values of b = kp 2/5 determined by k > 5 provide valid approximations of the true sampling distributions and hence of the true rejection regions for backtesting tests. We have shown that although estimation risk can be diversified by choosing a large in-sample size relative to out-of-sample, model risk cannot. Model risk is pervasive in unconditional backtests. Far East and South Asia Meeting of the Econometric Society