The Analysis of Power for Some Chosen VaR Backtesting Procedures - Simulation Approach

Transcription

1 The Analysis of Power for Some Chosen VaR Backtesting Procedures - Simulation Approach Krzysztof Piontek Department of Financial Investments and Risk Management Wroclaw University of Economics ul. Komandorska 118/120, Wroclaw, Poland krzysztof.piontek@ue.wroc.pl Summary. Everyone who measures the market risk using the Value at Risk (VaR) approach should test if the assumed model is correct. This procedure is called backtesting. There are many different tests available, but usually risk managers are not concerned about their power. The aim of this paper is to analyze some chosen backtesting methods focusing on the problem of power of the tests and limited data sets. The paper is organized as follows. At the beginning a financial aspect of the analyzed problem is presented very briefly. The second part gives information about some chosen, but (in the author s opinion) the most popular backtests. The main attention is paid to tests based on the frequency of failures and on multiple VaR levels. Next, the results of the simulations are presented. The last part summarizes obtained results and gives hints for the optimal backtesting. Key words: Risk measurement, Value at Risk, Backtesting, Power of tests 1 Introduction Value at Risk is one of the most popular risk measures used by financial institutions. Since a VaR model constitutes a background for estimating the total capital requirement (TCR), choosing the proper model is a task of a great importance [4]. The definition of VaR is, however, quite general [4, 7]. Value at Risk is such a loss in market value of a portfolio that the probability that it occurs or is exceeded over a given time period is equal to a prior defined tolerance level q. The sense of this definition is presented below: P (W W 0 V ar (q)) = q, (1) where: W 0 is a value of a financial instrument (portfolio) at present, W is a value at the end of the investment horizon (random variable) and q is a prior defined tolerance level.

2 2 Krzysztof Piontek Usually, econometric models are not expressed in terms of values but in terms of returns. Thus, VaR is often defined in a relative interpretation, as a conditional or unconditional quantile of the forecasted return distribution: P ( r t F 1 r,t (q) ) = q, V ar r,t (q) = F 1 r,t, (2) where r t is a rate of return (periodic or logarithmic) and Fr,t 1 is a quantile of loss distribution related to the probability of 1 q [7]. However, this definition does not inform in what way a VaR measure should actually be estimated. Because of it, there are many approaches which can give different values. The most popular methodologies for calculating VaR are [4]: the historical simulation, stochastic (Monte Carlo) simulation, variance-covariance approach, group of methods using the quantile of non-normal distribution, extreme-value-theory approach. The risk managers, however, never know a priori, which approach or model will be the best or even correct, and should use several models and then backtest them. The validation of risk models, during the backtesting procedure, should be the critical issue in the acceptance of internal models. Backtesting is an ex-post comparison of a risk measure generated by a risk model against actual changes in portfolio value over a given period of time, both to evaluate a new model and to reassess the accuracy of existing models [4]. Nowadays, in the author s opinion, the challenge is not to suggest a new method of VaR measuring but to distinguish between correct and incorrect models. Although the issue is important, no single backtesting technique has been established until now. The most popular tests for validation of VaR models can be classified into three groups [7, 3, 8]: 1. those based on the frequency of failures (Kupiec, Christoffersen) [4, 5], 2. those based on the adherence of a VaR model to the asset return distribution (on multiple VaR levels) (Crnkovic-Drachman, Berkowitz) [1], 3. those based on various loss functions (Lopez, Sarma) [6]. In the further part of this paper, the empirical examination is carried out only for some methods from the first and second group. If one is interested in an automization of backtesting, the third approach is probably the least suited and not so popular as others. For this reason this group is not analyzed in the empirical part of this work.

3 The Analysis of Power For Some Chosen VaR Backtesting Procedures 3 2 Tests Based on the Frequency of Failures The most popular tool for validation of VaR models (for the length of the backtesting time period equal to T units) is the failure (or hit) process [I t (q)] t=t t=1. The hit function is defined as follows [5, 4]: 1; r p,t Frp,t(q) 1 if a violation occurs I t (q) = (3) 0; r p,t > Frp,t(q) 1 if no violation occurs and tallies the history of whether or not the exceptions have been realized. Kupiec proposed the proportion-of-failures test, which is probably the most common one. Here it is examined how many times a financial institutions VaR is violated over a given span of time [5, 4, 3, 8, 7]. The null hypothesis for this test is that the empirically determined probability matches the given tolerance level of VaR: H 0 : ˆq = q (4) The test statistic is based on the likelihood ratio given by the formula (5) and it is asymptotically chi-square distributed with one degree of freedom. ( (1 q) T 0 ) q T1 LR P OF = LR uc = 2 ln χ 2 (1 ˆq) T0 ˆq T1 1 (5) where: ˆq = T 1 T 0 + T 1, T 1 = T t=1 I t(q), T 0 = T T 1. (6) If the VaR model is reliable, the exceptions should not follow any pattern, such as clustering, for example [4, 2, 3]. The most popular method for examining the independence of exceptions is the Christoffersen test [4, 7], given by the formula : where: ( ) (1 q) T00+T10 q T01+T11 LR ind = 2 ln (1 ˆq 01 ) T00 ˆq T01 01 (1 ˆq χ 2 1, (7) 11) T10 ˆq T11 11 ˆq ij = T ij, q = T 01 + T 11 T i0 + T i1 T 00 + T 01 + T 10 + T, (8) 11 and T ij is a number of i values followed by a j value in the hit series. For the null hypothesis the test statistic is based on the likelihood ratio of serial independence against the alternative of the first order Markov dependence. This test examines whether or not the likelihood of VaR violations depends on whether or not a VaR violation occurred on the previous day: H 0 : ˆq 01 = ˆq 11 = q. (9)

4 4 Krzysztof Piontek Although Christoffersen criticizes first order Markovian process as a limited alternative compared to other forms of clustering [7], the presented approach is easy to implement and, because of this, it is still the most popular one. It is also independent of the frequency-of-failures. It is important to recognize that the unconditional coverage and independence property of the hit sequence are separate and distinct and must both be satisfied by a correct VaR model [4]. The test that jointly examines the both properties has been proposed by Christoffersen and it is usually called the mixed test [4, 2, 7]: LR (1) mix = LR uc + LR ind χ 2 2. (10) Some discussions about joint tests might seem to suggest that joint tests are universally preferable to test of either the unconditional coverage property or independence property. But this is not true. The fact that one property is satisfied makes it more difficult for a joint test to detect the other inadequacy of the VaR measure. 3 Tests Based on Multiple VaR Levels All the backtests that have been discussed in Section 2 have focused on examining the behaviour of the hit sequence. Despite the fact that the hit function plays the prominent role in a variety of backtesting procedures, the information contained in the hit sequence is limited. There is no need to restrict attention to a single VaR level. The unconditional coverage and independence properties of correct VaR model should hold for any tolerance level q, so backtest procedures based on multiple VaR levels have been also suggested [1, 8].They examine the deviation of the empirical return distribution from the theoretical model distribution [7]. Usually [1], the observed portfolio returns r t are transformed into a series u t (see Eq. (11)), where F denotes the ex ante forecasted return distribution function (conditional or not). u t = F (r t ) = rt If the model is well calibrated it is expected that: f(y)dy (11) the series u t should be uniformly distributed over the unit interval [0,1] (this property is a direct parallel to the unconditional coverage), the series should be independently distributed (it is analogous to the statement that VaR violations should be independent from each other). These two properties are often combined into the single statement: u t i.i.d.u(0, 1).

5 The Analysis of Power For Some Chosen VaR Backtesting Procedures 5 A wide variety of tests using these conditions have been suggested. Some of them are based on the distance between the observed u t series distribution and the theoretical uniform distribution [7, 3]. However, the approach which becomes more and more popular is the transformation of u t series into z t series based on the inverse normal distribution function Φ 1 ( ): z t = Φ 1 (u t ). (12) Under the null hypothesis that the VaR model is correct, the z t series should be independent and identically distributed standard normal random values. Now it is easy to construct a quite powerful likelihood ratio tests [1, 8]. z t µ = ρ(z t 1 µ) + ε t, var(ε t ) = σ 2 (13) H 0 : (µ, σ, ρ) = (0, 1, 0) (14) A restricted likelihood can be evaluated and compared to an unrestricted one for testing analyzed properties. The properties of unconditional coverage and independence may be tested separately or jointly. Under the null hypotheses these test statistics are distributed like a chi-square with a corresponding number of degrees of freedom: LR b uc = 2 [LLF (ˆµ, ˆσ, ˆρ) LLF (0, 1, ˆρ)] χ 2 2, (15) LR b ind = 2 [LLF (ˆµ, ˆσ, ˆρ) LLF (ˆµ, ˆσ, 0)] χ 2 1, (16) LR b mix = 2 [LLF (ˆµ, ˆσ, ˆρ) LLF (0, 1, 0)] χ 2 3. (17) Backtesting Errors Tests Presented before are, usually, used for evaluating internal VaR models developed by financial institutions. One should be, however, aware of the fact that two types of errors can occur: a correct model can be rejected or a wrong one may be not rejected [4]. All tests are designed for controlling the probability of rejecting the VaR model when the model is correct. It means that the type I error is known. This type of wrong decisions leads to the necessity of searching for another model, which is just wasting time and money. But the type II error (acceptance of the incorrect model) is a severe misjudgement because it can result in the use of an inadequate VaR model that can lead to substantial negative consequences. Performance of the selected tests needs to be analyzed with concern to the type II error, in order to select the best one for different (but small) numbers of observations and model misspecifications.

6 6 Krzysztof Piontek 4 Empirical Research - Simulation Approach For evaluating the power of the tests it is necessary that the properties of the asset return generation process are well known. This data generating process can differ from the probability distribution of the assumed VaR model. We assume that the generated returns follow standardized Student distribution with the number of degrees of freedom between about 3 and 25. The returns can be independent of each other or not. The VaR model is also based on the standardized Student distribution, but the number of degrees of freedom is equal to 6. So, it can be incorrect, which leads to the unconditional coverage and independence inaccuracy. On this ground we can test the power of backtests. The data series of the length of 100, 250, 500, 750 and 1000 observations were simulated. For each kind of inaccuracy of the model and for each of the specified lengths of the data series Monte Carlo simulations with draws were done. It allowed for calculating test statistics and for estimation of the frequency at which the null hypotheses were rejected for incorrect models. The last may be treated as an approximation of the test power. Tab. 1 and Tab. 2 present the summary results of the Kupiec approach. The central column (for 0.05) represents the type I error, other columns - the power of the test for given strength of inaccuracy (incorrect frequency of failures). Table 1. Kupiec test, q = 0.05, α = 0.05, CV based on χ Table 2. Kupiec test, q = 0.05, α = 0.05, CV based on simulations When we use the chi-square distribution for determining the critical value (CV) for the typical length of series we can be wrong because of two reasons: the assumption of asymptotic convergence of the test statistic is not met

7 The Analysis of Power For Some Chosen VaR Backtesting Procedures 7 and the test statistic is not continuous but discrete. We have to note this. However, as it has been checked, it did not make a big difference if chi-square or simulated critical values were used for the tolerance level of 5%. What we see is that the power of the test is rather low. For example, in the case with 250 observations, an inaccurate model giving 3% or 7% of violations, instead of 5%, was rejected only in about 35% of draws. So, in 65% of cases we did not reject the wrong model at 5% significance level. For the tolerance level of 1% the results are presented in Tab. 3 and Tab. 4. It turns out that the asymptotic property of the test statistic might be a serious problem. We obtain some untypical and unexpected values of type I error and incorrect results of power of test (bolded in the tables). Table 3. Kupiec test, q = 0.01, α = 0.05, CV based on χ Table 4. Kupiec test, q = 0.01, α = 0.05, CV based on simulations If we use the critical values obtained by simulations we observe that the power of the test is getting even worse. But regardless of that - the results indicate that the test based on failure proportion is not adequate for small samples and even for 1000 observations. Even if we observe 40% more or less exceptions than we expect, the power of the test is about 20-25%. It is very bad news. There is a significant probability of not rejecting the null hypothesis when it is false. The Kupiec test should not be used for VaR models with tolerance level of 1% for typical length of the observed series. The power of the Berkowitz test for unconditional coverage was also examined. The results are presented in the Tab. 5 and Tab. 6.

8 8 Krzysztof Piontek Table 5. Berkowitz test (unconditional), q = 0.05, α = 0.05, CV based on χ Table 6. Berkowitz test (unconditional), q = 0.01, α = 0.05, CV based on χ The power of the Berkowitz test for the tolerance level of 5% is higher comparing to the Kupiec test, but only for the longer series and stronger inaccuracies the power of the Berkowitz test could be acceptable for risk managers. For VaR tolerance level of 1%, again, the Berkowitz test has a higher power than the Kupiec test, however, the power of this test is, in author s opinion, not sufficient for risk managers for these typical lengths of data series. We can summarize that: the more incorrect VaR models - the bigger superiority of the Berkowitz test against the Kupiec test, for the VaR tolerance level equal to 0.05 the superiority of the Berkowitz test can be observable for all lengths of the return series, for the VaR tolerance level of 0.01 the superiority of the Berkowitz test can be observable for the series length of 750 and 1000 observations, for the shorter series the conclusions are ambiguous. We examine both Christoffersen and Berkowitz approaches for testing also the simple independence property (see Eq. (7) and (16)). Now the frequency of failures is correct for all series, but the exceptions are not independent. For example, for the first case in the Tab. 7: P (I t = 1) = 0.05 P (I t+1 = 1 I t = 1) = or Some chosen results are presented in the Tab. 7. In the first distinguished case, the power of test is low, but for all other cases it is low in an unacceptable way. The Berkowitz test has a low power as a test of exception independence,

9 The Analysis of Power For Some Chosen VaR Backtesting Procedures 9 which makes it inadequate for this range of applications. The Christoffersen test can be only used for the distinguished tolerance level of 5%. Table 7. Results for simple Christoffersen and Berkowitz tests of independence Christoffersen, q= Berkowitz, q= Christoffersen, q= Berkowitz. q= Finally, we examine the power of the tests if both unconditional coverage and independence properties are not met. The correct model is still given by: P (I t = 1) = 0.05 P (I t+1 = 1 I t = 1) = The results for the chosen case when: P (I t = 1) = 0.04 P (I t+1 = 1 I t = 1) = 0.02 or are presented in the Tab. 8. Table 8. Chosen results for mixed inadequacies of the model 0.04 C1 C2 C3 C4 Berkowitz Christoff. B+C mixed B+C separ The columns C1 and C2 show the results for simple tests, the column C3 the results for the mixed test, when the both test statistics are summed, and

10 10 Krzysztof Piontek the last column C4 presents results for tests used separately, which means that the correct model has to be not rejected by the both tests. We observe that it is the best way for increasing the probability of rejecting the incorrect model. The examinations were done for different strength of inaccuracies. For each incorrect unconditional and conditional probability the conclusions based on the obtained results are the same. Because of a limited content of this paper, tables with other, less illustrative, results are not presented here. 5 Some final conclusions For tolerance level of 5% the best choice is using the Berkowitz and Christoffersen test separately. The results are a little better than for the mixed test and somewhat better than for the simple Berkowitz test. However, for tolerance level of 1%, the best choice is using just the simple Berkowitz test for testing unconditional coverage. In this case, testing for independence is ineffective. It comes out that it is necessary to determine how low the power of VaR backtests may be in some typical cases. It seems particularly important to discuss the acceptable minimum of the test power and to focus on the type II error. References 1. J. Berkowitz. Testing Density Forecasts with Applications to Risk Management. University of California, Irvine, 2000, jberkowi/back.pdf. 2. S. Campbell. A Review of Backtesting and Backtesting Procedures. Federal Reserve Board, Washington, 2005, 3. M. Hass. New Methods in Backtesting. CAESAR, 2001, pp 0010 haas pdf. 4. P. Jorion. Value at Risk 2nd edition. McGraw-Hill, P. Kupiec. Techniques for Verifying the Accuracy of Risk Measurement Models. Journal of Derivatives, 3: 73 84, J. A. Lopez. Methods for Evaluating Value-at-Risk Estimates. Federal Reserve Bank of New York Research Paper no. 9802, K. Piontek. A Survey and a Comparison of Backtesting Procedures (in Polish). In P. Chrzan, Matematyczne i ekonometryczne metody oceny ryzyka finansowego. pages Prace Naukowe AE w Katowicach, Katowice, A. da Silva, C. da Silveira Barbedo, G. Araujo, M das Neves. Internal Models Validation in Brazil: Analysisis of VaR Backtesting Methodologies. Financial Stability Report, May 2005, Volume 4 Number 1, , Banco Central Do Brasil.