Backtesting or Backestimating 1?

Transcription

1 Gerhard Stahl Backtesting or Backestimating 1? I Introduction Recently Kupiec articulated his critique of various approaches to Backtesting, by investigating formal statistical properties of related test statistics. His results culminate in the approval of their low power, i.e. the type II error. Spoken in nontechnical terms, we just need more information, to raise the power. To overcome this information dilemma, we propose to incorporate all available information, experimental one given by the data and nonexperimental one, given by a prior and the loss function, details are given below. These kinds of nonexperimental information are typical, what is called a bayesian framework. Expressing degrees of beliefs about different values of a parameter by a prior function, whereas the consequences of our decisions are measured by a loss function. Combining all these components to one decision or test rule, we maximise exploitation of information. The following investigations oppose Kupiec s opinions, showing possibilities out of the dilemma, he constructed. Results of foregoing research state the backtesting-problem in the framework of a Bernoulli process (Kupiec). These investigations take a classical approach to statistical problems into account, i.e. a frequentistic point of view, assuming n iid observations of a Bin(1,θ) distributed random variable X, where the true parameter θ is fix, but unknown. As usual Bin(1,θ), θ Θ, denotes the Binomial distribution and Θ the parameter space. The work of Crnkovic\Drachman generalising the Bernoulli approach is also formulated in the classical setting. In this paper we introduce and apply bayesian ideas and concepts to backtesting. In contrast to classical statistics bayesians introduce a new random quantity to the statistical battlefied, the so called prior distribution π( θ ) on Θ. As a consequence they interpret parametrical models pθ ( x) as conditional distributions p( x θ ). Using the idendity p( x,θ) = π( θ) p( x θ) = p( x) p( θ x) a second new element p( θ x) = π( θ) p( x θ) / p( x) the posterior distribution is introduced, where p(x) is defined by p x = π θ p x θ dθ. Θ The posterior, p( θ x ), contains all information provided by the data, so called experimental information, and all other non experimental information. Sometimes a loss function 1 I especially want to point out that opinions in this article are of pure private nature; none of these may be cited or interpreted as opinions of the Federal Banking Supervisory Office of Germany.

2 L( θ, θ ), is additionally given, where θ is an estimate of θ. In this lucky case we may combine all elements (prior, loss and posterior) to a bayesian decision problem. Let us point out our motivation for the bayesian approach. As a common feature of classical procedures the drawn conclusions are based on the information given by the data, neglecting other sources of non experimental information like a loss function or a prior on Θ. These are well-known possible drawbacks of the classical setting, reflecting that classical inferences are, for the most part, made without regard to the use to which they are to be put. Clearly testing concepts are usually worked out to solve confirmatory aspects. But attention should be drawn to the fact, that the estimated VaR figure is a target value and any deviation from the true value causes some losses in the one or the other sense. This situation is similar to recent developments in the area of quality control, where the revolutionary findings of Taguchi, preferred concepts of estimation to well known testing procedures, mirroring that any deviation from a target value will cause some loss (testing procedures are characterised by 0-1 loss functions, grouping deviations very roughly into two disjoint subsets of Θ). Overestimation will possibly lead to suboptimal allocation of capital resources (limits for example). Even worse is the underestimation of true risk which may cause more serious problems with capital charges. We stress the point that the loss function s asymmetry is not reflected at all in the classical concept. The paper is organised as follows: the first section is the introduction, as given above. The second summarises the results of Kupiec and Crnkovic\Drachman. The third section gives the bayesian version of the Bernoulli backtesting, viewed as a test problem. In the fourth section we develop a general bayesian decision framework, interpreting the backtesting problem merely as a estimation problem, then a testing problem. A short summary is given at the end of the paper. II Results of previous research In this section we shortly review Kupiec s paper and give main results of Crnkovic\Drachman. Kupiec criticises common backtesting procedures in a sound but theoretical way, whereas Crnkovic\Drachman introduce a new nonparametric procedure to measure accuracy of risk measurement techniques. This paper was worked out at JP Morgan, New York. For practitioners it worth noting that the authors of RiskMetrics felt a need to improve their methods of performance measurement given in RiskMetrics. Kupiec examines various possible backtesting methods for parametric and nonparametric VaR-models, where the emphasis is in the field of parametric models. His starting point are distributions (geometric, binomial, etc.) closely related to the Bernoulli process. These distribution are members of the one parameter exponential family. Hence, general theorems of test theory (Lehmann) apply and yield a monotone likelihood ratio (LR) test statistic, implying that the test based on the LR is a uniformly most powerful test for a given sample size. We consider the following problem.

3 An observation x is the number of successes in n independent trials with unknown probability θ of success in each. The α-level LR-test of the hypothesis θ θ 0 against the alternative θ > θ 0 is given by λ( x ) defined as follows. If x n θ 0, then p x; λ x ( 1 ) θ0 θ0 θ0 = = x p( x; x / n) ( x / n) ( 1 x / n) x n x n x 1 if x n > θ 0 then p( x; x / n) λ x x ( x / n) ( 1 x / n) = = p( x θ ) x n x ; θ 1 θ n x. An important large sample result yields that 2 log λ is asymptotically χ 2 distributed with an appropriate degree of freedom. This in mind we get Kupiec s representation of the LR-statistic in the binomial case. Besides theoretical analysis Kupiec studies type II error probabilities simulating numerical examples. From exhibit 6 of his paper, we extract the typical example: Null Alternative Type II Error Type II Error Type II Error Hypothesis Hypothesis Rate n= 255 Rate n= 510 Rate n= 1000 P = 0.01 P = Remember, the VaR (for any linear instrument) at the 99% level for a one day holding period is defined by VaRF = MV * F 1 ( 0. 99) * δ, where δ denotes the sensitivity factor, F 1 ( 0. 99) the 99%-quantile of the distribution F and MV the market value of the position. For distributions F with fat tails, the difference between F 1 ( 0. 99) and F 1 ( 0. 98) gets substantial (extreme choices give 100%). From the table above we conclude that there is a good chance that the underestimated true risk is not detected by the binomial test. This result carries over to other related tests and the nonparametric case. Type I and type II error rates are inversely related, so one might hope to rescue by lowering type I error. Kupiec points out that even this idea does not hit for α=0.75. Kupiec s investigations give an imperative to look for new methods. Crnkovic\Drachman s paper is very interesting in this respect. Starting point for their improvement of the LR s power is the idea to test all quantiles simultaneously by means of nonparametric tests instead of one quantile as in the binomial case. Hence their test is a fully and superior generalisation of the binomial case. As usual in the nonpaprametric framework it is difficult to achieve conclusive results about power or robustness properties of the test. For a given sequence of independent, not necessarily identical, distributed (id) random variables, X i F i, the transformed sequence F i ( X i ) is also id, but uniform on [0, 1], we denote this by U i. Hence the observed percentiles F i ( x i ) may be aggregated (here comes the id argument into play) and then be compared with the uniform distribution. The distance between the empirical distribution function, P n := (number of observed percentiles p that are less or equal to t) / n of the observed percentiles and the distribution function of U i is measured by the Kuiper statistic, K( P n, t):= max { P n -t} + max {t - P n } 0 x 1 0 x 1

4 a cousin of the Kolmogroff Smirnoff statistic (Durbin). Taking the importance of the tails of the VaR-distribution into account, the authors suggest to apply a weight function w(t) w(t) = -0.5 ln(t(1-t)) to the distributions P n and t. Some few words about the id assumption and the asymptotics. The authors prefer the BDS-test for checking. Details and merits of this test are explained in (Brock et al). An open question is whether the asymptotics of the BDS or the transformation w(t) causes the need for large (at most 500, better 1000) sample sizes. The presented works of this section have one point in common, they clarified the need for much empirical information (data) to end up with reliable conclusions about the accuracy of VaR estimates. The main result of these studies is that backtesting methods should be based on a historical period of four years. III Bayesian Backtesting At a first stage of experience, users of VaR-models may probably not to be able to foundate backtesting on history of four years, possibly necessary to get conclusive results for backtesting. Therefore, we propose bayesian methods, to break the lack of data information. Starting point are n realisations from iid random variables X i X i iid X, where the X Ber (θ). Their sum, denoted by n X n = X i= 1 i follows a binomial distribution, n k n k p( X n = k ) = θ θ 1 θ, k k denotes the number of success. In the bayesian framework it is convenient to suppose a beta prior, π( θ) = Beta ( α, β), π( θ ) = π ( θ α β Γ α + β ) α β θ α 1 θ β 1 ;, = 1 1 [ ] ( θ), 0, 1 Γ Γ when working with the binomiai. Two advantages motivate using the beta family. First, it is sufficiently rich in respect to their shape, enabling to express symmetric, skewed or even improper priors within. Second betas are conjugated to the binomials implying the posterior, p( θ x) itself, is a beta distribution p( θ X n = k ) = Beta( α + k β + n k ),. Furthermore we suppose a given loss function L L( θ, θ ) on the states of nature. To test H 1 :θ θ 0 against H 2 :θ 0 >θ we calculate the posterior probability of H 1 and H 2, P( H1 x) and P( H x) θ0 P( H1 x) = p( θ x) dθ 0 and P( H2 x) = 1- P( H x) 2, by 1. Interpreting backtesting as a test problem completely described by the table below,

5 States of nature Acts H 1 is appropriate H 2 is appropriate Expected Loss Choose H 1 0 L 12 L 12 P( H2 x) Choose H 2 L 21 0 L 21 P( H1 x) Probabilities P( H1 x) P( H2 x) we decide for H 1 by the following rule 1 < L 21 P( H1 x) / L 12 P( H2 x). Further considerations (see Berger) of this decision rule explore parallels to the LRtest, introduced in the last section. In the present bayesian case the decision rule reflects priors and losses and sample information, whereas the classical LR decision is mainly driven by the level of significance, which is difficult to relate in a concrete manner to the problem s economic substance. In this respect we conclude bayes methods superior to classical ones. IV Backtesting from a bayesian decision theoretical viewpoint Anew, remember the roots of VaR-models are originally economic decisions problems, far away from isolated statistical exercises. Assuming n iid realisations from bernoullian distributed random variables, X Ber (θ), we have to infer or to decide about θ, to measure the VaR-model s accuracy. In the last section we solved this problem with the help of a statistical test. But is this in the line with our intuition? Let us have a closer look at the loss function in the table above, choosing exemplary H 1 : θ against H 2 : 0. 95>θ. Judging the situations θ [0, 0.5] and θ [0.93, 0.95] with the loss function, yields the same loss. Every riskmanager would be confused about this answer. But what is wrong with his intuition? Taguchi told us in an analogue situation stemming from quality control that slipping from a testing interpretation of our statistical task to an estimating one could help. This means in the considered example, any deviation from the target value 0.95 would cause some loss, as mentioned earlier the loss function is asymmetric. Before we begin with some considerations to bayesian inference about θ some words how the prior may be generally determined seem worthwhile. So far, our presentation has been restricted to the case of beta priors. Of course this is not the only answer. First it seems natural to transform to prior to an appropriate interval [0.75, 1] for example. Or the user feels more comfortable if a prior is specified by means of some quantiles of a cumulative distribution function, where only some probabilities are to be known. Finally, we mention the method of moments to specify a prior. With these remarks in mind we turn back to our inference problem. First we calculate the posterior combining the binomial model, the data and the prior: p θ X = k. ( n ) We may use the posterior to report a point estimate θ for θ, applying a generalised maximum likelihood method to θ = max L(θ) = max p( θ X n = k ). θ Θ θ Θ Parallel to classical lines an error estimate should be adjoined to θ. A famous candidate is the posterior variance of θ defined by

6 ( θ θ) E p( θ x) 2. It may be also very vulnerable to calculate a credible set C, the bayesian analogue to classical confidence intervals, for θ from the posterior. 1 α p( C x) As usual these intervals are not uniquely determined, for details see Berger. Besides these inferential approaches we focus a decision theoretic setting using a loss function L. We are interested to determine a decision θ minimising posterior expected loss. To be specific, we only examine the case of an asymmetric linear loss function L, defined by K if L(, ( ) 0 θ θ θ θ 0 θ θ) = K ( θ θ ) if θ θ 1 < 0. Under the met assumptions any ( K0 / ( K0 + K1 )) -fractile of the posterior p( θ X n = k ) is a Bayes estimate of θ. A first look at this bayesian machinery shows that the greater K 0 is, the greater is the quantile. This coincides with the imagination that greater losses should increase the quantile. The bayesian estimate reflects the asymmetry between underestimation and overestimation of the accuracy of VaRmodels, our prior beliefs about θ - for example we know that using normal distributions tend to underestimate risk - and the experimental information provided by the data. These or other methods of backestimation should deserve further investigations. V Summary and conclusions Backtesting methods are important tools applying VaR-models. By means of the proposed bayesian methods all subjective and objective informations may be incorporated to draw a adequate picture of the risk, to be faced. Hence, these methods have a clear-cut advantage to established ones especially in situations of the beginning to use VaR-models. Furthermore we feel that bayesian methods are close to the heuristics of nonstatisticians. It seems to us that the developed decision and test rules are easier to understand, better linked to the economical problem and well communicable compared than the classical ones. The likelihood ratio tests stand for a good example. In any case the bayesian methods should be used parallel to classical ones and a comparison should give further insights to the driven conclusions. Clearly many questions remain open. So it seems not been investigated how to use backtesting methods to optimise model selection (GARCH, t-distributions,..). VI Bibliography Basle Committee on Banking Supervision (1996), Proposal to Issue a Supplement to the Basle Capital Accord to Cover Market Risks. Basle, December. Berger, J.O. (1985) Statistical Decision Theory and Bayesian Analysis. 2nd Ed. Springer New York Crnkovic, C. and J. Drachman (1995), A Universal Tool to Discriminate Among Risk Measurement Techniques. Preprint, to appear in RISK-Magazine. Brock, W.A., Hsieh, D.A. and LeBaron, B. (1993) Nonlinear Dynamics, Chaos and Instability: Statistical Theory and Econmic Evidence. MIT Press, Cambride, Massachusetts.

7 Durbin, J. (1973) Distribution Theory for Tests based on the Sample Distribution Function. SIAM, Philadelphia Group of Thirty (1993), Derivatives: Practices and Principles. Washington, D.C. Kupiec, P. (1995), Techniques for Verifying the Accuracy of Risk Measurement Models. Journal of Derivatives. Lehmann, E.L. (1986) Testing Statistical Hypotheses. Wiley, New York. Morgan Guarantly Trust Company, (1995) RiskMetrics TM - Technical Document. 3rd Edition. Zellner, A. (1990) Bayesian Inference. The New Palgrave in Statistics. Gerhard Stahl Ringslebenstr. 2 D Berlin