A New Procedure for Multiple Testing of Econometric Models

Transcription

1 A New Procedure for Multiple Testing of Econometric Models Maxwell L. King 1, Xibin Zhang, and Muhammad Akram Department of Econometrics and Business Statistics Monash University, Australia April 2007 (Preliminary Draft) Abstract: A significant role for hypothesis testing in econometrics involves diagnostic checking. A typical test procedure involves the use of a test statistic and critical values in order to control the probability of wrongly rejecting the null hypothesis. When checking the adequacy of a chosen model, researchers almost always employ a range of diagnostic tests, each of which is designed to detect a particular form of model inadequacy. A major problem is how best to control the overall probability of rejecting the model when it is true and multiple test statistics are used. This paper presents a new multiple testing procedure, which involves checking whether the calculated values of the diagnostic statistics are consistent with the postulated model being true. This is done through a combination of bootstrapping to obtain a multivariate kernel density estimator of the joint density of the test statistics under the null hypothesis and the Monte Carlo integration to obtain a p-value using this kernel density. The proposed testing procedure is applied to tests for serial correlation in an observed time series, for normality, and for the significance of coefficients in a dynamic linear regression model. We find that the sizes of our test are as good as the sizes of the Portmanteau test, the Jarque-Bera test and the F test, respectively. Moreover, the powers of our test are, respectively, better than the powers of these three tests. Key words: multivariate kernel density; bootstrapping; serial correlation; normality JEL classification: C01, C12, C14. 1 Corresponding Author. max.king@adm.monash.edu.au; telephone: ; address: Monash University, Victoria 3800, Australia.

2 1 Introduction Statistical hypothesis testing is an extremely important technique in the practice of econometrics, particularly with respect to diagnostic checking of model specification. This is how econometricians are best able to combat the severe problem of uncertainty in model specification. Such testing procedures need to be as accurate as possible due to constraints on data availability. Fortunately advances in computer power and simulation based methods have allowed greater scope in the design of high quality tests. The purpose of any test is to accurately control the probability of wrongly rejecting the null hypothesis (known as the size of the test), while at the same time ensuring a high probability of correctly rejecting the null hypothesis (known as the power of the test). There is a very large literature on diagnostic testing of all kinds of econometric models. Therefore, in order to check the adequacy of a chosen model, researchers can apply a range of diagnostic tests, each of which is designed to detect a particular form of model inadequacy. A major problem is how best to control the overall probability of rejecting the model when it is true. For example, five statistically independent tests applied at the 5% level will result in a 22.6% chance of at least one rejection when the null hypothesis model is true. Of course it is unlikely that five diagnostic tests applied to the same model will be mutually independent, so in actual fact, this probability could be higher or lower than 22.6%. The major issue is how we should conduct these tests in order to control the overall probability of rejecting the model when it is true. The aim of this paper is to develop a new procedure for testing based on multiple test statistics in a way that controls the overall probability of a false rejection. A typical approach to hypothesis testing is to construct the critical region through a test statistic denoted by t(y) : R n R, which is a mapping from the n-dimensional sample space to the real line and follows or at least 1

3 asymptotically follows a known distribution under the null hypothesis. If the sample falls in the critical region, the null hypothesis is rejected. Multiple hypothesis testing is the testing of two or more separate hypotheses simultaneously. A simple example of such a test is testing for significance of the serial correlation coefficients, denoted by (ρ 1, ρ 2,, ρ d ), for a given time series denoted by (y 1, y 2,, y n ), where the null hypothesis is H 0 : ρ 1 = ρ 2 = = ρ d = 0. The Portmanteau test of Box and Pierce (1970) can be employed for such a purpose, and the test statistic asymptotically follows the χ 2 distribution with d degrees of freedom. Of course, there are also some other test statistics that involve a certain degree of approximation for deriving relevant critical regions. When such approximations are poor, the performance of the size and power is often poor. This paper aims to propose an alternative approach to the above-mentioned hypothesis testing procedure based on constructing a mapping v(y) : R n R d (n > d), where the critical region or the p-value can be derived according to the estimated density of v(y). For example, such a mapping can be the estimate of ρ = (ρ 1, ρ 2,, ρ d ) in the above mentioned test for significance of the serial correlation coefficients. Even though the distribution of v(y) is usually unknown under the null hypothesis, we can estimate the density of v(y) through bootstrapping under the null hypothesis. Once the bootstrapped density of v(y) is obtained, we can derive the p-value through Monte Carlo simulations, and then the null hypothesis can be tested. One of the key issues in the new testing procedure is how we could satisfactorily estimate the density of v(y). We can use the multivariate kernel density estimator to estimate the density of v(y). It has been generally accepted that the performance of the kernel density estimator is mainly determined by the chosen bandwidths, and only in a minor way by the choice of a kernel function. According to Scott (1992) and Bowman and Azzalini (1997), when data are observed from the multivariate normal density and the diagonal bandwidth matrix, denoted by (h 1, h 2,, h d ) with d denoting the dimension of the data, is employed, the optimal bandwidth that minimizes the 2

4 mean integrated squared error (MISE) can be approximated by { } 1/(d+4) 4 h i = σ i, (d + 2)n where σ i is the standard deviation of the ith variate and can be replaced by its sample estimator in practice, for i = 1, 2,..., d. This bandwidth selector, which is referred to as the normal reference rule (NRR) in the literature, is often used in practice due to the absence of any other practical bandwidth selection methods, despite the fact that data might be non-gaussian. Recently, Zhang, King and Hyndman (2006) presented a Bayesian approach to bandwidth selection for multivariate kernel density estimation. Their bandwidth selector is applicable to data of any dimension, and has a better performance than NRR. In this paper, we shall use both bandwidth selectors to estimate the kernel density of v(y). This paper aims to provide a new testing procedure, where the multivariate kernel density estimation method is used to compute the p-value of a test. The rest of the paper is organized as follows. Section 2 provides the outline of a new testing procedure. We examine the performance of our new testing procedure using Monte Carlo simulations in Section 3, where we provide comparisons of performances in terms of size and power between our testing procedure and some other commonly used testing procedures. Section 4 concludes the paper. 2 A New Testing Procedure We shall begin by first describing the main ideas behind our new testing procedure. Assume that we wish to test a null hypothesis using d test statistics denoted as t i, for i = 1, 2,, d. Let t = (t 1, t 2,, t d ) denote the d 1 vector of these test statistics. For the moment, we assume that each of the component tests is a two-sided test based on accepting the null hypothesis if c 1i < t i < c 2i, 3

5 where c 1i and c 2i, for i = 1, 2,, d, are critical values. Let ˆt denote the calculated value of our test statistic vector t using the available data. Essentially we wish to ask the question, is our vector consistent with the null hypothesis being true? The p-value is a useful tool for answering this question. It is defined as the probability under the null hypothesis of finding a value of our statistic as extreme or more extreme as the value we have found from the data, namely ˆt. Thus if we have the joint density function of t, denoted f(t), under the null hypothesis, then the p-value is the probability of obtaining a value of t such that f(t) < f(ˆt) holds. Once calculated, the p-value can be used to conduct the test at any level of significance. For example, at the 5% significance level, if the p-value is less than 0.05 then the null hypothesis is rejected. Otherwise, it cannot be rejected. Our approach involves using Monte Carlo integration, which was introduced by Hyndman (1996) to solve a related problem, to calculate the p-value as follows: 1) Simulate m independent sets of data under the null hypothesis, and for each simulated data set, calculate t. The computed t values are denoted as t (i), i = 1, 2,, m. 2) Based on {t (i) : i = 1, 2,, m}, estimate the joint density of t through the multivariate kernel density method. Denote the estimated joint density as ˆf(t). 3) Simulate n independent sets of data under the null hypothesis and, for each simulated data set, calculate t. The computed t values are denoted as t (j), for j = 1, 2,, n. Note that the n simulated data sets should be independent from the m data sets simulated in step 1). 4) Calculate ˆf( t (j) ), j = 1, 2,, n, and count the number of times that ˆf( t (j) ) < ˆf(ˆt) holds. If this count is denoted by r, then the p-value is estimated by r/n. A key issue in this new testing procedure is to estimate the multivariate density of t, which we estimate by the kernel density estimator. Let {x 1, x 2,..., x n } denote an independent random sample drawn from f(x), a d-dimensional density function. The kernel estimator of f(x) is given 4

6 by (Scott, 1992; Wand and Jones, 1995; among others) ˆf H (x) = 1 n n K H (x x i ), i=1 where K H (x) = H 1/2 K(H 1/2 x), K( ) is a multivariate kernel function, and H is a symmetric positive definite d d matrix known as the bandwidth matrix. The practical implementation of the kernel density estimators requires the specification of bandwidths. In this paper, we employ two bandwidth selectors, which are the Bayesian bandwidth selector of Zhang, King and Hyndman (2006) and NRR discussed in Scott (1992) and Bowman and Azzalini (1997). Wand and Jones (1995) showed that a kernel density estimator with a small bandwidth is often noisy in tails. A small bandwidth often produces a jagged kernel density estimator, which means that the estimated density picks up particularities in the sample and does not allow variation across the sample. Such an estimated density is said to be undersmoothed. When choosing bandwidths for a set of d dimensional data, Scott (1992) suggested using the corrected rule given by h i = 2ˆσ i n 1/(4+d), (1) which is approximately the bandwidth selected through NRR multiplied by 2, for i = 1, 2,, d. In a given set of data, when observations are sparse near boundaries (in other words, observations are sparsely distributed in tails), the kernel density estimator is often affected by these remote observations. As a consequence, the kernel density is likely to overestimate the true density near boundaries where observations are sparse. Silverman (1986) and Scott (1992) argued that any data set would be sparse near boundaries as the dimension of data increases. However, we believe that enlarged bandwidths may result in a smoother kernel density, which is helpful in reducing the density of observations near boundaries overestimated by usual bandwidths selected through eligible criteria such as NRR or the Bayesian bandwidth selector. To our understanding, Scott s (1992) suggestion of multiplying the bandwidths chosen through NRR by a constant 2 may result in a smoother kernel density estimator, especially in both tails. 5

7 As the new testing procedure relies on a correctly estimated density of the vector t, especially in the tail areas, we shall use two types of bandwidths: 1) bandwidth selected by NRR and the Bayesian selector; and 2) bandwidths selected by the two methods and multiplied by 2. We shall also compare the performance of kernel densities computed through these two types of bandwidths, where the former type is denoted as MKD 1 and the latter is denoted as MKD 2. 3 Simulation Studies In this section, we shall apply the new testing procedure to three tests, which are the testing for significance of serial correlations of a time series, the omnibus test for normality, and the testing for significance of regression coefficients, respectively. In each test, we estimate the kernel density of the vector of interest based on a collection of a large number of such vectors calculated independently through data sets generated under the null hypothesis, where bandwidths are chosen through NRR and the Bayesian method denoted as Algorithm 1 and Algorithm 2, respectively. To estimate the size of the new testing procedure, we derive a second collection of a large number of such vectors calculated through data sets generated independently under the null hypothesis, where these data sets are independent from those used in estimating the kernel density of the vector. The density values computed on second collection of such vectors according to the estimated density are used to calculate the relative frequency of rejecting the null hypothesis. To estimate the power of the new testing procedure, we follow the same procedure for estimating the size with a third collection of a large number of such vectors calculated through data sets generated independently under the alternative hypothesis. In addition, we compare the finite-sample size and power of our testing procedure with those of several commonly used test statistics. When testing for significance of various order serial correlations of a time series, we compare our testing procedure with the Portmanteau test proposed 6

8 by Box and Pierce (1970) and the modified Portmanteau test proposed by Lobato, Nankervis and Savin (2001). When testing for normality, we compare our testing procedure with the normality test proposed by D Agostino (1971) denoted as the D test, the normality test of Jarque and Bera (1980, 1981) denoted as JB, and the modified JB test of Urzúa (1996) denoted as MJB. When testing for significance of regression coefficients, we compare our testing procedure with the F test. 3.1 Testing for Significance of Serial correlations Test statistics In empirical studies, a frequently encountered problem is to test the null hypothesis that a time series is white noise, or in other words, the first d serial correlations of a time series are zero. A typical testing procedure is to use the Portmanteau test, which has some asymptotic optimality properties for higher-order autoregressive processes and are often used in practice to examine serial correlations up to any given order. Lobato, Nankervis and Savin (2001) investigated the finite-sample properties of a modified Portmanteau statistic in testing the hypothesis that the underlying time series is uncorrelated without imposing statistical independence. Romano and Thombs (1996) showed that in some examples, the BP test can produce misleading inferences when the time series is uncorrelated but statistically dependent. In this section, we compare the finite-sample size and power of our testing procedure with the Portmanteau test denoted by Q d, and the modified Portmanteau test denoted by Q d based on the size-corrected critical values, where d is the maximum order of serial correlation. The model we consider is a simple time series model given by y t = α + u t, (2) for t = 1, 2,, T, where u t iid N(0, 1), α = 1, and T is the sample size. 7

9 Our interest is to test the null hypothesis H 0 : u t is white noise against the alternative H 1 : u t is autocorrelated. The time series y t is generated independently from N(α, 1). Fitting (2) to a set of generated data, we obtain the residuals denoted as û = (û 1, û 2,, û T ), based on which we compute the serial correlations up to the dth order denoted as ˆρ = (ˆρ 1, ˆρ 2,, ˆρ d ). To estimate the power of the new testing procedure in rejecting a false hull hypothesis, the error term of (2) is assume to follow the following specifications. Case 1: u t = ρu t 1 + ε t with u 1 = ε 1 / 1 ρ 2 ; Case 2: u t = ρu t 2 + ε t with u i = ε i / 1 ρ 2 for i = 1, 2; Case 3: u t = ρu t 3 + ε t with u i = ε i / 1 ρ 2 for i = 1, 2, 3; and Case 4: u t = ρu t 4 + ε t with u i = ε i / 1 ρ 2 for i = 1, 2, 3, 4. In all four cases, we assume that ε t iid N(0, 1) and ρ = 0.25 and 0.5, respectively. We consider testing for significance of serial correlations up to the 4th and 6th orders, respectively. The BP-test statistic for order d is defined as Q d = T d i=1 r 2 i where T is sample size, and r i = T t=i+1 ûtû t i T t=1 û2 t The modified BP-test statistic is defined by where ˆτ ii is Q d = T d i=1 r2 i ˆτ ii ˆτ ii = (1/T ) T i t=1 (u t ū) 2 (u t+i ū) 2 c(0) 2 8

10 and ū = T t=1 u t/t and c(0) = T t=1 (u t ū) 2 /T. Our new testing procedure is described as follows. i) Generate u 1, u 2,, u T from N(0, 1) and compute y 1, y 2,, y T according to (2); ii) Calculate the residuals denoted as û 1, û 2,, û T through the ordinary least squares (OLS). Compute the serial correlation coefficient of residuals as follows. r i = T t=i+1 ûtû t i T, t=1 û2 t for i = 1, 2,, d, where û t = y t ŷ t. Let r (1) 1 = (r 1, r 2,, r d ). iii) Repeat i) and ii) for m times and derive r (1) j, for j = 1, 2,, m. The derived m d matrix is denoted by x (1). iv) Choose a bandwidth matrix for x (1) through Algorithm 1 and Algorithm 2, where the latter is implemented through a Markov chain Monte Carlo (MCMC) simulation algorithm with the initial values being the ones obtained via NRR. v) Compute the kernel density of x (1) based on the chosen bandwidth matrix. The density value computed at the jth row of x (1) is denoted by vi) Sort { ˆf(r (1) j ), for j = 1, 2,, m. (1) ˆf(r j ) : j = 1, 2,, m} and obtain critical values corresponding to the nominal sizes of 0.01, 0.05 and 0.10, respectively. The critical values are denoted as ˆf p (x (1) ), for p = 0.01, 0.05, 0.1. vii) Repeat i) iii) to generate a second collection of vectors r (2) j = (r 1, r 2,, r d ), for j = 1, 2,, n. The derived n d matrix is denoted by x (2). viii) Calculate the density at each row of x (2) according to the density function obtained in iv), for j = 1, 2,, n. The density value of the jth row of x (2) is denoted as (2) ˆf(r j ), for j = 1, 2,, n. ix) Estimate p-values by the relative frequencies that ˆf(r (2) j ) < ˆf p (x (1) ) holds, for p = 0.01, 0.05, 0.1. In the above testing procedure, the sample sizes for computing the OLS residuals are, respectively, 9

11 30, 50, 100 and 200. The values of m and n are 20, 000. In addition, we consider the serial correlation up to d = 4 and d = 6 orders, respectively The finite-sample performance of size and power In this section, we conduct Monte Carlo simulations to investigate the sizes and powers of MKD 1 and MKD 2 in comparison with those of the Portmanteau and the modified Portmanteau tests. In terms of testing for serial correlations up to the 4th order for a given time series, we present the sizes and powers of the three tests in Tables 1 and 2, respectively. When bandwidths are chosen through MKD 1 that consists of the Bayesian bandwidth selector and NRR, the estimated sizes are larger than the nominal sizes, especially for small nominal size values. However, when bandwidths are chosen through MKD 2 where the chosen bandwidths through the Bayesian selector and NRR are multiplied by 2, the estimated sizes are almost the same as the nominal sizes. We found that the sizes of the Portmanteau and the modified Portmanteau tests are almost the same as the nominal sizes, where the size-corrected critical values were used for computing the probability values of rejecting a true null hypothesis. Hence, in terms of the accuracy of the estimated size, our testing procedure, in particular MKD 2, performs as good as the conventional Portmanteau and modified Portmanteau tests. Table 2 presents the probability values of rejecting a false null hypothesis, where the serial correlation coefficient of the error series is 0.25 and 0.5, respectively. We found that both MKD 1 and MKD 2 have obvious larger probability values than the Portmanteau and modified Portmanteau tests in rejecting the two false null hypotheses. As the empirical sizes of MKD 1 are larger than the nominal sizes, one might be suspicious that MKD 1 is likely to over-reject the null hypothesis. However, the empirical sizes of MKD 2 are almost the same as the nominal sizes, and MKD 2 is more powerful than the Portmanteau and modified Portmanteau tests in rejecting the two false null hypotheses. Thus, the simulation study shows that our new testing procedure, in particular 10

12 MKD 2, produces correct empirical sizes and has higher power probabilities than the conventional Portmanteau and modified Portmanteau tests. It is important to note that MKD 2 contains two different algorithms, namely Algorithm 1 and Algorithm 2, where the former chooses bandwidths using NRR and the latter chooses bandwidths using the Bayesian bandwidth selector, and the chosen bandwidths are multiplied by 2. Table 1 shows that both algorithms result in almost the same sizes, which are quite close to the nominal sizes, and that both algorithms result in higher powers than the Portmanteau tests do. In addition, we found that for sample sizes of 50 and 100, the powers resulted from Algorithm 2 is higher than that resulted from Algorithm 1, while for sample size of 200 and 500, Algorithm 1 is more powerful than Algorithm 2. In terms of testing for serial correlations up to the 6th order for a given time series, we present the sizes and powers of the three tests in Tables 3 and 4. When bandwidths are chosen through MKD 1, the estimated sizes are larger than the nominal sizes, especially for small nominal size values. However, when bandwidths are chosen through MKD 2 where the chosen bandwidths through the Bayesian selector and NRR are multiplied by 2, the estimated sizes are almost the same as the nominal sizes. Table 3 shows that the sizes of the Portmanteau modified Portmanteau tests are almost the same as the nominal sizes, where the size-corrected critical values were used. Hence, in terms of the accuracy of the estimated size, our testing procedure, in particular MKD 2, performs as good as the conventional Portmanteau and modified Portmanteau tests. Table 4 shows that both MKD 1 and MKD 2 have obviously larger power values than the Portmanteau and modified Portmanteau tests in rejecting the two false null hypotheses. Because the empirical sizes of MKD 1 are larger than the nominal sizes, one may think that MKD 1 is likely to over-reject the null hypothesis. However, the sizes of MKD 2 are almost the same as the nominal sizes, and we found that MKD 2 has higher power values than the Portmanteau and modified Portmanteau tests in rejecting the two false null hypotheses. Thus, the simulation reveals 11

13 that our new testing procedure, in particular MKD 2, is able to produce correct empirical sizes, and has higher power probabilities than the conventional Portmanteau and modified Portmanteau tests. Regarding the two algorithms used in MKD 2, we found that both algorithms produce almost the same sizes, which are quite close to the nominal sizes, and that both algorithms result in higher powers than the Portmanteau tests do. In addition, Table 4 reveals that the power values resulted from Algorithm 2 are higher than those resulted from Algorithm 1 for all sample sizes considered. This finding also confirms the advantage of the Bayesian bandwidth selector over its counterpart of NRR, as claimed by Zhang, King and Hyndman (2006). To conclude, we have found that our new testing procedure, in particular MKD 2 with the chosen bandwidths via the Bayesian selector and NRR being multiplied by 2, not only can produce correctly estimated sizes, but also has higher powers than the conventional Portmanteau tests. 3.2 Testing for Normality Test statistics In many statistical situations, random observations are often assumed to be normally distributed. Moreover, many statistical inferences have been established based on the assumption of normality. Therefore, testing for normality is an important issue (see, for example, Shapiro and Wilk, 1965; D Agostino, 1971, 1972; Bowman and Shenton, 1975; Pearson, D Agostino and Bowman, 1977; Jarque and Bera, 1980, 1987; Spiegelhalter, 1980; Thode, 2002). In this section, we compare the finite-sample properties of the size and power of our testing procedure with those of other three different normality tests, which are the normality test proposed by D Agostino (1971), the asymptotic version of the Jarque-Bera test proposed by Urzúa (1996), the modified Jarque-Bera test. Note that the latter three tests use size-corrected critical values. 12

14 D Agostino (1971) proposed a test for normality. Let x = (x 1, x 2,, x T ) represent a random sample and let (x (1), x (2),, x (T ) ) denote a vector of the ordered observations. The test statistic is D = S 1 /(n 2 S 2 ), where T S 1 = [i 0.5(n + 1)] x (i), S 2 2 = 1 T i=1 T (x i x) 2, where x is the sample mean. The standardized version of D is i=1 Z = n(d )/ , where Z is approximately a standardized variable with zero mean and unit variance. D Agostino and Stephens (1986), Urzúa (1996) and Thode (2002) discussed the construction of omnibus tests for normality that combine information from skewness and kurtosis denoted b 1 and b 2, respectively. The simplest construction is the Jarque-Bera test, where the skewness and kurtosis measures are used to construct a test statistic expressed as [ ( b1 ) 2 JB = T 6 + (b ] 2 3) 2, 24 which follows χ 2 (2) under the null hypothesis that the observed data are normally distributed. Urzúa (1996) proposed a modified version of the Jarque-Bera test, and the test statistic is given by MJB = [ ( b1 ) 2 var( b 1 ) + (b ] 2 E(b 2 )) 2, var(b 2 ) where E(b 2 ) = 3(n 1)/(n+1), var( b 1 ) = 6(n 2)/[(n+1)(n+3)] and var(b 2 ) = 24n(n 2)(n 3)/[(n + 1) 2 (n + 3)(n + 5)]. Simulation studies revealed that the size of the Jarque-Bera test is incorrect for small- and moderate-size samples (Poitras, 1992; Durfour, Farhat, Gardial and Khalaf, 1998; among other). 13

15 Moreover, compared with the Jarque-Bera test, the modified Jarque-Bera test slightly improves the size performance, but is still incorrectly sized. A straightforward solution to the incorrect size problem is to use Monte Carlo simulations to obtain correct critical values, which are used for the tests to produce correct sizes (see, for example, Dufour and Khalaf, 2001; Poitras, 2006). Instead of constructing a test statistic of the skewness and kurtosis measures, our new testing procedure is focused on estimating the joint density of the skewness and kurtosis measures through a procedure of two-stage Monte Carlo simulations. In each of the first-stage m simulations, we generate a sample of random numbers independently from the distribution specified by the null hypothesis and compute the skewness and kurtosis of the sample. Thus, we obtain m bivariate vectors of the skewness and kurtosis, which are used to estimate the kernel density of the skewness and kurtosis. In the second-stage simulations, when examining the size of our testing procedure, we generate a sample of random numbers independently from the distribution specified by the null hypothesis; when examining the power of the testing procedure, we generate a sample of random numbers independently from the distribution specified by the alternative hypothesis. The probability values of rejecting the null hypothesis can be approximated the relative frequencies of rejecting the null in the second-stage n simulations. In this procedure of two-stage simulations, the sample sizes are 30, 50, 100 and 200, respectively The finite-sample performance of size and power This section presents the finite-sample performance of sizes and powers of the JB, MJB and our testing procedure, where the size-corrected critical values were used for computing the sizes and powers of JB and MJB tests. Table 5 presents the sizes of our testing procedure, and JB and MJB tests. The bandwidths chosen through MKD 1 is based on the direct use of NRR and the Bayesian selectors, while bandwidths chosen through MKD 2 involves multiplying the above-chosen bandwidths by 2. However, 14

16 we found that the sizes derived through MKD 1 are not obviously different from the corresponding sizes derived through MKD 2. Moreover, the sizes obtained through both MKD 1 and MKD 2 are very close to the corresponding nominal sizes. We also found that the sizes of JB and MJB tests are very close to the corresponding nominal sizes. Thus, we may conclude that according to the Monte Carlo investigation, all three tests have correct sizes. When examining the powers of the three tests, we generate a sample of random numbers for a mixture of two normal densities denoted as 0.5N(0, 1) + 0.5N(0, 4). Table 6 shows that the power of MKD 1 is not obviously different from that of MKD 2 at each of the three significance levels. In addition, the power values computed through Algorithm 1 are not obviously different from the corresponding power values computed through Algorithm 2. When the sample size is small (T =30 and 50), the powers of MKD 1 and MKD 2 are slightly larger than those of JB and MJB. However, when the sample size becomes large (T =100 and 200), the powers of MKD 1 and MKD 2 are slightly less than those of JB and MJB. We would like to note that the power of our testing procedure is not obviously different from that of JB or MJB. Hence, we may conclude that the power of our testing procedure is as good as that of size-corrected JB or MJB tests. When we compare the size and power of our testing procedure with those of D test, our testing procedure aims to estimate the kernel density of (S 1, S2) 2 through the above-mentioned two-stage procedure of Monte Carlo simulations. Table 7 presents the sizes of both tests. We found that the sizes derived through MKD 1 are not obviously different from the corresponding sizes derived through MKD 1, and that both MKD 1 and MKD 2 produce correct sizes. Moreover, the Monte Carlo investigation shows that the sizes of the D test are very close to the corresponding nominal sizes. Thus, we may conclude that the size performance of our testing procedure is as good as that of the D test. 15

17 When investigating the powers of the D test and our testing procedure, we generated data, respectively, from two distributions, where the first distribution is a mixture of normals specified as 0.5N(0, 1) + 0.5N(0, 4), and the second is the log-normal density given by { 1 f(x) = x 2πσ exp 1 } (log x 2 µ)2, 2σ2 for x > 0. Regarding the mixture normal alternative, we found that both MKD 1 and MKD 2 of our testing procedure outperform the D test for all sample sizes and all nominal sizes, respectively. In addition, there is no obvious difference between Algorithm 1 and Algorithm 2 in terms of power performance. regarding the log-normal alternative, we found that our testing procedure has higher powers than the D test for sample sizes of 30 and 50, while our test performs as good as the D test for sample sizes of 100 and 200. In addition, we found that in terms the power performance, Algorithm 1 and Algorithm 2 perform similarly. 3.3 Testing for Significance of Regression Coefficients Test statistics It is a common practice that researchers test for significance of the estimated regression coefficients. Such a hypothesis takes the form of equality restrictions on some or all of the parameters. In this section, we compare our testing procedure with the F test in terms of finite-sample performance of size and power. The model we consider is an AR(d) model given by y t = α 0 + α 1 y t 1 + α 2 y t α d y t d + u t, (3) where u t iid N(0, 1), for t = 1, 2,, T. Our interest is to test the null hypothesis H 0 : α 1 = α 2 = = α d = 0 against the alternative H a : at least one of the α i, for i = 1, 2,, d, is nonzero. In order to test the significance of α i, for i = 1, 2,, d, our testing procedure aims to estimate the joint density of (α 1, α 2,, α d ), based on which we can derive the critical values, 16

18 and therefore the null hypothesis can be tested. One of the conventional tests for the significance of α i, for i = 1, 2,, d, is conducted via the F statistic given by F = SSR/d SSE/(n d 1), where SSR is the sum of squares due to the regression, and SSE is the sum of squared residuals. We use Monte Carlo simulations to investigate the size and power of our testing procedure in comparison with those of the F test. When examining the size of the two tests, we set α i = 0, for i = 0, 1,, d, and generate u t iid N(0, 1), for t = 1, 2,, T. When examining the power of the two tests, we set α 0 and α i nonzero, for i = 1, 2,, d under the alternative hypothesis The finite-sample performance of size and power To estimate the density of (α 1, α 2,, α d ), we generate a sample of an AR(4) process according to (3) with the sample size being, respectively, 50, 100, 200 and 500, and m = 20, 000 samples were generated for each sample size. In each sample, we estimated α i, for i = 0, 1,, d, through the OLS method, such that we obtained a collection of m vectors for (α 1, α 2,, α d ) for each sample size. The joint density of (α 1, α 2,, α d ) was estimated by its kernel density with bandwidths chosen through either Algorithm 1 or Algorithm 2. This procedure of estimating density is referred to as the stage 1 simulation. To examine the size of our testing procedure, we repeated the above procedure with a different seed of random number generator. For each sample size, we obtained a collection of n = 20, 000 vectors for (α 1, α 2,, α d ), at which we calculated the density according to the estimated density function. Comparing the computed density values with the critical value obtained in the above procedure, we derived the relative frequency of rejecting the null hypothesis as the approximate p-value of the test. This procedure of examining the size of a test is referred to as the stage 2 simulation. 17

19 To examine the size of the F test, we derived the size-corrected critical values for the F statistic for each significance level in the stage 1 simulation. At each of the n iterations in the stage 2, we calculated the F statistic, which was compared with the size-corrected critical values to compute the relative frequency of rejecting null hypothesis. Such relative frequencies are approximations of the corresponding p-values. Note that as (3) meets the assumption required by the F statistics to follow the F (d, T d 1) distribution, the F test should have very good size performance. Tables 10 and 11 presents the empirical sizes and powers of our testing procedure and the F test, respectively. We found that the sizes of MKD 1 are larger than the corresponding nominal sizes, suggesting that such a testing procedure is likely to over-reject the null hypothesis. The over-rejection of the null hypothesis might be due to non-smooth density at tails. However, the sizes of MKD 2 are almost the same as the corresponding nominal sizes. We would like to emphasise that MKD 2 involves multiplying the chosen bandwidths by 2 with the aim to produce a smooth density at tails. In addition, we found that there is no obvious difference between Algorithm 1 and Algorithm 2 in terms of the size performance. Regarding the F test, we found that the empirical sizes are almost the same as the corresponding nominal sizes. This finding is consistent with our former expectation. In terms of the power performance, we found that the powers of MKD 2 resulted from Algorithm 1 are slightly larger than the corresponding powers of the F test. The powers of MKD 2 resulted from Algorithm 2 are also slightly larger than the corresponding powers of the F test, except for the powers derived for nominal size of 0.01 and for sample size of 50. However, we found no obvious difference between Algorithm 1 and Algorithm 2 in terms of the power performance. To conclude the Monte Carlo investigation of size and power of our testing procedure and the F statistic in testing for significance of regression coefficients, we have found that our testing procedure, in particular MKD 2, performs as good as the F test, and that the former is more powerful than the latter. 18

20 4 Conclusion This paper presents a new testing procedure based on multiple testing statistics, where in stead of constructing a critical region through a test statistic that follows or at least asymptotically follow a know distribution under that null hypothesis, we estimate the joint density of the multiple testing statistics through the bootstrapping technique. Based on the bootstrapped density, we are able to draw a conclusion about whether the null hypothesis is rejected. The proposed test involves a two-stage procedure of Monte Carlo simulations, where the first stage is to estimate the density of the multiple testing statistics, while the second stage is to approximate the probability of rejecting the null hypothesis. In order to examine the size and power of the proposed testing procedure, we have conducted Monte Carlo simulation and compared it with, respectively, the Portmanteau test for testing serial correlations, the Jarque-Bera test for testing for normality, and the F test for significance of regression coefficients. We have found that the sizes of our testing procedure are close to the corresponding nominal sizes, and that the power of our testing procedure is better than, or in some cases as good as, the three competing tests, respectively. References Box, G.E.P., and Pierce, D.A. (1970), Distribution of Residual Autocorrelations in Autoregressive Integrated Moving Average Time Series Models, Journal of the American Statistical Association, 65, Bowman, A.W., and Azzalini, A. (1997), Applied Smoothing Techniques for Data Analysis, Oxford University Press, London. Bowman, K.O., and Shenton, L.R. (1975), Omnibus Test Contours for Departures from Normality Based on b 1 and b 2, Biometrika, 62,

21 D Agostino, R.B. (1971), An Omnibus Test of Normality for Moderate and Large Size Samples, Biometrika, 58, D Agostino, R.B. (1972), Small Sample Probability Points for the D test of normality, Biometrika, 59, D Agostino, R.B., and Stephens, M. (1986), Goodness-of-fit Techniques, Marcel Dekker, New York. Dufour, J., and Khalaf, L. (2001), Monte Carlo Test Methods in Econometrics, In: Baltagi, Badi (Ed.), Companion to Theoretical Econometrics, Blackwell, Oxford. Durfour, J., Farhat, A., Gardial, L., and Khalaf, L. (1998), Simulation-based Finite Sample Normality Tests in Linear Regressions, Econometrics Journal, 1, C154-C173. Hyndman, R.J. (1996), Computing and Graphing Highest Density Region, American Statistician, 50, Jarque, C.M., and Bera, A.K. (1980), Efficient Tests for Normality, Homoscedasticity and Serial Independence of Regression Residuals, Economics Letters, 6, Jarque, C.M., and Bera, A.K. (1981), Efficient Tests for Normality, Homoscedasticity and Serial Independence of Regression Residuals: Monte Carlo evidence, Economics Letters, 7, Jarque, C., Bera, A.K. (1987), A Test for Normality of Observations and Regression Residuals, International Statistical Review, 55, Lobato, I., Nankervis, J.C., and Savin, N.E. (2001), Testing for Autocorrelation Using a Modified Box-Pierce Q Test, International Economic Review, 42, Pearson, E.S., D Agostino, R.B., and Bowman, K.O. (1977), Test for Departure from Normality: Comparison of Powers, Biometrika, 64, 2, Poitras, G. (1992), Testing Regression Disturbances for Normality with Stable Alternatives: Further Monte Carlo Evidence, Journal of Statistical Computation and Simulation, 41, Poitras, G. (2006), More on the Correct Use of Omnibus Tests for Normality, Economics Letters, 90,

22 Romano, J.L., and Thombs, L.A. (1996), Inference for Autocorrelation under Weak Assumptions, Journal of the American Statistical association, 91, Scott, D.W. (1992), Multivariate Density Estimation: Theory, Practice, and Visualization, John Wiley & Sons, New York. Shapiro, S.S. and Wilk, M.B. (1965), An Analysis of Variance Test for Normality (Complete Samples), Biometrika, 52, Silverman, B.W. (1986), Density Estimation for Statistics and Data Analysis, Chapman & Hall, London. Spiegelhalter, D.J. (1980), An Omnibus Test for Normality for Small Samples, Biometrika, 67, Thode, H. (2002), Testing for Normality, Marcel Dekker, New York. Urzúa, C.M. (1996), On the Correct Use of Omnibus Tests for Normality, Economics Letters, 53, Wand, M.P., and Jones, M.C. (1995), Kernel Smoothing, Chapman & Hall, London. Zhang, X., King, M.L., and Hyndman, R.J. (2006), A Bayesian Approach to Bandwidth Selection for Multivariate Kernel Density Estimation, Computational Statistics and Data Analysis, 50,

23 Table 1: Estimated sizes of tests for autocorrelations up to the 4th order. Tests Sample Algorithm 1 Algorithm 2 Size MKD MKD Q Q

24 Table 2: Estimated powers of tests for autocorrelations up to the 4th order. Tests Sample Algorithm 1 Algorithm 2 Size ρ = 0.25 MKD MKD Q Q ρ = 0.5 MKD MKD Q Q

25 Table 3: Estimated sizes of tests for autocorrelations up to the 6th order. Tests Sample Algorithm 1 Algorithm 2 Size MKD MKD Q Q

26 Table 4: Estimated powers of tests for autocorrelations up to the 6th order. Tests Sample Algorithm 1 Algorithm 2 Size ρ = 0.25 MKD MKD Q Q ρ = 0.5 MKD MKD Q Q

27 Table 5: Estimated sizes of Jacque-Bera tests and our testing procedure. Tests Sample Algorithm 1 Algorithm 2 Size MKD MKD JB MJB Table 6: Estimated powers of the Jacque-Bera tests and our testing procedure. Tests Sample Algorithm 1 Algorithm 2 Size MKD MKD JB MJB

28 Table 7: Estimated sizes of the D test and our testing procedure. Tests Sample Algorithm 1 Algorithm 2 Size MKD MKD D Table 8: Estimated powers of the D test and our testing procedure. Tests Sample Algorithm 1 Algorithm 2 Size MKD MKD D

29 Table 9: Estimated powers of the D test and our testing procedure. Tests Sample Algorithm 1 Algorithm 2 Size MKD MKD D Table 10: Estimated sizes of our testing procedure and the F test. Tests Sample Algorithm 1 Algorithm 2 Size MKD MKD F

30 Table 11: Estimated powers of our testing procedure and the F test. Tests Sample Algorithm 1 Algorithm 2 Size MKD MKD F