LECTURES IN ECONOMETRIC THEORY. John S. Chipman. University of Minnesota

Transcription

1 LCTURS IN CONOMTRIC THORY John S. Chipman University of Minnesota Chapter 5. Minimax estimation 5.. Stein s theorem and the regression model. It was pointed out in Chapter 2, section 2.2, that if no a priori knowledge is specified concerning β, the criterion of minimization of a matrix-valued meansquare error is ill posed. Nevertheless, there are some cases in which the choice of a particular scalar-valued definition of mean-square error makes it possible to obtain estimators with lower mean-square error than the Gauss-Markoff estimator, for all β. In such cases, the Gauss-Markoff estimator is inadmissible in the sense of Wald (95). A situation of this kind was first discovered by Stein (956). Let us now consider Stein s formulation. In the model (5..) y = Xβ + ε, ε =, εε = σ 2 Ω let the risk of any estimator ˇβ of β be defined as the scalar-valued mean-square error (5..2) Risk ˇβ =( ˇβ β) X Ω X( ˇβ β). We also define the modified F -statistic (5..3) f = ( β β) X Ω X( β β) ε Ω ε which corresponds to the case Ψ = I k and α = β and differs from the F -statistic (3.48) by the factor (n k)/r where in this case r = k. In Stein s formulation it is essential that normality is assumed, i.e., y N(Xβ,σ 2 Ω), where Ω is assumed to be positive definite. Stein considered only the special case Ω = I n and X =[I k, k (n k) ]. The following adaptation of Stein s result to the regression model is based largely on that of Sclove (968). Theorem 5... Let the estimator ˆβ ν be defined by (5..4) ˆβν = β ν f ( β β) = ( ν ) β + ν f f β Typeset by AMS-TX

2 2 JOHN S. CHIPMAN where β = X y =(X Ω X) X Ω y and β β is an initial guess at β, and (5..6) ε = y X β =[I XX ]y =[I X(X Ω X) X Ω ]y. Let it be assumed that y N(Xβ,σ 2 Ω),k 3. LetRisk ˆβ ν be defined by (5..2), and f by (5..3). Then Risk ˆβ ν < Risk β for all ν in the interval (5..7) <ν< and Risk ˆβ ν is minimized when 2(k 2) n k +2, (5..8) ν = k 2 n k +2. Proof. The deviations of ˆβ ν and β from β are respectively ˆβ ν β = β β ν f ( β β) and β β = X ε =(X Ω X) X Ω ε. Hence, since β is an unbiased estimator, and taking account of the symmetry of Ω XX and idempotency of XX, (5..9) Risk β =( β β) X Ω X( β β) =ε X X Ω XX ε =ε Ω XX ε =tr [XX εε Ω ] = σ 2 tr [XX ]=σ 2 tr [X X] = σ 2 tr I k = σ 2 k. Likewise, Risk ˆβ ν = ( ˆβ ν β) X Ω X( ˆβ ν β) [ ] [ ν = β β f ( β β) X Ω X β β ν ] f ( β β) = σ 2 ( k 2ν β β) X Ω X( β β) f + ν 2 ( β β) X Ω X( β β) (5..) f 2. From (5..9) and (5..3) this yields (5..) Risk β Risk ˆβ ν =2ν ( ε Ω ε) ( β β)x Ω X( β β) ( β β)x Ω X( β β) ν 2 ( ε Ω ε) 2 ( β β) X Ω X( β β).

3 LCTURS IN CONOMTRIC THORY 3 Now, since β β = X ε and ε =[I XX ]ε, it follows that, since XX Ω is symmetric and X X = I, ( β β) ε =X εε [I XX ] = σ 2 X Ω[I X X ] = σ 2 X [I XX ]Ω =, hence β and ε, being normally distributed vectors, are independent. Consequently, any functions of these random variables are independent. (5..) therefore becomes: (5..2) Risk β Risk ˆβ ν =2ν( ε Ω ε) ( β β)x Ω X( β β) ( β β)x Ω X( β β) ν 2 ( ε Ω ε) 2 ( β β) X Ω X( β β). In what follows we show, firstly, that (5..3) ( ε Ω ε) =σ 2 (n k) (Lemma 5.. below); secondly, that (5..4) ( ε Ω ε) 2 = σ 4 (n k)(n k +2) (this follows from the corollary to Lemma below, since ε Ω ε/σ2 χ 2 (n k)); and thirdly (Lemma below the James-Stein lemma ) that (5..5) ( β β)x Ω X( β β) ( β β)x Ω X( β β) =(k 2)σ 2 ( β β) X Ω X( β β) From (5..3), (5..4), and (5..5) it follows that (5..2) becomes. (5..6) Risk β Risk ˆβ ν = σ 4 (n k)ν2(k 2) ν(n k +2) ( β β) X Ω X( β β), and thus Risk ˆβ ν < Risk β if and only if (5..7) holds. Minimizing (5..6) with respect to ν we obtain (5..8), and Risk ˆβ ν becomes Risk ˆβ min Risk ˆβ ν ν (5..7) = σ 4 (n k)(k 2)2 k n k +2 ( β β) X Ω X( β β).

4 4 JOHN S. CHIPMAN To form an idea of the likely quantitative relationship between the Stein estimator ˆβ and the least-squares estimator β, we may relate the f-ratio to the squared multiple correlation coefficient R 2, which may be written (5..8) R 2 = ε Ω ε y Ω y = y Ω XX y y Ω. y From the definition (5..3) of f we have, for the case β =, (5..9) f = y Ω [I XX ]y y Ω = R2 y R 2. Substituting (5..9) in (5..4) for the value (5..8) of ν it becomes ( (5..2) ˆβ = k 2 R 2 ) β n k +2 R 2 We see immediately from (5..2) that if the regression relation (5..) has a good fit, in the sense that it has a multiple correlation coefficient close to (or more generally from (5..3) and (5..4), if f is large, so that an f-test strongly rejects the null hypothesis that β = β), then the Stein estimator will differ very little from the least-squares estimator. On the other hand, if an f-test does not reject this null hypothesis, or if (in the case β =)R 2 is rather low, then the Stein estimator can be expected to differ substantially from the least-squares estimator; however, this is precisely the case in which one may have doubts concerning whether the model (5..) is correctly specified. Thus, Stein estimation is best suited to cases in which one has great confidence in the model specification but insufficient data to give rise to a significant correlation coefficient. In the following section the fundamental lemmas underlying the above result will be proved. It will be convenient first, however, to reduce the above formulation to the normalized form treated in section 5.2. Let K be a k k nonsingular matrix such that K X Ω XK = I k, and define (5..2) z = σ K ( β β) and ζ =z = σ K (β β), so that (5..22) z ζ = σ K ( β β) =σ K X ε. Then (5..23) z z = ( β β) X Ω X( β β) σ 2 and (5..24) z (z ζ) = ( β β) X Ω X( β β) σ 2. Now, (z ζ)(z ζ) =σ 2 K X εε X K = K X ΩX K = K (X Ω X) K = I k,

5 LCTURS IN CONOMTRIC THORY 5 hence z N(ζ,I). The James-Stein lemma (Lemma below) states that z (z ζ) (5..25) z =(k 2) z z. z From (5..24) and (5..25) we have clearly ( (5..26) β β) X Ω X( β β) z (z ζ) ( β β) X Ω X( β β) = z, z but (5..27) ( β β) X Ω X( β β) = σ 2 z, z hence from (5..25), (5..26), and (5..27), we obtain (5..5) above. Since the above formula (5..3) is elementary, and does not depend on the normality of ε, its derivation is given here: Lemma 5... Let ε be defined by (5..6); then formula (5..3) holds. Proof. From the idempotency of XX and the symmetry of Ω XX,wehave (5..28) ( ε Ω ε) =ε [I XX ] Ω [I XX ]ε =ε Ω [I XX ]ε =tr[i XX ]εε Ω = σ 2 tr [I XX ] = σ 2 (n k) Lemmas underlying Stein s theorem. We now prove the basic lemmas underlying Theorem 5... Lemma 5.2., due to Stein (974), underlies the James-Stein (96) lemma (Lemma 5.2.2). Lemma 5.3, due to fron and Morris (976), provides a corollary which furnishes a derivation of formula (5..4). In the statements and proofs of these lemmas we shall follow the convention of denoting random variables by upper-case letters and their realizations by lower-case letters. Lemma 5.2. (Stein). Let N(ζ,), andleth : R R be any absolutely continuous function with Lebesgue-measurable derivative h satisfying h(b) h(a) = b a h (z)dz for all a<band (5.2.) h () <. Then (5.2.2) Cov, h() =( ζ)h() =h (). Proof. It will be convenient to define x = z ζ, X = ζ, andg(x) =h(x + ζ). Then X N(, ), and (5.2.2) is equivalent to (5.2.3) g (X) =Xg(X),

6 6 JOHN S. CHIPMAN which will now be proved. From (5.2.) we have (5.2.4) g (X) =(2π) /2 g (x) e x2 /2 dx <. For <a<x, (5.2.5) g(x) g(a) = x a g (u)e u2 /2 e u2 /2 du /2 ex2 a g (u) e u2 /2 du, and from (5.2.4) it follows that for any ε> one can choose a sufficiently large so that (5.2.6) a g (u) e u2 /2 du < ε/2, and x sufficiently large so that (5.2.7) g(a)e x2 /2 <ε/2. Thus, from (5.2.5), (5.2.6), and (5.2.7), since g(x) g(a) g(x) g(a), (5.2.8) g(x)e x2 /2 < g(a)e x2 /2 + a g (u) e u2 /2 du < ε. Therefore, (5.2.9) lim x g(x)e x2 /2 =. A similar argument shows that (5.2.) lim x g(x)e x2 /2 =. Now, integrating by parts we obtain g (X) =(2π) /2 g (x)e x2 /2 dx =(2π) /2 [ g(x)e x2 /2 ] +(2π) /2 g(x)xe x2 /2 dx. The first term on the right vanishes by virtue of (5.2.9) and (5.2.), and the second term is just Xg(X). This proves (5.2.3). Stein s basic result follows from the following fundamental lemma due to James and Stein (96, pp ):

7 LCTURS IN CONOMTRIC THORY 7 Lemma (James & Stein). Let be a k random vector such that N(ζ,I k ).Then ( ζ) (5.2.) =(k 2). Proof. Let N(ζ,I k ) and define, for z =(z,z 2,...,z k ), h i (z) = z i z z. Denoting )i( =(, 2,..., i, i+,..., k ), we have by Lemma ( ) (i ζ i ) i ( i ζ i ) i = )i( ( ( ) ) i = i )i( ( ) = 22 i ( ) 2 )i( = 22 i ( ) 2. Thus, ( ζ) = k 2 ( ) 2 =(k 2). The following lemma, due to fron & Morris (976), underlies the derivation of formula (5..4). Lemma (fron & Morris). Let W be distributed according to the gamma density (5.2.2) f(w) = wa e w for <w<, a>, where Γ(a) =(a )! Γ(a) Then, for any absolutely continuous and continuously differentiable function h : R + R such that (5.2.3) h(w ) <, h (W ) <, Wh(W ) <, Wh (W ) <, we have (5.2.4) Cov ( W, h(w ) ) =(W W )h(w ) =Wh (W ). Proof. Integrating by parts, we have (5.2.5) Wh (W ) = wf(w)h (w)dw [ ] = wf(w)h(w) [f(w)+wf (w)] h(w)dw.

8 8 JOHN S. CHIPMAN The first term on the right vanishes, since Wh(W ) = wh(w)f(w)dw < by (5.2.3), hence lim w wh(w)f(w) =. From (5.2.2), (5.2.6) f(w)+wf (w) = Γ(a) wa e w + w[(a )w a 2 e w w a e w ] = Γ(a) wa e w (a w) = (w a)f(w). Therefore, from (5.2.5) and (5.2.6), (5.2.7) Wh (W ) = (w a)h(w)f(w)dw =(W a)h(w ). To establish (5.2.4) it remains only to verify that W = a. The momentgenerating function of (5.2.2) is, for t<, m W (t) =e tw = Defining y =( t)w, this may be written m W (t) =( t) a y a e y Γ(a) w a e ( t)w dw. Γ(a) dy =( t) a f(y)dy =( t) a. The mean of f is W = m W () = a, aswastobeshown. Corollary to Lemma Let U be distributed as chi-square with d degrees of freedom. Then U 2 = d(d +2). Proof. If U χ 2 (d) thenw = U/2 has the gamma distribution with mean a = d/2. Defining h(w )=W we have from (5.2.4): (5.2.8) W 2 =(W ) 2 +W = a(a +). Consequently, (5.2.9) U 2 =4W 2 =4a(a +)=d(d +2), as was to be shown.

9 LCTURS IN CONOMTRIC THORY 9 References Chipman, John S. Statistical Problems Arising in the Theory of Aggregation, in Paruchuri R. Krishnaiah, ed., Applications of Statistics. Amsterdam: North- Holland Publishing Company, 977, pp fron, Bradley, and Carl Morris. Families of Minimax stimators of the Mean of a Multivariate Normal Distribution, Annals of Statistics, 4 (January 976), 2. James, W., and Charles Stein. stimation with Quadratic Loss, Proceedings of the Fourth Berkeley Symposium on Mathematical Statistics and Probability, Vol. I (Berkeley and Los Angeles: University of California Press, 96), Perlman, M. S. Reduced Mean Square rror stimation for Several Parameters, Sankhyā [B], 34 (972), Sclove, L. Stanley. Improved stimators for Coefficients in Linear Regression, Journal of the American Statistical Association, 63 (June 968), Sclove, L. Stanley, Carl Morris, and R. Radhakrishnan. Non-optimality of Preliminary-Test stimators for the Mean of a Multivariate Normal Distribution, Annals of Mathematical Statistics, 43 (October 972), Stein, Charles. Inadmissibility of the Usual stimator for the Mean of a Multivariate Normal Distribution, Proceedings of the Third Berkeley Symposium on Mathematical Statistics and Probability, Vol. I (Berkeley and Los Angeles: University of California Press, 956), Stein, Charles. Multiple Regression, in Ingram Olkin (ed.), Contributions to Probability and Statistics. ssays in Honor of Harold Hotelling (Stanford, California: Stanford University Press, 96), Stein, Charles. An Approach to the Recovery of Interblock Information in Balanced Incomplete Block Designs, in F. N. David (ed.), Research Papers in Statistics: Festschrift for J. Neyman (New York: John Wiley & Sons, 966), Stein, Charles. stimation of the Mean of a Multivariate Normal Distribution, in Jaroslav Hájek (ed.), Proceedings of the Prague Symposium on Asymptotic Statistics, 3 6 September 973, Vol. II. (Prague: Charles University, 974), (Previously issued as Technical Report No. 48, June 26, 973, Department of Statistics, Stanford University, Stanford, California.) Theil, Henri. conomic Forecasts and Policy, 2nd edition. Amsterdam: North- Holland Publishing Co., 96. Wald, Abraham. Statistical Decision Functions. New York: John Wiley & Sons, 95.