Estimation, admissibility and score functions

Transcription

1 Estimation, admissibility and score functions Charles University in Prague

2 1 Acknowledgements and introduction 2 3 Score function in linear model. Approximation of the Pitman estimator 4 5

3 Acknowledgements The subject of score functions I discussed and worked on mainly with Xavier Milhaud and Pranab K. Sen. However, during the time I met many excellent colleagues from whom I learned much. I started to write the list, but then I was afraid that I could miss some name. I was happy to learn not only from colleagues in Czech Republic, but also from Australia, Belgium, Canada, Denmark, France, Germany, Greece, Hungary, India, Italy, Japan, the Netherlands, Poland, Russia, Switzerland and United States. I have also learned from my excellent PhD students. I would like to thank all these people, and it makes me happy to share the obtained knowledge with the students and who is interested.

4 Introduction The score function of the random vector Y 1,..., Y n with density p(y 1,..., y n, θ) = p(y, θ), θ Θ R p, is the vector function ( ) ln p(y, θ) l p (, θ) =, j = 1,..., p. (1) θ j The score function is in the 1:1 correspondence with the probability distribution, and it is one of its main characteristics. Using the score functions was recommended by Hampel (1973), Field and Hampel (1982), Field and Ronchetti (1990), among others. A natural argument for the use of the score function is that it is linear for the normal distribution, which is considered as a unit in the family of probability distributions.

5 Location and regression models Random variable X with density f (x θ), θ R 1 has score function ψ f (x θ) = f (x θ) f (x θ). In the linear regression model with observations Y = (Y 1,..., Y n ), when Y i f (y x i β), 1 i n, the score function of Y is n x iψ f (y i x i β). The score function plays a basic role in the statistical inference; let us mention the maximum likelihood estimation, score tests, locally or asymptotically optimal rank tests, etc. The role of ψ f (X ) under f is analogous to the role of X under the normal f. ψ f uniquely characterizes distribution f and ψ f corresponding to normal f is linear; hence the knowledge of the logarithmic derivative of the density of a statistic S is of interest; its curvature shows the distance of the distribution of S from the normal.

6 Score function of a statistic Lemma Let X 1,..., X n be i.i.d. with density f (x, θ), differentiable in the components of θ, θ Θ R p, the Fisher information matrix J f (θ) positive definite. Let S n be a statistic with density g(s, θ), differentiable in θ. Then g θ (S) g θ (S) = E θ ( log g(s, θ) θ 1,..., [ fθ (X) f θ (X) ] S a.s. Pθ S (likelihood ratio), ) log g(s, θ) { = E θ θ p } l(x, θ) S = s where E θ is the expectation with respect to n f (x i, θ).

7 Application: Expansion of the test power Theorem Let Φ(X) be an α-test of H 0 : θ = θ 0 against K : θ θ 0 and let f (x, θ) be three times differentiable in θ. The test criterion S let satisfy the conditions of Lemma. Then E θ Φ(X) = α + (θ θ 0 ) E θ0 {l(x, θ 0 )Φ(X)} (2) (θ θ 0) E θ0 {Φ(X)A(X, θ 0 )} (θ θ 0 ) + O( θ θ 0 3 ) where A(x, θ) = [ l j (x, θ)l k (x, θ) + l j (x,θ) θ k ] p j,k=1.

8 Hence, unless E θ0 {l(x, θ 0 )Φ(X)} = 0, (3) the second term of the right-hand side of of (2) may be < 0 for some θ θ 0. This means that the test generally is not unbiased in a neighborhood of θ 0. This phenomenon is often ignored, while it deserves a special attention. Although the unbiasedness is a basic property of a good test, many tests on vector parameters or on scalar parameters against two-sided alternatives are not finite-sample unbiased. This was already noticed by Sugiura (1965); he found a (skew) alternative against which the Wilcoxon test was not unbiased. More authors were aware of this fact: Grose and King (1991) imposed condition (3) when they constructed a locally unbiased two-sided version of the Durbin-Watson test.

9 Linear model: Approximation of Pitman estimator Consider the model Y = Xβ + e with an unknown parameter β, where e is a vector of i.i.d. errors with absolutely continuous density f > 0, the matrix X of rank p. Consider the problem of estimating β under the loss L(b, β) = b β 2, invariant with respect to the group of transformations G = {Y + Xb, b R p }. One possible maximal invariant is Z = Y X β, β being the LSE. A broad class of estimators T n of β admits a representation (e j is the j-th unit vector, ψ is a suitable function) T n (Y) = β + 1 γ (X X) 1 X n e j ψ(y j e j Xβ) + R n, j=1 R n = o p ( X X 1/2 ). (4)

10 If T n is the LSE, the function ψ in (4) is linear and (4) holds as an identity. Kagan, Linnik and Rao (1973) showed that the admissibility of LSE implies that IE 0 ( β n Z) = 0, and this in turn implies the normality of the distribution. Even in the general case, the vanishing of the conditional expectation of the leading term on the right hand side of (4), given the maximal invariant Z, characterizes the distribution of the errors, via the score function ψ f = f f. Bondesson (1974) already proved a similar characterization for location and scale parameters, but did not use the score function, which makes the characterization more transparent and intuitive. More precisely, we can prove the theorem

11 Theorem Let f have finite Fisher s information and non-zero characteristic function ϕ. Let ψ f = f f. Then (i) X IE 0 ( n j=1 e jψ f (Y j ) Z) 0. (ii) Let ψ : IR 1 IR 1 be square-integrable with respect to f, n 3 + p and n X IE 0 ( e j ψ(y j ) Z) 0. j=1 Then ψ(x) = k ψ f (x) = k f (x) f (x) a.e. x IR1, 0 k = const. This is trivially true for f normal, but ψ f is nonlinear for non-normal f.

12 Approximation of the Pitman estimator The minimum risk equivariant estimator T n (Pitman, MRE) in the linear model we obtain when we start with an arbitrary equivariant estimator T 0 n of β that has a finite quadratic risk: Then T n = T 0 n IE 0 (T 0 n Z) (5) where the conditional expectation is calculated with respect to F under β = 0. It enjoys the finite-sample optimality property, but its calculation is difficult, with an exception of few special cases. T n = (Tn1,..., T np) can be expressed in the following integral form (see Kakosyan et al. (1990)), j = 1,..., p :... n f (Y i p T nj = b j... k=1 x ikb k )db 1... db p n f (Y i p k=1 x. ikb k )db 1... db p

13 Several approximations of the MRE were proposed, using e.g. the Hájek-Hoeffding projection, the Hoeffding-van Zwet decomposition or the asymptotic representation of the initial estimator T 0 n (e.g. Kakosyan et al. (1990), Jureč. and Sen (2001)). However, only using the score function avoids a calculation of a conditional expectation of any entity (Jureč. and Picek (2009)): Theorem Assume that the density f of the errors has finite Fisher information and that Q n = 1 n X n X n Q as n where Q is a positively definite p p matrix. Let T 0 n be an equivariant n-consistent initial estimator of β with a finite risk. Then T n = T 0 n + 1 ni(f ) Q 1 n X n n e i ψ f (Y i e i X n T 0 n) + o p (n 1/2 ).

14 Some finite-sample distributions are difficult to derive. However, we cannot limit ourselves to the asymptotics, which can be slow, can stretch the truth and approximates only the central part, not the tails. We can sometimes use the score functions: Example 1: Let X 1,..., X n F (x θ), Fisher information finite. Let S n = S n (X 1,..., X n ) be a statistic whose distribution function H θ (s) is continuously differentiable in θ. Then we have the following identity: H θ (s) θ = = IE θ [ n S(x) s n ( f (x i θ) f (x i θ) ) n f (x i θ)dx 1... dx n ( f ) (X i θ).i [S(X 1,..., X n ) s] f (X i θ) ]. (6)

15 LetT n be a translation equivariant estimator of θ with density g θ (t). Excellent approximations of g θ, working well numerically for very small samples were proposed e.g. by Field and Hampel (1982), Field and Ronchetti (1990) small-sample asymptotics. The exact expression of the finite-sample density g θ (t) we get from (6): g θ (t) = = IE 0 { n T (x) t n f (x i θ) f (x i θ) n f (x k θ)dx 1... dx k k=1 f (X i ) [ ]} f (X i ) I T (X 1,..., X n ) t θ { n f (X i ) [ n ]} = IE 0 f (X i ) I ψ(x i (t θ)) 0, the last line holds if T n is a solution of the equation n ψ(x i t) = 0 with monotone ψ.

16 Score function and density of α-regression quantile Example 2: In the linear model, let x i1 = 0, i = 1,..., n. The regression α-quantile (0 < α < 1) minimizes the criterion n ρ α(y i x i b), b R p where ρ α (z) = z {αi [z > 0] + (1 α)i [z < 0]}, z R 1. Dual to β n (α) in the linear programming sense is the vector â n (α) = (â n1 (α),..., â nn α) of the regression rank scores, defined as a solution of the linear programming problem n â ni (α)y i = max under n x ij â ni (α) = (1 α) n x ij, 0 â ni (α) 1, i = 1,..., n. j = 1,..., p

17 Lemma The score function of (â(α), β(α)), b R p, a A n (α) is r(a, b, β) = n ( I x i {f (x [ai = 0] i [b β]) F (x i [b β]) I [a i = 1] ) 1 F (x + I [0 < a i < 1] f (x i [b β]) } i [b β]) f (x, i [b β] { n n A n (α) = a : 0 a i 1, x i a i I [a i > 0] = (1 α) 0 < a ij < 1, j = 1,..., p for 1 i 1,..., i p n, such that x i1,..., x ip is a basis of R p}. x i

18 The score function r(a, b, β) is obtained as the conditional expectation of the model score function given (â(α), β(α)). It also provides an alternative expression for the moments of the regression quantiles. Integrating r(a, b, β) over β, we obtain the joint distribution of the α-regression quantile and the α-regression rank scores, and this in turn leads to the marginal density of the α-regression quantile. In the location model with X = 1 n, density (7) reduces to the density of the order statistic Y n:k with k = nα. It illustrates the fact that the regression quantile is, indeed, an extension of the sample quantile to the regression model, and that not only asymptotically. Quite similarly we derive the density of the α-regression quantile for the non-i.i.d. errors including the heteroscedastic model, and the density of the extreme regression quantile.

19 Theorem: Density of RQ Theorem Let the regression matrix X be of rank p, with the first column equal to 1 n, let the errors e 1,..., e n be i.i.d. with distribution function F and density f, absolutely continuous and positive for z (A, B) where A = inf{z : F (z) > 0} and B = sup{z : F (z) < 1}. Then the α-regression quantile β(α), 0 < α < 1 has the density g(b; α) = n a A n(α) ) I [ai =1]( ( 1 F (x i [b β] [( ) I F (x [ai =0] i [b β]) f (x i [b β]) ) I [0<ai <1]] (7), b R p.

20 Let X = (X 1,..., X p ), p 3 have the density p f (x 1 θ 1,..., x p θ p ) = f i (x i θ i ) = exp{ p ρ i (x i θ i )}, where ρ i is an absolutely continuous function with derivative ψ i = ρ i, and f i has positive and finite Fisher s information I(f i ). Let g : IR p IR 1 be an absolutely continuous function. Denote i g = g(x) x i, i = 1,..., p, g = ( 1 g,..., p g), and g = p 2 i g (whenever it exists). Assume that IE θ g(x) <. Then the following identity is a slight extension of Stein s identity from the normal to general f : IE θ [ g(x)] = IE θ [g(x)(ψ 1 (X 1 θ 1 ),..., ψ p (X p θ p )) ].

21 Let g(x) : IR p (0, ) < be superharmonic, i.e. the value g(ξ) at the center ξ of the ball B(ξ, δ) R p with radius δ is greater or equal to the average of g over the boundary B(ξ, δ) of B(ξ, δ), ξ, δ. Then g(x) 0 x R p. We can extend the Corollary 1 of Stein (1981) to show the shrinkage phenomenon of (ψ 1 (X 1 θ 1 ),..., ψ p (X p θ p )) : Lemma Let g be superharmonic and IE θ {P p 2 i g(x) g(x) } <. Then IE θ (ψ 1 (X 1 θ 1 ),..., ψ p (X p θ p )) + log g(x) 2 = p { } g(x) I(f i ) + 4IE θ g(x) p I(f i ) = IE θ (ψ 1 (X 1 θ 1 ),..., ψ p (X p θ p )) 2.

22 Bondesson, L. (1974). Characterization of probability laws through constant regression. Z. Wahrscheinlichkeitsth. verw. Geb. 30, Field, C. A. and Hampel, F. R. (1982). Small-sample asymptotic distribution of M-estimators of location. Biometrika 69, Field, C. A. and Ronchetti, E. (1990). Small Sample Asymptotics. IMS Lecture Notes, Monograph Series, 13, Hayward, CA. Grose, S. D. and King, M. L. (1991). The locally unbiased two-sided Durbin-Watson test. Economics Letters 35, Gutenbrunner, C. and Jurečková, J. (1992). Regression rank scores and regression quantiles. Ann. Statist. 20,

23 Hampel, F.R. (1973). Some small-sample asymptotics. In: Prof. Prague Symposium on Asymptotic Statistics (J. Hájek, ed.), pp Charles University in Prague. Jurečková, J. (2007). Remark on extreme regression quantile. Sankhya 69, Part 1, Jurečková, J. (2010). Finite-sample distribution of regresion quantile. Statist.& Probab. Letters (to appear). Jurečková, J. and Milhaud, X. (1993). Shrinkage of maximum likelihood estimator of multivariate location. Asymptotic Statistics [Proc. 5th Prague Symp.] (P.Mandl and M.Hušková, eds.), pp Physica-Verlag, Heidelberg.

24 J. Jurečková and X. Milhaud (1998). Characterization of distributions in invariant models. J. Statist. Planning Infer. 75, Jurečková, J. and Milhaud, X. (2003). Derivative in the mean of a density and statistical applications. In: IMS Lecture Notes, Monograph Series, 42, (M.Moore, S. Froda, C. Léger, eds.), pp Hayward, CA. J. Jurečková and J. Picek (2009). Minimum risk equivariant estimators in linear regression model. Statistics & Decisions 27, /DOI /stnd

25 J. Jurečková and P. K. Sen (2001). Asymptotically minimum risk equivariant estimators. Data Analysis from Statistical Foundations Festschrift in honour of the 75th birthday of D.A.S. Fraser (A. K. Md. E. Saleh, ed.), Nova Science Publ., Inc., Huntington, New York. J. Jurečková and P. K. Sen (2006). Robust multivariate location estimation, admissibility and shrinkage phenomenon. Statistics & Decisions 24, Kagan, A. M., Linnik, Ju. V. and Rao, C.R. (1972). Characteristic Problems of Mathematical Statistics (in Russian). Nauka, Moscow. Kakosyan, A. V., Klebanov, L. B. and Melamed, I. A. (1990). Nonlinear statistical estimation of parameters of linear regression models (in Russian). YerINE, Yerevan.

26 Koenker, R. (2005). Quantile Regression. Cambridge University Press, Cambridge. Koenker, R. and Bassett, G. (1978). Regression quantiles. Econometrica Portnoy, S. and J. Jurečková (1999). On extreme regression quantiles. Extremes 2, Stein, C. (1981). Estimation of the mean of a multivariate normal distribution. Ann. Statist. 9, Sugiura, N. (1965). An example of the two-sided Wilcoxon test which is not unbiased. Ann. Inst. Statist. Math. 17,