Chi-squared tests Math 6070, Spring 2014

Transcription

1 Chi-squared tests Math 6070, Sprig 204 Davar Khoshevisa Uiversity of Utah March, 204 Cotets MLE for goodess-of fit 2 2 The Multivariate ormal distributio 3 3 Cetral limit theorems 5 4 Applicatio to goodess-of-fit 8 4 Testig for a fiite distributio 8 42 Testig for a Desity 9 Chi-squared tests refers to a family of statistical test whose large-sample asymptotics are related i oe way or aother to the χ 2 family of distributios These are our first examples of o-parametric procedures Let me metio oe particularly useful chi-squared test here before we get started properly Cosider a populatio whose distributio is believed to follow a kow desity fuctio f say, expoetial(2) The there is a chi-squared test that ca be used to check our hypothesis This chi-squared test is easy to implemet, ad works well for large-sized samples First, we study a related parametric estimatio problem The we fid a cosistet MLE Ad fially, we use our estimate to devise a test for our hypothesis This will be doe i several stages It is likely that you already kow somethig of the resultig χ 2 -test (page 8) But here you will fid a rigorous proof, i lie with the spirit of this course, of the fact that the said χ 2 -test actually works A [covicig but] o-rigorous justificatio of the same fact ca be foud i pages of the excellet moograph of P Bickel ad K Doksum [Mathematical Statistics, Holde Day, First editio, 977] There, a likelihood priciple argumet is devised, which usually works

2 MLE for goodess-of fit Suppose X,, X is a idepedet sample from a fiite populatio More precisely, let t,, t m be m distict umbers (or types ), ad θ := (θ,, θ m ) a m-dimesioal probability vector Cosider iid radom variables X,, X such that P{X = t i } = θ i for all i m That is, each X is of type t i with probability θ i We wish to fid a (actually, the ) MLE for the mass fuctio of the X i s That is, the MLE for the vector θ := (θ,, θ m ) Note that, here, Θ is the collectio of all θ such that 0 θ,, θ m ad θ + + θ m = This is a kid of parametric problem, of course, ad we proceed by computig the MLE Defie N := (N,, N m ) to be the cout statistics for the various types Usig colum-vector otatio, this meas that j= I {X j = t } N = () j= I {X j = t m } I words, N k deotes the umber of occuraces of type k i the sample X := (X,, X ) Note that m j= N j =, so the N j s are obviously depedet rv s Let us calculate their joit mass fuctio Let = (,, m ) be a m-vector of positive itegers such that m j= j = Such s are the possible values of N Moreover, P θ {N = } = P θ {X = s,, X = s }, (2) where the sum is take over all s = (s,, s ) such that of them are of type t, 2 of type t 2, etc For all such s, P θ {X = s,, X = s } = θ θm m (3) Moreover, the umber of all such s s is exactly ( )! :=,, m! m! (4) Therefore, we have derived the followig Lemma The radom -vector N has the multiomial distributio with parameters (, θ) That is, for all itegers,, m such that + + m =, ( ) P θ {N = } = θ,, θm m (5) m Cosider the special case = 2 The, N biomial(, θ), ad N 2 = N So the multiomial distributio is a extesio of the biomial i this sese We write N Multiomial(, θ) i place of the statemet that N is multiomial with parameters (, θ) 2

3 It turs out that we ca compute the MLE for θ by usig N as the basic data To do so, we use Lemma ad first fid the log-likelihood as follows: L(θ) = A + m N i l θ i, (6) where A does ot deped o θ The case where some θ i {0, } is trivial ad ca be ruled out easily Therefore, we may assume that 0 < θ i < for all i =,, m Thus, for all l =,, m, m ( Ni L(θ) = θ ) i (7) θ l θ i θ l As θ varies i Θ, θ,, θ m roam freely subject to the sigle coditio that θ m = m θ i I other words, ( θ i / θ l ) = 0 for all i =,, m differet from l, but ( θ m / θ l ) = This proves that ( L(θ)/ θ l ) = (N m /θ m ) (N l /θ l ) for all l =,, m Set it equal to zero (for MLE) to see that there must exist γ such that N j = γθ j for all j =,, m But m j= N j =, whereas m j= θ j = Therefore, γ = We have proved the followig Lemma 2 The MLE for θ based o N,, N m is give by N / ˆθ := (8) N m / [To prove that L/ θ l = 0 ideed yields a maximum, you take secod derivatives ad apply similar argumets] 2 The Multivariate ormal distributio We have see already the multiomial distributio, which is a probability distributio for discrete m-dimesioal radom vectors You should ote that the multiomial distributio is a geeralizatio of the biomial distributio Ad, as it turs out, multiomial distributios have a multivariate ormal approximatio that is a geeralizatio of the better-kow cetral limit theorem for uivariate radom variables I this sectio, I will hash out a rough outlie of the theory of multivariate ormal distributios [ad radom vectors] The mathematical uderpiigs of this theory is explaied i math 600, ad should ot be missed Defiitio 3 Let X := (X,, X m ) deote a m-dimesioal radom vector It mea vector is defied as E(X ) E(X) :=, E(X m ) 3

4 ad its covariace matrix is defied as the matrix whose (i, j)th etry is Cov(X i, X j ); sice Cov(X i, X i ) = Var(X i ), Var(X ) Cov(X, X 2 ) Cov(X, X m ) Cov(X 2, X ) Var(X 2 ) Cov(X 2, X m ) Cov(X) := Cov(X m, X ) Cov(X m, X 2 ) Var(X m ) Of course, we are tacitly assumig that the meas, variaces, ad covariaces are fiite If x is a m-dimesioal oradom vector, the x Cov(X)x is always a scalar Oe ca show that, i fact, that scalar is equal to the variace of the radom scalar quatity x X That is, Var (x X) = x Cov(X)x Sice the variace of a radom variable is always 0, it follows that if Σ := Cov(X), the x Qx 0 for all x Ay matrix that has the precedig property is called positive semidefiite, ad we just saw that all covariace matrices are positive semidefiite The coverse is also true, but harder to prove Propositio 4 Let Q be a symmetric m m matrix The Q is the covariace of a radom vector X if ad oly if Q is positive semidefiite Also, a symmetric m m matrix Q is positive semidefiite if ad oly if all of eigevalues are 0 We say that Q is positive defiite if x Qx > 0 for all x Propositio 5 A symmetric m m matrix Q is positive defiite if ad oly if all of its eigevalues are > 0 I particular, such a matrix is ivertible Defiitio 6 Give a m-dimesioal vector µ ad a m m-dimesioal covariace matrix Q, the multivariate ormal pdf with mea vector µ ad covariace matrix Q is f(x) := exp (2π) m/2 det Q /2 ( ) 2 (x µ) Q (x µ) This is a proper pdf o R m We say that X N m (µ, Q), whe X = (X,, X m ), ad the joit pdf of (X,, X m ) is f The precedig fuctio f is ideed a proper joit desity i the sese that f(x) 0 for all x := (x,, x m ) ad So it makes sese to say X N m (µ, Q) f(x,, x m ) dx dx m = 4

5 Propositio 7 If X N m (µ, Q) for some vector µ := (µ,, µ m ) ad matrix Q, the µ = E(X), ad Cov(X) = Q Thus, we refer to µ ad Q respectively as the mea vector ad covariace matrix of the N m (µ, Q) distributio The followig states that a radom vector is distributed as a multivariate ormal if ad oly if all of possible [oradom] liear combiatios of its compoets are distributed as [uivariate] ormals Propositio 8 X N m (µ, Q) if ad oly if t X N(t µ, t Qt) for all oradom m-vectors t := (t,, t m ) I other words, the precedig states that the property about the liear combiatios of the compoets characterizes completely the multivariate ormal distributio A advatage of the precedig characterizatio is that the osigularity of Q is ot metioed aywhere explicitly Therefore, we ca defie the followig slight geeralizatio of multivariate ormals to the case that Q is positive semidefiite [as opposed to beig positive defiite]: Defiitio 9 Let X be a radom m-vector We say that X N m (µ, Q), for some µ = (µ,, µ m ) ad some m m symmetric positive semidefiite matrix Q, whe t X N(t µ, t Qt) for all t := (t,, t m ) 3 Cetral limit theorems Choose ad fix a probability vector θ, ad assume it is the true oe so that we write P, E, etc i place of the more cumbersome P θ, E θ, etc We ca use () to say a few more thigs about the distributio of N Note first that () is equivalet to the followig: N = I {X j = t } I {X j = t m } j= We ca compute directly to fid that := Y j (9) j= EY = θ, (0) ad θ ( θ ) θ θ 2 θ θ m θ θ 2 θ 2 ( θ 2 ) θ 2 θ m CovY = θ m θ θ m θ 2 θ m ( θ m ) := Q 0 () 5

6 Because Y,, Y are iid radom vectors, the followig is a cosequece of the precedig, cosidered i cojuctio with the multidimesioal CLT Theorem 0 As, N θ d Nm (0, Q 0 ) (2) There is a variat of Theorem 0 which is more useful to us Defie (N θ )/ θ Ñ := (N m θ m )/ (3) θ m This too is a sum of iid radom vectors Ỹ,, Ỹ, ad has EỸ := 0 ad CovỸ = Q, (4) where Q is the (m m)-dimesioal covariace matrix give below: θ θ θ 2 θ θ m θ θ 2 θ 2 θ 2 θ m Q = θ m θ (5) θ m θ 2 θ m The we obtai, Theorem As, Ñ d N m (0, Q) (6) Remark 2 Oe should be a little careful at this stage because Q is sigular [For example, cosider the case m = 2] However, N m (0, Q) makes sese eve i this case The reaso is this: If C is a o sigular covariace matrix, the we kow what it meas for X N m (0, C), ad a direct computatio reveals that for all real umbers t ad all m-vectors ξ, [ ( )] E [exp (itξ X)] = E exp t2 2 ξ Cξ (7) This defies the characteristic fuctio of the radom variable ξ X uiquely There is a theorem (Ma Wold device) that says that a uique descriptio of the characteristic fuctio of ξ X for all ξ also describes the distributio of X uiquely This meas that ay radom vector X that satisfies (7) for all t R ad ξ R m is N m (0, C) Moreover, this approach does ot require C to be ivertible To lear more about multivariate ormals, you should atted Math 600 6

7 As a cosequece of Theorem we have the followig: As, Ñ 2 d Nm (0, Q) 2 (8) [Recall that x 2 = x x 2 m] Next, we idetify the distributio of N m (0, Q) 2 A direct (ad somewhat messy) calculatio shows that Q is a projectio matrix That is, Q 2 := QQ = Q It follows fairly readily from this that its eigevalues are all zeros ad oes Here is the proof: Let λ be a eigevalue ad x a eigevector Without loss of geerality, we may assume that x = ; else, cosider the eigevector y := x/ x istead This meas that Qx = λx Therefore, Q 2 x = λ 2 x Because Q 2 = Q, we have prove that λ = λ 2, whece it follows that λ {0, }, as promised Write Q i its spectral form That is, λ 0 0 Q = P 0 λ 2 0 P, (9) 0 0 λ m where P is a orthogoal (m m) matrix [This is because Q is positive semidefiite You ca lear more o this i Math 600] Defie λ λ2 0 Λ := P (20) 0 0 λm The, Q = Λ Λ, which is why Λ is called the square root of Q Let Z be a m- vector of iid stadard ormals ad cosider ΛZ This is multivariate ormal (check characteristic fuctios), E[ΛZ] = 0, ad Cov(ΛZ) = Λ Cov(Z)Λ = Λ Λ = Q I other words, we ca view N m (0, Q) as ΛZ Therefore, m N m (0, Q) 2 = d ΛZ 2 = λ j Zi 2 (2) Because λ i {0, }, we have proved that N m (0, Q) χ 2 r, where r = rak(q) Aother messy but elemetary computatio shows that the rak of Q is m [Roughly speakig, this is because the θ i s have oly oe source of liear depedece Namely, that θ + + θ m = ] This proves that Ñ 2 d χ 2 m (22) Write out the left-had side explicitly to fid the followig: Theorem 3 As, m (N i θ i ) 2 θ i d χ 2 m (23) It took a while to get here, but we will soo be rewarded for our effort 7

8 4 Applicatio to goodess-of-fit 4 Testig for a fiite distributio I light of the developmet of Sectio, ca paraphrase Theorem 3 as sayig that uder P θ, as, X (θ) d χ 2 m, where X (θ) := ) 2 m (ˆθi θ i (24) θ i Therefore, we are i a positio to derive a asymptotic ( α)-level cofidece set for θ Defie for all a > 0, { C (a) := θ Θ : X (θ) a } (25) Equatio (24) shows that for all θ Θ, lim P θ {θ C (a)} = P { χ 2 m a } (26) We ca fid (eg, from χ 2 -tables) a umber χ 2 (α) such that It follows that P { χ 2 m > χ 2 m (α) } = α (27) C ( χ 2 m (α) ) is a asymptotic level ( α) cofidece iterval for θ (28) The statistics X (θ) is a little awkward to work with, ad this makes the computatio of C (χ 2 m (α)) difficult Nevertheless, it is easy eough to use C (χ 2 m (α)) to test simple hypotheses for θ Suppose we are iterested i the test H 0 : θ = θ 0 versus H : θ θ 0, (29) where θ 0 = (θ 0,, θ 0m ) is a (kow) probability vector i Θ The we ca reject H 0 (at asymptotic level ( α)) if θ 0 C (χ 2 m (α)) I other words, we reject H 0 if X (θ 0 ) > χ2 m (α) (30) Because X (θ 0 ) is easy to compute, (29) ca be carried out with relative ease Moreover, X (θ 0 ) has the followig iterpretatio: Uder H 0, X (θ 0 ) = m (Obs i Exp i ) 2, (3) Exp i The statistic X (θ) was discovered by K Pearso i 900, ad is therefore called the Pearso χ 2 statistic Icidetally, the term stadard deviatio is also due to Karl Pearso (893) 8

9 where Obs i deotes the umber of observatios i the sample that are of type i, ad Exp i := θ 0i is the expected umber uder H 0 of sample poits that are of type i Next we study two examples Example 4 A certai populatio cotais oly the digits 0,, ad 2 We wish to kow if every digit is equally likely to occur i this populatio That is, we wish to test H 0 : θ = ( 3, 3, 3 ) versus H : θ ( 3, 3, 3 ) (32) To this ed, suppose we make, 000 idepedet draws from this populatio, ad fid that we have draw 300 zeros (ˆθ = 03), 340 oes (ˆθ 2 = 034), ad 360 twos (ˆθ 3 = 036) That is, Types Obs Exp (Obs Exp) 2 /Exp / / / Total (approx) Suppose we wish to test (32) at the traditioal (asymptotic) level 95% Here, m = 3 ad α = 005, ad we fid from chi-square tables that χ 2 2(005) 5999 Therefore, accordig to (30), we reject if ad oly if X (θ 0 ) is greater tha χ 2 2(005)/ Thus, we do ot reject the ull hypothesis at asymptotic level 95% O the other had, χ 2 2(0) 4605, ad X (θ 0 ) is greater tha χ 2 2(0)/ Thus, had we performed the test at asymptotic level 90% we would ideed reject H 0 I fact, this discussio has show that the asymptotic P -value of our test is somewhere betwee 005 ad 0 It is more importat to report/estimate P -values tha to make blaket declaratios such as accept H 0 or reject H 0 Cosult your Math text(s) To illustrate this example further, suppose the table of observed vs expected remais the same but = 2000 The, X (θ 0 ) is a good deal greater tha χ 2 2(005)/ ad we reject H 0 soudly at the asymptotic level 95% I fact, i this case the P -value is somewhere betwee 0005 ad (check this!) 42 Testig for a Desity Suppose we wish to kow if a certai data-set comes from the expoetial(2) distributio (say!) So let X,, X be a idepedet sample We have H 0 : X expoetial(2) versus H : X expoetial(2) (33) 9

10 Now take a fie mesh 0 < z < z 2 < < z m < of real umbers The, P H0 {z j X z j+ } = 2 zj+ z j e 2r dr = e 2zj e 2zj+ (34) These are θ 0j s, ad we ca test to see whether or ot θ = θ 0, where θ j is the true probability P{z j X z j+ } for all j =,, m The χ 2 -test of the previous sectio hadles just such a situatio That is, let ˆθ j deote the proportio of the data X,, X that lies i [z j, z j+ ] The (30) provides us with a asymptotic ( α)-level test for H 0 that the data is expoetial(2) Other distributios are hadled similarly, but θ 0j is i geeral z j+ z j f 0 (x) dx, where f 0 is the desity fuctio uder H 0 Example 5 Suppose we wish to test (33) Let m = 5, ad defie z j = (j )/0 for j =,, 5 The, { j θ 0j = P H0 0 X j } ( ) ( 6(j ) = exp exp 6j ) (35) That is, j θ 0j Cosider the followig (hypothetical) data: = 200, ad j θ 0j ˆθj The, we form the χ 2 -table, as before, ad obtai Types Exp Obs (Obs Exp) 2 /Exp Total (approx) 0

11 Thus, X (θ 0 ) You should check that 005 P -value 0 I fact, the P -value is much closer to 0 So, at the asymptotic level 95%, we do ot have eough evidece agaist H 0 Some thigs to thik about: I the totals for Exp, you will see that the total is ad ot Ca you thik of why this has happeed? Does this pheomeo reder our coclusios false (or weaker tha they may appear)? Do we eed to fix this by icludig more types? How ca we address it i a sesible way? Would addressig this chage the fial coclusio?