Chi-Squared Tests Math 6070, Sprig 2006 Davar Khoshevisa Uiversity of Utah February XXX, 2006 Cotets MLE for Goodess-of Fit 2 2 The Multiomial Distributio 3 3 Applicatio to Goodess-of-Fit 6 3 Testig for a Fiite Distributio 6 32 Testig for a Desity 8 Chi-squared tests refers to a family of statistical test whose large-sample asymptotics are related i oe way or aother to the χ 2 family of distributios These are our first examples of o-parametric procedures Let me metio oe particularly useful chi-squared test here before we get started properly Cosider a populatio whose distributio is believed to follow a kow desity fuctio f say, expoetial(2) The there is a chi-squared test that ca be used to check our hypothesis This chi-squared test is easy to implemet, ad works well for large-sized samples First, we study a related parametric estimatio problem The we fid a cosistet MLE Ad fially, we use our estimate to devise a test for our hypothesis This will be doe i several stages It is likely that you already kow somethig of the resultig χ 2 -test (page 7) But here you will fid a rigorous proof, i lie with the spirit of this course, of the fact that the said χ 2 -test actually works A [covicig but] o-rigorous justificatio of the same fact ca be foud i pages 34 36 of the excellet moograph of P Bickel ad K Doksum [Mathematical Statistics, Holde Day, First editio, 977] There, a likelihood priciple argumet is devised, which usually works
MLE for Goodess-of Fit Suppose X,, X is a idepedet sample from a fiite populatio More precisely, let t,, t m be m distict umbers (or types ), ad θ := (θ,, θ m ) a m-dimesioal probability vector Cosider iid radom variables X,, X such that P{X = t i } = θ i for all i m That is, each X is of type t i with probability θ i We wish to fid a (actually, the ) MLE for the mass fuctio of the X i s That is, the MLE for the vector θ := (θ,, θ m ) Note that, here, Θ is the collectio of all θ such that 0 θ,, θ m ad θ + + θ m = This is a kid of parametric problem, of course, ad we proceed by computig the MLE Defie N := (N,, N m ) to be the cout statistics for the various types Usig colum-vector otatio, this meas that j= I {X j = t } N = () j= I {X j = t m } I words, N k deotes the umber of occuraces of type k i the sample X := (X,, X ) Note that m j= N j =, so the N j s are obviously depedet rv s Let us calculate their joit mass fuctio Let = (,, m ) be a m-vector of positive itegers such that m j= j = Such s are the possible values of N Moreover, P θ {N = } = P θ {X = s,, X = s }, (2) where the sum is take over all s = (s,, s ) such that of them are of type t, 2 of type t 2, etc For all such s, P θ {X = s,, X = s } = θ θm m (3) Moreover, the umber of all such s s is exactly ( )! :=,, m! m! (4) Therefore, we have derived the followig Lemma The radom -vector N has the multiomial distributio with parameters (, θ) That is, for all itegers,, m such that + + m =, ( ) P θ {N = } = θ,, θm m (5) m Cosider the special case = 2 The, N biomial(, θ), ad N 2 = N So the multiomial distributio is a extesio of the biomial i this 2
sese We write N Multiomial(, θ) i place of the statemet that N is multiomial with parameters (, θ) It turs out that we ca compute the MLE for θ by usig N as the basic data To do so, we use Lemma ad first fid the log-likelihood as follows: L(θ) = A + m N i l θ i, (6) where A does ot deped o θ The case where some θ i {0, } is trivial ad ca be ruled out easily Therefore, we may assume that 0 < θ i < for all i =,, m Thus, for all l =,, m, θ l L(θ) = m ( Ni θ i θ ) i (7) θ l As θ varies i Θ, θ,, θ m roam freely subject to the sigle coditio that θ m = m θ i I other words, ( θ i / θ l ) = 0 for all i =,, m differet from l, but ( θ m / θ l ) = This proves that ( L(θ)/ θ l ) = (N m /θ m ) (N l /θ l ) for all l =,, m Set it equal to zero (for MLE) to see that there must exist γ such that N j = γθ j for all j =,, m But m j= N j =, whereas m j= θ j = Therefore, γ = We have proved the followig Lemma 2 The MLE for θ based o N,, N m is give by N / ˆθ := (8) N m / [To prove that L/ θ l = 0 ideed yields a maximum, you take secod derivatives ad apply similar argumets] 2 The Multiomial Distributio Choose ad fix a probability vector θ, ad assume it is the true oe so that we write P, E, etc i place of the more cumbersome P θ, E θ, etc We ca use () to say a few more thigs about the distributio of N Note first that () is equivalet to the followig: N = I {X j = t } I {X j = t m } j= We ca compute directly to fid that := Y j (9) j= EY = θ, (0) 3
ad θ ( θ ) θ θ 2 θ θ m θ θ 2 θ 2 ( θ 2 ) θ 2 θ m CovY = θ m θ θ m θ 2 θ m ( θ m ) := Q 0 () Because Y,, Y are iid radom vectors, the followig is a cosequece of the precedig, cosidered i cojuctio with the multidimesioal CLT Theorem 3 As, N θ d Nm (0, Q 0 ) (2) There is a variat of Theorem 3 which is more useful to us Defie (N θ )/ θ Ñ := (N m θ m )/ (3) θ m This too is a sum of iid radom vectors Ỹ,, Ỹ, ad has EỸ := 0 ad CovỸ = Q, (4) where Q is the (m m)-dimesioal covariace matrix give below: θ θ θ 2 θ θ m θ θ 2 θ 2 θ 2 θ m Q = θ m θ (5) θ m θ 2 θ m The we obtai, Theorem 4 As, Ñ d N m (0, Q) (6) Remark 5 Oe should be a little careful at this stage because Q is sigular [For example, cosider the case m = 2] However, N m (0, Q) makes sese eve i this case The reaso is this: If C is a o sigular covariace matrix, the we kow what it meas for X N m (0, C), ad a direct computatio reveals that for all real umbers t ad all m-vectors ξ, E [exp (itξ X)] = E [ exp ( t2 2 ξ Cξ )] (7) 4
This defies the characteristic fuctio of the radom variable ξ X uiquely There is a theorem (Ma Wold device) that says that a uique descriptio of the characteristic fuctio of ξ X for all ξ also describes the distributio of X uiquely This meas that ay radom vector X that satisfies (7) for all t R ad ξ R m is N m (0, C) Moreover, this approach does ot require C to be ivertible To lear more about multivariate ormals, you should atted Math 600 As a cosequece of Theorem 4 we have the followig: As, Ñ 2 d Nm (0, Q) 2 (8) [Recall that x 2 = x 2 + + x 2 m] Next, we idetify the distributio of N m (0, Q) 2 A direct (ad somewhat messy) calculatio shows that Q is a projectio matrix That is, Q 2 := QQ = Q It follows fairly readily from this that its eigevalues are all zeros ad oes Here is the proof: Let λ be a eigevalue ad x a eigevector Without loss of geerality, we may assume that x = ; else, cosider the eigevector y := x/ x istead This meas that Qx = λx Therefore, Q 2 x = λ 2 x Because Q 2 = Q, we have prove that λ = λ 2, whece it follows that λ {0, }, as promised Write Q i its spectral form That is, λ 0 0 Q = P 0 λ 2 0 P, (9) 0 0 λ m where P is a orthogoal (m m) matrix [This is because Q is positive semidefiite You ca lear more o this i Math 600] Defie λ 0 0 0 λ2 0 Λ := P (20) 0 0 λm The, Q = Λ Λ, which is why Λ is called the (Perose) square root of Q Let Z be a m-vector of iid stadard ormals ad cosider ΛZ This is multivariate ormal (check characteristic fuctios), E[ΛZ] = 0, ad Cov(ΛZ) = Λ Cov(Z)Λ = Λ Λ = Q I other words, we ca view N m (0, Q) as ΛZ Therefore, N m (0, Q) 2 d = ΛZ 2 = m λ j Zi 2 (2) Because λ i {0, }, we have proved that N m (0, Q) χ 2 r, where r = rak(q) Aother messy but elemetary computatio shows that the rak of 5
Q is m [Roughly speakig, this is because the θ i s have oly oe source of liear depedece Namely, that θ + + θ m = ] This proves that Ñ 2 d χ 2 m (22) Write out the left-had side explicitly to fid the followig: Theorem 6 As, m (N i θ i ) 2 θ i d χ 2 m (23) It took a while to get here, but we will soo be rewarded for our effort 3 Applicatio to Goodess-of-Fit 3 Testig for a Fiite Distributio I light of the developmet of Sectio, ca paraphrase Theorem 6 as sayig that uder P θ, as, X (θ) d χ 2 m, where X (θ) := ) 2 m (ˆθi θ i (24) θ i Therefore, we are i a positio to derive a asymptotic ( α)-level cofidece set for θ Defie for all a > 0, { C (a) := θ Θ : X (θ) a } (25) Equatio (24) shows that for all θ Θ, lim P θ {θ C (a)} = P { χ 2 m a } (26) We ca fid (eg, from χ 2 -tables) a umber χ 2 (α) such that It follows that P { χ 2 m > χ 2 m (α) } = α (27) C ( χ 2 m (α) ) is a asymptotic level ( α) cofidece iterval for θ (28) The statistic X (θ) was discovered by K Pearso i 900, ad is therefore called the Pearso χ 2 statistic Icidetally, the term stadard deviatio is also due to Karl Pearso (893) 6
The statistics X (θ) is a little awkward to work with, ad this makes the computatio of C (χ 2 m (α)) difficult Nevertheless, it is easy eough to use C (χ 2 m (α)) to test simple hypotheses for θ Suppose we are iterested i the test H 0 : θ = θ 0 versus H : θ θ 0, (29) where θ 0 = (θ 0,, θ 0m ) is a (kow) probability vector i Θ The we ca reject H 0 (at asymptotic level ( α)) if θ 0 C (χ 2 m (α)) I other words, we reject H 0 if X (θ 0 ) > χ2 m (α) (30) Because X (θ 0 ) is easy to compute, (29) ca be carried out with relative ease Moreover, X (θ 0 ) has the followig iterpretatio: Uder H 0, X (θ 0 ) = m (Obs i Exp i ) 2, (3) Exp i where Obs i deotes the umber of observatios i the sample that are of type i, ad Exp i := θ 0i is the expected umber uder H 0 of sample poits that are of type i Next we study two examples Example 7 A certai populatio cotais oly the digits 0,, ad 2 We wish to kow if every digit is equally likely to occur i this populatio That is, we wish to test H 0 : θ = ( 3, 3, 3 ) versus H : θ ( 3, 3, 3 ) (32) To this ed, suppose we make, 000 idepedet draws from this populatio, ad fid that we have draw 300 zeros (ˆθ = 03), 340 oes (ˆθ 2 = 034), ad 360 twos (ˆθ 3 = 036) That is, Types Obs Exp (Obs Exp) 2 /Exp 0 030 /3 333 0 3 034 /3 33 0 4 2 036 /3 23 0 3 Total 00 00055 (approx) Suppose we wish to test (32) at the traditioal (asymptotic) level 95% Here, m = 3 ad α = 005, ad we fid from chi-square tables that χ 2 2(005) 5999 Therefore, accordig to (30), we reject if ad oly if X (θ 0 ) 00055 is greater tha χ 2 2(005)/000 000599 Thus, we do ot reject the ull hypothesis at asymptotic level 95% O the other had, χ 2 2(0) 4605, ad X (θ 0 ) is greater tha χ 2 2(0)/000 0004605 Thus, had we performed the test at 7
asymptotic level 90% we would ideed reject H 0 I fact, this discussio has show that the asymptotic P -value of our test is somewhere betwee 005 ad 0 It is more importat to report/estimate P -values tha to make blaket declaratios such as accept H 0 or reject H 0 Cosult your Math 5080 5090 text(s) To illustrate this example further, suppose the table of observed vs expected remais the same but = 2000 The, X (θ 0 ) 00055 is a good deal greater tha χ 2 2(005)/ 000299955 ad we reject H 0 soudly at the asymptotic level 95% I fact, i this case the P -value is somewhere betwee 0005 ad 00025 (check this!) You ca fid may olie χ 2 -tables Oe such table ca be foud at http://wwwucedu/~farkouh/usefull/chihtml [The ufortuate misspellig usefull is ot due to me] 32 Testig for a Desity Suppose we wish to kow if a certai data-set comes from the expoetial(2) distributio (say!) So let X,, X be a idepedet sample We have H 0 : X expoetial(2) versus H : X expoetial(2) (33) Now take a fie mesh 0 < z < z 2 < < z m < of real umbers The, P H0 {z j X z j+ } = 2 zj+ z j e 2r dr = e 2zj e 2zj+ (34) These are θ 0j s, ad we ca test to see whether or ot θ = θ 0, where θ j is the true probability P{z j X z j+ } for all j =,, m The χ 2 -test of the previous sectio hadles just such a situatio That is, let ˆθ j deote the proportio of the data X,, X that lies i [z j, z j+ ] The (30) provides us with a asymptotic ( α)-level test for H 0 that the data is expoetial(2) Other distributios are hadled similarly, but θ 0j is i geeral z j+ z j f 0 (x) dx, where f 0 is the desity fuctio uder H 0 Example 8 Suppose we wish to test (33) Let m = 5, ad defie z j = (j )/0 for j =,, 5 The, { j θ 0j = P H0 0 X j } ( ) ( 6(j ) = exp exp 6j ) (35) 0 5 5 That is, j θ 0j 06988 2 0205 3 00634 4 009 5 00058 8
Cosider the followig (hypothetical) data: = 200, ad j θ 0j ˆθj 06988 07 2 0205 02 3 00634 0 4 009 0 5 00058 0 The, we form the χ 2 -table, as before, ad obtai Types Exp Obs (Obs Exp) 2 /Exp 06988 07 206 0 6 2 0205 02 000052 3 00634 0 002 4 009 0 009 5 00058 0 00058 Total 09976 004655 (approx) Thus, X (θ 0 ) 004655 You should check that 005 P -value 0 I fact, the P -value is much closer to 0 So, at the asymptotic level 95%, we do ot have eough evidece agaist H 0 Some thigs to thik about: I the totals for Exp, you will see that the total is 09976 ad ot Ca you thik of why this has happeed? Does this pheomeo reder our coclusios false (or weaker tha they may appear)? Do we eed to fix this by icludig more types? How ca we address it i a sesible way? Would addressig this chage the fial coclusio? 9