Statistics Review Part 3. Hypothesis Tests, Regression

Size: px

Start display at page:

Download "Statistics Review Part 3. Hypothesis Tests, Regression"

Elmer Mathews
5 years ago
Views:

1 Statstcs Revew Part 3 Hypothess Tests, Regresso

2 The Importace of Samplg Dstrbutos Why all the fuss about samplg dstrbutos? Because they are fudametal to hypothess testg. Remember that our goal s to lear about the populato dstrbuto. I practce, we are terested thgs lke the populato mea, populato varace, some populato codtoal mea, etc. We estmate these quattes a radom sample take from the populato. Ths s doe wth sample estmators lke the sample mea, the sample varace, or some sample codtoal mea. Kowg the samplg dstrbuto of our sample estmator (e.g., the samplg dstrbuto of the sample mea), gves us a way to assess whether partcular values of populato quattes (e.g., the populato mea) are lkely or ulkely. E.g., suppose we calculate a sample mea of 5. If the true populato mea s 6, s 5 a lkely or ulkely occurrece?

3 Hypothess Testg We use hypothess tests to evaluate clams lke: the populato mea s 5 the populato varace s 6 some populato codtoal mea s 3 Whe we kow the samplg dstrbuto of a sample statstc, we ca evaluate whether ts observed value the sample s lkely whe the above clam s true. We formalze ths wth two hypotheses. For ow, we ll focus o the case of hypotheses about the populato mea, but we ca geeralze the approach to ay populato quatty.

4 Null ad alteratve hypotheses Suppose we re terested evaluatg a specfc clam about the populato mea. For stace: the populato mea s 5 the populato mea s postve We call the clam that we wat to evaluate the ull hypothess, ad deote t H 0. H 0 : µ = 5 H 0 : µ > 0 We compare the ull hypothess to the alteratve hypothess, whch holds whe the ull s false. We wll deote t H. H : µ 5 (a two-sded alteratve hypothess) H : µ 0 (a oe-sded alteratve hypothess)

5 How tests about the populato mea work Step : Specfy the ull ad alteratve hypotheses. Step a: Compute the sample mea ad varace Step b: Use the estmates to costruct a ew statstc, called a test statstc, that has a kow samplg dstrbuto whe the ull hypothess s true ( uder the ull ) the samplg dstrbuto of the test statstc depeds o the samplg dstrbuto of the sample mea ad varace Step 3: Evaluate whether the calculated value of the test statstc s lkely whe the ull hypothess s true. We reject the ull hypothess f the value of the test statstc s ulkely We do ot reject the ull hypothess f the value of the test statstc s lkely (Note: thaks to Popper, we ever accept the ull hypothess)

6 Example: the t-test Suppose we have a radom sample of observatos from a N(µ,σ ) dstrbuto. Suppose we re terested testg the ull hypothess: H 0 : µ = µ 0 agast the alteratve hypothess: H : µ µ 0 A atural place to start s by estmatg the sample mea, We kow that f the ull hypothess s true, the the samplg dstrbuto of s ormal wth mea µ 0 ad varace σ /. We say: ~ N(µ 0,σ /) uder the ull (draw a pcture)

7 Example: the t-test (cotued) Because ~ N(µ 0,σ /) uder the ull, we kow that Z = µ 0 ~ N σ / ( 0, ) uder the ull (recall we ca trasform ay ormally dstrbuted RV to have a stadard ormal dstrbuto by subtractg off ts mea ad dvdg by ts stadard devato) If we kew σ, we could compute Z, ad ths would be our test statstc: If Z s far from zero, t s ulkely that the ull hypothess s true, ad we would reject t. If Z s close to zero, t s lkely that the ull hypothess true, ad we would ot reject t. Why Z? Because we ca look up ts crtcal values a table. Problems wth ths approach: we do t kow σ how do we quatfy close ad far?

8 Example: the t-test (cotued) Sce we do t kow σ, why ot estmate t? We kow that the sample varace s s a ubased estmator of σ. Ufortuately, Q = µ 0 doesot havea N s / Some facts about samplg from a N(µ,σ ): Fact : ( )s / σ ~ χ - Fact : s s depedet of ( 0, ) dstrbuto uder theull We kow already that f Z ~ N(0,) ad W ~ χ v ad Z ad W are depedet, the We ca put all ths together to compute a test statstc that has a t samplg dstrbuto wth - degrees of freedom. Z T = ~ W / v t v

9 Example: the t-test (cocluded) Puttg together the thgs we kow, uder the ull : Ths s a bt of a mess, ad t stll volves a ukow quatty (σ ). But we ca smplfy t wth a bt of algebra: ( ) [ ] ( ) ( ) ( ) ( ) [ ] ( ) 0 ~ / / / / / / = = t s s Z T σ σ µ σ quatty (σ ). But we ca smplfy t wth a bt of algebra: Notce that ths test statstc has two crucal propertes: t does t cota ay ukows (so we ca compute t) we kow ts samplg dstrbuto (so we ca look a table ad see f a partcular value s lkely or ulkely uder the ull) ( ) ( ) ( ) [ ] ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ~ / / / / / / / / / / / / = = = = = t s s s s s T µ σ σ µ σ σ µ σ σ µ σ σ µ

10 Ths s how ALL hypothess testg works. We form the ull ad alteratve hypotheses that we care about.. The we use the sample data to costruct a test statstc that has a kow samplg dstrbuto whe the ull hypothess s true. 3. The we decde whether or ot to reject the ull o the bass of whether the value of the test statstc that we observe s lkely or ulkely uder the ull hypothess. The rght way to thk about hypothess tests: A hypothess test s a rule for usg the data to decde whether or ot to reject the ull hypothess.

11 A ote about Normalty Whe we derved the samplg dstrbuto for our t statstc, we reled o some facts that are oly true whe samplg from a ormal dstrbuto. Ths gves us a ormal samplg dstrbuto for the sample mea uder the ull, depedece of the sample mea ad varace, ad the Ch-square result for s. What do we do f we re ot samplg from a ormal dstrbuto? (the usual case) Usually, we rely o a asymptotc approxmato. As the sample sze gets bg (techcally, as ), the s gets very close to σ (t s cosstet). Thus our T statstc gets very close to our Z statstc: µ 0 µ 0 as, T = = s / σ / Furthermore, we kow from the cetral lmt theorem that as, uder the ull hypothess the samplg dstrbuto of Z s approxmately N(0,) NO MATTER WHAT THE POPULATION DISTRIBUTION IS! So f our sample s bg eough (how bg?) we ca compute the T statstc as usual, ad use values from the stadard ormal dstrbuto to decde whether a specfc value s lkely or ulkely uder the ull hypothess. Z

12 How do we kow f a partcular value of the test statstc s lkely? Whe we kow the samplg dstrbuto of the test statstc, we kow the probablty that the value of the test statstc wll fall a gve terval whe the ull hypothess s true. Just the area uder the pdf of the test statstc s samplg dstrbuto. If T s our test statstc, for ay α we ca fd L,U such that Pr[L T U] = α whe H 0 s true So we ca defe a rage of acceptable values of the test statstc. e.g., we ca defe a terval [L,U] such that f the ull hypothess s true, the test statstc falls the terval wth probablty 0.95 (α=0.05) that s, f the ull s true ad we draw 00 radom samples, the test statstc wll fall ths terval about 95 tmes. f we observe a value of the test statstc outsde ths terval, we kow t s ulkely that the ull s true (e.g., t would oly happe 5% of radom samples) ad we ca therefore comfortably reject the ull. We call L ad U crtcal values at the 00α% sgfcace level, ad we ca look them up a table.

13 Makg mstakes: Type I ad Type II errors Whe testg a hypothess, there s always the possblty that we make a mstake. There are two kds of mstakes: Type I error: we erroeously reject the ull whe t s true. Type II error: we fal to reject the ull whe t s false We call the probablty of makg a Type I error the sgfcace level of the test, ad deote t α. We use β to deote the probablty of makg a Type II error. We call βthe power of a test. It s the probablty that we correctly reject the ull whe t s false. We usually choose a sgfcace level α that we re comfortable wth (0.05 s most commo), ad look for a powerful test (small β) There s a tradeoff: as α gets smaller, β must get bgger. (draw a pcture)

14 A example Suppose you survey radomly selected Caadas about the umber of tmes they wet to the moves last year. Usg the survey data, you calculate = 7.5, s = 75 You wat to test the hypothess that the populato mea of move attedace s 9: H 0 : µ = 9 H : µ 9 You fgure move attedace s probably ormally dstrbuted the populato, ad costruct the T statstc: µ T = = =.9056 s / 75 / You kow that T ~ t 0 f H 0 s true. I a table of crtcal values for the t 0 dstrbuto, you fd that the crtcal value at the 0% level of sgfcace (α = 0.) s.658; at the 5% level of sgfcace (α = 0.05) the crtcal value s.980. Ths meas that f H 0 s true, Pr[-.658 T.658] = 0.90 Pr[-.980 T.980] = 0.95 Therefore t = s qute ulkely. You would reject H 0 at the 0% level of sgfcace, but would ot reject t at the 5% level of sgfcace.

15 p-values There s aother (related) way to decde whether or ot to reject the ull hypothess. Suppose we costruct a test statstc called W that has a kow samplg dstrbuto whe H 0 s true. Suppose that our sample, the value we compute for the test statstc s w. We ca ask what s the probablty of observg a value of the statstc W as bg as w whe the ull hypothess s true?.e., Pr[ w W w] = p* (for a two-sded alteratve) Pr[W w] = p* (for a oe-sded alteratve) The probablty p* s called a p-value. It s a tal probablty of the samplg dstrbuto of W. It s the probablty of observg a value of W that s more extreme (uusual) tha w. It s also the probablty of makg a type I error f we reject the ull. If the p-value s small (say 0.05) we ca cofdetly reject H 0. Computers routely report p-values for commo hypothess tests, so you ca save yourself some tme by lookg at the p-value rather tha lookg up crtcal values a table. The p-value for the example above s betwee 5% ad 0%

16 Iterval Estmato We re doe talkg about hypothess testg for ow but t wll come up aga soo the cotext of lear regresso. We talked earler about estmators statstcs that we use to estmate a populato quatty. The examples we saw (the sample mea, sample varace, sample covarace, etc.) are all called pot estmators because they gve us a sgle value for the populato quatty. A alteratve to a pot estmator s a terval estmator. Ths s a terval that cotas a populato quatty wth a kow probablty. A terval estmator of a populato quatty Q takes the form [L,U], where L ad U are fuctos of the data (they re statstcs). We use the terval estmator [L,U] to make statemets lke: Pr[L Q U] = -α (look famlar yet?)

17 Example: Cofdece Iterval for the Populato Mea A 95% cofdece terval for the populato mea µ s a terval [L,U] such that: Pr[L µ U] = 0.95 How do we fd the terval [L,U] such that ths s true? A llustratve (but mpossble) way:. Pck a radom value µ ad costruct the T statstc to test H 0 : µ = µ vs. H : µ µ.. If we reject H 0, the µ s ot the terval. If we do ot reject H 0, the µ s the terval. 3. Pck aother value µ ad repeat. 4. Do ths for all possble values of µ (ths s why t s mpossble). Thakfully, there s a easer way.

18 The 95% cofdece terval for µ The easer way s to make use of the samplg dstrbuto of our T statstc. We kow that whe samplg from the ormal dstrbuto, µ T = ~ t s / So we ca always look up the crtcal value t -,α/ such that: Pr[- t -,α/ T t -,α/ ] = α For a 95% cofdece terval, we have Pr[- t -,0.05 T t -,0.05 ] = 0.95 Now we just plug the formula for our T statstc, ad rearrage thgs as ecessary (ext slde)

19 Cofdece Iterval Algebra [ ] = = / Pr Pr 0 95., -., -., -., - s s t s t t T t. µ + = = = Pr Pr Pr., -., -., -., -., -., - t s t s t s t s t s t s µ µ µ

20 The Last Word o Cofdece Itervals (for ow) So ow we kow how to buld a 95% cofdece terval for µ whe samplg from the ormal dstrbuto. It s easy to geeralze ths to 90% or 99% etc. cofdece tervals just chage the crtcal value you use. Whe we re ot comfortable assumg that we re samplg from the ormal dstrbuto, we ca replace crtcal values from the t dstrbuto wth crtcal values from the stadard ormal dstrbuto f the sample s bg eough We ca buld smlar tervals for other populato quattes the dea s always the same. We just eed a sample quatty (usually a test statstc) whose samplg dstrbuto we kow & we ca use t to buld the terval. Typcally, a cofdece terval for a populato quatty Q looks lke [L,U] = [q - somethg, q + somethg] where q s a pot estmator of Q, ad somethg depeds o the varace of the samplg dstrbuto of q.

21 Movg o To ths pot we ve focused o a sgle radom varable,. We ve talked about populato quattes (e.g., µ ad σ ). We ve dscussed how to compute sample statstcs to estmate populato quattes, the propertes of estmators (e.g., bas ad effcecy), ad how to test hypotheses about the populato. Thgs get much more terestg whe we cosder two or more radom varables. we care about the relatoshp betwee varables we ca use oe or more varables to predct a varable of terest, etc. We ca get a lot of mleage out of studyg the codtoal expectato of a varable of terest (Y) gve aother varable (or group of varables). That s, studyg E(Y ). Recall that aother ame for E(Y ) s the regresso fucto. We ll sped the rest of the semester talkg about regresso aalyss, whch s a very powerful tool for aalyzg ecoomc data. Regresso aalyss s based o E(Y ).

22 What s Regresso Aalyss? Regresso aalyss s a very commo statstcal/ecoometrc techque We use t to measure/expla relatoshps betwee ecoomc varables Example: casual observato wll reveal that more educated dvduals ted to have hgher comes. regresso methods ca be used to measure the rate of retur of a extra year of educato or, use regresso methods to estmate the relatoshp betwee come ad educato, geder, labour market experece, etc. Example: ecoomc theory tells us that f the prce of a good creases, dvduals wll cosume less of t. that s, demad curves slope dow but ecoomc theory does t predct how bg the chage cosumpto wll be for a gve prce chage we ca use regresso aalyss to measure how much dvduals reduce ther cosumpto respose to a prce crease (.e., we ca estmate the elastcty of demad)

23 The Regresso Model Regresso s a tool to coveetly lear about codtoal meas. The goal of regresso aalyss s to expla the value of oe varable of terest (the depedet varable) as a fucto of the values of other varables (the depedet or explaatory varables) Usually, the depedet varable s deoted Y The depedet varables are,, 3 etc. Sometmes we say we wat to expla movemet Y as a fucto of movemet the varables. That s, how much does Y chage whe the varables chage? I ths cotext, a better word for movemet s varato. We use a equato (sometmes more tha oe) to specfy the relatoshp betwee Y ad the varables. Ths equato s called the regresso model: E(Y,, 3 ) = f(,, 3 ) Example: Y s come, s years of educato, s geder, 3 s years of labour market experece, ad f s some fucto... Note: we are ot sayg that the varables cause Y

24 Smple Lear Regresso The smplest example of a regresso model s the case where the regresso fucto f s a le, ad where there s oly oe E[Y ] = β 0 + β Ths specfcato of the regresso fucto says that the depedet varable Y s a lear fucto of the depedet varable Ths s just the equato of a le We call β 0 adβ the regresso coeffcets β 0 s called the tercept or costat term. It tells us the value of Y whe s zero β s the slope coeffcet. It measures the amout that Y chages for a ut chage --t s the slope of the le relatg ad Y: dy = β d sometmes we call β the margal effect of o Y.

25 About Learty There are two kds of learty preset the regresso model Y = β 0 + β from the prevous slde. Ths regresso fucto s lear. couter-example: Y = β 0 + β Ths regresso fucto s lear the coeffcets β 0 ad β couter-example: Y = β 0 + β I geeral, ether kd of learty s ecessary. However, we wll focus our atteto mostly o what ecoometrcas call the lear regresso model. The lear regresso model requres learty the coeffcets, but ot learty. Whe we say lear regresso model we mea a model that s lear the coeffcets.

26 The Stochastc Error Term Ecoometrcas recogze that the regresso fucto s ever a exact represetato of the relatoshp betwee depedet ad depedet varables. e.g., there s o exact relatoshp betwee come (Y) ad educato, geder, etc., because of thgs lke luck There s always some varato Y that caot be explaed by the model. There are may possble reasos: there mght be mportat explaatory varables that we leave out of the model; we mght have the wrog fuctoal form (f), varables mght be measured wth error, or maybe there s just some radomess outcomes. These are all sources of error. To reflect these kds of error, we clude a stochastc (radom) error term the model. The error term reflects all the varato Y that caot be explaed by. Usually, we use epslo (ε) to represet the error term.

27 More About the Error Term Add a error term, ad our smple lear regresso model s: Y = β 0 + β + ε It s helpful to thk of the model as havg two compoets:.a determstc (o-radom) compoet β 0 + β.a stochastc (radom) compoet ε Bascally, we are decomposg Y to the part that we ca expla usg (.e., β 0 + β ) ad the part that we caot expla usg (.e., the error ε) The rght way to thk about t: β 0 + β s the codtoal mea of Y gve. That s, Y = E(Y ) + ε where E(Y ) = β 0 + β Remember regresso fucto meas E(Y ) Ths gves us aother way to thk about errors: ε = Y E(Y ) (draw some pctures)

28 A example Thk about startg salares for ew uversty graduates (Y). There s a lot of varato startg salares betwee dvduals. Some of ths varato s predctable: startg salary depeds o uversty, feld of study, dustry, occupato/ttle, frm sze, etc. Call all of ths The predctable part of startg salary goes to the determstc compoet of the regresso: E(Y ). We do t eed to mpose that eters learly, but we wll requre E(Y ) to be lear the βs. We choose the specfc fuctoal form of E(Y ) whe we buld the model. Much of the varato startg salary s upredctable: startg salary also depeds o luck, epotsm, tervew skll, etc. We ca t measure these thgs, so we ca t clude them E(Y ). The upredctable part eds up the error term, ε.

29 Eve Smpler Regresso Suppose we have a lear regresso model wth oe depedet varable ad NO INTERCEPT: Y = + β ε Suppose also that For all. ( ) E[ ε ] = 0 ad E[ ε ] = σ Now, defe a estmator as the umber βˆ that mmses the sum of the squared predcto error e = Y ˆ β M ( ) ( ) ( ) e = Y ˆ β = Y Y ˆ β + ˆ β βˆ = = = = =

30 Mmsato The squared Y leadg term does t have βˆ M βˆ ( ˆ ) ( ˆ ) Y β β + = = Frst-Order Codto ˆ ( Y ) β ( ) + = 0 = = ( Y ) β ( ) + ˆ = 0 = = ( ) = ( Y ) ˆ β 0 ˆ β = = = = ( ) = Y

31 OLS Coeffcets are Sample Meas The estmated coeffcet s a weghted average of the Y s: ˆ Y = = = β w = ( ) = ( ) = = It s a fucto of the data (a specal kd of sample mea), ad so t s a statstc. It ca be used to estmate somethg we are terested : the populato value of β Sce t s a statstc, t has a samplg dstrbuto that we ca evaluate for bas ad varace. w Y

32 Bas Preted s ot radom. Remember assumptos from above: Y = β + ε E[ ε ] = 0 ad E[ ε ] = σ ( ) Substtute to the estmator ad take a expectato: Y ( β + ε ) ˆ E β = = E E = = ( ) ( ) = = ( ) ε = = = β E + E = β + 0 = β ( ) ( ) = =

33 Varace ε V ˆ β E ˆ β E ˆ β E = = = E ε ( ) ( ) ( ) = = ( ) = = = = E ( ) = = [ ε ε ε ε... ε ε ε ε ] ( ) E = E ( ε ) = = ( ) ( ε ) = ( ) = ( ) = σ

34 Iferece You have everythg you eed to do make probablty statemets about OLS regresso coeffcets whe there s just varable (ad o tercept). It s bascally the same whe you have more varables: The OLS coeffcet s a weghted sum of Y s It s a statstc wth a samplg dstrbuto The samplg dstrbuto depeds o the data, ad you ca thus use the data to make statemets about the populato parameter.

35 Exteded Notato We eed to exted our otato of the regresso fucto to reflect the umber of observatos. As usual, we ll work wth a d radom sample of observatos. If we use the subscrpt to dcate a partcular observato our sample, our regresso fucto wth oe depedet varable s: Y = β0 + β + ε for =,,..., So really we have equatos (oe for each observato): Y Y = β + β = β + β 0 0 Y = β0 + β Notce that the coeffcets β 0 ad β are the same each equato. The oly thg that vares across equatos s the data (Y, ) ad the error ε. + ε + ε + ε

36 Extedg the Notato Further If we have more (say k) depedet varables, the we eed to exted our otato further. We could use a dfferet letter for each varable (.e.,, Z, W, etc.) but stead we usually just troduce aother subscrpt o the. So ow we have two subscrpts: oe for the varable umber (frst subscrpt) ad oe for the observato umber (secod subscrpt). Y = β + β + β + β + + β + ε 0 What do the regresso coeffcets measure ow? They are partal dervatves. That s, β Y So, β measures the effect o Y of a oe ut crease holdg all the other depedet varables, 3,..., k costat. Y 3 3 = β = βk = Y k k k

37 What s Kow, What s Ukow, ad What s Assumed It s useful to summarze what s kow, what s ukow, ad what s hypotheszed. Kow: Y ad,,..., k (the data) Ukow:β 0, β, β,..., β k ad ε (the coeffcets ad errors) Hypotheszed: the form of the regresso fucto, e.g., E(Y ) = β 0 + β + β + β k k We use the observed data to lear about the ukows (coeffcets ad errors), ad the we ca test the hypotheszed form of the regresso fucto. We ca hope to lear a lot about the βs because they are the same for each observato. We ca t hope to lear much about the ε because there s oly oe observato o each of them.

38 Estmated Regresso Coeffcets Thk of the regresso fucto we ve developed to ths pot as the populato regresso fucto. As always ecoometrcs, we collect a sample of data to lear about the populato. We do t kow the (populato) regresso coeffcets (β), so we estmate them. We ll dscuss the detals of how to do ths ext day. For ow, we ll just troduce a otato for the estmated coeffcets. The estmated coeffcets are: ˆ β, ˆ β, ˆ β,..., 0 The estmated coeffcets are sample statstcs that we ca compute from our data. Because they are sample statstcs, they are RVs, have samplg dstrbutos, etc., just lke all sample statstcs. ˆ β k

39 Predcted Values ad Resduals Oce we have estmated the regresso coeffcets, we ca calculate the predcted value of Y. It s a sample estmate of the codtoal expectato of Y gve all the s: Yˆ = ˆ β ˆ ˆ 0 + β + β + + ˆ β It s our best guess of the value of Y gve the value of the s. Predcted values le o the estmated regresso le. Of course, Y ad ts predcted value are rarely equal. We call the dfferece betwee Y ad ts predcted value the resdual e : Resduals are the sample couterpart to the (ukow) errors ε = Y E(Y ). We ca wrte the estmated regresso fucto as: e = Y (draw a pcture true & estmated regresso les, resduals, ad errors) Yˆ Y = ˆ β ˆ ˆ + ˆ β + e 0 + β + β + k k k k

40 What we do wth regresso aalyss We use regresso models for lots of dfferet thgs. Sometmes we care most about predcted values & use the estmated regresso to predct (forecast) thgs. e.g., estmate a regresso of stock prces (Y ) o leadg dcators (uemploymet rate, cp, etc.) to forecast future stock prces. Sometmes we care most about the coeffcet values & use the estmated regresso to develop polcy, etc. e.g., estmate a regresso of labour eargs o years of educato, experece, geder, etc. ( the ktche sk ) the estmated coeffcet o years of educato gves a estmate of the rate of retur to a extra year of educato ceters parbus the estmated coeffcet o geder gves a estmate of the male-female wage dfferetal ceters parbus (see lecture ) both of these are mportat for desgg govermet polcy

Lecture 7. Confidence Intervals and Hypothesis Tests in the Simple CLR Model

Lecture 7. Confidence Intervals and Hypothesis Tests in the Simple CLR Model Lecture 7. Cofdece Itervals ad Hypothess Tests the Smple CLR Model I lecture 6 we troduced the Classcal Lear Regresso (CLR) model that s the radom expermet of whch the data Y,,, K, are the outcomes. The