STATISTICAL SIGNIFICANCE

Transcription

1 STATISTICAL SIGNIFICANCE EFFECT SIZE More to life than tatitical ignificance Reporting effect ize Turn out a lot of reearcher do not know what preciely p <.05 actually mean Cohen (1994) Article: The earth i round (p<. 05) What it mean: "Given that H 0 i true, what i the probability of thee (or more extreme) data? Trouble i mot people want to know "Given thee data, what i the probability that H 0 i true?" ALWAYS A IFFERENCE With mot analye we commonly define the null hypothei a no relationhip between our predictor and outcome(i.e. the nil hypothei) With ample data, difference between group alway exit (at ome level of preciion), correlation are alway non-zero. Obtaining tatitical ignificance can be een a jut a matter of ample ize Furthermore, the importance and magnitude of an effect are not accurately reflected becaue of the role of ample ize in probability value attained WHAT SHOUL WE BE OING? We want to make ure we have looked hard enough for the difference power analyi Figure out how big the thing we are looking for i effect ize CALCULATING EFFECT SIZE Though different tatitical tet have different effect ize developed for them, the general principle i the ame Effect ize refer to the magnitude of the impact of ome variable on another TYPES OF EFFECT SIZE Two baic clae of effect ize Focued on tandardized mean difference for group comparion Allow comparion acro ample and variable with differing variance Equivalent to z core Note ometime no need to tandardize (unit of the cale have inherent meaning) Variance-accounted-for Amount explained veru the total d family v. r family With group comparion we will alo talk about cae-level effect ize 1

2 COHEN S (HEGE S G) Cohen wa one of the pioneer in advocating effect ize over tatitical ignificance efined d for the one-ample cae X µ COHEN S Note the imilarity to a z-core- we re talking about a tandardized difference The mean difference itelf i a meaure of effect ize, however taking into account the variability, we obtain a tandardized meaure for comparion of tudie acro ample uch that e.g. a.0 in thi tudy mean the ame a that reported in another tudy COHEN S COHEN S IFFERENCES BETWEEN MEANS Now compare to the one-ample t-tatitic X µ X t = N So t t = d N and N Thi how how the tet tatitic (and it oberved p- value) i in part determined by the effect ize, but i confounded with ample ize Thi mean mall effect may be tatitically ignificant in many tudie (ep. ocial cience) Standard meaure for independent ample t tet X X 1 Cohen initially uggeted could ue either ample tandard deviation, ince they hould both be equal according to our aumption (homogeneity of variance) In practice however reearcher ue the pooled variance p EXAMPLE Average number of time graduate pych tudent cure in the preence of other out of total frutration over the coure of a day Currently taking a tatitic coure v. not X = 13 = 7.5 n= 30 ata: Xn = 11 = 5.0 n= 30 EXAMPLE Find the pooled variance and d Equal group o jut average the two variance uch that and p = =.8 6.5

3 COHEN S IFFERENCES BETWEEN MEANS GLASS S Δ Relationhip to t Relationhip to r pb 1 1 t n + n 1 For tudie with control group, we ll ue the control group tandard deviation in our formula X X 1 control Thi doe not aume equal variance n1 + n 1 1 rpb + 1 r pb n1 n r = d d + (1/ pq) P and q are the proportion of the total each group make up. If equal group p=.5, q=.5 and the denominator i d + 4 a you will ee in ome text COMPARISON OF METHOS EPENENT SAMPLES One option would be to imply do nothing different than we would in the independent ample cae, and treat the two et of core a independent Problem: Homogeneity of variance aumption may not be tenable They aren t independent EPENENT SAMPLES Another option i to obtain a metric with regard to the actual difference core on which the tet i run A d tatitic for a dependent mean contrat i called a tandardized mean change (gain) There are two general tandardizer: A tandard deviation in the metric of the 1. difference core (). original core EPENENT SAMPLES ifference core Mean difference core divided by the tandard deviation of the difference core 3

4 EPENENT SAMPLES The tandard deviation of the difference core, unlike the previou olution, take into account the correlated nature of the data Var1 + Var covar = (1 r ) p Problem remain however A tandardized mean change in the metric of the difference core can be much different than the metric of the original core Variability of difference core might be markedly different for change core compared to original unit Interpretation may not be traightforward 1 EPENENT SAMPLES Another option i to ue tandardizer in the metric of the original core, which i directly comparable with a tandardized mean difference from an independent-ample deign In pre-pot type of ituation where one would not expect homogeneity of variance, treat the pretet group of core a you would the control for Gla Δ p EPENENT SAMPLES CHARACTERIZING EFFECT SIZE Which to ue? Bae it on ubtantive theoretical interet If the emphai i really on change, i.e. the deign i intrinically repeated meaure, one might chooe the option of tandardized mean change In other ituation we might retain the tandardizer in the original metric, uch that the d will have the ame meaning a elewhere Cohen emphaized that the interpretation of effect require the reearcher to conider thing narrowly in term of the pecific area of inquiry Evaluation of effect ize inherently require a peronal value judgment regarding the practical or clinical importance of the effect HOW BIG? Cohen (e.g. 1969, 1988) offer ome rule of thumb Fairly widepread convention now (unfortunately) Looked at ocial cience literature and uggeted ome way to carve reult into mall, medium, and large effect Cohen d value (Lipey 1990 range in parenthee) 0. mall (<.3) 0.5 medium ( ) 0.8 large (.56-1.) Be wary of mindlely invoking thee criteria The wort thing that we could do i ubitute.0 for p =.05, a it would be a practice jut a lazy and fraught with potential for abue a the decade of poor practice we are currently trying to overcome SMALL, MEIUM, LARGE? Cohen (1969) mall real, but difficult to detect difference between the height of 15 year old and 16 year old girl in the US Some gender difference on apect of Wechler Adult Intelligence cale medium large enough to be viible to the naked eye difference between the height of 14 & 18 year old girl large groly perceptible and therefore large difference between the height of 13 & 18 year old girl IQ difference between Ph and college frehman 4

5 ASSOCIATION A meaure of aociation decribe the amount of the covariation between the independent and dependent variable It i expreed in an unquared tandardized metric or it quared value the former i uually a correlation*, the latter a variance-accounted-for effect ize A quared multiple correlation (R ) calculated in ANOVA i called the correlation ratio or etimated eta-quared (η ) ANOTHER MEASURE OF EFFECT SIZE The point-bierial correlation, r pb, i the Pearon correlation between memberhip in one of two group and a continuou outcome variable A mentioned r pb ha a direct relationhip to t and d When quared it i a pecial cae of etaquared in ANOVA An one-way ANOVA for a two-group factor: eta-quare R from a regreion approach = r pb ETA-SQUARE A meaure of the degree to which variability among obervation can be attributed to condition Example: η =.50 50% of the variability een in the core i due to the independent variable. ETA-SQUARE Relationhip to t in the two group etting η = t t + df SS η = = R treat pb SStotal OMEGA-SQUARE Another effect ize meaure that i le biaed and interpreted in the ame way a eta-quared SStreat ( k 1) MS ω = SS + MS total error error PARTIAL ETA-SQUARE A meaure of the degree to which variability among obervation can be attributed to condition controlling for the ubject effect that unaccounted for by the model (individual difference/error) partial η SStreat = SS + SS treat error Rule of thumb for mall medium large:.01,.06,.14 Note that in one-way deign SPSS label thi a PES but i actually eta-quared, a there i only one factor and no other to partial out 5

6 COHEN S F Cohen ha a d type of meauere for Anova called f ( X X ) f = k MS Cohen' f i interpreted a how many tandard deviation unit the mean are from the grand mean, on average, or, if all the value were tandardized, f i the tandard deviation of thoe tandardized mean e.. RELATION TO PES Uing Partial Eta-Squared PES f = 1 PES GUIELINES A eta-quared value are baically r value the feel for what i large, medium and mall i imilar and depend on many contextual factor Small eta-quared and partial eta-quare value might not get the point acro (i.e. look big enough to worry about) Might tranform to Cohen f or ue o a to continue to peak of tandardized mean difference Hi uggetion for f are:.10,.5,.40 which tranlate to.01,.06, and.14 for eta-quared value That i omething reearcher could overcome if they undertood more about effect ize OTHER EFFECT SIZE MEASURES Meaure of aociation for noncontinuou data Contingency coefficient Phi Cramer Phi d-family Odd Ratio Agreement Kappa Cae level effect ize CONTINGENCY COEFFICIENT PHI C = χ χ + N φ = χ N An approximation of the correlation between the two variable (e.g. 0 to 1) Problem- can t ever reach 1 and it max value i dependent on the dimenion of the contingency table Ued in X table a a correlation (0 to 1) Problem- get weird with more complex table 6

7 CRAMER S PHI φ = c χ Nk ( 1) Again think of it a a meaure of aociation from 0 (weak) to 1 (trong), that i phi for X table but alo work for more complex one. k i the leer of the number of row or column OS RATIOS Epecially good for X table Take a ratio of two outcome Although neither get the majority, we could ay which they were more likely to vote for repectively Odd Clinton among em= 564/636 =.887 Odd McCain among Rep= 450/550 = /.818 (the odd ratio) mean they d be 1.08 time a likely to vote Clinton among democrat than McCain among republican However, the 95% CI for the odd ratio i:.9 to 1.8 Thi ugget it would not be wie to predict either ha a better chance at nomination at thi point. Number coming from Feb 1-3 Gallup Poll daily tracking. Three-day rolling average. N=approx. 1,00 emocrat and emocratic-leaning voter nationwide. Gallup Poll daily tracking. Three-day rolling average. N=approx. 1,000 Republican and Republican-leaning voter nationwide. Ye No Total Clinton McCain KAPPA Meaure of agreement (from Cohen) Though two folk (or group of people) might agree, they might alo have a predipoition to repond in a certain way anyway Kappa take thi into conideration to determine how much agreement there would be after incorporating what we would expect by chance O and E refer to the oberved and expected frequencie on the diagonal of the table of Judge 1 v Judge K = O N E E Judgement by clinical pycholgit on the everity of uicide attempt by client. At firt glance one might think (10+5+3)/4 = 75% agreement between the two. However thi doe not take into account chance agreement. Judge 1 Judge 1 3 Total 1 10 (5.5) (3.67) (.88) K = = 57% CASE-LEVEL EFFECT SIZES Indexe uch a Cohen d and eta etimate effect ize at the group or variable level only However, it i often of interet to etimate difference at the cae level Cae-level indexe of group ditinctivene are proportion of core from one group veru another that fall above or below a reference point Reference point can be relative (e.g., a certain number of tandard deviation above or below the mean in the combined frequency ditribution) or more abolute (e.g., the cutting core on an admiion tet) CASE-LEVEL EFFECT SIZES Cohen (1988) meaure of ditribution overlap: U 1 Proportion of nonoverlap If no overlap then = 1, 0 if all overlap U Proportion of core in lower group exceeded by the ame proportion in upper group If ame mean =.5, if all group exceed group 1 then = 1.0 U 3 Proportion of core in lower group exceeded by typical core in upper group Same range a U OTHER CASE-LEVEL EFFECT SIZES Tail ratio (Feingold, 1995): Relative proportion of core from two different group that fall in the upper extreme (i.e., either the left or right tail) of the combined frequency ditribution Extreme i uually defined relatively in term of the number of tandard deviation away from the grand mean Tail ratio > 1.0 indicate one group ha relatively more extreme core Here, tail ratio = p/p1: 7

8 OTHER CASE-LEVEL EFFECT SIZES Common language effect ize (McGraw & Wong, 199) i the predicted probability that a random core from the upper group exceed a random core from the lower group z CL 0 ( X1 X) = + 1 Find area to the right of that value Range CONFIENCE INTERVALS FOR EFFECT SIZE Effect ize tatitic uch a Hedge g and η have complex ditribution Traditional method of interval etimation rely on approximate tandard error auming large ample ize General form for d d± t ( ) cv d CONFIENCE INTERVALS FOR EFFECT SIZE Standard error d N d / g = + ( df ) n n w Δ N Δ= + ( n ) nn 1 ependent Sample 1 PROBLEM However, CI formulated in thi manner are only approximate, and are baed on the central (t) ditribution centered on zero The true (exact) CI depend on a noncentral ditribution and additional parameter Noncentrality parameter What the alternative hype ditribution i centered on (further from zero, le belief in the null) d i a function of thi parameter, uch that if ncp = 0 (i.e. i centered on the null hype value), then 0 (i.e. no effect) d (1 r) d / g = + ( n 1) n d pop = ncp n + n nn 1 1 CONFIENCE INTERVALS FOR EFFECT SIZE Similar ituation for r and eta effect ize meaure Git: we ll need a computer program to help u find the correct noncentrality parameter to ue in calculating exact confidence interval for effect ize Statitica ha uch functionality built into it menu ytem while other allow for uch interval to be programmed (even SPSS cript are available (Smithon)) LIMITATIONS OF EFFECT SIZE MEASURES Standardized mean difference: Heterogeneity of within-condition variance acro tudie can limit their uefulne the untandardized contrat may be better in thi cae Meaure of aociation: Correlation can be affected by ample variance and whether the ample are independent or not, the deign i balanced or not, or the factor are fixed or not Alo affected by artifact uch a miing obervation, range retriction, categorization of continuou variable, and meaurement error (ee Hunter & Schmidt, 1994, for variou correction) Variance-accounted-for indexe can make ome effect look maller than they really are in term of their ubtantive ignificance 8

9 LIMITATIONS OF EFFECT SIZE MEASURES How to fool yourelf with effect ize etimation: 1. Examine effect ize only at the group level. Apply generic definition of effect ize magnitude without firt looking to the literature in your area 3. Believe that an effect ize judged a large according to generic definition mut be an important reult and that a mall effect i unimportant (ee Prentice & Miller, 199) 4. Ignore the quetion of how theoretical or practical ignificance hould be gauged in your reearch area 5. Etimate effect ize only for tatitically ignificant reult LIMITATIONS OF EFFECT SIZE MEASURES 6. Believe that finding large effect omehow leen the need for replication 7. Forget that effect ize are ubject to ampling error 8. Forget that effect ize for fixed factor are pecific to the particular level elected for tudy 9. Forget that tandardized effect ize encapulate other quantitie uch a the untandardized effect ize, error variance, and experimental deign 10. A a journal editor or reviewer, ubtitute effect ize magnitude for tatitical ignificance a a criterion for whether a work i publihed 11. Think that effect ize = caue ize RECOMMENATIONS Firt recall APA tak force uggetion Report effect ize Report confidence interval Ue graphic RECOMMENATIONS Report and interpret effect ize in the context of thoe een in previou reearch rather than rule of thumb Report and interpret confidence interval (for effect ize too) alo within the context of prior reearch In other word don t be overly concerned with whether a CI for a mean difference doen t contain zero but where it matche up with previou CI Summarize prior and current reearch with the diplay of CI in graphical form (e.g. w/ Tryon reduction) Report effect ize even for nonig reult RESOURCES Kline, R. (004) Beyond ignificance teting. Much of the material for thi lecture came from thi Ronow, R & Roenthal, R. (003). Effect Size for Experimenting Pychologit. Canadian JEP 57(3). Thompon, B. (00). What future Quantitative Social Science Reearch could look like: Confidence interval for effect ize. Educational Reearcher. 9