THE IMPACT OF NONNORMALITY ON THE ASYMPTOTIC CONFIDENCE INTERVAL FOR AN EFFECT SIZE MEASURE IN MULTIPLE REGRESSION

Transcription

1 THE IMPACT OF NONNORMALITY ON THE ASYMPTOTIC CONFIDENCE INTERVAL FOR AN EFFECT SIZE MEASURE IN MULTIPLE REGRESSION By LOU ANN MAZULA COOPER A DISSERTATION PRESENTED TO THE GRADUATE SCHOOL OF THE UNIVERSITY OF FLORIDA IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF DOCTOR OF PHILOSOPHY UNIVERSITY OF FLORIDA 007 1

2 007 Lou Ann Mazula Coope

3 ACKNOWLEDGMENTS I would like to take this oppotunity to thank the co-chais of my dissetation committee, D. David Mille and D. James Algina. They ae both exceptional teaches and without thei guidance, suppot, and patience, this wok would not have been possible. D. Mille, though his flexibility, appoachable manne, and beadth of knowledge has been an invaluable esouce thoughout the couse of my gaduate studies. D. Algina is pehaps the most geneous teache I have eve known and his doo was always open to me. His passion fo eseach and his incedible wok ethic had a pofound influence on me. I would also like to thank the othe membes of my committee fo thei help and encouagement. Special thanks go to D. Richad Davidson. I will always be gateful fo the oppotunity he povided to apply and expand my skills to expeimental design, data analysis, and psychometic issues in medical education eseach. Thank you fo giving me the most ewading and stimulating job I have eve had. D. Walte Leite, although not thee at the beginning, has povided a valuable sounding block and I look fowad to ou futue collaboations. I would also like to expess my gatitude to my family fo thei love and encouagement duing my mid-life caee change. I know it has not always been easy to live with me. To my daughtes, Abigail and Amanda, my hope is that I have povided a good example fo you in pusuing my life-long love of leaning. Finally, and most especially to Bian, my husband and best fiend, whose love and unwaveing belief in me made achieving this goal possible. 3

4 TABLE OF CONTENTS ACKNOWLEDGMENTS...3 LIST OF TABLES...6 LIST OF FIGURES...8 ABSTRACT...10 CHAPTER 1 INTRODUCTION...1 page Effect Sizes and Confidence Intevals in Multiple Regession Analysis...14 Asymptotic Confidence Intevals fo Coelations...15 The Impact of Nonnomality on Statistical Estimates...4 Statement of the Poblem...6 Pupose of the Study...6 METHODS...8 Study Design...8 Numbe of pedictos...8 Squaed multiple coelations...8 Sample size...9 Distibutions...9 Backgound and Theoetical Justification fo the Simulation Method...33 Data Simulation...37 Data Analysis RESULTS...47 Replication of Results fo Multivaiate Nomal Data...47 Simulation Pope...49 Analysis of Vaiance and Mean Squae Components...54 The Influence of Nonnomality on Coveage Pobability...58 Nonnomal pedictos...58 Nonnomal eo distibution...59 The Impact of Squaed Multiple Coelations on Coveage Pobability...59 The Impact of Sample Size on Coveage Pobability...64 Pobability Above and Below the Confidence Inteval...65 The Relationship between Estimated Asymptotic Vaiance, Empiical Sampling Vaiance of ΔR, and Coveage Pobability

5 4 DISCUSSION...14 Limitations...16 Futhe Reseach...18 Conclusion APPENDIX A B PROGRAM FOR COMPUTING MARDIA S MULTIVARIATE MEASURES OF SKEWNESS AND KURTOSIS IN SAS DATA SIMULATION PROGRAM IN SAS LIST OF REFERENCES...14 BIOGRAPHICAL SKETCH

6 Table LIST OF TABLES page -1 Study Design Madia s Multivaiate Skewness, b 1,k, fo the Nonnomal Distibutions Madia s Multivaiate Kutosis, b,k, fo the Nonnomal Distibutions Replication of Algina and Moulde s Results fo Multivaiate Data and Two Pedictos Replication of Algina and Moulde s Results fo Multivaiate Data and Six Pedictos Replication of Algina and Moulde s Results fo Multivaiate Data and Ten Pedictos Empiical Coveage Pobabilities fo Nomal Pedictos and Nomal Eos Empiical Coveage Pobabilities fo Nomal Pedictos and Nonnomal Eos Empiical Coveage Pobabilities fo Nonnomal Pedictos and Nomal Eos Empiical Coveage Pobabilities fo Pedictos Nonnomal and Eos Nonnomal Desciptive Statistics fo Coveage Pobability by Distibutional Condition Analysis of Vaiance, Estimated Mean Squae Components, and Pecentage of Total Desciptive Statistics fo Coveage Pobability by Distibution fo the Pedictos Desciptive Statistics fo Coveage Pobability by Distibution fo the Eos Coveage Pobability by Δ and the Distibution fo the Pedictos Coveage Pobability by Δ and the Distibution fo the Eos Coveage Pobability by 3-15 Coveage Pobability by 3-16 Coveage Pobability by 3-17 Coveage Pobability by 3-18 Coveage Pobability by and the Distibution fo the Pedictos and the Distibution fo the Eos and and and Δ fo X Distibuted Multivaiate Nomal Δ fo X Distibuted Pseudo-t 10 (g = 0, h =.058) Δ fo X Distibuted Pseudo-χ

7 3-19 Coveage Pobability by 3-0 Coveage Pobability by and and Δ fo X Distibuted Pseudo-χ Δ fo X Distibuted Pseudo-exponential Coveage Pobability by Sample Size and Δ Coveage Pobability by Sample Size and Numbe of Pedictos Analysis of Vaiance, Estimated Mean Squae Components, and Pecentage of Total Coveage Pobability as a Function of n, Selected Values fo and Δ, and Distibution fo the Pedictos

8 LIST OF FIGURES Figue page -1 Plot of the empiical cumulative distibution function fo a univaiate nonnomal distibution whee g = 0, h =.058 ovelaid with a nomal cuve with μ gh = 0, σ gh = Plot of the empiical cumulative distibution function fo a univaiate nonnomal distibution whee g =.301, h = ovelaid with a nomal cuve with μ gh =.150, σ gh = Plot of the empiical cumulative distibution function fo a univaiate nonnomal distibution whee g =.50, h = ovelaid with a nomal cuve with μ gh =.49, σ gh = Plot of the empiical cumulative distibution function fo a univaiate nonnomal distibution whee g =.760, h = ovelaid with a nomal cuve with μ gh =.378, σ gh = Compaison of Madia s multivaiate skewness fo the multivaiate nomal distibution to that of the distibutions investigated Madia s multivaiate kutosis fo the multivaiate nomal distibution and the nonnomal distibutions investigated Mean estimated coveage pobability by nomality vs. nonnomality in the pedictos, nomality vs. nonnomality in the eos, and sample size Empiical coveage pobability as a function of distibutional condition and sample size Box plots of the distibutions of coveage pobability estimates by distibution fo the pedictos (n i = 14,700) Box plots of the distibutions of coveage pobability estimates by distibution fo the eos (n i = 14,700) Main effect of the squaed semipatial coelation coefficient, Δ, and the effect of the inteaction of Δ and X on coveage pobability fo Δ > Main effect of the squaed semipatial coelation coefficient, Δ, and the effect of the inteaction of Δ and e on coveage pobability fo Δ > Effect of the inteaction between the size of the squaed multiple coelation in the educed model,, and the distibution fo the pedictos, X, on coveage pobability

9 3-8 Inteaction between the size of the squaed multiple coelation in the educed model,, and the distibution fo the eos, e, and its elationship to coveage pobability Effect of the 3-10 Effect of the X Δ inteaction on coveage pobability fo Δ > Δ inteaction on coveage pobability fo Δ > Inteaction between sample size, n, and the population squaed semipatial coelation, Δ, and the impact on coveage pobability fo Δ > Effect of the inteaction between sample size, n, and numbe of pedictos, k, on coveage pobability Ratio of mean estimated asymptotic vaiance to the vaiance in ΔR (MEAV/Va ΔR ) as a function of the distibution fo the pedictos, Δ, and Relationship between coveage pobability and the atio of mean estimated asymptotic vaiance to the empiical sampling vaiance of ΔR fo Δ > Relationship between coveage pobability and the atio of mean estimated asymptotic vaiance to the empiical sampling vaiance of ΔR fo Δ > 0 fo multivaiate nomal data (g = 0, h = 0) Relationship between coveage pobability and the atio of mean estimated asymptotic vaiance to the empiical sampling vaiance of ΔR fo Δ > 0 and X distibuted pseudo-t 10 (g = 0, h =.058) Relationship between coveage pobability and the atio of mean estimated asymptotic vaiance to the empiical sampling vaiance of ΔR fo Δ > 0 and X distibuted pseudo-χ (g =.50, h = -.048) Relationship between coveage pobability and the atio of mean estimated asymptotic vaiance to the empiical sampling vaiance of ΔR fo Δ > 0 and X distibuted pseudo-χ (g =.50, h = -.048) Relationship between coveage pobability and the atio of mean estimated asymptotic vaiance to the empiical sampling vaiance of ΔR fo Δ > 0 and X distibuted pseudo-exponential (g =.760, h = -.098) Coveage pobability as a function of sample size and seveal combinations of and Δ fo pedictos sampled fom a nomal distibution and (A) pseudo-t 10 ; (B) pseudo- χ ; (C) pseudo- χ ; and (D) pseudo-exponential distibutions

10 Abstact of Dissetation Pesented to the Gaduate School of the Univesity of Floida in Patial Fulfillment of the Requiements fo the Degee of Docto of Philosophy THE IMPACT OF NONNORMALITY ON THE ASMPTOTIC CONFIDENCE INTERVAL FOR AN EFFECT SIZE MEASURE IN MULTIPLE REGRESSION Chai: M. David Mille Co chai: James Algina Majo: Reseach and Evaluation Methodology By Lou Ann Mazula Coope May 007 The incease in the squaed multiple coelation coefficient, ΔR, associated with an individual pedicto in a egession analysis is a measue commonly used to evaluate the impotance of that vaiable in a multiple egession analysis. Pevious eseach using multivaiate nomal data had shown that elatively lage sample sizes ae necessay fo an acceptably accuate confidence inteval fo this egession effect size measue. The coveage pobability that an asymptotic confidence inteval contained the population squaed semipatial coelation, Δ, was investigated by simulating data fom a ange of nonnomal distibutions such that (a) the pedictos wee nonnomal, (b) the eo distibution was nonnomal, o (c) both pedictos and eos wee nonnomal. Additional factos manipulated included (a) the numbe of pedicto vaiables, (b) the magnitude of the population squaed multiple coelation coefficient in the oiginal model, population squaed semipatial coelation, Δ, and (d) sample size., (c) the magnitude of the This study showed that when nonnomality is intoduced, empiical coveage pobability was always less than the nominal confidence level, often damatically so. The degee of 10

11 nonnomality in the pedictos was the most impotant facto influencing poo coveage pobability. Although coveage pobability inceased as a function of sample size, when nonnomality in the pedictos was substantial, the confidence inteval is likely to be inaccuate no matte how lage a sample size is used. With multivaiate nomal data, coveage pobability impoved as both and Δ inceased. When pedictos ae sampled fom a nonnomal distibution, coveage pobability tended to decease as and Δ inceased and became even wose as the degee of nonnomality inceased. It was futhe demonstated that the asymptotic vaiance undeestimates the sampling vaiance of ΔR. This poduces standad eos that ae too small and esults in a confidence inteval that is too naow. Reliance on this confidence inteval as a measue of the stength of the effect size will lead us to undeestimate the impotance of an individual pedicto to the egession. 11

12 CHAPTER 1 INTRODUCTION Thee is a gowing consensus that the tadition of null hypothesis significance testing (NHST) has led to ove-eliance on statistical significance in evaluating eseach esults in the behavioal and social sciences. Accoding to Cohen (1994), the biggest flaw in NHST is that it does not tell us what we want to know. A statistical test evaluates the pobability of the sample esults given the size of the sample assuming that the sample is dawn fom a population whee the null hypothesis is exactly tue. In this famewok, the outcome of a significance test is a dichotomous decision whethe o not to eject the null hypothesis. As noted by Steige and Fouladi (1997, p. 5), this dichotomy is inheently dissatisfying to psychologists and educatos, who fequently use the null hypothesis as a statement of no effect, and ae moe inteested in knowing how big an effect is than whethe it is (pecisely) zeo. Fundamentally, we ae inteested in detemining how accuately the population effect has been estimated fom the sample data and whethe the obseved effect size has pactical significance. Statistical significance testing fails to povide the answes. Within the behavioal and social sciences, methodological ecommendations fo epoting eseach esults have inceasingly emphasized the impotance of epoting confidence intevals (Cumming & Finch, 001; Smithson, 001), effect sizes (Olejnik & Algina, 00; Vacha-Hasse & Thompson, 004), and confidence intevals fo effect sizes (Cohen, 1990; Steige & Fouladi, 1997; Thompson, 00) to complement the esults of hypothesis testing. Among the ecommendations of the APA s Task Foce on Statistical Infeence (Wilkinson & Task Foce on Statistical Infeence, 1999) was a poposal to move away fom outine eliance on NHST as a pimay means of analyzing data to exploing, summaizing and analyzing data using visual epesentations, effect-size measues, and confidence intevals. The most ecent edition of The 1

13 Publication Manual of the Ameican Psychological Association (001, p. 5-6) states, Fo the eade to fully undestand the impotance of you findings, it is almost always necessay to include some index of effect size o stength of elationship in you Results section The geneal pinciple to be followed, howeve, is to povide the eade not only with infomation about statistical significance but also with enough infomation to assess the magnitude of the obseved effect o elationship. The Manual also states that failue to epot an effect size is a defect (p. 5). In 1996, Thompson ecommended that Ameican Educational Reseach Association (AERA) jounals equie that effect sizes be epoted and intepeted in all studies. Ten yeas late the AERA Council ecommends that statistical esults should include an effect size measue as well as an indication of the uncetainty of that index of effect such as a confidence inteval. The ecently adopted Standads fo Repoting on Empiical Social Science Reseach in AERA Publications (AERA, 006) states that when quantitative methods ae employed, It is impotant to epot the esults of analyses that ae citical fo the intepetation of findings in ways that captue the magnitude as well as the significance of those esults (p. 37). Editos of ove 0 APA and othe social science jounals have published guidelines explicitly equiing authos to epot effect sizes (Ellis, 000; Hais, 003; Heldef Foundation, 1997; Hesko, 000; McLean & Kaufman, 000; Roye, 000; Snyde, 000; Thompson, 1994; Vacha-Haase, Nilsson, Rentz, Lance, & Thompson, 000) and the Edito of Jounal of Applied Psychology equies an autho to povide an explanation when an effect size is not epoted (Muphy, 1997). Although this is evidence that editoial pactices have evolved somewhat, effect size epoting is unlikely to become the nom until we move fom ecommendation and encouagement to equiement (Thompson, 1996; 1999). 13

14 Effect Sizes and Confidence Intevals A confidence inteval establishes a ange of paamete values that ae easonably consistent with the data obseved fom a sample. Because a confidence inteval gives a best point estimate of a paamete of inteest and an inteval about it eflecting an estimate of likely eo, it contains all the infomation to be found in a significance test and moe (Cohen, 1994). The likely ange of the paamete values povides eseaches with a bette undestanding of thei data. If the paamete estimated has meaningful units, a confidence inteval can be used to make statistical infeences that povide infomation in the same metic. Accoding to Cumming and Finch (001), thee ae fou main easons fo pomoting the use of confidence intevals: (a) they ae eadily intepetable, (b) ae linked to familia statistical tests, (c) can encouage eplication and meta-analytic thinking, and (d) give infomation about pecision. The tem effect size is boadly used to efe to any statistic that povides infomation that helps us judge the pactical significance of the esults of a study (Kik, 1996). Cohen (1990) ecommends that in addition to epoting an effect size, eseaches should povide confidence intevals fo effect sizes in ode to gauge the possible ange of values an effect size may assume. Absent a confidence inteval, it is difficult to evaluate the accuacy of the effect size estimate. This, in tun, has implications fo dawing meaningful conclusions. Unfotunately, despite the inceasing demand fo eseaches to do so, epoting effect sizes and confidence intevals has yet to become commonplace in educational and psychological jounals. Vacha-Hasse, Nilsson, Rentz, Lance, and Thompson (000) eviewed ten studies of effect size epoting in 3 jounals, and found effect size(s) to be epoted in oughly 10 to 50 pecent of aticles, notwithstanding the encouagement to do so fom the fouth edition of the APA manual (1994). Empiical studies show that even when effect sizes ae epoted, intepetation is often given shot shift (Finch et al, 00; Keselman et al., 1998). 14

15 It is likely that the emphasis on null hypothesis significance testing in gaduate couses in statistics and eseach methodology has contibuted to a geneal lack of knowledge concening confidence intevals. Moeove, techniques fo computing confidence intevals ae often neglected in popula statistics textbooks and ae not easily available in the statistical softwae that is outinely employed by applied eseaches in the social sciences (Smithson, 001). Even if these factos wee not opeating, eseaches might be eluctant to epot confidence intevals because as Steige and Fouladi (1997, p. 8) obseve, inteval estimates ae sometimes embaassing. Repoting confidence intevals can highlight the level of impecision of statistical estimates and exposes the tivial natue of many published studies. Smithson (001, p. 614) notes, Almost any liteatue eview o meta-analysis in psychology would give a vey diffeent impession fom that conveyed by NHST if we outinely econstucted CIs fo multiple R and elated GLM paametes. Asymptotic Confidence Intevals fo Coelations A confidence inteval establishes a ange of hypothetical paamete values that cannot be uled out given the obseved sample data. The pobability that the andom inteval includes, o coves, the tue value of the paamete is the coveage pobability of the inteval. When the exact distibution of a statistic is known, the coveage is equal to the confidence level and the inteval is said to be exact. A confidence inteval is exact if it can be expected to contain a paamete s tue value 100(1 α)% of the time. Often exact intevals ae not available o ae difficult to calculate, and appoximate intevals ae used instead. Confidence intevals ae based on the sampling distibution of a statistic. Due to the cental limit theoem, when sample size is sufficiently lage, the sampling distibution of statistic will become moe symmetic and eventually appea nealy nomal, even when the population itself is not nomally distibuted. Methods based on asymptotic theoy use appoximations to the 15

16 sampling vaiance of a statistic. If only the asymptotic distibution of the statistic is known, we can obtain an appoximate confidence inteval, which may o may not be easonably accuate in finite samples. If the asymptotic confidence inteval pocedue is fully adequate, unde epeated andom sampling unde identical conditions, a 95% confidence inteval would contain the tue population paamete 95% of the time. The accuacy of the appoximation depends on whethe thee is a lack of bias and the degee to which the sampling distibution deviates fom nomality. If a statistic has no bias as an estimato of a paamete, its sampling distibution is centeed at the tue value of a paamete. An unbiased confidence inteval is one whee the pobability of including any value othe than the paamete s tue value is less than o equal to 100(1 α)%. An inteval is said to be consevative if the ate of coveage is geate than 100(1 α)%, the nominal confidence level. If the coveage pobability is less than the nominal, the inteval is said to be libeal. In geneal, consevative intevals ae pefeed ove libeal ones (Smithson, 003). Wheneve a statistic based on asymptotic theoy has poo finite sample popeties, a confidence inteval based on that statistic has poo coveage. Multiple egession analysis is a common statistical application fequently used to pedict a dependent vaiable (outcome) fom two o moe independent vaiables (pedictos). The intepetation of esults would be enhanced by the epoting of confidence intevals and effect sizes. The sample statistic, R, which estimates the popotion of vaiance in the dependent vaiable that is explained by the set of pedictos, is commonly used to evaluate a multiple egession model. Published eseach studies fequently epot R values without any evidence of the pecision with which they have been estimated. It is unfotunate that a confidence inteval fo the population paamete,, is not computed by most popula statistical softwae packages. 16

17 Pehaps moe significant, the topic is not even discussed in many applied o theoetical statistics texts. In addition to the amount of vaiance explained by a given multiple egession model, eseaches ae often inteested in evaluating the contibution that one vaiable makes to the egession, ove and above a set of othe explanatoy vaiables. The incease in R, ΔR, when a vaiable (X j ) is added to a multiple egession model is a useful measue of the stength of the elationship between X j and the dependent vaiable, Y, contolling fo all othe independent vaiables in the model. The change in R that we obseve by including each new X j in the egession equation is the squaed semipatial coelation coesponding to a given egession coefficient. Typically, whethe X j has made a statistically significant contibution to pedicting Y is tested by conducting a t- o F-test on that egession coefficient. But, the squaed semipatial coelation itself is a useful measue of effect size and as ecommended by Cohen (1990) and Thompson (00), we should calculate a confidence inteval to evaluate the pecision with which it has been estimated and the ange of likely values. Hedges and Olkin (1981) pesented pocedues fo constucting a confidence inteval fo the squaed semipatial coelation based on calculating the asymptotic covaiance matix fo commonality components. Commonality analysis is a pocedue by which the vaiance accounted fo in the citeion is patitioned into two pats, the unique pat and the common pat. The unique pat is attibutable to the pedictos individually. This is essentially the patial contibution of each pedicto to the squaed multiple coelation with the citeion. The second pat is the common pat, attibutable to a combination of the pedictos, which is the contibution to the multiple coelation with the citeion that all of the pedictos in the combination shae. 17

18 Thus, commonality analysis is a way to measue the impotance of vaiables though the use of patial coelations. Hedges and Olkin s esults can be used to constuct a confidence inteval fo ΔR. Olkin and Finn (1995) deived explicit expessions fo asymptotic (lage-sample) confidence intevals fo functions of simple, patial, and multiple coelations. Since the focus of this study is on the squaed semipatial coelation, the following discussion will be limited to Olkin and Finn s Model A (p ). Model A is the special case fo use in detemining whethe an additional vaiable povides an impovement in pedicting the citeion. All of the pocedues fo compaing two sample coelation coefficients o two sample squaed coelation coefficients descibed by Olkin and Finn have the same geneal fom. Let A and B be the two sample coelations to be compaed and A and B denote thei coesponding population values. The lage-sample distibutional fom fo the diffeence in two coelations is whee [( A B) ( A B) ] ~ N( 0, ) σ (1.1) σ = va( ) + va( ) cov(, ) (1.) A B A B is the asymptotic vaiance of the diffeence of the two coelation coefficients; on the population coelations (Olkin & Finn, 1995, p. 156). 1. become σ is dependent When squaed coelation coefficients ae compaed, the expessions in Equations 1.1 and N σ (1.3) [( A B) ( A B)] ~ (0, ) and ( A B ) ( A B) ( A B ) σ = va cov,. (1.4) 18

19 Olkin and Finn pesent the geneal fom fo the lage-sample vaiance of functions of coelations σ f( ij, ik, jk ) = Φ a a (1.5) specialized to a function of thee coelations, ij, ik, and jk whee f( ) is a function of the coelations, Φ is the sampling vaiance-covaiance matix of the coelations, and vecto a contains a set of coefficients that depend on the function of the coelations to be evaluated. The vaiance of sample coelation ij is va( ij ) (1 ij ) / = n (1.6) and the covaiance of two coelations is 1 = (1.7) ( ) / n. cov( ij, ik ) ij kl ( ik il jk jl ) ik jl il jk ij ik il ji jk jl ki kj kl li lj lk When two coelations have one vaiable in common, Equation 1.7 simplifies to 1 3 cov( ij, ik ) = ( jk ijik )(1 ij ik jk ) +jk / n. (1.8) Lage-sample estimates ae obtained by eplacing the population paametes with values computed fom sample data. Using the delta method, it can be shown that if f( ij, ik. jk ) is a function of the thee coelations, then the vecto a consists of the patial deivatives f f f a =,, (1.9) In the simplest case, suppose that two vaiables X 1 and X ae used to pedict a thid vaiable, X 0. In ode to detemine whethe X, makes a significant contibution to the egession, we ae inteested in the diffeence, R 0(1) 01. Hee, we use a capital R to signify a multiple coelation athe than a bivaiate coelation, denoted by a lowe case. The symbol R 0(1) 19

20 denotes the squaed multiple coelation between X 0, X 1 and X, which is a function of the coelations among the vaiables 01, 0, and 1 given by R + ˆ (1) = 0(1) = 1 1 (1.10) The squaed coelation between X 0 and X 1 is epesented by 01. Theefoe, a confidence inteval fo R 0(1) 01 can be computed using Olkin and Finn s esults fo compaing two squaed multiple coelation coefficients. In ode to compae the population squaed multiple coelations 0(1) and 01, we use the estimates R, 0(1) 01, and σ ˆ, the estimated vaiance of the diffeence R - whee 0(1) 01 va( R ) =aφa. (1.11) 0(1) 01 The uppe tiangula of the symmetic population coelation matix is P = and the elements of the vecto, a, ae a = ( ), a = ( ), a = ( + ) (1 1) The vaiance-covaiance matix fo the sample coelations is φ11 φ1 φ13 va( 01 ) cov( 01, 0) cov( 01, 1) φ φ = va( ) cov(, ) φ 33 va( 1) (1.1) (1.13) (1.14) (1.15) (1.16) 0

21 The sample coelation matix, R, estimates P and the sample values in R can be used to compute the elements of a. Because the calculation of analytic deivatives becomes inceasingly complicated as the numbe of vaiables inceases, Olkin and Finn illustated thei method fo a multiple egession model with no moe than two pedictos. Gaf and Alf (1999) expanded Olkin and Finn s pocedues to moe geneal foms. Gaf and Alf substituted numeical deivatives and offeed two BASIC pogams fo calculating asymptotic confidence limits on the diffeence between two squaed multiple coelations and the diffeence between two patial coelations. These pogams, REDUX-AB, to compae two multiple coelations, and REDUX-CD, to compae two patial coelations, compute the Φ matix, the patial deivatives in vecto a, and a 95% confidence inteval. Alf and Gaf (1999) pesent a futhe simplification that does not employ numeical deivatives, is less computationally demanding, and poduces esults equivalent to the method descibed by Olkin and Finn. All computations ae based on sample estimates. The poblem is appoached by epesenting a multiple coelation as a zeo-ode coelation between the outcome vaiable and anothe single vaiable that is a weighted sum of the pedictos. Alf and Gaf defined AB 0B = (1.17) 0 A whee the subscipts A and B denote weighted sums of two sets of pedictos and AB is the coelation between the two composite vaiables. The confidence inteval fo the squaed semipatial coelation coefficient is detemined by the special case in which one set of pedictos is a pope subset of the pedictos in the othe coelation. The two squaed multiple coelations ae computed using the same sample and the 1

22 vaiables in the educed model ae a subset of the vaiables in the full model. Let and denote the population squaed multiple coelation coefficients coesponding to R and R. The subscipt, f, efes to the full model with all pedictos; the subscipt,, efes to the educed model. The educed model contains all pedictos with the exception of the vaiable of inteest. The asymptotic vaiance of ( f ) R f is 4 1 f Va( R f ) =. (1.18) n The asymptotic vaiance of ( ) R is 4 1 Va( R ) =. (1.19) n The asymptotic covaiance between R and R f is f f Cov R R ( f, ) ( )( ) 4.5 / 1 / +/ = n 3 3 f f f f f f. (1.0) Fo the squaed semipatial coelation, let Δ R = R f R. The asymptotic vaiance of Δ R is σ = Va( R ) + Va( R ) Cov( R, R ). (1.1) f f An asymptotically coect 100(1 - α)% confidence inteval fo Δ = is f Δ ± σ (1.) R z α/ˆ whee z α/ is the (1 - α/)th pecentile of the standad nomal distibution andσˆ is the estimate of σ. In pactice, the lage-sample vaiance is estimated by substituting R fo f f and R fo in Equations 1.18, 1.19, and 1.0. Equations 1.18 and 1.19 ae poblematic when the population squaed multiple coelations ae zeo because the implication is that the sampling vaiance of R is also zeo (Stuat, Od, &

23 Anold, 1999). Similaly, Equation 1.0 implies that the sampling covaiance is zeo if eithe population multiple coelation coefficient, f o, is zeo. If it wee known that both f and wee zeo and these values wee used to constuct a confidence inteval, we would incoectly conclude that the width of the esulting inteval is zeo. This computational poblem is unlikely to occu in pactice since we substitute sample multiple coelation coefficients fo thei population values and it is doubtful that eithe R o f R will eve be exactly zeo. The Alf and Gaf fomulas ely on asymptotic esults. As such, they ae only exactly coect fo infinitely lage samples. Thus, the accuacy of this appoximation is heavily dependent on sample size. Alf and Gaf (1999, p.74) concluded that the coelation between two multiple coelations will be extemely high when the vaiables in one multiple coelation ae a subset of the vaiables in anothe multiple coelation and to ensue that coveage pobability is equal to the nominal fo the confidence inteval on Δ, modeately lage to lage sample sizes ae necessay. In the absence of moe specific ecommendations on sample sizes, Algina and Moulde (001) conducted a simulation study to evaluate the empiical pobability that the inteval in Equation 1. includes Δ fo 95% confidence inteval. Algina and Moulde manipulated, the numbe of pedictos in the model (k), and the sample size (n). When the data ae distibuted multivaiate nomal, esults indicate that when Δ > 0, fo sample sizes epesentative of those used in psychology (i.e., n 600), coveage pobabilities fo a nominal 95% confidence inteval wee less than.95. This tends to be tue even with elatively lage sample sizes, i.e. between 600 and 100. f, 3

24 When = 0 all coveage pobabilities wee at least.999 fo all sample sizes studied. f That is, when does not incease when a pedicto is added to a multiple egession model, the confidence inteval is always too wide. Algina and Moulde (001) posited two easons fo this defect in the confidence inteval: (a) fo all conditions in which = 0 the asymptotic f vaiance oveestimated the sampling vaiance and (b) the distibution of R R is positively f skewed with a lowe limit of 0. Because the confidence inteval does not take this lowe limit into account, even if the asymptotic vaiance was not oveestimated, the lowe limit would tend to be smalle than zeo. Algina and Moulde (001) showed that coveage pobability tends to incease as inceases and as Δ inceases and tends to decease as the numbe of pedictos inceases. Futhe, when the inteval does not contain Δ, thee is a tendency fo the inteval to be entiely below Δ. Algina and Moulde conclude that using the Alf and Gaf method to compute a confidence inteval with an inadequate sample size will undeestimate the stength of the elationship between the pedicto and the outcome vaiable. The Impact of Nonnomality on Statistical Estimates Evey pocedue used to make statistical infeences is based on a set of coe assumptions. If the assumptions ae met, the test will pefom as theoized. Howeve, the esults may be misleading when the assumptions ae violated. The most common method fo estimating egession coefficients is odinay least squaes (OLS). Odinay least squaes yields unbiased, efficient, and nomally distibuted estimates when the following conditions ae met: (a) No measuement eo; () the mean of the esiduals is zeo; (3) the esiduals have constant vaiance; (4) the esiduals ae not inte-coelated; and (5) the esiduals ae nomally distibuted. 4

25 In tems of powe and accuate pobability coveage, standad analysis of vaiance (ANOVA) and egession methods ae affected by abitaily small depatues fom nomality. As ealy as 1960, Tukey found that nonnomality could have a sizeable impact on powe and measues of effect size could be misleading wheneve means ae being compaed. By sampling fom a contaminated nomal distibution, Tukey showed that classical estimatos ae quite sensitive to distibutions with heavy tails. The contaminated nomal distibution is a mixtue of two nomal distibutions, one of which has a lage vaiance; the othe distibution is standad nomal. This esults in a distibution with heavie tails than the Gaussian. Heavy-tailed distibutions ae chaacteized by unusually lage o small values. Both heavy-tailed and skewed distibutions ae commonplace in applied wok (Miccei, 1989). The pesence of these chaacteistics in the data can diminish the chances of detecting tue associations among andom vaiables and obtaining accuate confidence intevals fo the paametes of inteest (Wilcox, 1998). Afte eviewing ove 400 lage data sets fom educational and psychological eseach, Miccei (1989) found the majoity did not follow univaiate nomal distibutions. Appoximately two-thids of ability measues and ove 80% of the psychometic measues examined exhibited at least modeate asymmety. Fo all data sets studied, 31% of the distibutions showed skewness, γ 1, geate than.70 and 5% of psychometic measues demonstated exteme to exponential asymmety, γ 1 >.00. Psychometic measues also exhibited heavie tails than ability measues. Kutosis estimates anged fom 1.70 to To put this in some pespective, the kutosis fo the double exponential distibution is 3.0. Beckle (1990) consideed 7 aticles in pesonality and social psychology jounals and found that in analyses elying on the assumption of multivaiate nomality, only 19% of authos 5

26 acknowledged this assumption and less than 10% consideed whethe it had been violated. Keselman and his colleagues (1998) eviewed aticles in pominent educational and behavioal sciences eseach jounals published duing 1994 and 1995 and concluded (a) the majoity of eseaches conduct statistical analyses without consideing the distibutional assumptions of the tests they ae using and theefoe use analyses that ae not obust; (b) eseaches aely epoted effect sizes; and (c) eseaches failed to pefom powe analyses in ode to infom sample size decisions. Statement of the Poblem Methods fo constucting confidence intevals based on asymptotic theoy, such as those poposed by Olkin and Finn and Alf and Gaf, have the potential to be vey attactive to applied eseaches. In the case of the equations pesented by Alf and Gaf, a hand calculato can be used to compute a confidence inteval using the appopiate estimates fom the esults of data analysis obtained using standad statistical analysis softwae. Howeve, as Algina and Moulde demonstated, even unde the best case scenaio, whee data ae dawn fom a multivaiate nomal distibution, the coveage pobability of the asymptotic confidence inteval fo Δ is less than optimal, and when sample size is elatively small, e.g., < 00, would be consideed unacceptable by most eseaches. Since multivaiate nomal data is ae, the pefomance of Alf and Gaf s pocedue unde eal wold conditions waants futhe investigation. Pupose of the Study My dissetation will extend the wok of Algina and Moulde (001) and investigate the effect of the magnitude of population squaed multiple coelation coefficients, and f, as well as the numbe of pedictos, on the asymptotic confidence inteval fo Δ unde a ange of nonnomal conditions. The study will investigate coveage pobability when (a) the pedicto 6

27 vaiables ae not distibuted multivaiate nomal; (b) the esiduals ae not nomal; and (c) both pedictos and esiduals ae nonnomal. Empiical coveage pobabilities will be compaed to nominal coveage pobabilities ove a wide ange of sample sizes. My eseach will addess the following questions: How adequate is Alf and Gaf s asymptotic confidence inteval pocedue fo the squaed semipatial coelation coefficient when used with sample sizes typically employed in eseach in education, psychology and the behavioal sciences unde conditions of nonnomality? Is thee a minimum sample size fo which this method meets established standads fo accuacy ove a wide ange of situations such that ecommendations can be made fo the use of this pocedue in epoting the esults of applied eseach? 7

28 CHAPTER METHODS In conducting a simulation study, especially when the goal is to infom the pactice of eseaches, it is impotant to ensue that the elevant factos ae manipulated and that the levels of these factos eflect those outinely obseved. To that end, six factos wee manipulated in a factoial design using values typical of those obseved in applied eseach: the numbe of pedictos, the size of the squaed multiple coelation in the educed model, the size of the squaed semipatial coelation, sample size, the distibution fo the pedictos, and the distibution fo the eo. These factos, and the levels of these factos, ae detailed in Table -1. Numbe of pedictos Study Design Algina, Moulde, and Mose (00) examined sample size equiements fo accuate estimation of squaed semipatial coelation coefficients and found a modest effect on the distibution of ΔR due to the numbe of pedictos included in the multiple egession model. Theefoe, it follows that the sample size equied fo the confidence inteval on Δ to be obust, i.e. to have the coveage pobability equal to the nominal confidence level, will likewise depend on the numbe of pedictos. The numbe of pedictos in the initial set of pedictos (k 1) anged fom to 10 in incements of. This allowed investigation of the pefomance of the asymptotic confidence inteval fo a easonable ange of model sizes. Squaed multiple coelations Algina, Moulde, and Mose also showed that the sampling distibution of ΔR stongly depends on the population squaed multiple coelations in both the full and educed models, f and. Based on a suvey of all APA jounal aticles published in 199 epoting multiple 8

29 egession esults, Jaccad and Wan (1995) found the median squaed multiple coelation in these studies to be.30. The 75 th pecentile fo squaed multiple coelations was appoximately.50. Based on these esults, the values fo the squaed multiple coelation coefficients fo the pedictos in the initial set ( ) anged fom.00 to.60 in steps of.10 (7 levels of the facto). Cohen (1988) poposed, as a convention, that.0,.13, and.6 epesent small, medium, and lage effect sizes fo squaed semipatial coelations. By manipulating the squaed multiple coelation coefficient fo the entie set of pedictos ( ), such that it anged fom f to +.30 in steps of.05, values fo Δ that anged fom.00 to.30 in steps of.05 wee poduced (7 levels of the facto). The values fo Δ ae easonably epesentative of likely effect sizes and the values selected fo and f cove a compehensive ange of population squaed multiple coelations fo multiple egession models fom =.00 to =.90. Sample size Jaccad and Wan also epoted typical sample sizes fo studies using egession analysis. The median sample size was 175; a sample size of 400 was at the 75 th pecentile. Howeve, Algina and Moulde found with multivaiate nomal data empiical estimates of the coveage pobability wee smalle than.95 even with a sample size as lage as 100. Since we expected empiical coveage pobabilities to be wose fo nonnomal data, lage sample sizes than ae usually obseved in psychological eseach wee included. Sample size anged fom 100 to 1000 in steps of 100 and fom 1000 to 000 in steps of 50 (14 levels of the facto). Distibutions The distibutions chosen fo study epesent vaying levels of nonnomality and wee selected to: (a) allow examination of the effects of skewness and kutosis; and (b) be epesentative of the types of univaiate nonnomality commonly encounteed in applied eseach 9

30 in education and psychology. The method descibed in Hoaglin (1985) and Matinez and Iglewicz (1984) using the g-and-h distibutions was used to geneate data that is chaacteized by vaying degees of skewness (γ 1 ) and kutosis (γ ). A g-and-h distibution is geneated by a single tansfomation of the standad nomal distibution and allows fo asymmety and a vaiety of tail weights. In the case of the standad nomal distibution, g = h = 0 and γ 1 = γ = 0. When g = 0, a distibution is symmetic. Distibutions with positive skew typically have γ 1 > 0 and in distibutions with negative skew, γ 1 < 0. The tails of the distibution become heavie as h inceases in value. Long-tailed distibutions, such as the t-distibution, ae chaacteized by γ > 0. Shot-tailed distibutions, such as the unifom distibution, have γ < 0. The distibutions selected fo this study and thei skewness and kutosis ae pesented in Table -1. Distibution 1 is the multivaiate nomal case. Distibution is symmetic and long-tailed and has the same skew and kutosis as a t-distibution with 10 degees of feedom. Distibution 3 is both asymmetic and leptokutotic with the same skew and kutosis as a χ distibution with 10 degees of feedom. Since distibutions and 3 have simila kutosis, but diffe with espect to asymmety, this allowed us to evaluate the elative impotance of skewness and kutosis on the coveage pobability of the confidence inteval. Distibution 4 has the same skew and kutosis as χ 4. Distibution 5 is extemely skewed with heavy tails and has skew and kutosis equal to the exponential distibution. Nonnomality was manipulated in eithe (a) the pedictos, (b) the esiduals, o (c) in both the pedictos and the esiduals. The eo distibution is a univaiate distibution. The empiical cumulative distibution functions fo the fou nonnomal distibutions selected fo this study, geneated by sampling 1,000,000 andom vaiates fom each g-and-h distibution, ae depicted in Figues -1 to -4. In addition, the deviation fom nomality is shown by including the nomal cuve with mean equal 30

31 to μ gh and standad deviation equal to σ gh fo each distibution. The population mean and standad deviation fo each g-and-h distibution wee calculated using the fomulas given by Hoaglin (1985, p ). In multiple egession, the pedictos ae multivaiate. Multivaiate nomality, howeve, is a stonge assumption than univaiate nomality. Univaiate nomality of each of the vaiables is necessay, but not sufficient, and a nonnomal multivaiate distibution can have nomal maginals. Theefoe, a peliminay step in evaluating multivaiate nomality is to study the easonableness of assuming maginal nomality fo the obsevations on each of the vaiables (Gnanadesikan, 1997). In addition to gaphical appoaches, a common method fo evaluating the nomality of univaiate obsevations is by means of skewness and kutosis coefficients, b 1 and b : n ( ) n x x i i= 1 1 n 3/ b = ( xi x) i= 1 3 (.1) and b n ( ) n x x i i= 1 n = ( xi x) i= 1 4. (.) These ae sample estimates of the population skewness and kutosis paametes β1 and β, espectively. When the population is nomal, β 1 = 0 and β = 3. If β < 3, thee is negative kutosis; if β > 3, thee is positive kutosis. Population skewness and kutosis ae also 31

32 commonly descibed by γ 1 and γ (Hoaglin, 1985) whee and γ 1 = β 1 (.3) γ =β 3. (.4) Madia (1970) poposed indices fo assessing multivaiate nomality that ae genealizations of the univaiate skewness and kutosis measues b 1 and b. Let X 1,,X n be a andom sample fom a population with mean vecto μ and covaiance matix Σ. The sample mean vecto and covaiance matix ae denoted by X and S, espectively. The skewness and kutosis, β 1,k and β,k, fo a multivaiate population, as defined by Madia, ae ( ) ( ) 3 β 1 1,k = E xi Σ x j μ μ (.5) and ( ) ( ) β 1, k = E xi Σ x j. μ μ (.6) Accoding to Renche (1995), since thid ode cental moments fo the multivaiate nomal distibution ae zeo, β 1,k = 0 when X ~ N(μ,Σ). Futhemoe, it can be shown that fo multivaiate nomal X β, k = kk ( + ) (.7) whee k is equal to the numbe of vaiables. Sample estimates of β 1,k and β,k ae given by 1 b = X X S X X (.8) n n 1 1, ( ) k i j n i= 1 j= 1 ( ) 3 3

33 and 1 b X X S X X 1, k = ( i ) ( j ). n (.9) i Multivaiate skewness and kutosis wee calculated by simulating 1,000,000 andom vaiates sampled fom each g-and-h distibution fo each level of k unde investigation and then applying equations.8 and.9 to obtain estimates of Madia s multivaiate measues, b 1,k and b,k. The SAS pogam used to estimate these indices is included in Appendix A. Madia s multivaiate skewness estimates ae pesented in Table - and Table -3 pesents Madia s multivaiate kutosis estimates. Figues -5 and -6 ae gaphic pesentations that compae the coefficients fo the nonnomal distibutions to the values expected unde multivaiate nomality fo the numbe of pedictos unde investigation in this study. The design fo the study is a 5 (data geneating distibution fo the pedictos) 5 (data geneating distibution fo the eos) 7 ( ) 7 (Δ ) 5 (k) 14 (n) fully cossed factoial. This esulted in a total of 85,750 unique conditions. Each combination of factos was eplicated 10,000 times and fo each eplication, a 95% confidence inteval was constucted using the Alf and Gaf method. Backgound and Theoetical Justification fo the Simulation Method The multiple egession model can be witten as Yj =β 0 +β 1X1j +β X j β kxkj +ε j. (.10) In the standadized multiple egession model, in the population with k 1 pedictos and one citeion, all vaiables ae standadized to mean zeo and unit vaiance so an intecept is not needed. This model is Y =β X +β X β X +ε = β X +ε (.11) j 1 1j j k kj j i ij j i= 1 k 33

34 whee β i is the population standadized egession coefficient associated with the ith pedicto; e ij ~N(0,σ ); i = 1,, k; j = 1,, n. Assuming that we ae opeating on the population and that the model is coect, pedicted values ae given by Yˆ k = β X (.1) j i ij i= 1 and the squaed coelation between the obseved (Y) and the pedicted (Ŷ ) values is denoted as YYˆ. In the sample, this is estimated by R. When the pedictos ae uncoelated, the sum of the squaed coelations is equal to the vaiation accounted fo by all the pedictos k YX = ˆ. i YY (.13) i= 1 A simplifying tansfomation (Bowne, 1975) holds that fo any set of pedictos that has a squaed multiple coelation,, with Y, it is always possible to tansfom the pedictos so that (a) the tansfomed pedictos ae mutually uncoelated, (b) have unit vaiance, and (c) the egession coefficients ae equal to any set of values such that k β j =σy. (.14) j= 1 The quantity Δ is a function of the elements of the covaiance matix fo the pedictos and the citeion. In ode to illustate the application of Bowne s esults to the cuent simulation, let denote the vecto of standadized pedicto vaiables, with k x k coelation matix P and k 1 vecto of coelation coefficients between the pedictos and the citeion vaiable, y. The squaed multiple coelation coefficient fo all k vaiables is denoted by f and fo the fist k 1 vaiables is denoted by. We seek a tansfomation of the pedictos to x such that the new 34

35 vaiables ae standadized and uncoelated, and the egession coefficients elating y to the vaiables in x ae i 0 β = fo the fist k vaiables and β = and k 1 β =, fo the k f last two vaiables, espectively. The tansfomation can be constucted in two steps. It is well known that the vaiables in the vecto x= Ax, whee A is a k k matix, will be uncoelated dependent on an appopiate choice of A. Fo example, A can be selected as the invese of the left Cholesky facto of 1 T R ( i.e., R = A A, whee T A indicates the invese of A '). The vecto of coelation coefficients between the tansfomed pedictos and the citeion is A and because the tansfomed vaiables ae uncoelated, β = A is the vecto of egession coefficients elating the citeion vaiable to the vaiables in x%. Because the citeion is a standadized vaiable and x% = Ax is a nonsingula tansfomation, is unchanged by the tansfomation, and = ββ. We next seek a tansfomation x= Tx, whee T is k k, such that the vaiables in x ae f standadized and uncoelated and so that the egession coefficients fo the vaiables in x ae f β = 0 fo the fist k vaiables and i β = and k 1 β = fo the last two vaiables, k f espectively. We see that ββ = f. Because the vaiables in x ae standadized and uncoelated, the matix T must be othogonal so that the vaiables in x will be standadized and uncoelated. With an othogonal tansfomation, β = Tβ. The matix T can be constucted as follows (M. W. Bowne, pesonal communication with J. Algina, 1999): Let u = β β. Then, ( ) 1 ββ = ββ, u β = uu 1, and = T = I u u u u is an othogonal matix, and β = Tβ. Because β Tβ it follows that Tβ= β ( β β) u β. Thus, if the vaiables in uu 35