A Regression Solution to the Problem of Criterion Score Comparability

Transcription

1 A Regressin Slutin t the Prblem f Criterin Scre Cmparability William M. Pugh Naval Health Research Center When the criterin measure in a study is the accumulatin f respnses r behavirs fr an individual ver time, there may be a prblem in btaining cmparable scres if all individuals cannt be bserved fr the same amunt f time. Past slutins t this prblem have included prrating the criterin scre and deleting individuals lacking a prtin f the criterin infrmatin. In the present paper the prblem f criterin scre cmparability is viewed as a missing data prblem, and a slutin based upn linear regressin is prpsed. The regressin slutin was evaluated by cmparing it t the prrating and deletin methds in a mnte carl analysis. The ppulatin data were an actual data set cllected during a study f illness abard U.S. Navy ships, and a series f samples were created by randmly deleting individuals recrds and prtins f recrds. Results f the analyses suggested that the regressin methd is mre efficient than the alternative methds. Applicatin f the regressin methd and the results btained t ther research settings is discussed. Frequently, the dependent r criterin variable in an investigatin is cllected ver an extended perid f time. Fr example, in a study f jb perfrmance the number f prductin errrs each wrker made during a mnth might be cunted. Or, in an epidemilgical investigatin, the number f illness episdes fr an individual may be accumulated ver a year r even five years. In such situatins it is likely that individuals will enter and leave the study ppulatin during the curse f the investigatin, and when this ccurs, scres fr individuals are nt cmparable because they reflect different perids f expsure as well as the prpensity t make errrs r t becme ill. There are tw cmmn remedies taken t vercme this prblem with criterin scre cmparability. One is t cntrl fr the length f time peple were bserved by eliminating any individual nt present during the entire study frm the data analysis. The ther remedy is t divide each criterin cunt by the length f the bservatin perid, the result ften being expressed in terms f a rate (e.g., illness rate r errr rate). The purpse f the present paper is t shw that the prblem f criterin scre cmparability can be viewed as a missing data prblem in a multivariate analysis. This perspective was gained by cnsidering the ttal bservatin perid as a cntinuum that can be divided int several intervals. Viewed in this way, each persn s criterin data cnsist f the set f criterin tallies crrespnding t the time APPLIED PSYCHOLOGICAL MEASUREMENT Vl. 5, N.1. Winter 1981, pp Q Cpyright 1981 Applied Psychlgical Measurement Inc. 113

2 114 intervals. Taking this psitin is useful because it prvides a basis fr evaluating the tw slutins discussed abve; and frm the vantage pint btained, the drawbacks f bth prcedures becme apparent. On the ne hand, deleting all the data btained fr a persn because f ne r tw missing data pints can result in the waste f large amunts f infrmatin (Frane, 1976; Gleasn & Staelin, 1975) and, as argued by Galdi and Bnat (1977), that exclusin can intrduce bias int the criterin. On the ther hand, the practice f cmputing rates r prrating data appears flawed when viewed frm the missing data perspective. Functinally, the prrating prcedure treats data frm cmplete bservatins as a perfect predictr f the unbserved data. Althugh past and future bservatins may be related, rarely wuld a perfect relatinship be expected. In additin t prviding a basis fr evaluating the deletin and prrating prcedures, viewing the prblem f criterin cmparability as a missing data prblem sets the stage fr applying prcedures designed t handle the prblem f incmplete data in a multivariate analysis. Generally, the techniques that have been develped (Buck, 1960; Edgett, 1956; Frane, 1976; Glasser, 1964; Gleasn & Staelin, 1975; Rubin 1977; Timm, 1970) use the available infrmatin t cmpensate fr the incmplete data by taking advantage f the redundancy amng measures. In tw studies (Gleasn & Staelin, 1975; Timm, 1970) designed t cmpare the perfrmance f different techniques fr handling missing data, it was fund that regressin techniques, such as the ne prpsed by Buck (1960), were effective. It was nted, hwever, that the regressin methd in which a linear equatin is used t predict missing data pints frm the available data tended t be slw and cmputatinally difficult. Befre applying ne f these methds t the present prblems, it shuld be pinted ut that, fr the mst part, the existing techniques were designed t handle the prblem f missing data amng a set f predictr variables; and cautin shuld be used in applying them in a situatin where the criterin data are missing. Fr instance, methds t estimate hw an individual wuld have respnded t a questinnaire frm backgrund infrmatin available n a persn may be justifiable, but t estimate criterin scres frm the individual s backgrund data is a questinable prcedure. Methd Because the data set being cnsidered in this paper was generated by dividing the ttal bservatin perid int several intervals and cmputing a criterin tally within each interval, the resulting measures wuld be expected t crrelate, thereby making the regressin apprach attractive. Hwever, in rder t avid the cmputatinal difficulties that wuld be encuntered when the amunt f missing infrmatin varied frm persn t persn, a mdified methd was develped in which the regressin weights varied as a functin f the amunt f data available and the amunt missing. Thus, a single equatin can be used, regardless f the amunt f missing data. This mdified regressin prcedure was cmpared t the deletin and prrating prcedures that are ften used when the criterin is treated as a single value fr each persn rather than as a set f repeated measures. Each methd was used t adjust individual scres in a data set designed t simulate infrmatin btained frm individuals having varying degrees f expsure. The simulated data were generated by mdifying actual data cllected during a study f mrbidity amng U.S. Navy enlisted men. Starting with individuals wh had cmplete data (and therefre equal amunts f expsure) t define the initial ppulatin, a series f sample data sets were generated using a mnte carl prcedure t randmly delete prtins f the illness data. Thus, the sample sets simulated data btained frm naval persnnel wh had different amunts f expsure as a result f transfers t and frm the research setting, i.e., Navy ships. Using these data, the three methds fr adjusting criterin data were cmpared in terms f the degree that estimates btained with the mdified criterin reprduced results btained with the initial ppulatin data.

3 115 Ratinale fr Adjusting Criterin Scres by Means f Linear Regressin Cnsider a study in which the sample is cmpsed f tw grups: Grup 1 cnsists f individuals present fr the entire study and Grup 2 cnsists f individuals present fr nly the first half f the study. In this situatin the data frm Grup 1 culd be used t generate a regressin equatin fr predicting criterin scres btained during the secnd half f the study frm the value btained during the first half. This equatin can be expressed as fllws: where Y2, Yh and Ware the predicted scre fr the secnd half f the study, the actual scre frm the first half, and the regressin weight, respectively. The prblem, f curse, is that individuals are generally lst frm a study at different pints during the investigatin. Therefre, a regressin equatin tailred t the circumstances f a particular individual prbably wuld nt fit the next persn, and creating a separate equatin fr each persn wuld be impractical at best. T develp a mre practical methd fr predicting the criterin scre crrespnding t m2 bservatins frm the scre btained during ml bservatins, a methd was develped t estimate Yml, Ym2, and W-1-2 frm Yi, Y2, and W, respectively. Recnsidering the data fr Grup 1, the criterin scre measured during the first half f a study can be expressed as the separate criterin cunts made during each bservatin perid summed ver the k bservatin perids in the first half f the study, i.e., Similarly, the scre fr the secnd half f the study wuld be If it is assumed that the criterin variance frm ne bservatin perid t anther is cnstant and that the crrelatin f the criterin cunts frm ne time t the next is als cnstant, then it can be shwn that where Sx2 is the criterin variance during ne bservatin perid, and rxx is the crrelatin between the criterin scres btained n tw different ccasins. In additin, when the abve cnditins are assumed, the cvariance f Yi and Y2 is The reader may recgnize Equatin 7 as the Spearman-Brwn frmula fr the reliability f a lengthened test (cf. McNemar, eq , 1969 ; Lrd & Nvick, eq ,1968).

4 116 Slving Equatin 7 fr rxx gives Nw, the abve prcess will be reversed. Hwever, this time, instead f using Equatin 7, the frmula fr tw tests f lengths m, and m2 will be used (McNemar, eq , 1969) t allw fr the pssibility f different lengths f expsure amng individuals. This prcedure yields the fllwing equatin fr the crrelatin f the criterin cunts made during m, bservatins with thse made during m2 bservatins: T find ~V~ ~ where Equatins 4 and 9 can be cmbined t yield T btain estimates f Y,&dqu;1 the criterin mean per sample per bservatin is and Ym2 it will be assumed that there is n verall grup trend s that where Y is the mean criterin scre fr the entire study. Therefre, the mean criterin scres fr m, i and m2 bservatins are Finally, an estimate f the mean criterin scre assessed during m2 bservatins (Y,,,,) frm the scre determined frm m, bservatins (Y&dqu;,1) can be derived by substituting W m m2 fr W, ~~, fr k,, and Ym2 fr Y2 in Equatin 1 t yield Therefre, when sme individuals have fewer criterin assessments than thers, the deficiency can be estimated by first cmputing ryl Y2 and Y fr thse peple present during the entire study. Then, using Equatin 8, an estimate f r- can be cmputed which, when used in cmbinatin with Y in Equatin 15, yields Ym2-an estimate f the data nt bserved. Thus, the adjusted criterin scre is i.e., the bserved criterin scre plus the estimate fr the unbserved perid.

5 117 Data Data btained during a study f illness incidence abard three U.S. Navy amphibius ships were used in the mnte carl analysis. These data were gathered during an verseas deplyment which lasted apprximately 6 mnths fr each ship. Because f small differences in the length f deplyment, nly illness data btained during the first 160 days were used. In additin t the illness data, three ther variables were included. These cnsisted f jb demands (1 = lw physical demands, 2 = medium demands, and 3 = high physical demands), pay grade (pay level), and age. These variables were used t assess predictr-criterin crrelatins n the basis f previus studies (Pugh & Gundersn, 1979; Rahe, Gundersn, Pugh, Rubin, & Arthur, 1972), which fund them t be significantly related t illness. Frm a ttal f 1,241 individuals abard the three ships (fr at least ne day f the deplyment), thse wh had cmplete data n all f the abve variables were selected fr analysis. This resulted in a study ppulatin f 470 men. These data were then altered t simulate the effect f transferring crew members t r frm a ship. The methd f altering data fr an individual invlved tw steps. First, a day frm 2 t 159 was selected at randm; and secnd, a randm decisin was made as t whether the day selected was t be the day the persn transferred t r frm the ship. If the day selected was t be the day &dqu;reprted abard,&dqu; illness data prir t that day were remved; if the day selected was cnsidered t be the &dqu;detached&dqu; day, the data after that day were deleted. Each time an individual s data were altered, a prrated criterin scre (Yp) was created as fllws: and a regressin adjusted criterin scre was derived using Equatin 16. T evaluate trends related t the amunt f missing data, six different cnditins were assessed. In the first cnditin 10% f the sample had sme illness data remved, in the secnd 20% had data remved, in the third 30% had data remved, in the furth 50%, in the fifth 70%, and in the sixth 90% f the individuals had sme illness data remved. Fr each cnditin 30 samples were created by randmly selecting the designated percentage f individuals, creating the tw adjusted criterin scres, then returning thse individuals t the sample, and repeating the prcess. It is imprtant t nte that the abve percentage fr each cnditin refers t the prprtin f individuals with missing data. With respect t the ttal amunt f data, hwever, the percentages were abut ne-half f the abve values because nly a prtin f each individual s data was altered. Fr example, under the first cnditin 10% f the persns in the sample had data altered, but nly abut 5% f all illness data were remved, since a prtin f each persn s data remained intact. Analyses were designed t determine whether adjusting the illness data by prrating, regressin, r simply deleting subjects with incmplete data generated statistics mst like thse fund when cmplete data fr all 470 individuals were used. Criterin Characteristics Results The effect f adjusting the illness cunts n estimates f the criterin mean and standard deviatin are shwn in Table 1. Inspectin f the mean value f the 30 sample criterin mean scres fr each methd under each cnditin shws that all three methds yielded estimates near the ppulatin value. Hwever, the last three clumns in Table 1 shw that the regressin-derived values were mst frequently the clsest t the ppulatin value. Therefre, the regressin methds appear smewhat mre efficient than the ther methds.

6 118 c 0.H -H lj c0 U ~! 0ru m» t8 M v 4J cz 4-) m lil I 0 v # (D CU r-~ ~- x -H ru» > v ~-) 0 en c 0.r-4 ru -H ~ j ~» c l# ct 4-) OD ~) Cru M C Et S CS 4-) c ~ ~! ~. b0 CM ~1 r-~ t 4J t t V ~r-1 ~ 8~1!=! <D ~ (U ~ ~ c. H I C H O g.h 4-) c 0 n ~) t M C3a O N C3a Pn Có O > N C bd O r r-q ~~ cv5 D V P T U ~! ~ C3a U 0 N N L4 ~~

7 119 Cmparing the three methds with regard t estimates f the criterin standard deviatin reveals sme divergent results. Regardless f the missing data cnditin, the deletin methd prduced the best estimates f the ppulatin value, becming prgessively smaller as the amunt f missing data increased. Prrating prduced estimates farthest frm the ppulatin value. These estimates were cnsistently greater than the ppulatin value and increased at a relatively rapid rate as the amunt f missing data increased. Criterin scres generated by the regressin and prrating methds were further evaluated by cmputing the amunt f errr intrduced int the data. Errr was assessed by subtracting the adjusted criterin scre fr an individual frm the actual value, squaring the difference, summing the squared deviatin ver individuals, dividing the result by the number f individuals (N 470), and = then taking the square rt f the mean squared deviatin. The results f this prcedure are shwn in Table 2. As wuld be expected, the mean errr fr bth methds increased as the amunt f missing data increased. Hwever, the t values shwn in Table 2 indicated that under every cnditin the amunt f errr intrduced by prrating was significantly mre than the amunt due t the regressin methd. Table 2 Amunt f Criterin Errr Intrduced by Regressin and Prratinga Criterin errrl,is measured in terms f E fr each sample, individual, and y is the adjusted E = [ (y_y)2/n]2, y is the actual criterin scre f the criterin scre. bpercent f ppulatin given missing data. N = 470. Predictr-Critedn Relatinships The crrelatins amng the three predictr variables-jb demands, pay grade, and age-and actual illness cunts fr the study ppulatin are shwn in Table 3. Nte that past illness did predict future illness in this ppulatin (r =.359), that the crrelatins f jb demands and pay with illness were significant, but that age did nt crrelate t a significant degree with illness. T determine the effect that the varius methds f dealing with missing criterin data had n crrelatinal analyses, the crrelatins between the three predictr variables and the varius criterin scres were cmputed. The 30 values f each cefficient were cnverted t z scres; the mean z scre fr each cefficient was determined and cnverted back t a crrelatin cefficient. These data are shwn in Table 4. Several trends are ntably similar fr the three predictrs. First, the regressin and prrating methds cnsistently underestimated the ppulatin value, whereas the estimates btained

8 120 Table 3 Crrelatins amng Actual Illness Cunts _ and Three Predictr Measures thrugh deletin were abve and belw the ppulatin value, depending upn the level f missing data. Secnd, the mean predictr-criterin crrelatin estimates frm bth regressin and prrating grew smaller as the amunt f missing data increased, with prrating declining at a faster rate. Third, mean estimates generated by the regressin methd were cnsistently clser t the ppulatin value than the mean estimates prduced by the prrating methd. Finally, the regressin methd appeared mst efficient, prducing estimates clser t the ppulatin value than either deletin r prrating in 17 ut f 18 cases. The ne ccasin where regressin was nt clsest was when age was being crrelated with the criterin scres under the 20% missing data cnditin and then prrating was mst ften clsest. In rder t summarize the abve data and t illustrate the salient prperties f each methd, multiple crrelatins f the three predictrs with each criterin were cmputed. Means and standard deviatins fr these multiple crrelatins were cmputed fr each methd under each cnditin, and the results are depicted in Figure 1. The dtted lines indicate the critical value fr a multiple crrelatin at the.01 prbability level. It can be seen that as the amunt f missing data increased (1) the values generated by the deletin methd tended t vary abut the ppulatin mean until extreme levels f missing data were reached, (2) the values generated by the regressin and prrating methds shwed a steady decline with means fr prrating having the steeper gradient, and (3) the regressin values which had the smallest amunt f variatin remained abve the critical level (a =.01) mre ften than the values generated by either the deletin r prrating methds. Discussin and Cnclusins Viewing the criterin cmparability prblem as a missing data prblem appears t be useful because the results btained in the mnte carl analyses indicated that the prcedure designed t estimate missing infrmatin, the regressin prcedure, cmpared favrably with the deletin and prrating prcedures. In particular, with respect t predictr-criterin crrelatins, the regressin methd appeared t be mre efficient than either f the ther methds because the values btained with the regressin methd mst ften were the best estimates f the ppulatin cefficient. Therefre, in a situatin where data have been cllected fr crrelatinal analysis and a prtin f the criterin infrmatin was nt bserved fr sme individuals, the regressin methd may be a mre attractive manipulatin than that prvided by either prrating r deletin. Althugh the crrelatins

9 121 n H ~ n u ~ U# M» O M C H E #» i 0 I N U r-t ~ n CO 0 E-!. r-!» d) H U ~ f 4-J U H TJ Q)H 4~ ~4-) O M U Ez H ~M M c0 (D 4J WE c~ T3 II 0.0.!=! t C I) H Q) x t t t V..,.r.. ~! flj E 4-J I cz C flj h V h 0:>rn H 0-. :>- 0.0 c P ) c~j r0 0 Xr-l~ U).m p.. U) 0 Q) C3an < P c0 ~)- c 0 > Q) a V C u 0 (Z -H 04-J4-J % < E a II UC1 ~ <D 0 x P-4 P-4 x U

10 122 Figure 1 The Mean and Standard Deviatin f Sample Multiple Crrelatins Obtained After Missing Criterin Data Were Adjusted Using Each f Three Methds PERCENTAGE OF PEOPLE WITH INCOMPLETE DATA

11 123 btained tended t underestimate slightly the value that wuld be btained frm cmplete data, they wuld be expected t be clser than the cefficients generated by either f the ther methds. Further, by retaining individuals in the analysis, enugh pwer can be gained t mre than cmpensate fr the criterin bias. Hwever, the regressin estimates are biased; and if circumstances are such that it is mre imprtant fr the estimatr t be unbiased than efficient, then the deletin methd wuld be preferred. The abve cnclusins shuld be qualified, hwever. Because individuals were selected at randm t receive missing data, cautin shuld be used in extending the results t situatins where the lack f data ccurs in a systematic fashin. Mrever, many assumptins abut the hmgeneity f variances, cvariances, and means were made when deriving the regressin frmulas. T the degree that a data set fails t meet these assumptins, it can be expected that the perfrmance f the regressin methd will degenerate. Nevertheless, reservatins abut the perfrmance f the regressin methd in a &dqu;real&dqu; study may be relieved smewhat by nting that the riginal data set (i.e., ppulatin data) was gathered fr a study f illness amng Navy men. In this way many f the deviatins that actual data take frm idealized states were represented in the mnte carl analyses. Finally, the present interpretatins depend upn the assumptin that estimating what happened when a persn was present, but nt bserved, is equivalent t predicting what wuld have happened if the persn had been present when, in fact, he/she was nt. With due regard fr the abve pints, it is felt that the regressin methd can be a useful technique fr adjusting criterin data when sme individuals have less expsure time than thers. It shuld be pinted ut that the regressin technique requires a sufficient number f individuals with cmplete data in rder t estimate the degree t which past behavir is assciated with future behavir. In additin, individuals with incmplete data shuld have enugh actual infrmatin t use as a basis fr estimating the missing infrmatin. If it is fund that many individuals with missing data have a minimal amunt f criterin data (i.e., nt enugh data t prject frm), little wuld be gained by retaining them in the analysis. In such circumstances, the deletin methd may be preferred. Of curse, when there are excessive amunts f missing infrmatin-large numbers f individuals with large amunts f missing data pints-n methd f analysis wuld be advisable. Althugh illness data have been used t illustrate and evaluate the varius prcedures, researchers studying ther criteria may als find the regressin methd useful. Fr example, studies f wrker perfrmance in an industrial setting might emply the regressin methd fr analysis f data retained frm individuals wh transfer t a new cmpany. This may be particularly desirable if these individuals had participated in an extensive pretesting and/r training prgram prir t the cllectin f perfrmance data. Befre using the regressin methd, hwever, an investigatr may wish t inspect the split-half criterin crrelatin. As this crrelatin becmes unity, Equatin 16 reduces t Equatin 17, and prrating becmes equivalent t regressin. Therefre, if the split-half criterin crrelatin is fund t be very high, the available criterin data can simply be prrated. On the ther hand, if the split-half criterin crrelatin appraches zer, the regressin slpe becmes zer and the missing data are replaced by a prrated mean scre. Hwever, if it is fund that the split-half crrelatin is in fact near zer, then the criterin measures wuld have n reliability-a mst challenging bject fr study. Finally, in actual practice the researcher shuld cnsider trying all the methds and cmparing the results btained frm each ne. In additin, when evaluating the usefulness f the regressin prcedure fr a particular prblem, decisins regarding (1) the adequacy f the split-half criterin crrelatin, (2) the randmness f missing data, (3) the amunt f missing data per persn and per sample, and (4) the hmgeneity f means, standard deviatins, and crrelatins must be made in the cntext f the csts f varius types

12 124 f errrs. One must cnsider, fr example, if the amunt f bias intrduced by adjusting the data causes an imprtant and valid hypthesis t be rejected. Perhaps, instead f adjusting the criterin, mre data shuld be cllected. Als t be cnsidered is the cst f the wasted data if individuals are drpped frm an analysis. The prcedures, analyses, and results f the present paper were designed t aid the researcher in answering such questins. Because the circumstances f each study are different, the answers will tend t vary. References Buck, S. F. A methd f estimatin f missing values in multivariate data suitable fr use with an electrnic cmputer. Jurnal f the Ryal Statistical Sciety, Series B, 1960, 22, Edgett, G. L. Multiple regressin with missing bservatins amng the independent variables. Jurnal f the American Statistical Assciatin, 1956, 51, Frane, G. W. Sme simple prcedures fr handling missing data in multivariate analysis. Psychmetrika, 1976, 41, Galdi, J., & Bnat, R. R. Biasing effects f missing data n a family illness criterin. Psychlgical Reprts, 1977, Glasser, M. Linear regressin analysis with missing bservatins amng the independent variables. Jurnal f the American Statistical Assciatin, 1964, Gleasn, T. C., & Staelin, R. A prpsal fr handling missing data. Psychmetrika, 1975, Lrd, F. M., & Nvick, M. R. Statistical theries f mental test scres. Reading, MA: Addisn-Wesley, McNemar, Q. Psychlgical statistics (4th ed.). New Yrk: Jhn Wiley & Sns, Pugh, W. M., & Gundersn, E. K. E..Envirnmental factrs in the nset f illness abard Navy ships (Reprt N. 79-4). San Dieg, CA: Naval Health Research Center, Rahe, R. H., Gundersn, E. K. E., Pugh, W. M., Rubin, R. T., & Arthur, R. J. Illness predictin studies: Use f psychlgical and ccupatinal characteristics as predictrs. Archives f Envirnmental, Health 1972, 25, Rubin, D. B. Frmalizing subjective ntins abut the effect f nnrespndents in sample surveys. Jurnal f the American Statistical Assciatin, 1977, 72, Timm, N. H. The estimatin f variance-cvariance and crrelatins matrices frm incmplete data. Psychmetrika, 1970, 35, Acknwledgments This paper was Reprt Number 79-51, supprted by Naval Medical Research and Develpment Cmmand, Department f the Navy, under Research Wrk ~r~ Unit ~~MF5&52~.C2~-6 C/7. MF y/!c~~prm~t~ The views presented in this paper are thse f the authr. N endrsement by the Department f the Navy has been given r shuld be inferred. The authr acknwledges the assistance f Ms. Mary Paul, wh with enthusiasm and dedicatin, prvided the cmputer supprt required fr the data analyses reprted n in this paper and t Ms. Pat Plak fr her preparatin f the manuscript. Authr s Address Send requests fr reprints r further infrmatin t William M. Pugh, Naval Health Research Center, San Dieg, CA