The Stata Journal. Editors H. Joseph Newton Department of Statistics Texas A&M University College Station, Texas

Transcription

1 The Stata Journal Editors H. Joseph Newton Department of Statistics Texas A&M University College Station, Texas Nicholas J. Cox Department of Geography Durham University Durham, UK Associate Editors Christopher F. Baum, Boston College Nathaniel Beck, New York University Rino Bellocco, Karolinska Institutet, Sweden, and University of Milano-Bicocca, Italy Maarten L. Buis, University of Konstanz, Germany A. Colin Cameron, University of California Davis Mario A. Cleves, University of Arkansas for Medical Sciences Michael Crowther, University of Leicester, UK William D. Dupont, Vanderbilt University Philip Ender, University of California Los Angeles James Hardin, University of South Carolina Ben Jann, University of Bern, Switzerland Stephen Jenkins, London School of Economics and Political Science Ulrich Kohler, University of Potsdam, Germany Frauke Kreuter, Univ. of Maryland College Park Peter A. Lachenbruch, Oregon State University Stanley Lemeshow, Ohio State University J. Scott Long, Indiana University Roger Newson, Imperial College, London Austin Nichols, Abt Associates, Washington, DC Marcello Pagano, Harvard School of Public Health Sophia Rabe-Hesketh, Univ. of California Berkeley J. Patrick Royston, MRC CTU at UCL, London, UK Mark E. Schaffer, Heriot-Watt Univ., Edinburgh Philippe Van Kerm, LISER, Luxembourg Vincenzo Verardi, Université Libre de Bruxelles, Belgium Ian White, MRC CTU at UCL, London, UK Richard A. Williams, University of Notre Dame Jeffrey Wooldridge, Michigan State University Stata Press Editorial Manager Lisa Gilmore Stata Press Copy Editors Adam Crawley, David Culwell, and Deirdre Skaggs The Stata Journal publishes reviewed papers together with shorter notes or comments, regular columns, book reviews, and other material of interest to Stata users. Examples of the types of papers include 1) expository papers that link the use of Stata commands or programs to associated principles, such as those that will serve as tutorials for users first encountering a new field of statistics or a major new technique; 2) papers that go beyond the Stata manual in explaining key features or uses of Stata that are of interest to intermediate or advanced users of Stata; 3) papers that discuss new commands or Stata programs of interest either to a wide spectrum of users (e.g., in data management or graphics) or to some large segment of Stata users (e.g., in survey statistics, survival analysis, panel analysis, or limited dependent variable modeling); 4) papers analyzing the statistical properties of new or existing estimators and tests in Stata; 5) papers that could be of interest or usefulness to researchers, especially in fields that are of practical importance but are not often included in texts or other journals, such as the use of Stata in managing datasets, especially large datasets, with advice from hard-won experience; and 6) papers of interest to those who teach, including Stata with topics such as extended examples of techniques and interpretation of results, simulations of statistical concepts, and overviews of subject areas. The Stata Journal is indexed and abstracted by CompuMath Citation Index, Current Contents/Social and Behavioral Sciences, RePEc: Research Papers in Economics, Science Citation Index Expanded (also known as SciSearch), Scopus, and Social Sciences Citation Index. For more information on the Stata Journal, including information for authors, see the webpage

2 Subscriptions are available from StataCorp, 4905 Lakeway Drive, College Station, Texas 77845, telephone or , fax , or online at Subscription rates listed below include both a printed and an electronic copy unless otherwise mentioned. U.S. and Canada Elsewhere Printed & electronic Printed & electronic 1-year subscription $124 1-year subscription $154 2-year subscription $224 2-year subscription $284 3-year subscription $310 3-year subscription $400 1-year student subscription $ 89 1-year student subscription $119 1-year institutional subscription $375 1-year institutional subscription $405 2-year institutional subscription $679 2-year institutional subscription $739 3-year institutional subscription $935 3-year institutional subscription $1,025 Electronic only Electronic only 1-year subscription $ 89 1-year subscription $ 89 2-year subscription $162 2-year subscription $162 3-year subscription $229 3-year subscription $229 1-year student subscription $ 62 1-year student subscription $ 62 Back issues of the Stata Journal may be ordered online at Individual articles three or more years old may be accessed online without charge. More recent articles may be ordered online. The Stata Journal is published quarterly by the Stata Press, College Station, Texas, USA. Address changes should be sent to the Stata Journal, StataCorp, 4905 Lakeway Drive, College Station, TX 77845, USA, or ed to sj@stata.com. Copyright c 2018 by StataCorp LLC Copyright Statement: The Stata Journal and the contents of the supporting files (programs, datasets, and help files) are copyright c by StataCorp LLC. The contents of the supporting files (programs, datasets, and help files) may be copied or reproduced by any means whatsoever, in whole or in part, as long as any copy or reproduction includes attribution to both (1) the author and (2) the Stata Journal. The articles appearing in the Stata Journal may be copied or reproduced as printed copies, in whole or in part, as long as any copy or reproduction includes attribution to both (1) the author and (2) the Stata Journal. Written permission must be obtained from StataCorp if you wish to make electronic copies of the insertions. This precludes placing electronic copies of the Stata Journal, in whole or in part, on publicly accessible websites, fileservers, or other locations where the copy may be accessed by anyone other than the subscriber. Users of any of the software, ideas, data, or other materials published in the Stata Journal or the supporting files understand that such use is made without warranty of any kind, by either the Stata Journal, the author, or StataCorp. In particular, there is no warranty of fitness of purpose or merchantability, nor for special, incidental, or consequential damages such as loss of profits. The purpose of the Stata Journal is to promote free communication among Stata users. The Stata Journal (ISSN X) is a publication of Stata Press. Stata,, Stata Press, Mata,, and NetCourse are registered trademarks of StataCorp LLC.

3 The Stata Journal (2018) 18, Number 1, pp validscale: A command to validate measurement scales Bastien Perrot University of Nantes, University of Tours, INSERM SPHERE U1246 methods in Patient-centered outcomes and HEalth ResEarch Nantes, France bastien.perrot@univ-nantes.fr Emmanuelle Bataille University of Nantes, University of Tours, INSERM SPHERE U1246 methods in Patient-centered outcomes and HEalth ResEarch Nantes, France emmanuelle.anthoine@chu-nantes.fr Jean-Benoit Hardouin University of Nantes, University of Tours, INSERM SPHERE U1246 methods in Patient-centered outcomes and HEalth ResEarch Nantes, France jean-benoit.hardouin@univ-nantes.fr Abstract. Subjective measurement scales are used to measure nonobservable respondent characteristics in several fields such as clinical research, educational sciences, or psychology. To be useful, the scores resulting from the questionnaire must be validated; that is, they must provide the psychometric properties validity, reliability, and sensitivity. In this article, we present the validscale command, which carries out the required statistical analyses to validate a subjective measurement scale. We have also developed a dialog box, and validscale will soon be implemented online with Numerics by Stata. Keywords: st0512, validscale, subjective measurements scales, score, psychometrics, validity, reliability 1 Introduction Several domains require measurement scales to measure concepts, but no technical instrument in these domains allows one to obtain a measure. This is the case, for example, in education sciences, when we want to measure the ability of students; in social and human sciences, when we want to measure characteristics of individuals like personality traits or behaviors; and in health, when we want to measure perceived health, quality of life, pain, or mental disorders. Generally, these concepts are measured using a questionnaire composed of items. Questionnaires can be unidimensional (they measure only one concept) or multidimenc 2018 StataCorp LLC st0512

4 30 validscale: A command to validate measurement scales sional (they measure several concepts), so they can lead to one or more measures able to measure the concept or concepts of interest. Historically, the model of measurement is defined in the framework of classical test theory (CTT). If other theories of measurement coexist today like item response theory (IRT) or Rasch measurement theory (RMT), CTT continues to be largely used in several domains (psychology, health, etc.) to validate scales. The success of CTT can be explained by the simplicity of obtaining the measures, because, in this framework, the measure of each concept can be obtained using scores computed as a combination (for example, sum or mean) of responses to the items. To construct a valid and reliable questionnaire, one must provide its psychometric properties: Validity is the degree to which an instrument measures the concept or concepts of interest accurately. Reliability is the degree to which an instrument measures the concept or concepts of interest consistently. Validity and reliability are assessed by checking their respective facets. Validity is composed of content validity, construct validity, and criterion validity. Reliability is composed of internal consistency, test-retest reliability, and scalability. Content validity should be evaluated by experts. In this step, the experts define the number of concepts measured by the scale, the definition of each of these concepts, and the assumed relationships between the concepts and the questionnaire items. This facet of validity is based on qualitative methods, but all the other facets can be assessed using statistical analyses to statistically confirm the experts opinions. However, there is currently no statistical software package to perform all of these analyses in a userfriendly way. The objective of the validscale command is to perform the necessary analyses to validate a measurement scale in the framework of CTT. 2 Psychometric properties assessed by validscale The concepts described below are based on the terminology used by Fayers and Machin (2007). Construct validity: Convergent and divergent validity test whether the items of the questionnaire measure the constructs they are designed to measure. Convergent validity tests whether an item is sufficiently correlated to the score computed with items of the same dimension. Divergent validity tests whether an item is poorly correlated to the scores computed in the other dimensions (for multidimensional scales). Structural validity tests the dimensional structure of the questionnaire (the number of dimensions).

5 B. Perrot, E. Bataille, and J.-B. Hardouin 31 Criterion validity: Reliability: Concurrent validity is assessed by comparing the score or scores of the questionnaire with previously validated instruments or a gold standard measuring approximately the same concept or concepts. Known-groups validity tests whether the scores differ according to known groups in a predictable way. Internal consistency refers to how unidimensional the dimension is and whether it is composed of enough items. Reproducibility refers to how well the items and scores are stable when the individual s state is stable. 3 The validscale command 3.1 Description The validscale command computes elements to provide structural validity, convergent and divergent validity, concurrent validity, reproducibility, known-groups validity, internal consistency, and scalability. The command is intended to be used with questionnaires that comprise dichotomous (that is, two response categories) or polytomous items (that is, more than two response categories, for example, the Likert-type scale). The user defines the items used to compute the scores. The second parameter required (partition()) is the repartition of the items in the different dimensions of the questionnaire. 3.2 Syntax validscale varlist, partition(numlist) [ scorename(string) scores(varlist) categories(numlist) impute(method) noround compscore(method) descitems graphs cfa cfamethod(method) cfasb cfastand cfanocovdim cfacov(string) cfarmsea(#) cfacfi(#) cfaor convdiv tconvdiv(#) convdivboxplots alpha(#) delta(#) h(#) hjmin(#) repet(varlist) kappa ickappa(#) scores2(#) kgv(varlist) kgvboxplots kgvgroupboxplots conc(varlist) tconc(#) ] varlist contains the variables (items) used to compute the scores. The first items of varlist compose the first dimension, the following items define the second dimension, and so on.

6 32 validscale: A command to validate measurement scales validscale requires the commands delta (Hardouin 2007a), detect2 (included with the software for this article), imputeitems (Hardouin 2007b), mi twoway (Hamel 2014), kapci (Reichenheim 2004), loevh (Hardouin 2004), and lstrfun (Blanchette 2010). 3.3 Options partition(numlist) defines the number of items in each dimension. The number of elements in this list indicates the number of dimensions. partition() is required. scorename(string) defines the names of the dimensions. By default, the dimensions are named Dim1, Dim2,... unless scores(varlist) is used. scores(varlist) selects scores already computed in the dataset. varlist must contain as many variables as there are dimensions in the questionnaire. scores(varlist) and scorename(string) cannot be used together. This option is useful when the scores result from combinations of items that are more complex than the combinations available in the compscore() option (unweighted sum, unweighted mean, or standardization between 0 and 100). In that case, the scores must be computed prior to using validscale. categories(numlist) specifies the minimum and maximum possible values for item response categories. If all the items have the same response categories, the user may specify these two values in numlist. If the item response categories differ from a dimension to another, the user must define the minimum and maximum values of items responses for each dimension. So the number of elements in numlist must be equal to the number of dimensions times 2. Eventually, the user may specify the minimum and maximum response categories for each item. In this case, the number of elements in numlist must be equal to 2 times the number of items. By default, the observed minimum and maximum values are assumed to be the minimum and maximum for each item. impute(method) imputes missing-item responses with person-mean substitution (pms) or the two-way imputation method applied in each dimension (mi). Both methods are applied in each dimension. When pms is specified, missing data are imputed only if the number of missing values in the dimension is less than half the number of items in the dimension. By default, imputed values are rounded to the nearest whole number, but with the noround option, imputed values are not rounded. If impute() is absent, then noround is ignored. noround specifies that imputed values are not rounded. By default, imputed values are rounded to the nearest whole number. If impute() is absent, then noround is ignored.

7 B. Perrot, E. Bataille, and J.-B. Hardouin 33 compscore(method) defines the method used to compute the scores. method may be mean (default), sum, or stand (set scores from 0 to 100). compscore() is ignored if scores() is used. descitems displays a descriptive analysis of the items. The option displays missing data rate per item and distribution of item responses. It also computes for each item the Cronbach s alphas obtained by omitting each item in each dimension. Moreover, the option computes Loevinger s H j coefficients and the number of nonsignificant H jk. See Hardouin, Bonnaud-Antignac, and Sébille (2011) for details about Loevinger s coefficients. graphs displays graphs for items and dimensions descriptive analyses. It provides histograms of scores, a biplot of the scores, and a graph showing the correlations between the items. cfa performs a confirmatory factor analysis (CFA) using the sem command. It displays estimations of parameters and several goodness-of-fit indices. cfamethod(method) specifies the method to estimate the parameters. method may be ml (maximum likelihood), mlmv (ml with missing values), or adf (asymptotic distribution free). cfasb produces Satorra Bentler-adjusted goodness-of-fit indices by using the option vce(sbentler) from sem. cfastand displays standardized coefficients for the CFA. cfanocovdim asserts that the latent variables are not correlated. cfacov(string) adds covariances between measurement errors. cfacov(item1*item2) estimates the covariance between the errors of item1 and item2. To specify more than one covariance, use cfacov(item1*item2 item3*item4). cfarmsea(#) automatically adds the covariances between measurement errors found with the estat mindices command until the root mean square error of approximation (RMSEA) of the model is less than #. More precisely, the basic model (without covariances between measurement errors) is fit; then we add the covariance corresponding to the greatest modification index. The model is then fit again with this extra parameter, and so on. The option adds only the covariances between measurement errors within a dimension and can be combined with cfacov(). The specified value # may not be reached if all possible within-dimension measurement errors have already been added. cfacfi(#) automatically adds the covariances between measurement errors found with the estat mindices command until the comparative fit index (CFI) of the model is greater than #. More precisely, the basic model (without covariances between measurement errors) is fit; then we add the covariance corresponding to the greatest modification index. The model is then fit again with this extra parameter, and so on. The option adds only the covariances between measurement errors within a dimension and can be combined with cfacov(). The specified value # may not

8 34 validscale: A command to validate measurement scales be reached if all possible within-dimension measurement errors have already been added. cfaor is useful when both cfarmsea() and cfacfi() are used. By default, covariances between measurement errors are added, and the model is fit until both RMSEA and CFI criteria are met. If cfaor is used, the estimations stop when one of the two criteria is met. convdiv assesses convergent and divergent validities. The option displays the matrix of correlations between items and rest scores (that is, the scores computed after omitting the item being examined). If scores(varlist) is used, then the correlation coefficients are computed between items and scores of varlist. tconvdiv(#) defines a threshold for highlighting some values. # is a real number between 0 and 1. The default is tconvdiv(0.4). Correlations between items and their own score are displayed in red if it is less than #. Moreover, if an item has a smaller correlation coefficient with the score of its dimension than those computed with other scores, the coefficient is displayed in red. convdivboxplots displays boxplots for assessing convergent and divergent validities. The boxes represent the correlation coefficients between the items of a given dimension and all scores. Thus, the box of correlations coefficients between items of a given dimension and the corresponding score must be higher than other boxes. There are as many boxplots as dimensions. alpha(#) defines a threshold for Cronbach s alpha. # is a real number between 0 and 1. The default is alpha(0.7). Alpha coefficients less than # are displayed in red. delta(#) defines a threshold for Ferguson s delta coefficient (see delta). Delta coefficients are computed only if compscore(sum) is used and scores() is not used. # is a real number between 0 and 1. The default is delta(0.9). Delta coefficients less than # are displayed in red. h(#) defines a threshold for Loevinger s H coefficients (see loevh). # is a real number between 0 and 1. The default is h(0.3). Loevinger s coefficients less than # are displayed in red. hjmin(#) defines a threshold for Loevinger s H j coefficients. The displayed value is the minimal H j coefficient for an item in the dimension (see loevh). # is a real number between 0 and 1. The default is hjmin(0.3). If the minimal Loevinger s H j coefficient is less than #, then it is displayed in red, and the corresponding item is displayed. repet(varlist) assesses reproducibility of scores by defining in varlist the variables corresponding to responses at time 2 (in the same order than for time 1). Scores are computed according to the partition() option. Intraclass correlation coefficients (ICC) for the scores and their 95% confidence interval are computed with Stata s icc command. kappa computes the kappa statistic for items with Stata s kap command.

9 B. Perrot, E. Bataille, and J.-B. Hardouin 35 ickappa(#) computes confidence intervals for kappa statistics using kapci. # is the number of replications for the bootstrap used to estimate confidence intervals if items are polytomous. If they are dichotomous, an analytical method is used. See kapci for more details about calculation of confidence intervals for kappa s coefficients. If the kappa option is absent, then the ickappa() option is ignored. scores2(varlist) allows selecting scores at time 2 from the dataset. kgv(varlist) assesses known-groups validity according to the grouping variables defined in varlist. The option performs an analysis of variance (ANOVA) that compares the scores between groups of individuals constructed with variables in varlist. A p-value based on a Kruskal Wallis test is also given. kgvboxplots draws boxplots of the scores split into groups of individuals. kgvgroupboxplots groups all boxplots into one graph. If kgvboxplots is absent, then kgvgroupboxplots is ignored. conc(varlist) assesses concurrent validity with variables precised in varlist. These variables are scores from one or several other scales in the dataset. tconc(#) defines a threshold for the correlation coefficients between the computed scores and those from other scales defined in varlist. Correlation coefficients greater than # in absolute value (the default is tconc(0.4)) are displayed in bold. 4 Output 4.1 Data used in the examples ThedatausedintheoutputareasampleofresponsestotheFrenchversionoftheImpact Of Cancer version 2 questionnaire (Crespi et al. 2008). This questionnaire measures the impact of cancer on survivors lives. It consists especially of four positive-impacts subscales named altruism and empathy (AE), health awareness (HA), meaning of cancer (MOC), and positive self-evaluation (PSE) and four negative-impacts subscales named appearance concerns(ac), body change concerns(bcc), life interferences(li), and worry (W). There is also one subscale for employment and relationship concerns(not used here). The questionnaire is composed of 37 items (ioc1 ioc37) scored on a 5-point scale from 1 ( strongly disagree ) to 5 ( strongly agree ). They are grouped into eight dimensions, and the scores result from the sum of the responses of the corresponding items. The questionnaire was answered by 371 patients.

10 36 validscale: A command to validate measurement scales 4.2 Examples Minimal output. use data_ioc. validscale ioc1-ioc37, partition( ) > scorename(ha PSE W BCC AC AE LI MOC) compscore(sum) Items used to compute the scores HA : ioc1 ioc2 ioc3 ioc4 PSE : ioc5 ioc6 ioc7 ioc8 W : ioc9 ioc10 ioc11 ioc12 ioc13 ioc14 ioc15 BCC : ioc16 ioc17 ioc18 AC : ioc19 ioc20 ioc21 AE : ioc22 ioc23 ioc24 ioc25 LI : ioc26 ioc27 ioc28 ioc29 ioc30 ioc31 ioc32 MOC : ioc33 ioc34 ioc35 ioc36 ioc37 Number of observations: 371 Reliability n alpha delta H Hj_min HA (item ioc1) PSE W BCC AC AE LI (item ioc26) MOC Thisistheminimaloutputproducedbyvalidscale. Inthisexample, weseethatall of Cronbach s alphas are acceptable according to the threshold specified with alpha(#) (0.7 by default), except for scales HA and PSE (0.67 and 0.69, respectively). Loevinger s H coefficients for the 8 scales are 0.3, which indicates good scalability. However, the Hioc1 HA coefficient is < 0.3 (0.25), which means that item ioc1 might not be consistent with scale HA. The Hioc26 LI coefficient is also < 0.3 for item ioc26. For other subscales, no Hitem subscale is < 0.3. By default, Cronbach s alphas < 0.7 and Loevinger s H coefficients < 0.3 are displayed in red. These thresholds can be changed with the options alpha(#) and h(#), respectively.

11 B. Perrot, E. Bataille, and J.-B. Hardouin 37 The descitems option. validscale ioc1-ioc37, partition( ) > scorename(ha PSE W BCC AC AE LI MOC) compscore(sum) descitems Items used to compute the scores (output omitted) Description of items Missing N Response categories Alpha Hj # of - item NS Hjk ioc1 3.77% % 12.61% 24.65% 33.05% 19.61% ioc2 1.08% % 8.72% 10.90% 39.78% 37.60% ioc3 2.16% % 5.79% 11.02% 44.63% 36.09% ioc4 2.43% % 8.56% 18.51% 43.09% 26.52% ioc5 2.96% % 15.28% 22.78% 28.06% 24.44% ioc6 2.96% % 15.28% 24.17% 33.61% 16.67% ioc7 2.43% % 8.01% 22.10% 42.27% 22.65% ioc8 2.16% % 19.83% 33.06% 20.66% 11.85% ioc9 2.43% % 22.65% 14.64% 28.18% 19.06% ioc % % 27.58% 20.89% 12.26% 5.85% ioc % % 9.62% 13.74% 42.03% 29.12% ioc % % 18.94% 19.22% 37.05% 16.16% ioc % % 24.79% 18.11% 30.36% 13.09% ioc % % 26.30% 14.25% 28.49% 18.90% ioc % % 19.62% 18.26% 39.78% 15.53% ioc % % 21.86% 12.57% 30.33% 25.14% ioc % % 22.19% 13.97% 32.33% 21.64% ioc % % 34.88% 12.53% 21.80% 9.81% ioc % % 27.58% 19.50% 24.79% 13.37% ioc % % 29.36% 19.67% 14.68% 9.14% ioc % % 19.13% 16.39% 26.50% 17.76% ioc % % 12.26% 20.71% 45.50% 15.26% ioc % % 2.45% 5.71% 50.54% 39.67% ioc % % 8.52% 25.00% 42.03% 21.70% ioc % % 16.57% 33.98% 29.28% 14.92% ioc % % 38.84% 16.53% 9.64% 6.61% ioc % % 37.40% 11.08% 15.79% 7.20% ioc % % 35.71% 10.99% 7.97% 5.22% ioc % % 34.52% 11.23% 13.42% 7.40% ioc % % 31.15% 12.57% 19.13% 9.56% ioc % % 35.79% 10.93% 12.02% 4.92% ioc % % 17.22% 17.78% 33.89% 16.94% ioc % % 24.38% 23.27% 31.30% 12.47% ioc % % 26.39% 30.83% 24.72% 11.11% ioc % % 26.40% 28.65% 22.47% 8.99% ioc % % 32.68% 24.86% 15.64% 5.87% ioc % % 31.20% 22.84% 19.22% 8.36% (output omitted)

12 38 validscale: A command to validate measurement scales The column Alpha-item shows the Cronbach s alpha coefficients of the scales when the considering item is removed. For example, removing ioc1 from the first subscale (HA) would increase the alpha coefficient for this subscale from 0.67 (see previous table) to 0.71, whereas removing ioc2 would decrease the alpha coefficient from 0.67 to The column Hj corresponds to the Loevinger s coefficients of item-scale consistency. We note that the first Hj coefficient of the column (0.25) corresponds to the value of Hioc1 HA displayed in the previous table. The last column indicates the number of H jk coefficients (Loevinger s coefficients between the considering item and each of the other items of the dimension) that are not significantly different from zero. If one H jk coefficient would appear not to be significantly different from zero, it would indicate a problem in item-scale consistency. The graph option. validscale ioc1-ioc37, partition( ) > scorename(ha PSE W BCC AC AE LI MOC) graph (output omitted) Percent HA Percent PSE Percent W Percent BCC Percent AC Percent AE Percent LI Percent MOC Figure 1. Histograms of scores

13 B. Perrot, E. Bataille, and J.-B. Hardouin BCC LI AC W HA AE MOC PSE Figure 2. Correlations between scores ioc8 ioc25 ioc6ioc24 ioc36 ioc7 ioc35 ioc34 ioc37 ioc1 ioc3 ioc5 ioc22 ioc2 ioc23 ioc33 ioc4 ioc31 ioc28 ioc18 ioc20 ioc10 ioc19 ioc29 ioc30 ioc17 ioc32ioc16ioc27 ioc9 ioc13 ioc21 ioc11 ioc15 ioc12 ioc14 ioc HA PSE W BCC AC AE LI MOC Figure 3. Correlations between items The graph option produces histograms of scores (figure 1), a biplot 1 of the scores (figure 2), and a biplot of the items (figure 3). The histogram allows one to examine the distribution of the scores, particularly to check normality and to identify potential floor or ceiling effects (that is, high proportion 1. Only variables are represented in the biplots.

14 40 validscale: A command to validate measurement scales of subjects scoring minimum score or maximum score, respectively). Figures 2 and 3 are produced with the biplot command. In figure 2, the cosine of the angle between arrows approximates the correlation between the scores. This allows one to graphically assess the correlation between the scores. For example, appearance concerns (AC), body change concerns (BCC), and life interferences (LI) seem very correlated with each other but uncorrelated with the meaning of cancer (MOC). In figure 3, the cosine of the angle between arrows approximates the correlation between the items. Moreover, in figure 3, items within the same dimension are represented in the same color. Thus, we expect items of the same color to be close to each other. In this example, we note that item ioc5, although theoretically grouped with items ioc6, ioc7, and ioc8, seems more correlated with ioc2 and ioc3. The cfa option. validscale ioc1-ioc37, partition( ) > scorename(ha PSE W BCC AC AE LI MOC) cfa cfacov(ioc1*ioc3) (output omitted) Confirmatory factor analysis Warning: some items have less than 7 response categories. If multivariate normality assumption does not hold, maximum likelihood estimation might not be appropriate. Consider using cfasb in order to apply Satorra-Bentler adjustment or using cfamethod(adf). Covariances between errors added: e.ioc1*e.ioc3 Number of used individuals: 292 Item Dimension Factor Standard Intercept Standard Error Variance of loading error error variance dimension ioc1 HA ioc2 HA ioc3 HA ioc4 HA ioc5 PSE ioc6 PSE ioc7 PSE ioc8 PSE ioc9 W ioc10 W ioc11 W ioc12 W ioc13 W ioc14 W ioc15 W ioc16 BCC ioc17 BCC ioc18 BCC ioc19 AC ioc20 AC ioc21 AC

15 B. Perrot, E. Bataille, and J.-B. Hardouin 41 ioc22 AE ioc23 AE ioc24 AE ioc25 AE ioc26 LI ioc27 LI ioc28 LI ioc29 LI ioc30 LI ioc31 LI ioc32 LI ioc33 MOC ioc34 MOC ioc35 MOC ioc36 MOC ioc37 MOC Covariances between dimensions: HA PSE W BCC AC AE LI MOC HA PSE W BCC AC AE LI MOC Goodness of fit: chi2 df chi2/df RMSEA [90% CI] SRMR NFI [0.049 ; 0.059] (p-value = 0.000) RNI CFI IFI MCI The cfa option uses the official Stata command sem to perform a CFA. The option cfacov(ioc1*ioc3) allows one to consider a covariance between the errors of ioc1 and ioc3. Goodness-of-fit indices computed by Stata or from Gadelrab (2010) are given below the tables of estimates. In this example, the RMSEA is < 0.06, which indicates acceptable fit. However, the CFI is only To improve the CFI, we could specify direct effects between some items by using cfacov() or cfacfi() (see Options for details). A warning is displayed because items have only five response categories. In that case, we could use cfasb to apply Satorra Bentler adjustment after maximum likelihood estimation.

16 42 validscale: A command to validate measurement scales The convdiv option. validscale ioc1-ioc37, partition( ) > scorename(ha PSE W BCC AC AE LI MOC) convdiv convdivboxplot Items used to compute the scores (output omitted) Correlation matrix HA PSE W BCC AC AE LI MOC ioc ioc ioc ioc ioc ioc ioc ioc ioc ioc ioc ioc ioc ioc ioc ioc ioc ioc ioc ioc ioc ioc ioc ioc ioc ioc ioc ioc ioc ioc ioc ioc ioc ioc ioc ioc ioc Convergent validity: 33/37 items (89.2%) have a correlation coefficient with the score of their own dimension greater than 0.400

17 B. Perrot, E. Bataille, and J.-B. Hardouin 43 Divergent validity: 33/37 items (89.2%) have a correlation coefficient with the score of their own dimension greater than those computed with other scores. Correlations between items of HA and scores HA PSE W BCC AC AE LI MOC Figure 4. Correlations between items of HA and scores Correlations between items of PSE and scores HA PSE W BCC AC AE LI MOC Figure 5. Correlations between items of PSE and scores convdiv assesses convergent and divergent (discriminant) validities through the examination of a correlation matrix. The elements of this matrix are the correlation coefficients between items and rest scores.

18 44 validscale: A command to validate measurement scales Elements on the diagonal (correlations between an item and the rest score of its dimension) are displayed in bold font in Stata. On the diagonal, values less than tconvdiv(#) (0.4 by default) are displayed in red, indicating lack of convergent validity. For each row, off-diagonal values greater than values on the diagonal are displayed in red, indicating lack of divergent validity. In this example, the value of the correlation coefficient between ioc1 and HA would be displayed in red because < 0.4. The value of the correlation coefficient between ioc1 and W and between ioc1 and BCC would also be displayed in red because > and > 0.266, respectively. The convdivboxplot option produces boxplots representing the values of the correlation matrix. In this example, because there are eight subscales, eight graphs composed of eight boxes are generated. Figures 4 and 5 correspond to two of these graphs (the six remaining graphs are not shown). In figure 4, we expect the first boxplot to be the highest in the graph because it represents the correlations between HA and its own items. We also expect these correlations to be 4. In figure 5, we expect the second boxplot to be higher than others because it represents the correlations between PSE and its own items.

19 B. Perrot, E. Bataille, and J.-B. Hardouin 45 The repet() option. set seed foreach v of varlist ioc1-ioc37 { 2. generate `v _2 = round(rnormal(`v,0.5)) 3. } (output omitted). validscale ioc1-ioc37, partition( ) > scorename(ha PSE W BCC AC AE LI MOC) repet(ioc1_2-ioc37_2) kappa ickappa(500) (output omitted) Reproducibility Dimension n Item Kappa 95% CI for Kappa ICC 95% CI for ICC (bootstrapped) HA 368 ioc [ 0.52 ; 0.66] 0.96 [ 0.96 ; 0.97] ioc [ 0.46 ; 0.59] ioc [ 0.45 ; 0.58] ioc [ 0.50 ; 0.64] PSE 367 ioc [ 0.46 ; 0.59] 0.97 [ 0.97 ; 0.98] ioc [ 0.54 ; 0.66] ioc [ 0.51 ; 0.64] ioc [ 0.57 ; 0.69] W 366 ioc [ 0.45 ; 0.58] 0.99 [ 0.98 ; 0.99] ioc [ 0.51 ; 0.63] ioc [ 0.46 ; 0.59] ioc [ 0.53 ; 0.65] ioc [ 0.55 ; 0.67] ioc [ 0.51 ; 0.62] ioc [ 0.49 ; 0.61] BCC 369 ioc [ 0.50 ; 0.64] 0.98 [ 0.97 ; 0.98] ioc [ 0.49 ; 0.61] ioc [ 0.54 ; 0.66] AC 366 ioc [ 0.55 ; 0.67] 0.98 [ 0.97 ; 0.98] ioc [ 0.48 ; 0.61] ioc [ 0.49 ; 0.61] AE 368 ioc [ 0.54 ; 0.66] 0.96 [ 0.96 ; 0.97] ioc [ 0.46 ; 0.59] ioc [ 0.52 ; 0.65] ioc [ 0.51 ; 0.64] LI 366 ioc [ 0.51 ; 0.64] 0.98 [ 0.98 ; 0.99] ioc [ 0.52 ; 0.64] ioc [ 0.50 ; 0.63] ioc [ 0.55 ; 0.67] ioc [ 0.54 ; 0.66] ioc [ 0.48 ; 0.61] ioc [ 0.51 ; 0.63] MOC 361 ioc [ 0.57 ; 0.69] 0.98 [ 0.97 ; 0.98] ioc [ 0.52 ; 0.64] ioc [ 0.51 ; 0.63] ioc [ 0.56 ; 0.69] ioc [ 0.53 ; 0.65]

20 46 validscale: A command to validate measurement scales The repet() option computes ICC and its corresponding confidence intervals with the Stata command icc to assess the reproducibility of scores. varlist is the list of items measured at time 2. The kappa option computes kappa statistics with the Stata command kap to assess the reproducibility of items. Computation of confidence intervals for kappa coefficients are possible with the ickappa() option based on the community-contributed command kapci. If the scores() option is used, one would probably want to also use the scores2() option to indicate the variables corresponding to the scores computed at time 2. In that case, ICCs are based on these scores rather than on a combination of items, as defined by compscore(). Because there was only one time of measurement for this questionnaire, responses to items ioc1-ioc37 at the second time of measurement have been simulated for the sake of this example. Kappas > 0.6 would indicate good reproducibility of items, and ICCs > 0.75 would indicate good reproducibility of scores. The kgv() option. validscale ioc1-ioc37, partition( ) > scorename(ha PSE W BCC AC AE LI MOC) kgv(chemo) kgvboxplot kgvgroup (output omitted) Known-groups validity chemo mean sd p-value HA 0 (n=106) (KW: 0.060) 1 (n=245) PSE 0 (n=105) (KW: 0.029) 1 (n=245) W 0 (n=105) (KW: 0.471) 1 (n=244) BCC 0 (n=105) (KW: 0.011) 1 (n=247) AC 0 (n=105) (KW: 0.014) 1 (n=245) AE 0 (n=104) (KW: 0.095) 1 (n=247) LI 0 (n=104) (KW: 0.215) 1 (n=246) MOC 0 (n=103) (KW: 0.190) 1 (n=242)

21 B. Perrot, E. Bataille, and J.-B. Hardouin 47 HA PSE W 0 1 chemo (p=.101) 0 1 chemo (p=.042) 0 1 chemo (p=.535) BCC AC AE 0 1 chemo (p=.009) 0 1 chemo (p=.011) 0 1 chemo (p=.187) LI MOC 0 1 chemo (p=.157) 0 1 chemo (p=.213) Figure 6. Known-groups validity: chemotherapy/no chemotherapy The kgv() option assesses the known-groups validity of the questionnaire. In this example, an ANOVA is performed to compare the scores between women who had chemotherapy and those who had not. A Kruskal Wallis test is also performed in case ANOVA assumptions were not met. We can see that respondents who received chemotherapy scored higher than others on the BCC (body change concerns) dimension (3.20 versus 2.87, p = 0.009). kgvboxplots produces boxplots of scores by the group variable; kgvgroupboxplots groups the boxplots into a single graph (figure 6). The conc() option. validscale ioc1-ioc37, partition( ) > scorename(ha PSE W BCC AC AE LI MOC) conc(sf12mcs sf12pcs) (output omitted) Concurrent validity sf12mcs sf12pcs HA PSE W BCC AC AE LI MOC

22 48 validscale: A command to validate measurement scales In this example, the conc() option assesses the concurrent validity by examining the correlations between the scores of the Impact Of Cancer version 2 and the SF- 12 (Ware, Kosinski, and Keller 1996). Correlations coefficients 0.40 or 0.40 are displayed in bold font in Stata. This threshold can be changed with tconc(#). For instance, the worry (W) score is (negatively) correlated with the mental component summary score (sf12mcs) of the SF-12 (ρ = 0.44) but only moderately correlated with the physical component summary score (sf12pcs) of the SF-12 (ρ = 0.21). 4.3 Complex syntax of validscale An example of a complex syntax with most of the available options is given below. validscale ioc1-ioc37, partition( ) /// scorename(ha PSE W BCC AC AE LI MOC) categories(1 5) impute(pms) /// noround compscore(sum) descitems graphs cfa cfamethod(ml) cfastand /// cfacov(ioc1*ioc3 ioc2*ioc4) convdiv tconvdiv(0.4) convdivboxplots /// alpha(0.7) delta(0.9) h(0.3) hjmin(0.3) repet(ioc1_2-ioc37_2) /// kappa ickappa(500) kgv(chemo radio) kgvboxplots kgvgroupboxplots /// conc(sf12mcs sf12pcs) tconc(0.4) 5 Implementation of validscale 5.1 Dialog box A dialog box is available if you want to use validscale more intuitively. The db validscale command displays the dialog box shown in figure 7 (only the first tab is shown here). Figure 7. Dialog box for validscale

23 B. Perrot, E. Bataille, and J.-B. Hardouin Online implementation with Numerics by Stata validscale will soon be implemented as a command in an online application called the PRO-online project, which is dedicated to the analysis of patient-reported outcomes. The PRO-online project aims at proposing to students or researchers a way to perform analyses in the field of classical test or item response theories on their own data in a user-friendly way without complex handling of the data. Analyses are performed with Numerics by Stata. 6 Conclusion validscale is a command performing analyses to assess the psychometric properties of subjective measurement scales in the framework of CTT in a user-friendly way. It provides information on structural validity, convergent and discriminant validities, reproducibility, known-groups validity, internal consistency, and scalability. Other theories of measurement coexist in psychometry, particularly IRT and RMT. In these two theories, a latent construct is defined from the relationships between items. Stata provides several commands for IRT (by using theirtcommand). RMT can beused in Stata by using the community-contributed command raschtest (Hardouin 2007c) for dichotomous items or pcmodel (Hamel et al. 2016) for polytomous items. Finally, validscale can be used in a user-friendly way with a dialog box and will soon be implemented in an online application with Numerics by Stata. 7 References Blanchette, D lstrfun: Stata module to modify long local macros. Statistical Software Components S457169, Department of Economics, Boston College. Crespi, C. M., P. A. Ganz, L. Petersen, A. Castillo, and B. Caan Refinement and psychometric evaluation of the impact of cancer scale. Journal of the National Cancer Institute 100: Fayers, P., and D. Machin Quality of Life: The Assessment, Analysis and Interpretation of Patient-Reported Outcomes. 2nd ed. Chichester, UK: Wiley. Gadelrab, H Evaluating the Fit of Structural Equation Models: Sensitivity to Specification Error and Descriptive Goodness-of-Fit Indices. Saarbrücken, Germany: Lambert Academic Publishing. Hamel, J.-F mi twoway: Stata module for computing scores on questionnaires containing missing item responses. Statistical Software Components S457901, Boston College Department of Economics. ideas.repec.org/ c/ boc/ bocode/ s html.

24 50 validscale: A command to validate measurement scales Hamel, J.-F., V. Sébille, G. Challet-Bouju, and J.-B. Hardouin Partial credit model: Estimations and tests of fit with pcmodel. Stata Journal 16: Hardouin, J.-B loevh: Stata module to compute Guttman errors and Loevinger H coefficients. Statistical Software Components S439401, Department of Economics, Boston College. ideas.repec.org/ c/ boc/ bocode/ s html a. delta: Stata module to compute the Delta index of scale discrimination. Statistical Software Components S456851, Department of Economics, Boston College b. imputeitems: Stata module to impute missing data of binary items. Statistical Software Components S456807, Department of Economics, Boston College. econpapers.repec.org/ software/ bocbocode/ s htm c. Rasch analysis: Estimation and tests with raschtest. Stata Journal 7: Hardouin, J.-B., A. Bonnaud-Antignac, and V. Sébille Nonparametric item response theory using Stata. Stata Journal 11: Reichenheim, M. E Confidence intervals for the kappa statistic. Stata Journal 4: Ware, J.E., Jr., M.Kosinski, ands.d.keller A12-itemshort-formhealthsurvey: Construction of scales and preliminary tests of reliability and validity. Medical care 34: About the authors Bastien Perrot is a PhD student in biostatistics at the SPHERE U1246 methods in Patientcentered outcomes and HEalth ResEarch unit at the University of Nantes, France. Part of his research deals with developing and validating measurements scales with applications in health. Emmanuelle Bataille is a biostatistician at the SPHERE U1246 methods in Patient-centered outcomes and HEalth ResEarch unit at the University of Nantes, France. Jean-Benoit Hardouin is a teacher researcher in biostatistics at the SPHERE U1246 methods in Patient-centered outcomes and HEalth ResEarch unit at the University of Nantes, France. His research deals with psychometrics and especially with applying item response theory in clinical trials.