Reliability of Three-Dimensional Facial Landmarks Using Multivariate Intraclass Correlation

Transcription

1 Reliability of Three-Dimensional Facial Landmarks Using Multivariate Intraclass Correlation Abstract The intraclass correlation coefficient (ICC) is widely used in many fields, including orthodontics, as a measure of reliability of quantitative data. The main goals of this study were to determine the multivariate intraclass correlation, a measure of intra-rater reliability, for the selection of 6 3D mid-facial soft tissue landmarks and compare the multivariate ICC with other measures of landmark reliability. 3D stereophotogrammetric images of 15 randomly selected subjects were landmarked twice by the same rater using 3dMD software. The multivariate ICC was found to be as high or higher than the individual coordinate measures in each of the landmarks examined in this study. The reliability of all six soft-tissue landmarks in this study was excellent (ICC >0.98). When the within-coordinate and Euclidean distance criteria were applied, the reliability of the landmarks was found to be acceptable. However, the two selections of the landmark pronasion were found to be significantly different in the X coordinate. The multivariate ICC identified this same landmark as its lowest estimate of ICC. This indicates that the multivariate ICC may be important in quantifying reliability and warrants investigation in future craniofacial landmark reliability studies. Introduction The intraclass correlation coefficient is widely used in many fields, including orthodontics, as a measure of reliability of quantitative data. With the availability of technical software capable of 1

2 recording two and three-dimensional landmark coordinate data for cone beam computerized tomography (CBCT) and digital cephalometric data, the intraclass correlation coefficient is used to establish intra- and inter-rater reliability or reliability of landmarks across software or machines. Recent applications of the intraclass correlation to assess reliability of landmark placement have been used in the study of soft tissue landmarks using CBCT (Fourie, 2010), hard tissue landmarks using CBCT (Lagravere, 2009), and cephalometry (Chien, 2009). The most commonly used method of reporting reliability is a summary measure of the distribution (mean or range) of intraclass correlation coefficients calculated within each coordinate. The estimates of the intraclass correlation coefficient are often obtained using a one-way or two-way random effects ANOVA model (Fisher, 1958; Haggard, 1958; Shrout and Fleiss, 1979) from which confidence intervals and significance values can also be computed. However, statistical methods for calculating a multivariate estimate of intraclass correlation exist and have been used extensively in the fields of biology, epidemiology, and genetics to estimate the degree of resemblance between members of a family with respect to several characteristics (Mian, 1997). In the study of reliability of craniofacial landmark selection, an estimate of multivariate intraclass correlation would represent how closely a family of measures (usually 2 measures) resemble each other with respect to the X, Y, and Z coordinate information. The unified estimate across coordinates enable a global measure, rather than a summary of multiple estimates, on which to assess reliability of landmark selection. A measure of multivariate intraclass correlation was first introduced by Rao (1945, 1953) in which asymptotic distribution and methods of significance testing were established in the case 2

3 that each group had the same number of members. This early work was expanded by Donner and Koval (1980) who proposed an analysis of variance estimator. Srivastava (1984) developed non-iterative methods based on weighted sums of squares to obtain unbiased estimators of covariance matrices for a single characteristic and their asymptotic properties for the case in which groups have a different number of members. Other estimates based on maximum likelihood methods have been proposed (Rosner, 1977), however it has been shown through simulation studies that inferences based on non-iterative methods perform as favorable or more favorable than methods based on maximum likelihood estimators (Rosner, 1979; Konishi, 1982, 1985; Donner and Eliasziw, 1988). Non-iterative approaches are free of the possible pitfalls of methods of estimation based on the maximization of implicit functions of multiple parameters, which are sensitive to local extrema and not guaranteed to converge. Konishi and Khatri (1991) generalized Srivastava s estimator and the Donner and Koval (1980) estimator to the multivariate situation of more than one characteristic and proposed a unified estimator based on the maximum canonical correlation (interclass) and eigenvalue (intraclass) of the covariance matrix as measures of degree of resemblance (Konishi, 1991). This method assumes multivariate normal distribution for asymptotic properties of the estimates. The main goals of this study were to determine the multivariate intraclass correlation, a measure of intra-rater reliability, for the selection of 6 three-dimensional mid-facial soft tissue landmarks (nasion, pronasion, right and left alare, labiale superius, labiale inferius, and subnasale (used for superimposition) see Figure 1), and compare the multivariate intraclass correlation coefficient with other measures of landmark agreement. This study is innovative in the application of these methods of calculating the estimate of multivariate intraclass 3

4 correlation coefficient to assess the reliability of 3D landmark selection and in the comparison of this estimate to conventional within-coordinate intraclass correlation coefficients. Materials and Methods Three-dimensional stereophotogrammetric images of 15 randomly selected subjects were landmarked using 3dMD software (Atlanta, GA) and coordinates of each landmark were selected twice by the same rater with approximately 2 weeks between landmark selections. Six facial landmarks were selected for this study because they represent the nasal and mouth area of the face which are pronounced three- dimensional facial features. Coordinates were exported and the sets of landmarks were superimposed based on translation of each set of first subnasale landmark coordinates to the origin. The sets of landmarks were then standardized with respect to rotation (X into Y holding Z constant) and facial tilt (Y into Z holding X constant) using rotation matrices with the coordinates of the first measure of nasion located on the Y-axis for all subjects after adjustment (Lele and Richtsmeier, 2001). Standardization of head position and coordinate location was not done in the imaging process, but is important in assessing reliability to accurately reflect the variation of each landmark; which impacts the intraclass correlation coefficient. Therefore the head position was standardized using rotation matrices after the coordinates were exported. Conventional within-coordinate intraclass correlation coefficients, confidence intervals, and significance probabilities, obtained from a one-way ANOVA model, were computed for the X, Y, and Z planes for each landmark. Since the intraclass correlation coefficient is a parametric procedure, the residuals from each model were examined for normality using the Shapiro-Wilk test. Only left alare in the Y coordinate was there evidence of non normality (Shapiro-Wilk p=0.0172). However since the one-way ANOVA 4

5 model is generally robust against minor departures from normality, the use of ICC for this coordinate was deemed appropriate. The intraclass correlation coefficient gives the proportion of variance attributable to between-group differences, and the null hypothesis for significance testing is that this coefficient is equal to zero. The ICC ranges from 0 to 1, with 1 indicating perfect agreement. A commonly adopted minimum acceptable univariate intraclass correlation coefficient is 0.80 (Shrout and Fleiss, 1979), with 0.90 generally considered excellent agreement. However, since the estimate of the intraclass coefficient is subject to sampling error, Lee (1989) proposes that the lower bound of the 95% confidence interval of the estimate be at least A method of multivariate intraclass correlation (Konishi, 1991) was used to determine the level of agreement across X, Y, and Z coordinates. Distributional assumptions were validated using the Shapiro-Wilk test within each dimension. This multivariate method, as discussed above, produces a unified estimator of intra-rater agreement between the dual landmark selections with respect to X, Y, and Z coordinate information. Lee (1989) proposes a three-pronged approach for establishing acceptable agreement. In addition to a 1) lower bound of the intraclass correlation coefficient 95% confidence interval of at least 0.75, there must be 2) no systematic bias between the measures, and 3) no significant differences between the measures. Estimation of intraclass correlation does not distinguish between systematic and random error. To identify any possible bias and detect significant differences in the sets of landmarks, the difference was calculated for each set of landmark coordinates (first-second) within each plane. The normality of each within-coordinate difference variable was assessed using the Shapiro-Wilk test under the null hypothesis that the distribution is normal. Difference variables shown to be normal were tested using the paired t- 5

6 test under the null hypothesis that the mean difference between the two measures was equal to zero. For variables in which the distributions of the difference were shown to be nonnormal, a Wilcoxon Signed-Rank test was used to determine if the median difference between the coordinates from the two landmarks was equal to zero (assuming symmetry). To assess differences in landmark placement that may be of clinical importance, the Euclidean distance between the two sets of 3D coordinates was calculated. Descriptive statistics are given to summarize the actual distance in space between the two sets of landmarks. Calculation of the multivariate intraclass correlation coefficient is not available in standard commercially available statistical software. An R program was created by the author to perform the calculations of the estimates for the multivariate ICC reported in this study and is available upon request. Analysis was done using SAS Enterprise Guide 4.2 and R statistical software, with a specified Type I error of Results Within-coordinate reliability analysis was performed on the selection of these six soft tissue landmarks. The intraclass correlations coefficients for the landmarks are shown in Table 1. The within-coordinate coefficients ranged from in the Y coordinate of left alare, to in the Y coordinate of nasion and the X coordinate of right alare. No lower bounds of the 95% confidence intervals were below the 0.75 standard proposed by Lee; with all lower bounds >0.95. In addition, all coordinates and all landmarks had intraclass coefficients significantly greater than zero (p< for all). The mean ICC for all coordinates within a landmark was 6

7 calculated, this value ranged from in left alare to in nasion. It is worth noting here that the first measure of nasion was a landmark used to standardize the rotation and tilt of the set of facial landmarks. The multivariate intraclass correlation (Konishi, 1990), encompassing information for all three dimensions, was calculated for each landmark. These unified landmark values ranged from for pronasion to for nasion, right and left alare. The difference between the first and second landmark selections (first second) were calculated for each dimension and each landmark. These differences were tested using the Student s t test or Wilcoxon signed rank test to determine if mean or median respectively was equal to zero. The p-values for these tests are given in Table 2. A p value <0.05 indicates that there is a significant difference in the two landmark selections in that coordinate. A significant difference occurred between the two selections of pronasion in the X coordinate with significant differences (mean=-0.22mm, t-test p=0.0014). Note that no adjustment was made for multiple comparisons. The Euclidean distance (in mm) was calculated for each landmark to assess the actual distance between the two landmark selections in three-dimensional space. The summary of the Euclidean distances is given in Table 3. The largest mean Euclidean distances were found to exist in the left alare landmark (mean=0.84mm, st.dev=0.73mm) followed by the labiale superius landmark (mean=0.78mm, st.dev=0.51mm). Pronasion (mean=0.49mm, st.dev=0.24mm) and right alare (mean=0.48mm, st.dev=0.29mm) had the smallest mean Euclidean distances between landmark selections. However, when medians were considered 7

8 the largest distance occurred in the labiale superius landmark (median=0.70mm). The left alare landmark had the largest variance in the Euclidean distance of the landmarks examined in this study. Discussion This study applied a technique of calculating multivariate intraclass correlation coefficients, a widely used measure in other scientific fields, to measure the reliability of two sets of 3D softtissue facial landmark selections. The multivariate measure was then compared to the conventional methods used currently in reliability studies in the field of craniofacial landmarking, specifically within-coordinate intraclass correlation coefficients and summary measures by landmark. In contrast with these summary measures of coordinate-specific intraclass correlations, the multivariate measure was as high or higher than the individual coordinate measures in each of the landmarks examined in this study. As the maximum eigenvalue of the variance-covariance matrix, the multivariate estimate takes into account both the variances of all three dimensions simultaneously and their interrelationships as well; a possible explanation of the differences between the multivariate and univariate procedures. This does raise the question of direct numeric comparability of the measures and, further, the interpretability of the multivariate measures in terms of a threshold value that would indicate a clinical need for re-calibration of landmark selection. The reliability of all six soft-tissue landmarks in this study was excellent (ICC >0.98). Lagravere (2010) warns that landmarks with a mean difference of 1-2mm are clinically acceptable, however landmarks with mean differences >2mm should be used with caution. All the 8

9 landmarks examined in this reliability study met this clinically acceptable criterion with the maximum mean Euclidean distance of 0.84mm (left alare). When Lee s (1989) criteria were applied, no landmark had the lower bound of the 95% confidence interval of the ICC below the 0.75 threshold. However, a significant difference existed between the two landmark selections in the X dimensions of pronasion (p=0.0014). Also, the significant difference was in the negative direction indicating a possible bias of the second X coordinate being systematically greater than the first. Using Lee s criteria, the reliability of the landmark pronasion may not be acceptable and may warrant review of selection protocol and re-calibration. Left alare had the largest variance of the Euclidean distance between selections of all the landmarks in the study. The lowest mean within-coordinate ICC was found in the left alare landmark in response to two of the lower ICCs in the Y and Z coordinates, left alare also had the highest mean Euclidean distance between landmark selections. However the lowest multivariate ICC was found in the pronasion landmark which was the only landmark with a significant difference between the coordinates (X dimension). It is possible that the univariate and multivariate ICCs are sensitive to different characteristics within the landmark data. Further work is needed to determine the relationship and comparability of the within-coordinate measures of agreement and the unified multivariate estimator of agreement as well as the interpretability and clinically acceptable level of the multivariate intraclass correlation coefficient. 9

10 References Adams, G. L., Gansky, S. A., Miller, A. J., Harrell, W. E., & Hatcher, D. C. (2004). Comparison between traditional 2-dimensional cephalometry and a 3-dimensional approach on human dry skulls. American Journal of Orthodontics and Dentofacial Orthopedics, 126(4), doi:doi: /j.ajodo Bland, J. M., & Altman, D. G. (1990). A note on the use of the intraclass correlation coefficient in the evaluation of agreement between two methods of measurement. Computers in Biology and Medicine, 20(5), doi:doi: / (90)90013-F Chien, P., Parks, E., Eraso, F., Hartsfield, J., Roberts, W., & Ofner, S. (2009). Comparison of reliability in anatomical landmark identification using two-dimensional digital cephalometrics and three-dimensional cone beam computed tomography in vivo. Dentomaxillofacial Radiology, 38(5), doi: /dmfr/ Donner, A., & Bull, S. (1984). A comparison of significance-testing procedures for parent-child correlations computed from family data. Journal of the Royal Statistical Society.Series C (Applied Statistics), 33(3), pp Donner, A., & Eliasziw, M. (1991). Methodology for inferences concerning familial correlations: A review. Journal of Clinical Epidemiology, 44(4-5), doi:doi: / (91)90084-M Donner, A., & Eliasziw, M. (1991). Methodology for inferences concerning familial correlations: A review. Journal of Clinical Epidemiology, 44(4-5), doi:doi: / (91)90084-M Fourie, Z., Damstra, J., Gerrits, P. O., & Ren, Y.Evaluation of anthropometric accuracy and reliability using different threedimensional scanning systems. Forensic Science International, In Press, Corrected Proof doi:doi: /j.forsciint Haggard, EA. Intraclass Correlation and the Analysis of Variance. New York: Dryden Konishi, S. (1982). Asymptotic properties of estimators of interclass correlation from familial data Springer Netherlands. doi: /bf Konishi, S. (1985). Testing hypotheses about interclass correlations from familial data. Biometrics, 41(1), pp Lagravère, M. O., Gordon, J. M., Guedes, I. H., Flores-Mir, C., Carey, J. P., Heo, G., & Major, P. W. (2009). Reliability of traditional cephalometric landmarks as seen in three-dimensional analysis in maxillary expansion treatments. The Angle Orthodontist, 79(6), doi: / r.1 Lagravère, M. O., Low, C., Flores-Mir, C., Chung, R., Carey, J. P., Heo, G., & Major, P. W. (2010). Intraexaminer and interexaminer reliabilities of landmark identification on digitized lateral cephalograms and formatted 3-dimensional cone-beam computerized tomography images. American Journal of Orthodontics and Dentofacial Orthopedics, 137(5), doi:doi: /j.ajodo Lee, J., Koh, D., & Ong, C. N. (1989). Statistical evaluation of agreement between two methods for measuring a quantitative variable. Computers in Biology and Medicine, 19(1), doi:doi: / (89)90036-X Lele S, Richtsmeier J. An Invariant Approach to Statistical Analysis of Shapes. Chapman & Hall: Boca Raton, Mian, I. U. H., & Shoukri, M. M. (1997). Statistical analysis of intraclass correlations from multiple samples with applications to arterial blood pressure data. Statistics in Medicine, 16(13), doi: /(sici) ( )16:13<1497::aid-sim569>3.0.co;2-7 Shrout, P. E., & Fleiss, J. L. (1979). Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin, 86(2), doi: /

11 Figure 1. Facial Soft Tissue Landmarks Nasion Pronasion Right Alare Left Alare Sub Nasale Labiale Superius Labiale Inferius Table 1. Intraclass Correlation Coefficients for Each Landmark By Coordinate and Multivariate Estimate (Adjusted Measures) Multivariate Landmark ICC 95% CI P value Mean ICC 1 Left Alare 2 Labiale Superius 3 Nasion 4 Pronasion 5 Right Alare 6 Labiale Inferius X ( , ) < Y ( , ) < Z ( , ) < X ( , ) < Y ( , ) < Z ( , ) < X ( , ) < Y ( , ) < Z ( , ) < X ( , ) < Y ( , ) < Z ( , ) < X ( , ) < Y ( , ) < Z ( , ) < X ( , ) < Y ( , ) < Z ( , ) < All p values <0.0001; these are significance probabilities associated with the test of the null hypothesis that the intraclass correlation is equal to zero. 11

12 Table 2. Summary of Within-coordinate Differences (mm) Between Dual Landmark Selections Landmark Coordinate Mean St. Dev. Min Max Median 1 Left Alare 2 Labiale Superius 3 Nasion 4 Pronasion 5 Right Alare 6 Labiale Inferius 7 Subnasale T-test or Wilcoxon* P-value X Y * Z X Y Z X Y Z X Y Z X Y Z X Y Z X Y Z Significance probabilities associated with the t-test of the null hypothesis that the mean is equal to zero, or the Wilcoxon* signed rank that the median difference is equal to zero (assuming symmetry). Table 3. Summary of Euclidean Distances Between Dual Landmark Selections Landmark N Mean St Dev. Median Minimum Maximum Left Alare Nasion Pronasion Right Alare Labiale Inferius Labiale Superius