C-LTPP Surface Distress Variability Analysis

Transcription

1 Canadian Strategic Highway Research Program Programme stratégique de recherche routière du Canada C-LTPP Surface Distress Variability Analysis Prepared by: Stephen Goodman, B.A.Sc. C-SHRP Technology Transfer Manager June 2000

2 Disclaimer: The opinions expressed in this report are those of the author and do not necessarily represent those of the Canadian Strategic Highway Research Program (C-SHRP) or its sponsoring agencies. Canadian Strategic Highway Research Program (C-SHRP) 2323 St. Laurent Blvd. Ottawa, Ontario, Canada K1G 4J8 Tele: (613) Fax: (613) ISBN

3 ACKNOWLEDGEMENTS The author of this report would like to thank Mr. Wael Bekheet and Dr. Juan Salinas of Carleton University for their assistance in the completion of this investigation. i

4 EXECUTIVE SUMMARY In 1997, the C-LTPP program capitalized on an opportunity to have surface distress surveys completed twice at more than half of the C-LTPP test sections. In addition to the regular surface distress surveys completed by transportation agency representatives, a second set of surveys was completed concurrently with Falling Weight Deflectometer (FWD) testing during the same year. This second set of surveys was completed to provide some insight as to the variability of surface distress surveys completed by different raters. As only two distress surveys for each test section were available, traditional analysis techniques to evaluate association such as analysis of variance (ANOVA) were not applicable. Instead, an analysis technique referred to as Cohen s Weighted Kappa Statistic was utilized to directly compare the agreement between the distress surveys of the baseline rater and agency raters. The kappa statistic is extremely powerful as it considers the likelihood of chance agreement and also allows the introduction of penalty values for individual cases of disagreement. The total linear distance of cracking measured by the baseline rater and the combined agency raters over the 35 sections was metres and metres, respectively, a difference of only 3.5 percent. Overall, the performance of the test sections to date has been quite good, with most of the cracking measured classified as either slight or very slight severity after 7 to 9 years in service. Agreement between the baseline rater and agency raters based on the total amount of cracking grouped by crack type was very high while less agreement was observed for severity level. Agreement analyses completed on crack type by severity level (simultaneously) yielded very low values of the kappa statistic, indicating poor agreement between the baseline and agency rater. However, interagency agreement observed indirectly through the baseline rater appeared small indicating that the variability between individual agency raters may be lower than indicated in the agreement analyses. Agreement increased significantly with data aggregation, indicating that much less variability will be present for network level analyses than for project level analyses. This fact questions the ability of accurate crack growth modelling at the individual crack level. Based on overall agreement, ranking of the crack types by highest to lowest agreement was as follows; Wheel Track, Transverse, Centreline, Edge, Midlane, Meander and Block. The effect of rater experience was also investigated, although no firm conclusions could be made. However, it appeared that a rating pair consisting of 2 inexperienced raters generated equal agreement as that of a single experienced rater. Methods to reduce the variability of future distress surveys were also presented. These include the reduction of the number of severity levels from 5 to 3, reduction of the number of individual agency raters, more frequent training of raters and the use of rating teams. Also, it was noted in this study that variability between raters was affected by the number of days between replicate surveys, most likely due to climatic effects. However, the typical annual surveys completed for the C-LTPP program are completed using the previous year s survey as a guide, which should reduce some of the variability associated with climatic effects. In general, the results of the C-LTPP distress variability analysis were in agreement with previous studies. ii

5 TABLE OF CONTENTS Acknowledgements... i Executive Summary... ii Table of Contents... iii List of Tables and Figures... v 1.0 Introduction Background Objectives and Scope of Analysis Review of Previous Studies US-LTPP Distress Data Variability Study Saskatchewan Surface Condition Rating Data Quality Study C-LTPP Surface Distress Data Variability Analysis Introduction Data Source and Preparation Procedure for Assessing Surface Distress Data Variability Traditional ANOVA Cohen s Weighted Kappa Statistic Selected Error Matrices Results and Discussion Total Cracking Extent Measured Overall Distribution of Crack Type and Severity Level Agreement by Crack Type Agreement by Severity Level Agreement by Crack Type and Severity Level Ranking of Cracking Type Agreement by Agency Rater/Pair and Experience Level Surface Distress Results with Respect to Overall Long Term Performance Conclusions and Recommendations References Appendix A: C-LTPP Surface Distress Mapping Form (SD1) and Distress Code List iii

6 Appendix B: Sample Distress Summary Table Appendix C: Master Input Data Sheet Appendix D: Agreement Analyses vs. Combined Agency Raters by Crack Type Appendix E: Agreement Analyses vs. Combined Agency Raters by Severity Level iv

7 LIST OF TABLES AND FIGURES Table 1: Breakdown of Test Sections Rated by Agency Raters/Pairs... 5 Table 2: Distress Summary Tables Created from Raw Surface Distress Data... 6 Table 3: Cracking Severity Levels... 6 Table 4: Classification of 100 Metres of Cracking by Two Raters... 8 Table 5: Chance Matrix for Table 4 Data... 8 Table 6: Comparison of Observed and Expected Frequencies... 8 Table 7: Kappa Statistic vs. Strength of Agreement... 9 Table 8: Weighted Error Matrix Table 9: Example Observed Crack Extent Matrix Table 10: Error Matrix for Severity Level Table 11: Error Matrix for Crack Type Table 12: Error Matrix for Group Comparison (Total Extent) Table 13: Agreement Analyses for Crack Type Table 14: Agreement Analyses for Severity Level Table 15: Agreement Analyses of Crack Type by Severity Level for each Section Group.. 17 Table 16: Rank of Cracking Distress based on Agreement Analyses Table 17: Agreement Analyses for Agency Raters/Pairs Figure 1: Distribution of Total Crack Extent by Crack Type ( Rating) Figure 2: Distribution of Total Crack Extent by Crack Type (Agency Rating) Figure 3: Distribution of Total Crack Extent by Crack Severity ( Rating) Figure 4: Distribution of Total Crack Extent by Crack Severity (Agency Rating) Figure 5: Agreement Index vs. Time Between and Agency Rating v

8 1.0 INTRODUCTION 1.1 Background The Canadian Long Term Pavement Performance (C-LTPP) project was established in 1989 as a fifteen-year experiment to extend the US-LTPP concept to consider factors of particular interest to Canada. The overall goal of the C-LTPP project is to increase pavement life through the development of cost effective pavement rehabilitation procedures, based upon systematic observation of in-service pavement performance. In 1997, the C-LTPP program capitalized on an opportunity to have surface distress surveys completed twice at more than half of the C-LTPP test sections. In addition to the regular surface distress surveys completed by transportation agency representatives, a second set of surveys was completed concurrently with Falling Weight Deflectometer (FWD) testing during the same year. This second set of surveys was completed to provide some insight as to the variability of surface distress surveys completed by different raters. This report is the product of the variability analysis. 1.2 Objectives and Scope of Analysis The success of the C-LTPP and US-LTPP programs depends on reliable data for pavement performance model development, calibration and validation. Therefore, reducing the variability of all collected data will help ensure that accurate and reliable data is available for use with confidence during modelling or analysis efforts. Unlike other variability studies, there was no attempt to determine a correct distress rating for each test section either through the use of an expert panel or using the mean of a large number of raters. For this investigation, the surface distress surveys completed by the FWD operator served as a consistent baseline across the 35 sections. Therefore, variability between the various agency raters was observed indirectly through comparison with the baseline rater. The specific project objectives for this investigation are as follows: 1. Determine the inherent variability in the extent ratings for each major distress type; 2. Determine the inherent variability in the severity ratings for each major distress type; 3. Rank the distresses in order of their reliability; 4. Observe the effect of rater experience on calculated variability; 5. Suggest ways that the inherent error can be accommodated in performance modelling or other analyses using surface distress data; 6. Identify major sources of error and suggest potential improvements to C-SHRP s surface distress mapping process and data entry process; 1

9 2.0 REVIEW OF PREVIOUS STUDIES Two previous studies were reviewed prior to the completion of the C-LTPP surface distress analysis. The first was completed by the United States Federal Highway Administration (FHWA) for the US- LTPP program, while the second was completed by the Saskatchewan Department of Highways and Transportation. 2.1 US-LTPP Distress Data Variability Study A report entitled Study of LTPP Distress Data Variability Final Report was published in March of 1998, documenting a comprehensive analysis of surface distress data variability completed for the U.S. Long Term Pavement Performance Program [FHWA 1998]. Data for the analysis was collected through nine LTPP rater accreditation workshops held throughout the United States between 1992 and Distress ratings from between 6 and 16 raters were collected per workshop on the same test section on the same day. At each workshop, reference surveys were also completed by workshop instructors using a consensus rating method to provide a ground truth rating from which to compare individual raters surveys. Three main analyses were completed to observe variability with the manual surface distress data. The first analysis observed global trends in distress data variability by plotting the quantity (extent) of individual distresses at each severity level for each distress type. For each graph, the reference value (as determined by the workshop instructors) was compared to the group mean (the average of the individual raters) as well as the minimum and maximum values observed in the individual rater surveys. The following observations were made based on the plots: Although the magnitude of variability for any given distress type and severity level varied from workshop to workshop, in general, the variability was large and the scatter of data tended to increase in magnitude with an increase in distress quantity. Coefficients of variation of 30% were common. For total distress summed across all severity levels for each distress type, the group means were generally close to the reference value and the between rater scatter was smaller that for the individual severity levels. This was indicative of the greater variability in classification of severity level than distress type. For closely related distress types, such as fatigue cracking and longitudinal cracking in the wheelpaths, compensatory differences between the group ratings and reference values were observed. For example, group ratings indicated a higher quantity of fatigue cracking and a lower quantity of longitudinal cracking as compared to reference values. There was no apparent significant positive or negative bias in the data; i.e. no tendency to consistently rate all distress type and severity level combinations higher or lower. Due to the high variability between the distress rater values, the reference value was almost always with the range of data scatter for all distress type and severity level combinations. 2

10 The report noted that the most of the above results were expected, although the high level of rater variability was somewhat surprising and could potentially have a significant impact upon the usefulness of the data. The second analysis investigated the effect that variable distress data would have on Pavement Condition Index (PCI) since many transportation agencies use a composite distress statistic such as the PCI in their pavement management systems. As expected, the variability observed between PCI values was much smaller than observed with individual distress types and severity levels. Reduced variability was observed both between the instructors and the raters, as well as between individual raters. The third and final analysis performed in the LTPP study concerned the quantification of bias and precision. Bias and precision analyses of group mean values, reference values, standard deviations and coefficients of variation provided the following observations: The apparent bias for most distress type-severity level combinations was small, confirming the earlier observation that the group means were close to the reference values. No uniform bias in the data was apparent indicating that there was no tendency to consistently rate all distress type and severity level combinations higher or lower. The precision of distress data relative to the group mean was not deemed acceptable for research quality data as the coefficient of variation extended to over 300% for some distress/severity levels. There was a general trend for a decrease in coefficient of variation with increased distress quantity. The precision for total distress quantities (extent) was generally much better than that for the individual severity levels, again indicating a greater variability in distinguishing severity levels. Overall, the authors believed that the question of acceptable level of distress data variability depends on the intended use of the data. Although the variability resulting from the workshops appeared to be acceptable for pavement management, particularly at the network level, the authors contended that the level of variability is not acceptable for research use. Clearly this conclusion presents a serious problem to the LTPP program since the main purpose of the LTPP data is to provide pavement performance models. The main recommendation posed by the study was to replace individual surveys with group consensus surveys. 2.2 Saskatchewan Surface Condition Rating Data Quality Study Like many highway agencies, the Saskatchewan Department of Highways and Transportation uses a condition rating of its provincial highways to develop its annual maintenance and rehabilitation budgets. In 1995, the Saskatchewan Surface Condition Rating Data Quality Study was completed to develop an understanding of the error associated with manual surface distress surveys and how that uncertainty may impact the Department s decisions [Sask Highways 1995]. The investigation cited five main objectives. The first objective was to determine the inherent variability of each major distress using a sound statistical approach. The approach used for the 3

11 investigation was a modified version of the approach developed by the Australian Bureau of Statistics based on Cohen s weighted kappa statistic. The kappa statistic involves the calculation of an agreement coefficient for ratings of a set of segments (test sites) by two different raters. The agreement coefficient ranges from 1 (complete disagreement) to +1 (complete agreement). The experiment involved the rating of 18 test sections by 9 separate teams of raters across a wide crosssection of distress conditions. Overall, the agreement index for cracking and rutting ranged between 0.2 and 0.6, indicating fair to moderate agreement between raters. Only slight agreement between raters was observed for local surface defects. In general, agreement between crack type and extent was much greater than severity. The second objective of the investigation was to assess the impact of the rating variability on the Department s decisions. Results of a sensitivity analysis indicated that the budgeted amounts for individual rehabilitation levels (routine maintenance, light preservation, moderate preservation and heavy preservation) could be in error of up to 21%, the error in the total budgeted amount across all rehabilitation levels was only 1% due to the cancellation of errors. Ranking the individual distresses by reliability level was the third objective of the study. Overall categories were ranked in the following order from best agreement to worst agreement; Cracking Extent, Rutting Extent, Rutting Severity, Cracking Severity, Surface Condition and Local Surface Defects. Agreement ranged from moderate for cracking and rutting extent to slight for surface condition and local surface defects. The fourth objective of the study was to compare the ratings of experienced raters to inexperienced ones. Based on comparison of reliability scores, there was no clear pattern suggesting that experience lead to increased reliability. In fact, the inexperienced group had a higher average reliability score than the experience group, indicating that frequent training of the raters was more effective than experience alone. The final objective of the Saskatchewan study was to identify major sources of error. It was concluded that improving Local Defect, Surface Condition and Transverse Cracking Severity would provide substantial increases in reliability. The largest single source of error identified was location of the gauging length. 4

12 3.0 C-LTPP SURFACE DISTRESS DATA VARIABILITY ANALYSIS 3.1 Introduction The C-LTPP procedure for surface distress monitoring is found in the Pavement Research Technical Guidelines published by C-SHRP prior to the commencement of the C-LTPP program [C-SHRP 1990]. Distresses are mapped on standard forms (form SD1) and submitted to C-SHRP annually for entry into the C-LTPP database. A sample SD1 form is included as Appendix A. Five SD1 forms are completed for each 150-metre test section. As shown, localized defects are drawn on a scaled area representing the 30-metre subsection using various distress codes also included in Appendix A. Uniform distresses covering more than 75 percent of the subsection area are not drawn, but described in a separate space in the SD1 form with an associated severity and extent. 3.2 Data Source and Preparation In 1997, duplicate surface distress surveys were completed for 35 of the 65 C-LTPP sections, representing 5 provinces. Five individual agency raters and two pairs of agency raters completed the standard sets of agency surveys. Therefore, seven distinct section groups were available for analysis as shown in Table 1. A single rater, known throughout this report as the baseline rater, subsequently surveyed all 35 sections during FWD testing. For this investigation, rater experience was separated into experienced (more than 5 years) and inexperienced (less than 5 years). The baseline rater was classed as inexperienced, although he was trained immediately prior to completing the surveys. Table 1: Breakdown of Test Sections Rated by Agency Raters/Pairs Section Group No. of Agency Experience Raters Level Group 1 2 Both Inexperienced Group Experienced, 1 Inexperienced Group 3 1 Inexperienced Group 4 1 Inexperienced Group 5 1 Experienced Group 6 1 Inexperienced Group 7 1 Experienced Entry of the data from both the agency and baseline sources was completed by a single person to minimize bias during the interpretation of the distresses drawn on the SD1 forms. Generally, the data entry person must simply measure the dimensions of the distresses. However, many distresses (cracking in particular) were either not labelled, or were of such complexity that interpretation was necessary. For this investigation, a person with first-hand knowledge of the surface distress mapping procedure and familiarity with pavement distress mechanisms completed the data entry. Before the raw surface distress data is entered into the database, it is subjected to a set of range checks designed to identify invalid entries. Examples of invalid entries include individual longitudinal linear (single and multiple) distresses that are longer than the section length (or twice 5

13 the section length if in the wheel paths), transverse distresses that are longer than the lane width, and distress severities that are not within the specified range. Once the data has passed the range checks, a series of seven distress summary tables are created by the C-LTPP software. The distress summary tables are listed in Table 2 with their respective distresses. At this time, only distress summary tables are generated for cracking distresses in the C-LTPP database. Information concerning area distresses such as ravelling, flushing, potholes, as well as maintenance treatments such as patching, spray sealing and seal coats is collected and entered into the database, however, summary information is not generated. Table 2: Distress Summary Tables Created from Raw Surface Distress Data Distress Summary Table (filename) Block Crack (blockcrk) Centreline Crack (Ctrlncrk) Edge Crack (Edgecrk) Meander Crack (Meandcrk) Midlane Longitudinal Crack (Midlacrk) Transverse Crack (Transcrk) Wheel Track Crack (Wheelcrk) Distresses Included (code) Random and Map Cracking (C11) Centreline Single and Multiple (C3) Centreline Alligator (C4) Pavement Edge Single and Multiple (C5) Pavement Edge Alligator (C6) Meander Cracking (C9) Midlane Longitudinal Cracking (C10) Transverse Half, Full and Multiple (C7) Transverse Alligator (C8) Wheel Track Single and Multiple (C1) Wheel Track Alligator (C2) Each summary table includes a summation of the respective distress by each of the six severity levels (Extent Sealed and Severity Level 1 through Severity Level 5) for each 150-metre test section. Severity is rated in terms of crack width as shown in Table 3. Table 3: Cracking Severity Levels Severity Level (Class) Crack Width (mm) Sealed 1 (Very Slight) < 5 2 (Slight) (Moderate) (Severe) (Very Severe) > 30 6

14 A sample distress summary table is included as Appendix B for reference. It should be noted that alligator cracking is entered as a linear distress in the C-SHRP database, not as an area distress. Distress summary tables created separately for the baseline surveys and the agency surveys served as the basis for comparison of rater variability in this investigation. 3.3 Procedure for Assessing Surface Distress Data Variability Traditional ANOVA Various methods are available for observing the variability between individual data records or multiple datasets. For individual records, perhaps the simplest method involves the calculation of a mean (central or average) value and the standard deviation of that mean. The standard deviation is a very useful measure of variation because it is given in the same units as the mean [Miller et al 1990]. Once standard deviation is known, the coefficient of variation (COV) may be calculated by dividing the standard deviation by the mean. COV is a convenient measure for gauging variability because it is a measure of relative variation, allowing the comparison of several sets of data. Analysis of variance (ANOVA) may also be used to compare multiple mean values to determine whether or not those values come from the same overall population, i.e. whether or not they are statistically the same. The above analysis techniques are very useful when a well-defined standard for characterizing a particular subject is available. For example, crack extent (i.e. length measured with a measuring tape or distance wheel) is a fairly standard and accurate method of characterizing a crack. Therefore, the calculation of mean crack length, standard deviation, etc. between multiple raters is valid for analysis of variance. However, crack type and severity level are not as easily defined and require a certain amount of judgement, despite the guidelines developed by C-SHRP and other agencies for the measurement of surface distress. For example, block cracking may easily be classified as a combination of transverse and longitudinal cracks. Furthermore, ratings between individual raters can be influenced by the distribution of the surface distress [Sask Highways 1995]. Consider a section of road containing a large amount of transverse cracking with varying levels of severity as well as a much smaller amount of longitudinal cracking with similar severity. For this example, the raters would be much more likely to agree on the ratings for the longitudinal cracks than the transverse cracks since there is less opportunity (chance) for disagreement. This phenomenon is referred to as the floor effect. The opposite effect, called the ceiling effect occurs in situations where it is easy to consistently agree on ratings for badly distressed sections [Sask Highways 1995]. These biases are not accounted for by standard ANOVA, but may be accurately captured by another type of analysis involving an agreement index, also known as Cohen s Weighted Kappa Statistic Cohen s Weighted Kappa Statistic The advantage of the kappa statistic over traditional ANOVA for the comparison of subjective ratings can be best illustrated with the following example modified from [Fleiss 1973]. Consider the scenario where one hundred metres of cracking is rated by two raters, with each crack classified into one of three mutually exclusive crack types as shown in Table 4. 7

15 Table 4: Classification of 100 Metres of Cracking by Two Raters Rater A Edge Midlane Wheel Total Edge Rater B Midlane Wheel Total A typical analysis of variance would determine the magnitude of the chi-square statistic to determine whether the ratings are associated. To do so, one must first determine what values are expected in Table 4 assuming no association between the raters at all in other words, completely by chance. In Table 4, Rater A classified 50 metres of the full 100 metres as Edge cracking while Rater B classified 40 metres of the total 100 metres as Edge cracking. Therefore, the probability that the two raters agreed by chance is (50/100)*(40/100)=0.2, or 20%. Since the total amount of cracking was 100 metres, the expected number of metres rated solely by chance as Edge cracking is simply 20 metres (the product of the chance probability and the number of applications). Table 5 displays the resulting chance matrix. Table 5: Chance Matrix for Table 4 Data Rater A Edge Midlane Wheel Total Edge Rater B Midlane Wheel Total The next step involves comparing the number of observed frequencies (n ij ) with the number of expected (chance) frequencies (n ij (e)) using the following expression: [1] d ij ( n = ij n n ij ij ( e)) ( e) 2 Thus d 11 =(20-20) 2 /20=0, etc. as shown in Table 6. Table 6: Comparison of Observed and Expected Frequencies Rater B Rater A Edge Midlane Wheel Edge Midlane Wheel

16 Finally, the chi-square statistic is calculated by summing all values of d ij : [2] χ 2 = = 9.91 In this hypothetical case, the value of chi-square is significant at the 95% confidence level, indicating that an association does exist between the ratings of the two raters. However, the chi-square statistic measures association of any kind, not specifically agreement. To measure the actual agreement, attention must be made to the cells along the diagonal of Tables 4 and 5. Notice that for each of the three crack types, the length of cracking agreed upon by the two raters (20, 6 and 9 metres respectively) is identical to the length of cracking expected solely on the basis of chance, as indicated in Table 5. Therefore, the actual degree of agreement between Rater A and Rater B is no better than that predicted by chance. The chi-square statistic, however, indicated that the ratings were reliable. The kappa statistic investigates the amount of agreement beyond chance according to the following formula: [3] po p κ = 1 p c c where p o is the observed proportion of agreement and p c is the proportion of agreement expected by chance alone. Therefore, if p o =p c, the kappa statistic is zero, and the conclusion would be that all of the observed agreement could be attributed to chance. The kappa statistic ranges between +1 and 1 with the associated descriptions of resulting agreement as shown in Table 7 [Dunn 1989]. Table 7: Kappa Statistic vs. Strength of Agreement Kappa Statistic Strength of Agreement Complete disagreement < 0.00 Disagreement (worse than random) 0.00 Chance agreement 0.01 to 0.20 Slight agreement 0.21 to 0.40 Fair agreement 0.41 to 0.60 Moderate agreement 0.61 to 0.80 Substantial agreement 0.81 to 0.99 Almost perfect agreement 1.00 Perfect agreement The kappa statistic may also be weighted by assigning penalty factors to the observed disagreement values. This is completed by multiplying the observed cracking lengths (Table 4) by an error matrix such as the one shown in Table 8. 9

17 Table 8: Weighted Error Matrix Rater B Rater A Type 1 Type 2 Type 3 Type Type Type As shown, the cells along the diagonal of the error matrix contain zeros as no penalty is associated to situations of agreement. Penalty values are assigned to the remaining cells based on the relative error associated with the particular misclassification. The ability to customize the relative weight of each misclassification is beneficial, particularly since some misclassifications are more unlikely than others. For example, classification of a crack as being transverse when in reality it is longitudinal in direction is a gross misclassification, whereas classification of a crack as Severity 1 when it is actually Severity 2 is much more tolerable. The error matrix may accommodate all misclassifications by assigning relative penalty values when they occur Notes on Comparing the and Agency Ratings As previously mentioned, the data used for agreement analyses were obtained from the 7 distress summary tables produced by the C-LTPP software. Each value in the summary tables represents the summation of the extent of each crack type by severity level for each 150-metre section (35 in total). However, even at this level of aggregation, many of the cells in the tables were blank indicating no distress (of that particular crack type or severity level) was present for many of the 150-metre sections. From a performance point of view, this result was encouraging since the overlays were approximately 7 years old at the time of the surveys. However, to provide more data for the analyses, the data was further aggregated by summing all of the 150-metre sections for each individual agency rater or rater pair (7 groups in total). The resulting dataset is attached as Appendix C for reference. Therefore, the agreement analyses for this investigation were completed on values representing the summation of each crack type by severity level for each section group. Ideally, comparison would be completed on a crack-bycrack basis at a single section displaying a wide range of cracking distresses and severity levels. Another problem stemming from this initial aggregation of the data concerned the handling of cases of disagreement. For example, if the baseline rater measured 100 metres of Midlane cracking over the 150 metre section, while the agency rater measured only 75 metres of Midlane cracking, it was impossible to determine what type of cracking the remaining 25 metres were rated as (Edge, Wheel, etc), or if they were rated at all since the comparison was not at the individual crack level. To accommodate this, a separate column entitled was added to each matrix. In the above example, the amount of agreement between the baseline and agency was 75 metres, while the disagreement was 25 metres. These 25 metres were entered in the column as shown in Table 9. 10

18 Table 9: Example Observed Crack Extent Matrix Agency Rater Midlane Edge Wheel 25 Midlane 75 Edge Wheel The same methodology was applied for severity level. Therefore, in reality, the agreement indices were only calculated based on a binary choice (agreement vs. other) for crack type and severity level not an ideal situation - however, this technique was necessary given the data available and was deemed acceptable for purposes of comparing the agency raters/pairs Selected Error Matrices Judicious selection of the weighting factors within the error matrix was critical to the agreement analysis and therefore deserves further discussion. Three distinct sets of agreement analyses were completed to achieve the investigation objectives, thereby requiring three distinct error matrices as shown in Tables 10 through 12. To compare baseline and agency ratings for severity level, the error matrix shown in Table 10 was used. Penalty values of 1.5 were used for each cell in the first column and first row representing the category. The penalty of 1.5 was based on the assumption that if disagreement between the baseline and agency raters was observed for any severity rating, the difference was most likely within one or two severity levels. For example, if the baseline rater measured 100 metres of cracking with Severity Level 1 and the agency rater only measured 75 metres of Severity Level 1 cracking, it was reasonable to assume that the other 25 metres of cracking were rated at either Severity Level 2 or (at most) Severity Level 3. The remaining cells in the error matrix were populated by penalty values equal to the difference in severity levels measured - i.e. the error for disagreement between a Level 1 and a Level 2 crack is a penalty value of 1. As mentioned in the previous section, these penalty values did not directly affect the observed ratings since the observed matrix was populated only in the diagonal and cells. However, the penalty values were critical for determining the chance matrix. Table 10: Error Matrix for Severity Level Agency Rater Sealed Severity 1 Severity 2 Severity 3 Severity 4 Severity Sealed Severity Severity Severity Severity Severity

19 Table 11 displays the error matrix used to calculate the agreement indices based on crack type. As in Table 10, the penalty value of 1.5 was also used for cells in the category. This inherently implies that the penalty for misclassification by crack type was the same as misclassification by severity level, which was a reasonable assumption for aggregate analysis, but certainly not at the individual crack level. The remaining cells were populated with penalty values more representative of misclassification by crack type. For example, perhaps the grossest misclassification would be the classification of a transverse crack as being longitudinal in direction (edge, midlane, etc). Therefore a penalty of 5 was assigned to these scenarios. Lesser penalties of 2 and 3 were assigned to those situations where misclassification was more likely such as the classification of an edge crack as being in the wheelpaths. Table 11: Error Matrix for Crack Type Agency Rater Block Centreline Edge Meander Midlane Transverse Wheel Block Centreline Edge Meander Midlane Transverse Wheel The third and final error matrix, shown in Table 12, was used to compare the total extent of cracking measured (by type and/or severity) by each individual agency rater or rating pair and the baseline rater. The penalty value of 1.5 was again used for the cells, however, all remaining cells were assigned a penalty of 5. This high penalty value was imposed since it, for these analyses, it was impossible to have entries outside of the diagonal or cells. For example, disagreement between Rater 3 and the baseline rater could not be entered in any other rater cell since Rater 3 was the only agency rater to rate cracks on his/her respective sections. Table 12: Error Matrix for Group Comparison (Total Extent) Agency Rater Group 1 Group 2 Group 3 Group 4 Group 5 Group 6 Group Group Group Group Group Group Group Group

20 3.4 Results and Discussion Total Cracking Extent Measured The total linear distance of cracking measured by the baseline rater and the combined agency raters over the 35 sections was metres and metres, respectively, a difference of only 3.5 percent. This minute difference clearly displayed the diligence of the raters toward accurate measurement of cracking length Overall Distribution of Crack Type and Severity Level Figures 1 and 2 display the percentage distribution of total crack extent by crack type for the baseline and combined agency raters, respectively. The percentages shown were calculated using the individual cracking totals ( metres or metres, respectively) as opposed to an average of the two. As shown, the baseline and combined agency raters measured the same quantity (extent) of transverse cracking (27% of the total extent) and midlane cracking (8%). Edge cracking was also very close between the baseline and agency raters at 18% and 17%, respectively. However, the baseline rater measured no block cracking and very little meander cracking while the agency raters measured 1% block cracking and 2% meander cracking. The block and meander cracking measurements by the agency raters were apparently at the expense of centreline cracking as the baseline rater measured 23% while only 19% was measured by the agency raters. More wheel track cracking was observed by the agency raters (26% versus 24%), most likely accounting for the smaller amount of edge cracking measured by the agency raters. Agreement analysis completed at this highest level of data aggregation resulted in a kappa statistic of 0.95, indicating almost perfect agreement between the baseline and combined agency raters for crack type. Figures 3 and 4 display the percentage distribution of total crack extent by severity level for the baseline and combined agency raters, respectively. Much larger differences were seen compared to crack type. This observation was expected as crack type should be more easily determined, and therefore more consistent, than the severity level. As shown, the baseline rater appeared to be more harsh regarding the lowest severity level, rating only 29% of the total cracks at Severity 1 compared to 37% rated by the agency raters. The reverse occurred for Severity level 2 as the baseline rater measured 39% of the cracks at Severity 2 while the agency raters measured 26%. Summing the percentages of Severity 1 and 2 yielded much more similar results between the baseline rater and the agency raters (68% versus 63%, respectively). The percentages of Severity level 3 were much closer than levels 1 or 2, and it appeared that more consistency between raters is observed at higher severity levels, especially levels 4 and 5. This result is reasonable since crack widths at low severity are small and more difficult to measure/rate than larger crack widths. Almost no difference in total extent was seen for sealed cracks, which is also expected since sealed cracks should be much more easily identified than individual severity levels. Agreement analysis at this highest level of data aggregation resulted in a kappa statistic of 0.67, indicating substantial agreement, but much lower than observed for crack type. 13

21 Wheel 24% Block 0% Centreline 23% Transverse 27% Midlane 8% Meander 0% Edge 18% Figure 1: Distribution of Total Crack Extent by Crack Type ( Rating) Wheel 26% Block 1% Centreline 19% Edge 17% Transverse 27% Midlane 8% Meander 2% Figure 2: Distribution of Total Crack Extent by Crack Type (Agency Rating) 14

22 Severity 3 19% Severity 5 Severity 4 Sealed 1% 6% 6% Severity 1 29% Severity 2 39% Figure 3: Distribution of Total Crack Extent by Crack Severity ( Rating) Severity 3 25% Severity 4 5% Severity 5 1% Sealed 6% Severity 1 37% Severity 2 26% Figure 4: Distribution of Total Crack Extent by Crack Severity (Agency Rating) 15

23 3.4.3 Agreement by Crack Type Agreement analyses by individual crack type were completed to compare the total extent of each type measured by the baseline rater and each of the 7 agency raters/pairs. The actual analyses are included in Appendix D for reference while the resulting kappa statistics are summarized in Table 13. Associated descriptions of agreement strength are also provided. Table 13: Agreement Analyses for Crack Type Crack Type Kappa Statistic Strength of Agreement Block 0.00 Chance Centreline 0.88 Almost Perfect Edge 0.95 Almost Perfect Meander 0.37 Fair Midlane 0.70 Substantial Transverse 0.90 Almost Perfect Wheel Track 0.96 Almost Perfect As shown, the agreement between the baseline rater and the agency raters for total extent by crack type was high for all crack types with the exception of Block and Meander cracking. The highest agreement was observed with Wheel Track cracking, followed closely by Edge, Transverse, Centreline and Midlane. The low agreement for Meander and Block cracking was not surprising, since such cracks are more ambiguous and could easily be rated as another type. For example, block cracking may be rated as a combination of transverse and longitudinal cracking Agreement by Severity Level Another set of agreement analyses were completed to compare the total extent of each severity level measured by the baseline rater and each of the 7 agency raters/pairs. The analyses are presented in Appendix E and the resulting kappa statistics are displayed in Table 14, again with the associated description of agreement strength. Table 14: Agreement Analyses for Severity Level Severity Level Kappa Statistic Strength of Agreement Sealed 0.88 Almost Perfect Severity Substantial Severity Almost Perfect Severity Moderate Severity Substantial Severity Slight 16

24 Not surprisingly, the kappa statistics for severity level were lower than those observed for crack type. Sealed cracks were most readily agreed upon followed by a general decrease in agreement with increasing severity, although the low kappa statistic of 0.13 for Severity 5 was likely due to the small amount of cracking measured (52.8 metres vs metres in total for the baseline and agency raters, respectively) rather than simply by disagreement alone. This observation is in general disagreement with the US-LTPP study, although the analysis techniques were different for the two investigations. A further analysis was completed for the summation of Severity Levels 1 and 2, yielding a kappa statistic of 0.89, a significant improvement over the individual levels Agreement by Crack Type and Severity Level As mentioned, the statistics displayed in Tables 13 and 14 were the result of comparisons between the agency and baseline raters for total extent of cracking measured (first by crack type and then by severity level). Therefore, it was expected that relatively high agreement would be observed since disagreement at this level of data aggregation can be averaged out. Table 15 displays the results of analyses comparing the agreement for each crack type based on individual severity level (as opposed to total extent), for each section group. As shown, the values of the kappa statistic greatly decreased, representing the decreased amount of agreement for both crack type and severity level at the same time. Based on the results of Table 15, the greatest average kappa statistics were observed with Transverse and Wheel Track cracking (0.21 and 0.20, respectively), followed by Edge, Centreline, Meander, Midlane and Block. Individual agreement indices varied considerably, ranging from 0.30 to 0.68, with many zero values, indicating chance agreement with the baseline rater. Table 15: Agreement Analyses of Crack Type by Severity Level for each Section Group Kappa Statistics Section Wheel Group Block Edge Midlane Meander Centreline Transverse Average Track Group Group Group Group Group Group Group Average Ranking of Cracking Type As displayed in Table 16, the actual rank of crack type based on agreement is dependent upon what quantities were used in the analysis. For example, based on total extent (Table 13), Wheel Track Cracking was most readily agreed upon followed closely by Edge and Transverse cracking. However, based on crack type by severity level (Table 15), the order is changed 17

25 somewhat such that Edge cracking became one of the least agreed upon. As an exercise, the average kappa statistics of Table 13 and Table 15 were calculated and termed Overall statistics. As shown, the resulting order indicated that Wheel Track, Transverse and Centreline cracking were the most consistent, followed by Edge, Midlane, Meander and Block. Table 16: Rank of Cracking Distress based on Agreement Analyses Total Extent Extent by Severity Overall Crack Type Kappa Kappa Average Rank Rank (Table 13) (Table 15) Kappa Rank Block Centreline Edge Meander Midlane Transverse Wheel Track Agreement by Agency Rater/Pair and Experience Level Table 15 also displays the average kappa statistic for each section group. As shown, average values range from 0.00 (chance agreement) to 0.40 (fair agreement). While the results in Table 15 were not particularly promising, it was interesting to note that the differences between the kappa statistics for each section group (individual agency rater or pair) were much more consistent. Therefore, although the agency raters did not agree with the baseline rater to a high degree, it was hypothesized that the interagency rater variability may be lower than initially indicated in Table 15. Indeed, this hypothesis appeared to be supported by the results of analyses completed for each section group based on all cracking types and severity levels as shown in Table 17. While moderate and substantial overall agreement with the baseline rater was observed, the difference between the kappa statistics for each agency rater was quite low with the exception of Section Group 3. Unfortunately, it was impossible to examine this in more detail since all 35 sections were not rated by each of the 7 agency raters/pairs. Table 17: Agreement Analyses for Agency Raters/Pairs (Extent by Crack Type and Severity Level) Section Group Kappa Strength of No. of Statistic Agreement Raters Rater Experience Group Moderate 2 Both Inexperienced Group Moderate 2 1 Experienced, 1 Inexperienced Group Chance 1 Inexperienced Group Moderate 1 Inexperienced Group Moderate 1 Experienced Group Substantial 1 Inexperienced Group Substantial 1 Experienced 18

26 Furthermore, some interesting observations were made with regard to rater experience. Perhaps the most obvious concerned the low (chance) agreement between the agency and baseline raters for Section Group 3. As shown in Table 17, a single inexperienced agency rater completed the survey for these sections. Although it appeared that the low kappa statistic could be attributed largely to rater inexperience, the agency raters from Section Groups 4 and 6 were also inexperienced and yielded similar or higher agreement than the experienced raters. This finding was in general agreement with the Saskatchewan study. Another interesting observation concerned the consistency between agency raters/pairs and experience level. As shown, although Section Group 1 incorporated two inexperienced agency raters, they achieved the same kappa statistic as the single experienced rater from Section Group 4. This result seemed to confirm the positive effect of multiple raters that was observed with the US-LTPP study [FHWA 1998] although as mentioned above, the individual inexperienced raters for Section Groups 4 and 6 yielded similar or higher agreement. Therefore, the influence of rater experience upon agreement was not clearly apparent based on the results of this investigation Surface Distress Results with Respect to Overall Long Term Performance Considering the surface distress surveys of all raters, some interesting observations were made about the 35 sections in general. First, approximately two-thirds of all rated cracks have a severity rating of either slight (Severity 2) or very slight (Severity 1) after 7 or 8 years in service, indicating good overall performance. Furthermore, slightly more than one quarter of all cracks measured were transverse cracks, also known as thermal or low temperature cracks - a testament to Canada s cold winters. This result indicated that a significant amount of cracking might be reduced or even eliminated through better selection of asphalt binders for cold temperatures. The Superpave performance-graded binder specification therefore holds much promise as asphalt binders are selected to resist low temperature cracking for local environmental conditions. Such cracks may have also reflected from the original pavement. Furthermore, another one quarter of the cracking observed was wheel track cracking, a fatigue related distress caused by repetitive truck traffic loading. Improved pavement designs based on data from C-LTPP may yield improved performance at lower life cycle cost. Finally, approximately one fifth of the cracking measured was centreline cracking, most likely along the joint in the asphalt pavement between lanes. Improved jointing techniques such as the use of hot-jointing devices may reduce the frequency of this distress Sources of Error and Possible Improvements With the exception of more ambiguous crack types such as block and meander cracking, it was clear that variability between raters based on crack type is acceptably low. Therefore, efforts to reduce rater variability should be put toward more consistent rating of severity level. One method to improve consistency would be to reduce the number of severity levels perhaps from five to three. However, this alternative would likely compromise the ability to model crack growth since more time would be required to change severity levels. Another option would be to reduce the number of agency raters or even utilize as single distress rater to rate the all of the C-LTPP sections. However, based on the vast distance between sites and associated costs, this 19