EXPLORING AN ALTERNATIVE APPROACH TO irap STAR RATING VALIDATION. CDV Transport Research Centre, Líšeňská 33a, Brno, Czech Republic

Transcription

1 Ambros, Borsos, and Sipos EXPLORING AN ALTERNATIVE APPROACH TO irap STAR RATING VALIDATION Jiří Ambros, PhD (corresponding author) CDV Transport Research Centre, Líšeňská a, 00 Brno, Czech Republic Tel. +0 ; jiri.ambros@cdv.cz Attila Borsos, PhD Széchenyi István University, Egyetem tér, 0 Győr, Hungary Tel. + () -; borsosa@sze.hu Tibor Sipos KTI Institute for Transport Sciences, Thán Károly u. -, Budapest, Hungary Tel. + (0) -; sipost@kti.hu Submitted: August, 0 Revised: November, 0 0 Submitted for presentation at the th Annual Meeting of the Transportation Research Board and publication in Transportation Research Record. Total number of words:, (abstract, text, references) + figures/tables 0 =,

2 Ambros, Borsos, and Sipos ABSTRACT Star Rating protocol, applied by International Road Assessment Program (irap), awards to stars depending on the level of safety which is built-in to the road. Although this procedure is known and used world-wide, there have been only a few validation studies, showing the relationship between Star Ratings and crashes. In addition, the past studies relied on crash rates or crash costs per kilometer, derived from Police-reported data only. This approach is not the best, since it incorrectly assumes a linear relationship between crash frequency and the measure of exposure, and does not control for the confounding effect of regression to the mean. The objective of the paper is to fill the gap and conduct validation study using state-of-the-art empirical Bayes approach, combining observed crash frequency with expected crash frequency based on a crash prediction model. For the purpose of this exploratory study, Star Ratings from the Hungarian rural road network were used. The proposed approach proved to be feasible, with results confirming the relationship between increasing Star Ratings and decreasing crash frequencies. However, some limitations are also reported, which should be addressed by further and improved validation studies. Keywords: International Road Assessment Program, Star Rating, validation

3 Ambros, Borsos, and Sipos INTRODUCTION The International Road Assessment Program (irap) is a world-wide activity aimed at road safety improvement. irap activities include inspecting roads and developing four protocols: () Risk Maps, () Star Ratings, () Safer Roads Investment Plans (SRIPs), and () Performance Tracking. While Risk Maps and Performance Tracking are based primarily on crashes, Star Ratings and SRIPs are risk-based, using road inspection data and providing a simple measure of safety. One to five stars are awarded depending on the level of safety which is built-in to the road. The safest roads (- and -star) have attributes appropriate for the prevailing traffic speeds, while the opposite is true for the least safe roads (- and -star) (). Road traffic accidents are among the leading causes of fatalities with low- and middleincome countries (LMIC's) hardest hit with 0% of global road traffic deaths (). At the same time, LMIC's have not fully established crash databases reducing their ability to identify and measure road safety problems (). Therefore a risk-based approach, using non-crash measures, is an alternative for cost-effective and sustainable road safety improvement. This approach has been recognized not only in LMIC's as irap Star Ratings are used to set upgrade targets for national road safety policies in high-income countries as well,, such as Australia, UK and the Netherlands (). Since Star Rating is often used when there are no crash data available, it is reasonable to ask how well the Star Rating correlates to safety. To this end several validation studies have been conducted, which generally concluded that Star Ratings provide a valid measure of injury risk (for a summary see ). However there have still been critiques of Star Rating procedure, for example in Germany, where crash-based analyses were favored instead (). These contradictions from previous research work confirm (as indicated for example in ) that there is a need for more studies and different approaches. Previous studies from the US (), the Netherlands () and Austrailia() relied on indicators of fatal and serious crash rates per kilometer traveled, or one of its derivatives such as crash costs per kilometer traveled. However, crash rates per kilometer traveled are not a suitable indicator, since by definition they incorrectly assume a linear relationship between crash frequency and the degree of exposure ( ). In addition relying only on observed (Police-reported) crashes is not recommended, since it does not account for the confounding effect of regression to the mean, i.e. random fluctuations around long-term mean value ( ). For these reasons, an empirical Bayes approach, combining observed crash frequency with expected crash frequency according to crash prediction model (safety performance function), has been recommended. More information on previous related applications in Hungary can be found in. In order to better understand the possible disconnect between different irap validation methodologies, the authors pursued a new approach. As opposed to relying on biased indicators derived from rates of observed crashes, empirical Bayes estimates of expected crash frequencies were employed. The exploratory study used Star Ratings from about,0 km (00 mi) of the Hungarian rural road network. The following sections

4 Ambros, Borsos, and Sipos describe data collection and analysis methods (crash prediction model and empirical Bayes approach), results, discussion and conclusions. DATA COLLECTION AND ANALYSIS METHODS The study utilized Star Ratings collected in Hungary within SENSoR project (South East Neighbourhood Safe Routes). This international project was realized in 0 0 across Southern and Eastern European countries, including Hungary (for more information see SENSoR project partner for Hungary was KTI Institute for Transport Sciences Nonprofit Ltd. The Hungarian network survey contained selected motorways and national roads, totaling, km (approx., mi) see Figure. The road survey took place in February 0 and was conducted by a Czech company AF-CITYPLAN. The survey included data collection and processing. For more details, see the technical report (). 0 FIGURE Map of Hungary with irap-covered road network. From the collected and coded data, a Star Rating Score was calculated for each 0-meter (0.0-mile) road segment. These scores were then allocated to Star Rating bands to determine the Star Rating for each 0 meters of road. However such short segments often lead to variations in Star Ratings. For example Koorey () recommended grouping short segments with adjacent segments or at least not terminating them until they have reached a specified minimum length; Cook et al. () found a trend showing as segment length increased, so did the statistical reliability of the average annual crash frequency; and Cafiso et al. () compared several segmentation approaches and noted that the ones resulting in very short segments yielded poor models. Smoothing (averaging) into longer segments was thus applied (for details see ).

5 Ambros, Borsos, and Sipos 0 0 For the present study, KTI provided the irap upload file, containing all collected road attributes. In addition to Smoothed Star Ratings, as described above, the attributes included land use, speed limit, median type or lane width (for a complete list see ). For the purpose of exploratory study the following attributes were chosen: Carriageway label (divided or undivided road) Area type (rural or urban area) Speed limit ( values between 0 and 0 km/h) Vehicle flow (AADT) Using a geographical information system, the dataset was merged with geo-located injury crash data including fatal (fatality within 0 days as a result of the crash), serious (injury healing beyond days) and light (recover within days) injury crashes from -year period (0 0), which were provided by the Hungarian Transport Administration. Property damage crashes were not taken into consideration. During data preparation it was decided to reduce the dataset for two reasons:. Less than % of the surveyed network consisted of urban roads.. Crashes on undivided roads should be split between two carriageways (driving directions), which is not a trivial task, for example at intersections, where traffic streams from all driving directions interact. The final dataset was thus restricted only to rural undivided roads, which covers approx. 0% of the surveyed network. Data were combined in a pivot table with the following segment structure: Speed limit AADT Star Rating Length Crash frequency where lengths and crash frequencies were aggregated from 0-m sub-segments. For simplicity, speed limit values were re-categorized into two categories: () up to 0 km/h (% cases) and () between 0 and 0 km/h (% cases). The remaining cases (%) with speed limit over 0 km/h were discarded. The process resulted in 0 segments of total length approx.,0 km (00 mi). Descriptive characteristics (min., max., mean, standard deviation, frequencies, %) are reported in Table. TABLE Descriptive Characteristics of Variables Used for Crash Prediction Model Min. Max. Mean SD Star Rating Freq. % Crash frequency AADT [veh/day],00,0,.,. 0. Length [km] Total 0 0.0

6 Ambros, Borsos, and Sipos 0 Using this data, a crash prediction model (safety performance function) was developed. Consistent with the literature (for example 0), a model with exposure variables expressed as a power was adopted: exp exp _ _ () where is injury crash frequency;,, _ and _ are explanatory variables; are regression parameters to be estimated (calibrated) during modeling. Generalized linear modeling procedure in IBM SPSS was applied, using a negative binomial error structure with a logarithmic link function; explanatory variables AADT and Length thus took form of natural logarithms (ln and ln ). The linearized model form is shown in Eq. : ln ln ln _ _ () Parameters of the resulting model are reported in Table. TABLE Parameters of a Crash Prediction Model % Confidence Interval Parameter B SE Lower Upper Sig. (Intercept) ln_aadt ln_length Speed_Limit = Star_Rating (Overdispersion) Note: B regression parameters, SE standard errors, Sig. level of statistical significance. Characteristics of Speed_Limit = 0 are to be compared to reference value Speed_Limit = 0. AADT and length are highly significant predictors of crashes. In Table AADT has an exponent less than, which is consistent with international research results (the relationship between crash frequency and AADT is non-linear, i.e. shows a decreasing slope with increasing AADT) (e.g. ). Speed limit has a negative coefficient suggesting that in the second category with speed limits between 0 and 0 km/h crash frequencies tend to decrease. Compared to other predictors, star rating has a marginal coefficient, exerting minor influence on crash frequency; in addition it unexpectedly has a positive sign. Achieved significance levels were % and % for speed limit and Star Rating, respectively. For the purpose of the exploratory study the achieved values (around %) were accepted. The model (Eq. ) was used for prediction of mean crash frequency for each segment. In order to increase the precision of estimation and correct for the regression-to-mean bias, empirical Bayes adjustment was applied. Empirical Bayes (EB) estimate of expected crash frequency is calculated, using the weighted average of crash predictions ( ) and recorded crash history ( ). The weight depends on the strength of the crash record (how many crashes are to be expected), and on the reliability of the crash prediction model (how different is the

7 Ambros, Borsos, and Sipos safety of a specific site compared to the model average). The following formulas were applied (for further explanation see ): () () () where are computed EB estimates using weighted average (with weights ) of predicted and reported crash frequencies ( and ). The overdispersion parameter ( ) is computed as dependent on segment length ( ) (). RESULTS AND DISCUSSION Validation results are presented as bar graphs of EB averages for Star Rating categories, including % confidence interval ranges (Figure ). As expected, the graph shows decreasing EB average (i.e. increasing safety) with improving Star Rating. Since confidence intervals for - and -star roads are partially overlapping, differences between categories were confirmed at % confidence level by independent sample t-tests (): vs stars statistically significant difference, t() =., p = 0.00 vs stars statistically significant difference, t(0) =., p = vs stars statistically significant difference, t(0) =., p = FIGURE Graph of mean empirical Bayes (EB) estimates of expected crash frequency for individual Star Rating bands, with error bars indicating % confidence intervals. For the last category (-star roads) the confidence interval could not be calculated, because it contains only segment. This rarity was common also in other studies (, ).

8 Ambros, Borsos, and Sipos As mentioned in the introduction, previous Star Rating validation studies often used crash rate indicators. For illustration, crash rates were computed for the segments in the analyzed sample, according to the standard formula: () where is calculated using crash frequency, number of (i.e. years), and. The graph of mean of calculated crash rates for individual Star Ratings is in Figure. However, the results do not allow for sensible interpretation, because:. with increasing Star Rating, also crash rates are increasing, which is against logic, and. results for -, - and -star roads are almost identical, with fully overlapping confidence intervals. 0 FIGURE Graph of mean crash rates for individual Star Rating bands, with error bars indicating % confidence intervals. This example shows using crash rates can lead to mixed results. In contrast to the empirical Bayes approach which led to results that clearly confirm the valid relationship between Star Ratings and crashes. This confirms the validity of the EB approach to irap star rating. Nevertheless, the authors are aware of some limitations, including: Crash data aggregation. Validation studies should ideally consider individual Star Rating components, related to head-on, run-off and intersection crashes separately (). However in this study Star Rating was considered as a total score and linked to aggregated crash frequency, without distinguishing between crash types. Treatment of motorway data. As mentioned, for divided sections (motorways) crash frequency would need to be separated to individual directions; and AADT values should be directionally split. Such distinction was not done within the presented study.

9 Ambros, Borsos, and Sipos Data time frames. Ideally time frames of data used should be identical. This was not the case in the current study, since survey was done in 0, AADT was counted in 0, and crash data covered period 0 0. CONCLUSIONS The main goal of the exploratory study was to present an example of validation, using a stateof-the-art empirical Bayes technique for objective safety quantification. To the best of authorsʼ knowledge this approach has not been used for this purpose previously. Only a US validation study () attempted negative binomial regression analysis (i.e. crash prediction model, as used in this study), but concluded (without providing further details) that this modeling effort did not provide any meaningful relationships between crash frequency and star rating levels (p. ). In addition, previous studies have relied on crash rates, whose methodological insufficiency has been known for several decades. An illustrative comparison confirmed that using crash rates can lead to mixed results. In contrast the empirical Bayes approach led to results, which clearly confirm the relationship between increasing Star Ratings and decreasing crash frequencies. However, it should be noted that Star Rating as an explanatory variable in the crash prediction model turned out to have a minor influence and an unexpectedly positive relationship with crash frequency. In summary, the proposed evaluation approach proved to be feasible and should be followed by further and improved validation studies, taking into account the above mentioned limitations, mainly considering individual crash types and distinguishing between driving directions on divided roads. ACKNOWLEDGMENTS The authors are grateful to Gyula Orosz and Anita Veress-Szabó (Hungarian Transport Administration) for providing crash data. The study benefited from using research infrastructure of Transport R&D Centre (CZ..0/..00/0.00). REFERENCES. irap Methodology Fact Sheet #: Overview. irap, Basingstoke, 0.. Global status report on road safety 0. World Health Organization, Geneva, 0.. Road Safety Manual: A Guide for Practitioners. World Road Association, Paris, 0.. irap Star Rating Policy Targets Discussion Paper. irap, Basingstoke, 0.. Lawson, S. Crash rate Star Rating comparisons: Review of available evidence. Working Paper 0.. EuroRAP, Brussels, 0.. ESN vs. EuroRAP. Unfallforschung der Versicherer, Berlin. autobahn/landstrasse/verkehrssicherheitsarbeit/esn-vs-eurorap. Lawson, S. Validation how well do the irap models reflect crash costs or crash rates? Presented at irap Innovation Workshop, Washington, D.C., September, 0.. Harwood, D. W., K. M. Bauer, D. K. Gilmore, R. Souleyrette, and Z. N. Hans. Validation of U.S. Road Assessment Program Star Rating Protocol: Application to Safety Management of U.S. Roads. In Transportation Research Record: Journal of the

10 Ambros, Borsos, and Sipos Transportation Research Board, No., Transportation Research Board of the National Academies, Washington, D.C., 0, pp... Vlakveld, W. P., and W. J. R. Louwerse. The relation between Road Protection Scores (RPS) and the casualty rate at road sections in the province of Utrecht. Report R-0-. SWOV Institute for Road Safety Research, Leidschendam, 0.. McInerney, R., and M. Fletcher. Relationship between Star Ratings and crash cost per kilometre travelled: the Bruce Highway, Australia. Working Paper 0.0. irap, Basingstoke, 0.. Hauer, E. On exposure and accident rate. Traffic Engineering & Control, Vol.,, pp... Elvik, R., A. Høye, T. Vaa, and M. Sørensen. The Handbook of Road Safety Measures, Second Edition. Emerald, Bingley, 00.. Highway Safety Manual. First Edition. American Association of State Highway and Transportation Officials, Washington, D.C., 0.. Hauer, E. Observational Before-After Studies in Road Safety: Estimating the Effect of Highway and Traffic Engineering Measures on Road Safety. Pergamon Press, Oxford,.. Persaud, B. N. Statistical Methods in Highway Safety Analysis: A Synthesis of Highway Practice. NCHRP Synthesis. Transportation Research Board, Washington, D.C., 00.. Sipos, T., G. M. Gál, and I. Fütyü. Modeling the Traffic Accidents Occurrence Probability and the Efficiency of Transport Safety Interventions. In IEEE th International Conference on Intelligent Engineering Systems (INES 0), Lisbon, June, 0, pp... Borsos, A., J. N. Ivan, and G. Orosz. Development of safety performance functions for two-lane rural first-class main roads in Hungary. Presented at th Transport Research Arena, Paris, April, 0.. Borsos, A., and C. Koren. Accident prediction models for Hungarian two-lane rural main roads using geometry data. Presented at th International Symposium on Highway Geometric Design, Vancouver, B.C., June, 0.. Sipos, T., and K. Tánczos. Measuring Road Safety Improvement Potential for the Hungarian Road Network. In Modern Safety Technologies in Transportation (MOSATT 0), Zlatá Idka, September, 0, pp.. 0. Borsos, A., S. Cafiso, C. D Agostino, and D. Miletics. Comparison of Italian and Hungarian black spot ranking. Transportation Research Procedia, Vol., 0, pp... Ambros, J., J. Novák, A. Borsos, E. Hóz, M. Kieć, Š. Machciník, and R. Ondrejka. Central European comparative study of traffic safety on roundabouts. Transportation Research Procedia, Vol., 0, pp SENSoR Star Rating Report Hungary. SENSoR project consortium, 0. Koorey, G. Road Data Aggregation and Sectioning Considerations for Crash Analysis. In Transportation Research Record: Journal of the Transportation Research Board, No.

11 Ambros, Borsos, and Sipos 0 0 0, Transportation Research Board of the National Academies, Washington, D.C., 00, pp... Cook, D., R. Souleyrette, and J. Jackson. Effect of Road Segmentation on Highway Safety Analysis. Presented at 0 th Annual Meeting of the Transportation Research Board, Washington, D.C., January, 0.. Cafiso, S., C. D Agostino, and B. Persaud. Investigating the influence of segmentation in estimating safety performance functions for roadway sections. Presented at nd Annual Meeting of the Transportation Research Board, Washington, D.C., January, 0.. irap Methodology Fact Sheet #: Smoothed Star Ratings. irap, Basingstoke, 0.. irap Star Rating Upload File Specification. irap, Basingstoke, 0.. Walmsley, D. A., I. Summersgill, and C. Binch. Accidents on modern rural singlecarriageway trunk roads. Report. Transport Research Laboratory, Crowthorne,.. Reurings, M., and T. Janssen. Accident prediction models for urban and rural carriageways (based on data from The Hague region). Report R-00-. SWOV Institute for Road Safety Research, Leidschendam, Ambros, J., and J. Sedoník. A Feasibility Study for Developing a Transferable Accident Prediction Model for Czech Regions. Transportation Research Procedia, Vol., 0, pp Hadi, M. A., J. Aruldhas, L.-F. Chow, and J. A. Wattleworth. Estimating Safety Effects of Cross-Section Design for Various Highway Types Using Negative Binomial Regression. In Transportation Research Record: Journal of the Transportation Research Board, No. 00, Transportation Research Board of the National Academies, Washington, D.C.,, pp... Cafiso, S., A. Di Graziano, G. Di Silvestro, G. La Cava, and B. Persaud. Development of comprehensive accident models for two-lane rural highways using exposure, geometry, consistency and context variables. Accident Analysis and Prevention, Vol., 0, pp... Qin, X., J. N. Ivan, and N. Ravishanker. Selecting exposure measures in crash rate prediction for two-lane highway segments. Accident Analysis and Prevention, Vol., 00, pp... Hauer, E., D. W. Harwood, F. M. Council, and M. S. Griffith. Estimating Safety by the Empirical Bayes Method A Tutorial. In Transportation Research Record: Journal of the Transportation Research Board, No., Transportation Research Board of the National Academies, Washington, D.C., 00, pp... Hauer, E. Overdispersion in modelling accidents on road sections and in Empirical Bayes estimation. Accident Analysis and Prevention, Vol., 00, pp. 0.. Field, A. Discovering statistics using IBM SPSS Statistics, th Edition. SAGE, Thousand Oaks, 0.. Harwood, D. W., D. K. Gilmore, K. M. Bauer, R. Souleyrette, and Z. N. Hans. U.S. Road Assessment Program (usrap) Pilot Program Phase III. Final Report. AAA Foundation for Traffic Safety, Washington, D.C., 0.