Examining the Effects of Site Selection Criteria for Evaluating the Effectiveness of Traffic Safety Countermeasures

Transcription

1 Exaining the Effects of Site Selection Criteria for Evaluating the Effectiveness of Traffic Safety Countereasures Doinique Lord* Associate Professor and Zachry Developent Professor I Zachry Departent of Civil Engineering Texas A&M University 336 TAMU College Station, TX Tel. (979) Fax. (979) Eail: d-lord@tau.edu Pei-Fen Kuo Research Assistant Zachry Departent of Civil Engineering Texas A&M University 336 TAMU College Station, TX Tel. (979) Fax. (979) Eail: p-kuo@ttiail.tau.edu Noveber 0, 0 Revised Version Paper accepted for publication in AA&P *Corresponding Author

2 ABSTRACT The priary objective of this paper is to describe how site selection effects can influence the safety effectiveness of treatents. More specifically, the goal is to quantify the bias for the safety effectiveness of a treatent as a function of different entry criteria as well as other factors associated with crash data, and propose a new ethod to iniize this bias when a control group is not available. The study objective was accoplished using siulated data. The proposed ethod docuented in this paper was copared to the four ost coon types of before-after studies: the Naïve, using a control group (CG), the epirical Bayes (EB) ethod based on the ethod of oent (EB MM ), and the EB ethod based on a control group (EB CG ). Five scenarios were exained: a direct coparison of the ethods, different dispersion paraeter values of the Negative Binoial odel, different saple sizes, different values of the index of safety effectiveness ( ), and different levels of uncertainty associated with the index. Based on the siulated scenarios (also supported theoretically), the study results showed that higher entry criteria, larger values of the safety effectiveness, and saller dispersion paraeter values will cause a larger selection bias. Furtherore, aong all ethods evaluated, the Naïve and the EB MM ethods are both significantly affected by the selection bias. Using a control group, or the EB CG, can utually eliinate the site selection bias, as long as the characteristics of the control group (truncated data for the CG ethod or the non-truncated saple population for the EB CG ethod) are exactly the sae as for the treatent group. In practice, finding datasets for the control group with the exact sae characteristics as for the treatent group ay not always be feasible. To overcoe this proble, the ethod proposed in this study can be used to adjust the naïve estiator of the index of safety effectiveness, even when the ean and dispersion paraeter are not properly estiated. Key words: regression-to-the-ean, biased estiate, safety index, site selection effects, before-after study, Epirical Bayes Method

3 INTRODUCTION Evaluating the effects of an intervention or a countereasure on the nuber and severity of crashes is a very iportant topic in highway safety. In fact, this topic has been researched thoroughly over the last 30 years (Hauer, 980a, 980b, 997; Abbess et al., 98; Hauel et al., 983; Danielsson, 986; Wright et al., 988; Davis, 000; Miranda- Moreno et al., 009; Maher and Mountain, 009; Rock, 995; Haed et al., 999), where researchers have developed and applied various ethods to iniize known biases associated with crash data. Developing precise and reliable ethods to evaluate countereasure effectiveness is crucial, since erroneously easuring the safety effects could have iportant consequences both in ters of lives saved and wasted funds. Over the years, we have seen a variety of ethods that have been proposed for evaluating safety interventions. They include the Naïve before-after study, the before-after study with control group, the before-after study using the epirical Bayes (EB) ethod, and ore recently the before-after study using the full Bayes approach (Hauer and Persaud, 984; Hauer, 997; Persaud and Lyon, 007; Li et al., 008; Park et al., 00). As an alternative to the before-after study, soe people have suggested using a cross-sectional study (usually via a regression odel) (Tarko et al., 998; Noland, 003). However, the forer ethods are still considered the ost appropriate ethodology by ost researchers, since it can directly account for changes that occurred at the sites investigated (Hauer, 997). One of the ost iportant biases that have been docuented in the literature which negatively influence the evaluation of treatents is the regression-to-the-ean (RTM). The RTM dictates that when observations characterized by very high (or low) values in a given tie period and for a specific site (or several sites) ( N before ), it is anticipated that observations occurring in a subsequent tie period ( N ), are ore likely to regress towards the long-ter ean of a site ( N ) (Hauer and Persaud, 983) (see Figure ). Not including it could over-estiate the effects of the treatent (see, e.g., Persaud, 00). Although a large body of work has been devoted to the RTM, very few studies have exained the selection bias on the effects of a treatent, at least analyzing it as a distinct bias (see, e.g., Hauer, 980a, b; Davis, 000). As discussed by Cook and Wei (00), Davis (000) and ore recently by Park and Lord (00), the site selection effects and RTM are distinct biases and influence the overall effectiveness of a treatent differently. Cook and Wei (00) indicated that not including the site-selection bias could overestiate the effects of a treatent and under-estiate the variance associated with the before-after study (which deterines whether the treatent is statistically significant or not). after

4 N before Average Accidents per Site N N after Figure. Graphical representation of the Regression-to-the-ean phenoena. The priary objective of this research was to describe how site selection effects can influence the evaluation of treatents. More specifically, the goal is to quantify the bias for the safety effectiveness of a treatent as a function of different entry criteria and other factors associated with crash data and propose a new ethod to iniize this bias when a control group is not available. The study objective was accoplished using siulated data (but is supported by theoretical derivations docuented in Appendix A). The bias was estiated for the four ost coon before-after ethods: Naïve, using a control group (CG) and for the EB ethod estiated using the ethod of oent (EB MM ) and EB ethod estiated using a control group (EB CG ). For the EB CG, a NB regression odel is usually estiated fro data collected fro the control group (Hauer, 997). For this study, the characteristics of the control group were directly used for estiating the EB CG value. Five scenarios were exained: a direct coparison of the ethods, different dispersion paraeter values of the NB odel, different saple sizes, different values of the index of safety effectiveness ( ), and different levels of uncertainty associated with the index (i.e., standard deviation). BACKGROUND This section provides relevant background related to the RTM and site-selection biases. REGRESSION-TO-THE-MEAN Tie The RTM was first noted ore than a century ago by Francis Galton (Stingler, 997). At the tie, Galton reported that tall parents would produce children which were, on average, Although theoretical equations to estiate the site-selection bias are available, siulation is needed for three reasons: () the theoretical derivations proposed by Cook and Wei (00) had to be odified because it included several typos, and the siulation results are used to verify the accuracy of these odified equations; () the theoretical derivations assued that the saple is infinite, which does not reflect how these equations would be applied using observed data; (3) and, siulation was also utilized to exaine the effects of using dissiilar control groups and the accuracy of the bias-estiation equations when the paraeters are unknown. All these topics are discussed in greater details in the paper.

5 shorter than either parent (Chuang-Stein and Tong, 997). Intrigued by this phenoenon, he further exained this attribute using peas and found the sae outcoe as what he noticed in huans. At the tie, he did not copletely grasp the atheatical relationship associated with this phenoenon, but he clearly understood the effect associated with the RTM. It was not for another 0 years that soe atheaticians were able to explain it using atheatical principles (Stingler, 997). In order to explain how the RTM effects work on traffic safety, let us assue that Y (first easureent of speed or reaction tie of driver i, etc.) and Y (second easureent of speed, reaction tie of the sae driver i ) are two rando variables with alost exactly the sae distribution, but the conditional expectation E Y Y is not equal to Y (Copas, 997). The conditional expectation is however very close to the overall ean (i.e., longter ean). It can be shown that the conditional expectation can be defined as a jointly noral distribution (Copas, 997): EY Y Y () Where is the correlation between Y and Y and is the coon ean (recall that COV Y, Y ). Equation () shows that the RTM effect is a function of the Y, Y correlation. When the correlation is equal to, there is no RTM, since E Y Y Y. On the other hand, when the correlation is not equal to, we observe the presence of the RTM. Equation () shows that saller values of is associated with larger RTM effects. Using the characteristics of Equation (), the agnitude of the RTM can be coputed by taking the difference between E Y Y and Y, i.e. EY Y Y Y. This can be represented graphically, as shown in Figure. Equation () also shows that the RTM will always exist as long as the correlation is iperfect, regardless how large or Y E Y Y ). sall the value of the observations is (unless 3

6 E Y Y and Y. Figure. Relationship between The discussion above focused on data that follow a noral distribution. With the exception of speed data (see Park and Lord, 00; Kuo and Lord, 0), ost of the data used in highway safety are classified as discrete nonnegative events (e.g., the nuber of crashes per unit of tie). Cook and Wei (00) developed an approach for analyzing discrete data that is analogous to one used for norally distributed data described above. Using their notation, let N i denote the count of events during before period and N i denote the nuber of observed counts for the after period for subject i, i,, and tie period k, k for the before period and k for the after period. Assuing a subject-specific rando effect ui 0, the count N ik for the i th subject is generated fro a Poisson distribution with a ean u i k, where k k, k is defined as the rate of events and is the tie period that is assued to be equal in the before and after periods (note: if they are unequal, an offset could be used for either periods). If ui is generated fro a distribution with a ean equal to and a variance equal to, then ENik k and Var Nik k, for k, and cov N k i, Ni. If the ean i follows a one-paraeter gaa (, ),the arginal and joint probability ass function (pf) can be obtained as follows and P N ik N N ik ik k N! k ik k k, P N, N,,, for k, () Ni Ni Ni Ni i i Ni! Ni! (3) i i N N 4

7 The odel in Equation (3) can be solved via the axiu likelihood ethod. The steps used for solving this equation can be found in Cook and Wei (00). Equation (3) can be used for analyzing the effects of an intervention, as long as an entering criterion is not included in the study. This study used a one-paraeter gaa distribution (instead of a two-paraeter gaa) to ensure that all odel paraeters are identifiable (Miaou et al., 003). It should be noted that using a one-paraeter dispersion paraeter did not liit the applications described in this study. If an entry criterion is eployed, further anipulations are needed and soe of the are discussed in the next section. The odel described in Equation (3) is very siilar to the approach proposed by Hauer (997), in which the RTM and the evaluation of the treatent are estiated via the EB ethod that is itself based on a Poisson-gaa odel. The difference here is that the odel proposed in Equation (3) is odeled as a joint distribution. The EB ethod, on the other hand, uses a ulti-step process. SITE SELECTION EFFECTS The general idea of site selection effects is that setting entry criteria will transfer the original population distribution to a truncated saple distribution resulting in changes in the characteristics of the new distribution, such as the ean and variance/dispersion paraeter. Hence, ignoring these changes will create a biased estiator for the safety effectiveness (Figure 3), the preise being that setting a higher entry criterion will cause higher site selection effects (discussed further below). 5

8 Figure 3. The population distribution for coplete and truncated saple. When selecting sites for treatent, transportation agencies can select these sites based on a required iniu the nuber of crashes. For instance, Warrant 7 that is used for justifying the installation of traffic signals base solely on safety in the Manual on Unifor Traffic Control Device (MUTCD) states that a site ust experience five or ore failure-to-yield crashes within a -onth period (FHWA, 00). Hence, when such traffic signals are evaluated to address a safety proble, the saple should theoretically be extracted fro sites that et Warrant 7 of the MUTCD. Setting a iniu entry criterion is not liited to practitioners, as researchers have also used entry criteria for evaluating safety projects (Miranda-Moreno et al., 009). Furtherore, in any other cases, the selection effect exists but is never explicitly spelled out, because the safety evaluation of a treatent often includes sites that are greater than 0, even if the sites selected for treatent were not chosen specifically for safety reasons. When the entry criterion is sall, say when all the sites experience at least one crash (N i > 0), there is a bias, as explained further below. Unlike the extensive discussions related to the RTM, estiating the agnitude of the site selection effects directly by their entry criteria values is a relatively new topic in traffic safety. As discussed above, selection bias has been sporadically studied over the last thirty years ago (Hauer, 980a,b; Abbess et al., 98; Davis, 000), but has not been 6

9 explicitly incorporated into the evaluation of treatents, at least not as a distinct effect as the one attributed to the RTM, with one exception. Using the work of Rubin (977) and Pearl (009) aong others, Davis (000) discussed how site selection effects could influence the estiation of a treatent by assigning probabilities to sites that have zero or ore than zero observations (). The priary ethod proposed by Davis does not specifically include an entry criterion variable for different values, which eans that the bias cannot be directly estiated by the value of entry criterion, as it is done in this paper. Furtherore, the estiator based on Rubin s work requires extra inforation, such as the need for collecting data for the control group, which is not needed for the approach proposed in this paper (although it ay be useful to collect and include such data for iproving the estiates of the safety effects of a treatent). The use of entry criteria in before-after studies, however, has been coonly adopted in other areas, especially in edical studies (e.g., clinical trials). Cook and Wei (00), for instance, discussed the possible ipacts of the selection effects for testing new edicines in clinical studies and derived the atheatical equations to account for the bias created by setting an entry criterion. They developed the ethodology for data that follow a noral distribution and for count data. Their ost iportant findings included the following:. Norally distributed responses: when scientists set the entry criteria for choosing experient subjects, the original unbiased estiators of treatent effectiveness becoe biased. Even when there is no any relationship between the response in the before and after periods, 0, the analysis can be biased. Furtherore, when the treatent does not work, using the naïve before-after ethod ay lead to a positive estiate of treatent effectiveness, especially for low value responses. The selection bias will exist until the correlation (ρ) is or the entry criteria (C) tend to - (see Equation (4)). However, when a control group with the exact sae characteristics is used in the before-after study, the biases related to the RTM and site selection will cancel out. f d Fd E Y C, i,, (4) Where, Y Y is an unbiased estiator of ; Y i and Y i are the response variables before ( k ) and after ( k ) the treatent, for subject, i,, ; Yik k ui ik with ~ 0, ik ~ N 0, ; ui N and Pearl (009) derived useful boundary equations for describing possible casual effects for the evaluation of treatents. However, there are several liitations about applying Pearl s equations in traffic safety. They include () the boundaries only show the probability of that the event happened instead of actual ipact value, () the boundaries are too wide to copare treatents, (3) the definitions of event that happened are subjective, and (4) very detailed inforation are necessary to copute the effects of a treatent. Due to space constraints, these issues are not discussed here, but the curious reader is referred to Kuo (0) for additional discussion about these liitations. 7

10 Y Y is an unbiased estiate of k k k i ik correlation between tie period k and f d and Fd Fd distribution., 0 is the ; k ; d c ; and, are the density and CDF is the standard noral. Count data responses: The results are analogous as for the Noral Distribution response. The bias for estiating the treatent perforance increased when setting higher entry criteria, but the bias always exist until the entry criteria is less than 0 (C<0). Details about the atheatical equations are presented in the next section. Table suarizes the atheatical equations used for quantifying the RTM and siteselection effects (i.e., data are left-truncated). It should be noted that the probability of the data being characterized by the site-selection effects in the after period is equal to the truncated noral distribution ultiplied by the conditional noral distribution of the after period (Cook and Wei, 00). Table. Equations Describing Site-Selection and RTM Effects. Effects Before After Site selection P( Y Yi C), i : site i PY ( Yi C) PY ( Yi) PY ( i C) Regression-to-the-Mean PY ( Y), Y PY ( Y), Y i i i i In su, Cook and Wei (00) showed that setting higher entry criteria ay cause a larger site selection bias for estiating the perforance of the treatent and using a naïve before-after approach tends to overestiate the treatent perforance when the response nuber is low. The overall indication is that site-selection effects can play an integral role in iproving traffic safety, as the distributions of crash frequency are often characterized by a low saple ean (Lord and Mannering, 00). It is iportant to note, however, that Cook and Wei s (00) proposed ethodology is not without its flaws. There are three ajor probles that need to be resolved before the ethodology proposed by Cook and Wei (00) can be utilized in traffic safety. First, there are several typographic errors in the bias-estiation equations related to the effectiveness and variance, which needed to be corrected for this study. Second, the calculation based on the Control Group ethod for count data in edical studies is different fro those used in traffic safety studies. In edicine, the treatent and control groups have siilar characteristics with the exception of the variable being investigated (i.e., effect of a drug). As a results, Cook and Wei (00) did not consider the dissiilar characteristics for the control group a very coon characteristic found in safety studies nor did they exaine the accuracy of their bias equation when paraeters (e.g., 8

11 ,, ) are unknown. To these ends, the following sections show the updated equation for estiating the bias caused by the setting an entry criterion. COMPUTATION OF SITE SELECTION BIAS This section describes the ethodology used for estiating the selection biased for count data. As discussed above, soe of the equations shown further below have been revised, since the original ones described in the paper by Cook and Wei (00) contained several typographical errors related to the estiators of paraeters. Thus, to ensure a coplete description, all the equations are reproduced in this paper. ESTIMATOR OF θ WITH ENTRY CRITERIA Suppose that site i (i=,,, ) experience Ni crashes during the before period (tie length = t ) and Ni crashes in the after period (tie length = t ). Let N ik follow the Poisson distribution ( Nik ~ Poisson ( u i k )), where is a subject-specific rando effect, and is the average crash rate ( k i tk). The ter refers to the instant rate of crash. Let, often defined as the index of safety effectiveness (Persaud and Lyon, 007), and, the difference in the nuber of crashes; for this paper, we will only focus on. Equation (5) describes the difference between the estiator and the true value for the easure of effectiveness. This equation was slightly odified to account for a typo found in the original equation proposed by Cook and Wei (00): ˆ ( ) li E( Ni c, i,, ) (5) ( ) Equation (5) can be siplified by dividing both the denoinator and nuerator by µ : li E ˆ N c, i,, (6) i c ( ) ( ) With Equation (6), it becoes ore obvious that setting higher criteria will cause a larger bias because the truncated expected value (µ ) increases. Larger values of the index ( ) will also increase the bias. It should be noted that µ is the function of the entry criteria (C), the dispersion paraeter ( ), and ean response rate in the before period ( ). Equation (6) is the foundation behind the new ethod proposed in this study. Siulated data will be used to confir this finding further below, but theoretical derivations that support the siulated that can be found in Appendix A. 9

12 ESTIMATOR OF θ WITH ENTRY CRITERIA AND CONTROL GROUP Using a siilar approach as the one described in the section above, only one superscript needs to be added to distinguish the coparison fro the control group. Let T stands for the treatent group, while C is used for the control group. All other conditions and assuptions will be sae as the section above. Hence, the responses still follow a T T C C Poisson distribution ( Nik ~ Poission( ui k ), Nik ~ Poission( ui k ) ) with u i which follows the gaa distribution ( T T C C, ). k k k and k k k are the population average count for k th TC, period. The estiator of average crash rate ( TC, ) is N / where is the nuber of subjects in each group. Following the sae steps, the estiator of becoes (adapted fro Cook and Wei, 00): k i ik T T T T T T ˆ true C C C C C C (7) T C T C Let use the optial solution where and, then Equation (7) can be siplified as follows: true T T ˆ 0 C C (8) Based on Equation (8), it can be assued that using a control group ay ake the T C estiator of to becoe asyptotically unbiased only when and T C. These relationships ean that the treatent and control groups ust have the exact sae characteristics: saple ean and dispersion paraeter. However, for crash data, it is very rare that the saple ean will be the sae between the treatent and control groups, since if they are, the sites in the control group should also be treated. The sae applies for the dispersion paraeter (need to be the sae for both groups) (this actually applies to the EB ethod). 0

13 BOUNDARY CONDITIONS FOR As discussed above, the RTM and site selection effects create two different biases. Siilar to continuous data, the RTM does not exist when the correlation ( ) between N and N is equal to one. In this case, either or. When this happens, all the bias associated with estiating disappears However, in practice, observing perfect correlation where the RTM is equal to zero rarely exists or ever happens. The site selection bias on will becoe zero when, as shown in Equation (6). In other words, site selection bias will always exist until C<0, even when 0 (the responses in the before and after period are independent or 0 ). This result is consistent with Davis (000) findings (see Eq. 6 in his paper), but only when the selection effect or entry criterion is equal to C<0. Thus, when C is a nonnegative integer (0,,, ), the site selection bias will constantly be present. This also eans that for crash data, this bias of will exist even when sites with a iniu of crash (as long as ) are included. It should be pointed out that for count data, the site selection effect will cause different biases for different estiators:,,, and, as pointed out earlier. Soe of these will be discussed in subsequent publications or can also be found in Kuo (0). METHODOLOGY AND SIMULATION PROTOCOL This section describes the ethodology and siulation protocol used for estiating the bias for the following scenarios. Appendix B presents a glossary of the ters used in this paper. SCENARIO ANALYSIS Five scenarios were used to exaine possible factors related to site-selection bias. Scenario copared the four ost coon types of before-after studies: the Naïve ethod, using a control group (CG), the EB ethod based on the ethod of oent (EB MM ), and the EB ethod based on a control group (EB CG ). Then, these were copared with the new proposed ethod to adjust the naïve estiator. Scenario exained different values for the dispersion paraeter: fro 0.5 (sall heterogeneity) to 7.0 (very large heterogeneity). Scenario 3 studied the effects of saple size on the bias: 0 (sall), 30 (ediu), and 00 (large). More specifically, this scenario assesses whether the estiator, based on Equation (6), is influenced when the saple size does not tend towards infinity. Scenario 4 analyzed the agnitude associated with the safety effectiveness for three values: 0.90 (high), 0.70 (ediu), and 0.50 (low). Finally, Scenario 5 looked at the effects related to the standard deviation of the index of safety effectiveness on the bias: 0.05 (sall), 0.0 (ediu), and 0.0 (large). For all scenarios, the entry criteria were assued to vary fro (i.e., i N ) to. The data generation procedure followed a Poisson-gaa distribution to generate the

14 ean response rate and observed nuber of counts. (i= to ) observations were selected randoly for the treatent group only when the response was larger than the t T T entry criteria ( N j ij C). These sites are labeled as N ij and N ij. The saple size was equal to 00 and the safety effectiveness was equal to 0.50 unless the scenarios required these two variables to change. The safety effectiveness for Scenario 5 was equal to 0.90, in order to avoid having equal to zero. Note that Scenarios and were analyzed siultaneously. The equations for the four ethods and the one proposed in this study are as follows: Naïve ethod: ˆ CG ethod: ˆ t t i j n n t T N ij Nij C N i j i j ij n n t T ( N ij C) N i j ij n n t ˆ T i N j ij i i n n ˆ c n n C ˆ t t N i T ij i ˆ c N j ij j C i i i i i Nij Epirical Bayes ethod: ˆ n t i j n t i j n t i j N M Var n t i j T ij T ij M M T ij T ij (9) (0) () Proposed Adjustent ethod (based on Equation (6) and the estiators fro Geyer (007)): ˆ ˆ adjusted naive ( ) ˆ naive C PN ( C) ( ) PN ( C) ˆ C ˆ PN ( C) ( ˆ ) ( ) PN ( C) ()

15 n t Where, ˆ T N (3) n i j ij E(( Ni ) Ni C) EN ( i Ni C) EN ( N C) i i (4) Note: The index for all the ethods were not adjusted for the saple size by the approach proposed by Hauer (997). SIMULATION PROTOCOL The siulated data was generated using the software R (R Developent Core Tea, 006). The general steps were as follows: ) For each dispersion paraeter, generate the subject-specific rando effect ( u i ), which follows a gaa distribution gaa ~, and generate a crash count with a Poisson ean ( Ni ~ Poissoni ui i ) for each site i, where i, the saple ean, was equal to. Saple ean values equal to 3, 5 and 0 were also tested, but the results are not presented here due to space constraints. All the results were consistent with the values presented in this paper, expect at the boundary when is alost equal to zero (to be discussed in another paper). ) Generate three years of counts in the before period using i for each site. Generate the data for = 5,000 sites, but randoly select 00, 30 and 0 sites depending on the scenario. 3) Only for Scenario 5: Generate the treatent effectiveness for each site using a noral distribution, ~ N,, where = 0.90 and = 0.05, 0.0 or 0.0. (note: the index could theoretically show an increase) 4) Estiate the crash ean rate ( ˆ i ) for each site in after period equal to the product of the above atrixes ( ˆ i ). Then, generate three years of count using the Poisson distribution, Ni ~ Poisson i. 5) Then, n sites are selected as the saple whose observed crash nubers are larger than the entry criteria (0,, 5), and its effectiveness can be estiated using Equations (9), (0), (), or (). 6) When the control group is used, θ is equal to. In other words, there is no different for ean rate between before and after period. 7) Repeat steps to 6 for a total of,000 ties, and estiate the various biased estiates ˆ

16 SIMULATION RESULTS This section describes the results based on the siulation. They are presented for each scenario separately. SCENARIO - RESULTS Figure 4 shows the site selection bias for the Naïve, CG, EB MM,EB CG and the Adjusted ethods. Overall, this figure shows that the bias goes down as the dispersion paraeter increases, except when is alost equal to zero (at least for C < 3) (recall that if C 0, Ni, etc.). This was expected given the characteristics of Equation (6). The greater the entry criteria, ore biased the estiate will be. Aong the four ethods, the Naïve (Figure 4a) and the EB MM (Figure 4c) ethods are the ones that are the ost affected by the site selection bias; can be over-estiated by as uch as 36%. As discussed by Cook and Wei (00), unless the RTM is copletely non-existent ( ), will be biased if an entry criteria is used (e.g., the bias never equals zero when 7 ). Readers ay be surprised to see that the EB MM ethod does not reduce or eliinate the RTM when site selection effects are included. This is caused by the fact that the MM estiate is calculated using the characteristics of the truncated saple rather than the full population or non-truncated saple. Appendix A describes in greater details the conditions when the EB ethod (both for EB MM and EB CG ) can be biased. When a control group is used, the bias can be theoretically eliinated. For the CG ethod (Figure 4b), the control group needs to have the sae characteristics, i.e., the sae saple ean and variance (which can be used for obtaining the dispersion paraeter) as the truncated saple used for the Naïve ethod (see right-hand side of Figure 3). As explained above, it ay be difficult to find datasets with the exact sae characteristics as for the treatent group in practice. For the EB CG ethod (Figure 4d), the control group needs to have the sae characteristics as the full saple (or saple population) fro which the truncated data were used for calculating the Naïve or EB MM estiates (see left-hand side of Figure 3). Again, the reader is referred to Appendix A for the conditions when the EB CG can be biased. The application of the control group is further explored in Figure 5. Figure 5a is the sae as Figure 4b, and is used to copare the results with the other figure below. Figure 5b shows that when the control group does not have the sae characteristics, in this case the sae saple ean, the site selection bias is still present, although it is still saller than using the Naïve before-after ethod. Furtherore, the bias is also in the opposite direction (under-estiate ). It should be noted that the estiator based on Equation (6) can estiate the site-selection bias accurately when the true value of the ean and variance are known. Even when a control group is not available, the siulation results show that the estiator can still reduce the bias by approxiately 50% siply by utilizing the truncated saple ean and variance, as shown in Figure 4e and Equation (). This characteristic has been validated using observed data, as docuented in Kuo (0). 4

17 Bias Bias (a) Naïve Dispersion paraeter (b) CG Dispersion paraeter C= C= Bias (c) EB MM C= Dispersion paraeter (d) EB CG Bias Dispersion Paraeter C= Bias (e) Adjustent Method C= Dispersion Paraeter Figure 4. Site selection bias for the Naïve, CG, EB and Adjusted ethods. 5

18 (a) CG Bias Dispersion paraeter C= (b) CG Bias Dispersion paraeter C= Figure 5. Site selection bias for the CG ethod for the following characteristics: (a) sae ean, (b) 0.75 ean. SCENARIO RESULTS As discussed in the first scenario, the site selection bias decreases when the dispersion paraeter increases, but the rate at which the bias decreases becoes alost flat for values above 5 (Figure 4). With the Naïve ethod, the bias is never eliinated, copared to the CG ethod. It should also be noted that when the dispersion paraeter tends towards zero (which now alost becoes a Poisson odel), the site selection bias still exists, as pointed out by Cook and Wei (00). 6

19 SCENARIO 3 RESULTS Figure 6 shows that the saple size related to the treatent group does not affect the bias considerably. When the saple size is over 30, the bias obtained by using the Naïve ethod, Equation (9), and the one estiated with Equation (6) are very close. Furtherore, there is a slight difference between the siulated bias and the estiated one when the dispersion paraeter is less than (ost often observed in crash data). However, the axiu difference (saple size=0, ) is about 0.04 when the safety effectiveness is equal to Hence, Equation (6) which was derived by assuing that the saple size is close to infinity ( ) ay be used for estiating site selection biases even the saple size is sall. 7

20 (a) Saple Size=00 (b) Saple Size=30 Bias Dispersion paraeter C= Bias Dispersion paraeter C= (c) Saple Size=0 (d) Estiated Bias Bias C= bias C= Dispersion paraeter Dispersion paraeter Figure 6. Site selection bias for the Naïve ethod when the saple size is equal to (a) 00, (b) 30, (c) 0 or (d) estiated using Equation (6). SCENARIO 4 RESULTS Figure 7 shows that the value of the index of effectiveness ( ) changes the value of the bias, because the bias increases when the countereasure effectiveness gets saller ( closer to ). This figure also shows that the ratio between the site selection bias and the safety effectiveness sees fixed, which is consistent with the characteristics of Equation (6). The results illustrate that the selection bias ay influence the final decision ade by the engineer or transportation safety specialist when different treatents are evaluated, but only one or a liited nuber of treatents can be selected. For instance, when the effectiveness of two treatents is very close but their entry criteria are very different (because different sites are used), the selection bias ay overestiate the perforance for one of the treatents over the others, which could result in the wrong selection of the 8

21 treatent. Looking at the first line (orange line with the dashed circled node) in Figure 7(a) and the last line (blue line with the dashed circled node) in Figure 7(b), the biased estiator in Figure 7(a) is equal to 0.6 (= ) which is lower than the estiator (0.65= ) found in Figure 7(b). This eans that the treatent identified in Figure 7(a) would be selected over the treatent identified in Figure 7(b), since the forer one looks like it reduces ore crashes, although it in fact reduces less crashes than the one shown in Figure 7(b). (a) θ=0.90 (b) θ=0.70 Bias C= Bias C= Dispersion paraeter Dispersion paraeter (c) θ= C= Bias Dispersion paraeter Figure 7. Site selection bias for the Naïve ethod when the safety countereasure is equal to (a) 0.90, (b) 0.70 and (c)

22 SCENARIO 5 RESULTS Figure 8 shows that the standard deviation associated with the countereasures effectiveness does not change the value of the bias. However, if the standard deviation is relatively large, such as when it is equal to 0.0, the siulated results becoe unreliable, because the ean value for the after period ay be equal to zero. When this occurred, the observations were reoved fro the analysis (such as Figure8 (a)). This scenario is ore of a theoretical exercise, since the ter is usually estiated fro the data. This scenario could also be used to replicate the application of a crash reduction factor (CRF) or crash odification factor (CMF) characterized with different levels of uncertainty. (a) θsd=0.0 (b) θsd=0.0 Bias C= Bias C= Dispersion paraeter Dispersion paraeter (c) θsd=0.05 (d) Estiated Bias Bias C= bias C= Dispersion paraeter Dispersion paraeter Figure 8. Site selection bias for the Naïve ethod when the standard deviation is equal to (a) 0., (b) 0., (c) 0.05 or (d) the bias is estiated using Equation (6). 0

23 SUMMARY AND CONCLUSIONS This study has exained how setting an entry criterion influences the estiation of traffic safety countereasures. The goal consisted in estiating how the bias changes for four coonly types of before-after studies: the Naïve, the CG and the EB MM and EB CG ethods. Those ethods were then copared to a new proposed ethod that could be used to adjust the naïve estiator when a control group is not available. Five scenarios were evaluated: a direct coparison of the ethods, different saple sizes, different dispersion paraeter values, different safety effectiveness values, and different standard deviation values associated with the safety effectiveness. The analysis was carried out using siulated data, but theoretical derivations were also presented in Appendix A to support results docuented in this study. The study results showed that aong all ethods evaluated, the Naïve and the EB MM ethods are the ones that are the ost significantly affected by the selection bias. Using a control group (CG) or the EB CG can eliinate the site selection bias, as long as the characteristics of the control group are exactly the sae as for the treatent group. For the CG ethod, the characteristics of the CG need to be the sae as the characteristics of the truncated saple (see right-hand side of Figure 3). For the EB CG, the characteristics of the control group need to be the sae as the full or non-truncated saple (see left-hand side of Figure 3). In practice, this ay not be possible. However, using the approach proposed in this study to adjust, but not eliinate, the naïve estiator can partially eliinate site-selection bias, even when biased estiators of the ean and dispersion paraeter of a truncated Negative Binoial distribution are used. To do this, the researcher or safety analyst siply needs to copute the naïve estiator, calculate the saple ean and the variance of the data in order to get the dispersion paraeter, and incorporate the values into Equation (6) to get an estiate of the bias. Then, once the bias is estiated, the index can be adjusted using Equation (). Based on the siulated scenarios (also supported theoretically), the study results also showed that higher entry criteria, larger values of the index ( ), and saller dispersion paraeter values will cause a higher site selection biased estiate. It should be pointed out that the assuptions ade in this study did not influence the validity of the analysis. In fact, the analyses presented in this paper have been validated using observed data, as docuented in Kuo (0). Given the nature of the work docuented in this paper, there are several avenues for further work. First, since the estiator of the site-selection bias, Equation (6), only estiates about 50% of the bias when a control group is not available, it ay be beneficial to apply ore advanced techniques to estiate the paraeters of a truncated Negative Binoial odel (or truncated Noral distribution for continuous data) in order to obtain ore precise estiates. Second, the site selection effect ay be influenced by issing data. The estiator of the selection bias ay not be affected when the issing data are randoly distributed since the saples are still representative of the population. On the other hand, if the issing data are systeatic, such as those associated with property daage only or possible injury (classified as injury C) crashes or for certain categories of collision types, the estiator ay becoe ore biased. Hence, further work

24 needs to be conducted on this topic. Third, guidelines should be developed to define what the entry criterion should be used when it is not known (e.g., iniu value in saple, MUTCD, etc.). Fourth, one should exaine site selection effects close to the boundary when 0 as a function of different ean values for the before period. Fourth, statistical tests or ethodologies should be developed to ensure that the data collected for the control group when the EB ethod is used is the sae as the full data fro which the truncated distribution is used (which ay not be possible to find out). Although the EB CG ethod has been (and still is) frequently used aong transportation safety analysts, very few ever copare the characteristics of the treatent and control groups. Researchers autoatically assue that the NB regression odels estiated fro the control group has the sae characteristics as the sites selected for potential treatent. Finally, a sipler approach for displaying the site selection bias should be done. Tables based on the saple ean ( ), entry criteria, and the level of dispersion could be provided in the Highway Safety Manual (00) or any siilar types of docuents. ACKNOWLEDGEMENTS The authors would like to thank Dr. Gary A. Davis fro the University of Minnesota and Dr. Ezra Hauer, Professor Eeritus at the University of Toronto, for their input. The authors also thank Dr. Jeff Hart fro Texas A&M University for his review and suggestions related to Appendix A. REFERENCES Abbess, C., Jarrett, D, Wright, C., 98. Accidents at blackspots: Estiating the effectiveness of reedial treatent, with special reference to the regression-to-ean effect, Traffic Engineering and Control, Aerican Association of State Highway and Transportation Officials, 00. Highway Safety Manual st Edition, Washington, DC. Chuang-Stein, C., Tong, D., 997. The ipact and iplication of regression to the ean on the design and analysis of edical investigations. Statistical Methods in Medical Research 6, 5-8. Cook, R., Wei, W., 00. Selection effects in randoized trials with count data, Statistics in Medicine, Copas, J., 997. Using regression odels for prediction: shrinkage and regression to the ean. Statistical Methods in Medical Research 6, Danielsson, S., 986. A coparison of two ethods for estiating the effect of a countereasure in the presence of regression effects. Accident Analysis and Prevention 8,3 3.

25 Davis, C.E., 976. The effect of regression to the ean in epideiologic and clinical studies. Aerican Journal of Epideiology,Vol.04, pp Davis, G., 000. Accident reduction factors and causal inference in traffic safety studies: a review. Accident Analysis & Prevention, 3, FHWA, 00. Manual on Unifor Traffic Control Devices. Federal Highway Adinistration, Washington, D.C. Geyer, C.J., 007. Lower-Truncated Poisson and Negative Binoial Distributions. Working paper written for the software r. University of Minnesota, MN. (available: Haed M.M., Al-Eideh B.M., Al-Sharif M.M Traffic accidents under the effect of the Gulf crisis, Safety Science, 33, Hauer, E., 980a. Bias-by-Selection: Overestiation of the Effectiveness of Safety Countereasures Caused by the Process of Selection for Treatent, Accident Analysis & Prevention,, 3-7 Hauer, E., 980b. Selection for treatent as a source of bias in before-and-after studies. Traffic Engineering and Control, Hauer, E., 997. Observational Before-After Studies in Road Safety. Pergaon Publications, London. Hauer, E., Byer, P., Joksch, H., 983. Bias-by-selection: The Accuracy of an unbiased estiator. Accident Analysis & Prevention 5, Hauer, E. Persaud, B., 983. Coon bias in before-and-after accident coparisons and its eliination. Transportation Research Record 905, Hauer, E., Persaud, B., 984. Proble of Identifying Hazardous Locations Using Accident Data. Transportation Research Record 975, Kuo, P.-F., 0. Exaining the Effects of Site Selection Criteria for Evaluating the Effectiveness of Traffic Safety Countereasures. Ph.D. Dissertation. Zachry Departent of Civil Engineering, Texas A&M University, College Station, TX. Kuo, P.-F., Lord, D., 0. Accounting for Site-Selection Bias in Before-After Studies for Continuous Distributions: Characteristics and Application Using Speed Data. Paper presented at the 9st Annual Meeting of the Transportation Research Board, Washington, D.C. Lawless J., Fong D., 999. State duration odels in clinical and observational studies. Statistics in Medicine 8,

26 Li W., Carriquiry, A., Pawlovich, M., Welch, T., 008. The Choice of Statistical Models in Road Safety Countereasure Effectiveness Studies in Iowa. Accident Analysis and Prevention 40, Lord, D., and F. Mannering, 00. The Statistical Analysis of Crash-Frequency Data: A Review and Assessent of Methodological Alternatives. Transportation Research - Part A, Vol. 44, No. 5, pp Maher, M., Mountain, L., 009. The sensitivity of estiates of regression to the ean. Accident Analysis & Prevention 4, Miaou, S.-P., J. J. Song, and B. K. Mallick Roadway Traffic-Crash Mapping: A Space-Tie Modeling Approach. Journal of Transportation and Statistics 4, Miranda-Moreno, L., Fu, L., Ukkusuri, S., Lord, D., 009. How to Incorporate Accident Severity and Vehicle Occupancy into the Hotspot Identification Process? Transportation Research Record 0, Noland, R., 003. Traffic fatalities and injuries: the effect of changes in infrastructure and other trends. Accident Analysis and Prevention 35, Park, E., Park, J., Loax, T., 00. A fully Bayesian ultivariate approach to before after safety evaluation. Accident Analysis & Prevention 4, 8-7. Park, P., and D. Lord., 00. Investigating Regression-to-the-Mean in Before-and-After Speed Data. Transportation Research Record 65, Pearl, J., 009. Causality: Models, Reasoning, and Inference. nd edition Cabridge University Press, New York. Persaud, B., 00. Statistical Methods in Highway Safety Analysis, NCHRP Synthesis of Highway Practice 95, Transportation Research Board, National Research Council, Washington, DC. Persaud B., Lyon, C., 007. Epirical Bayes before after studies: lessons learned fro two decades of experience and future directions, Accident Analysis and Prevention 39, R Developent Core Tea, 006. R: A Language and Environent for Statistical Coputing. R Foundation for Statistical Coputing. Vienna, Austria, ISBN Rock, S. M., 995. Ipact of the 65 ph speed liit on accidents, deaths, and Injuries in Illinois. Accident Analysis and Prevention, 7,

27 Robbins, H., 956. An Epirical Bayes Approach to Statistics. Proceedings of the Third Berkeley Syposiu on Matheatical Statistics and Probability, Volue : Contributions to the Theory of Statistics, Rubin, D., 977. Assignent of treatent group on the basis of a covariate. Journal of Educational and Behavioral Statistics, -6. Sauels, M.L., 99. Statistical Reversion Toward the Mean: More Universal than Regression Toward the Mean. The Aerican Statistician 45 (4), Stingler, S., 997. Regression towards the ean, historically considered. Statistical Methods in Medical Research 6, Tarko, A., Eranky, S., Sinha, K., 998. Methodological considerations in the developent and use of crash reduction factors. Preprint CD. 77th Annual Meeting of the Transportation Research Board, Washington, DC. Wright, C., Abbess, C., Jarrett, D., 988. Estiating the regression-to-ean effect associated with road accident black spot treatent: Towards a ore realistic approach. Accident Analysis & Prevention, 0,

28 Appendix A This appendix describes the conditions when the epirical Bayes (EB) estiator of the index of safety effectiveness ( ) is asyptotically unbiased and biased. The bias is defined as the difference between and the expected value of. The following paragraphs show the EB estiators for three different cases: () Without entry criteria, () With entry criteria and with perfect 3 control group data; and (3) With entry criteria but without perfect control group data. The first one is the ost coon estiator, and previous studies have already shown that it is unbiased (Robbins, 956; Hauer, 997). For the second estiator, it is also unbiased, and the results are consistent with the Davis (000) study. For the third estiator (EB MM ), which is the one used in this research, we deonstrate when the estiator can be as biased as for the Naïve ethod. To siplify the coparison, all three estiators are shown below. It should be noted that the oent estiators, axiu likelihood estiators, or other E( N N C) estiators based on conditional data consistently estiate ( ˆ i i E ) rather E( N N C) i i E( N than ) ˆ i E( N ). All notations in this appendix are the sae as in the ain text, i and E Ni Ni C is for notational convenience.. Experients without effective entry criteria For this case, the crash frequency of site i in the before period (N i ) can be any nonnegative integers (e.g. 0,,, ). When there are no entry criteria, C is equal to - or less. The EB estiator for Case is given as follows: 3 The perfect reference or control group has the exact sae ean and dispersion paraeter (or the variance) as those of the treatent group. This situation rarely occurs in the real world, because the true ean of the control group is ost likely unknown. 6