Session 6: and Evaluation. M c Farland Management, LLC

Transcription

1 Session 6: Performance Measurement and Evaluation State of the Practice Overview 1

2 Why Evaluate WRTM Strategies Understand system performance and benefits Specific benefits directly attributable to WRTM Support research of deployments to address needs Prioritize resources and justify future investments Support business case for WRTM deployments Establish WRTM benefits database Encourage wider deployment Serve national transportation goals 2

3 Evaluation Approaches and Procedures to Assess the Benefits of WRTM Report Chapter 5 Evaluation plan, approach, and structure Challenges Evaluation guidance for selected WRTM strategies t Evaluation examples 3

4 Evaluation Planning Scope and objectives Specify hypotheses Design and methods Potential challenges Data collection Analysis Evaluation report Pathway WRTM Strategy (Broadly Defined) Specific Strategy/System Implementation System Outputs Expected Benefits/ Outcomes Achievement of National ITS Goals EXAMPLE Active Warning System Automated system to warn travelers that ice may be present on a bridge at a specific location and to proceed with caution Accuracy of warning relative to conditions Timeliness of warning initiation removal Reduction in vehicle speeds Reduction in crashes, injuries, and fatalities Changes in driver perceptions of understandability and usefulness of the warnings Improved safety Improved driver satisfaction 4

5 Challenges (examples) WRTM Evaluation Challenges Seasonality issues comparing results across winters or seasons given the unpredictability and differences of the weather. Weather and Road Weather Information is lacking Agencies lack the comprehensive road weather information in the location being considered for WRTM strategy deployment/evaluation, leading to an inability to assess and attribute strategy performance. Mitigation Approaches When using a before-after design, compare data for similar weather events in the two time periods. Collect eventspecific data not only on strategy performance but also on the actual weather conditions experienced. Identify desired performance targets appropriate for different weather conditions and measure against those standards. It is critical to a successful evaluation that the necessary road weather information is available prior to strategy implementation. The evaluation plan needs to identify all road weather information necessary to support the research design. If additional weather sensing or weather information sources are needed, these elements should be included in the strategy implementation. 5

6 Challenges (examples) Pathway to benefits is often unclear, especially for traveler behavior and response the linkage to final outcomes as experienced by the traveler is difficult to establish given confounding factors. Consider conducting focus groups with representative travelers to help understand how they are likely to be impacted by and their attitudes toward the new WRTM strategy and use these insights to design the evaluation data collection. Weak link between attitudes and behavior how travelers Include in the evaluation data both stated preference say they will react and how they actually behave are often questions for travelers and revealed preference measures very different. based on their actual use of the relevant elements of the strategy. Lack of good baseline information good data on traffic conditions and safety during weather events are rare and difficult to obtain. Before the evaluation, gain a clear understanding of historical data availability, specifically as linked to weather events. It may be necessary to use expert interviews to help fill some gaps in recorded quantitative data sources. Interviewing or surveying travelers it is often difficult to find travelers who have experienced the systems being evaluated and having them agree to be surveyed. Candidate subjects need to be pre-screened to assure they have the appropriate experience. Alternatively, subjects can be recruited and exposed to specific elements of the strategy. For other self-selected samples, include questions that help classify respondents appropriately in terms of their experience with and use of the strategy elements. 6

7 Evaluation Guidance Selected Strategies WRTM Strategy Active Warning Systems Pre-trip and En-route Weather and Pavement Information Systems Speed Advisories and Enforceable Variable Speed Limit Systems Vehicle size, height, and profile restriction systems Seasonal Load Restriction Systems Road and Lane Use Restriction Systems Traffic Signal Control Systems Service and Courtesy Patrol Systems Agency Coordination and Integration 7

8 Evaluation Guidance - Components Description Examples Key inputs Key output MOEs and data require- ments Key outcome MOEs and data require- ments Example results Dry and Clear Storm Begins to Impact Roadway VSL In Effect Storm Clears (VSL In Effect) 8

9 Evaluation Guidance Summary Table Description: Active warning systems could supplement passive warning signs with flashing beacons to alert travelers that the conditions specified on the static sign are currently in effect. Additionally, an active warning system could include a changeable message sign activated automatically or manually from an operations center. The flashing beacons may be activated either manually by operators in a traffic management center or by field personnel based on observed conditions, or automatically if tied to a road weather monitoring system (such as a flood detection stream gauge or a high wind detection system). Active Warning Systems Examples of Strategy Applications: Ice on a bridge High winds in a defined location Static sign with flashers warning about a specific condition in a specific location CMS with specific weather-related messages Manual or automated system implementation Key Inputs: Operational procedures that guide manual operation of the system Operator training Sensors that measure and report on road weather conditions or automatically trigger a message sign Time stamped archived road weather condition information (appropriate to site locations, and before, during and after warning time periods) Traffic condition information Key Output MOEs: 1. Timeliness of issuance of warning 2. Timeliness of removal of warning 3. Accuracy of warning relative to conditions 4. Time lag between when weather condition threshold exceeded and operator action taken Key Outcome MOEs: 1. Reduction in vehicle speeds in area of warning 2. Reduction in number of crashes, injuries and fatalities (in proximity to warning system) 3. Changes in driver perceptions of understandability and usefulness of warning Data Requirements: System record (or operator log) of all warnings issued in a defined period (message content) System record (or operator log) of dates and times warnings were issued System record (or operator log) of dates and times warnings were removed Reports from field, or traveler surveys/interviews, of warning appropriateness for conditions System record of date and time of first receipt of condition indication that triggered the warning Data Requirements: Traffic information at warning site (type, speed, etc.) from field sensors or reports State records of crashes, injuries, fatalities, in area of warning, under defined weather conditions Traveler survey/interviews that measure whether warning was properly interpreted and understood, and how useful drivers found the warning 9

10 Evaluation Examples WRTM Strategy Active Warning Systems Examples Described Oregon Wind Warning Systems Pre-trip and En-route Weather and Pavement Information Systems Speed Advisories and Enforceable Variable Speed Limit Systems Vehicle size, height, and profile restriction systems Various 511 Systems and Highway Signage with Weather Information; Idaho Storm Warning System Wyoming Variable Speed Limit None identified Seasonal Load Restriction Systems Montana, North and South Dakota Seasonal Load Restriction Tool Road and Lane Use Restriction Systems Traffic Signal Control Systems Various States Winter Weather Road Closure Systems None identified Service and Courtesy Patrol Systems Agency Coordination and Integration Denver, CO Courtesy Patrol Pilot Utah DOT Business Case for Weather Operations 10

11 Summary WRTM evaluation/benefits database is lacking Key element to promote WRTM Strategy deployment Available information is contained in WRTM Report Encourage you to consider conducting an evaluation of your WRTM deployment Share results with other stakeholders 11