February 2011 Cold Snap Event Recommendations January 23, 2011 y, Earl Shockley, Director of Reliability Risk Management
Recommendations - Electric February 1 5, 2011 event occurred August 15, 2011 joint report released There were a total of 26 electric recommendations issued: Planning and reserves (5); Coordination with generator owners/operators (5); Winterization (10); Communications (4); and Load Shedding (2) There were also six gas recommendations 2 RELIABILITY ACCOUNTABILITY
Theme -Three P s August 2003 Blackout delivered three distinct themes called the three T s. Trees Tools Training The February 2011 Cold Snap event delivered three distinct themes Predict Planning Preparation 3 RELIABILITY ACCOUNTABILITY
Predict First P Predict The storm was not without precedent. There wereprior severe cold weather events in the Southwest in 1983, 1989, 2003, 2006, 2008, and 2010. Good analyses were conducted Good reports were written Good recommendations were developed No institutionalization of the recommendations and lessons Can we now predict the same circumstances? 4 RELIABILITY ACCOUNTABILITY
Planning Second P Planning On site visits with GO/GOPs revealed: Some plants had inadequate plans and safeguards in place to address severe weather. o Lack of formal plans, procedures to address severe weather o Lack of adequate knowledge of plant temperature design limits and the equipment most effected by freezing o Failure to adequately apply and institutionalize knowledge and recommendations from previous severe winter weather events Someplants had rigorous plans and safeguards in place, some required validation. Can you predict which plants were most successful during the event? 5 RELIABILITY ACCOUNTABILITY
Preparation Third P Preparation Generators were generally reactive as opposed to being proactive in their approach to winterization and preparedness. p Many generators failed to adequately prepare for winter, including the following: Failed or inadequate heat traces, Missing or inadequate wind breaks Inadequate or lack of insulation Failure to have or to maintain heating elements and heat lamps in instrument cabinets Failure to have necessary resources and supplies 6 RELIABILITY ACCOUNTABILITY
BY FAILING TO PREPARE, YOU ARE PREPARING TO FAIL. BENJAMIN FRANKLIN 7 RELIABILITY ACCOUNTABILITY
Frozen Sensor Aerator Sensor 3/8 inch tubing exposed R 14 GO/GOPs Ensure that t adequate maintenance and inspection of freeze protection elements is conducted on a timely and repetitive basis. 8 RELIABILITY ACCOUNTABILITY
Inadequate Insulation R 16 GO/GOPs inspect and maintain thermal i l ti on insulation all units. 9 RELIABILITY ACCOUNTABILITY
Lack of Insulation Notice burn marks (from torch) on bare tube R 18 GO/GOPs Develop and annually conduct winter specific and plantspecificoperator awareness and maintenance training. Outside exposure 10 RELIABILITY ACCOUNTABILITY
Wind Break Design Wind break was too Short Feedwater Sensor Froze R 17 GO/GOPs Plan to erect adequate wind breaks and enclosures, where needed. Wind Break 11 RELIABILITY ACCOUNTABILITY
Corroded Freeze Protection Panel R 15 GO/GOPs Inspect and maintain heat tracing equipment on all generating units. 12 RELIABILITY ACCOUNTABILITY
Oil Burning Wands R 6 TOs, BAs, and GO/GOPs Verify that units that have fuel switching capabilities can periodically demonstrate those capabilities 13 RELIABILITY ACCOUNTABILITY
Fuel Transfer Valves Valve Froze 14 RELIABILITY ACCOUNTABILITY
Lessons Learned NERC received a total of 55 lessons learned from entities impacted by the cold snap event. The lessons learned were reviewed by NERC, categorized and consolidated into 22 individual lessons that align with the recommendations dti outlined in the joint tferc/nerc report Outages and Curtailments during the Southwest Cold Weather Event published on August 15, 2011. On September 30, 2011 the first two cold snap lessons learned were posted to correspond with the fall maintenance program for winter weather preparation. Two additional lessons learned have been posted. 15 RELIABILITY ACCOUNTABILITY
Events Happen Lab Tests Confirm Salmonella Source 16 RELIABILITY ACCOUNTABILITY
Reliability Risk Management Concepts Severity Avoid Learn and Reduce Inverse Cost Benefit Reporting Threshold Trend lower tiered events identify emerging reliability risk trends. 17 RELIABILITY ACCOUNTABILITY
Drifting to Failure Concept* Hi Expectations: Desired approach to work (as imagined) Normal Practices: Work as actually performed Managements Stated Expectations liability Real Margin for Error Drift Error Normal Practice Re RISK CLUSTERS Programmatic deficiencies, deficiencies in barriers and defenses, Latent organizational weaknesses and conditions Errors in human performance and contextual factors Equipment design and/or maintenance issues Latent Errors unnoticed at the time made; often deeply or embedded within system. Lo Time * Adapted from Muschara Error Management Consulting, LLC 18 RELIABILITY ACCOUNTABILITY
Cold Snap Event Drifting to Failure* Hi Expectations: Desired approach to work (as imagined) Normal Practices: Work as actually performed Stated Expectations liability Real Margin for Error Drift Error Normal Practice Re Linked Causal factors = examples above Latent Error unnoticed at the time made; often deeply or embedded within system. Lo Time * Adapted from Muschara Error Management Consulting, LLC 19 RELIABILITY ACCOUNTABILITY
Defense in Depth Concept* When the wrong set of circumstances line up, major events occur Freezing weather, high winds Defense 4 Defense 2 Defense 3 Defense 1 Event * Adapted from Dr. James Reason, Human Error 1990 20 RELIABILITY ACCOUNTABILITY
Summary We do not want to lose our perspective of what happened during the winter of 2011 Consider the three P s Examine your safeguards We should consider the cost benefit as well as the risk to reliability and reputation: Institutionalize recommendation and learning's Show that we not only can identify problems but also put the necessary fixes in place. Have a passionate focus on reliability 21 RELIABILITY ACCOUNTABILITY
Questions? 22 RELIABILITY ACCOUNTABILITY