Final Report On the October 12, 2010 Thimbleberry – Heart Lake

 Final Report On the October 12, 2010 Thimbleberry – Heart Lake 69kV Transmission Line Outage Evaluation and Recommendations December 2010 CONTENTS SUMMARY: .......................................................................................................................... 3 SYSTEM EVENT: ................................................................................................................... 6 CITY & BOROUGH OF SITKA RESPONSE ............................................................................ 10 ELECTRIC DEPARTMENT ....................................................................................................... 10 CUSTOMER SERVICE ............................................................................................................ 13 PUBLIC WORKS .................................................................................................................. 14 COMMUNITY IMPACTS ..................................................................................................... 14 COMMUNICATIONS/MEDIA ................................................................................................. 14 ROLLING BLACKOUT IMPACTS ............................................................................................... 15 ECONOMIC IMPACTS ........................................................................................................ 16 DIRECT COSTS ................................................................................................................... 16 INDIRECT COSTS ................................................................................................................. 16 RECOMMENDATIONS ....................................................................................................... 17 Figure 1. Fallen Conductors, Thimbleberry ‐ Heart Lake 69kV Transmission Line ............. 3 Figure 2. System Average & Peak Loads ............................................................................. 4 Figure 3. Overview of 69kV Transmission Line ................................................................... 6 Figure 4. Gulf of Alaska Weather October 12, 2010 ........................................................... 7 Figure 5. Tree Contact ‐ Thimbleberry ‐ Heart Lake Transmission Line.............................. 8 Figure 6. Damaged Structure ‐ 69kV Transmission Line ..................................................... 9 Figure 7. Transmission Line Corridor ‐ Blue Lake Powerhouse to Sawmill Creek Rd. ...... 11 Appendices (A) – FERC Incident Report………………………………………………………………………….19 (B) – Electrical System One‐Line Diagram…………….……………………………………20 (C) – Blue Lake Outage Response……………………………………………………………….21 (D) – Sawmill Creek Road Transmission Line Report…………………………………..30 (E) – Customer Emergency Generator Report……………………………………….……37 (F) – Public Service Announcements…………………………………………………………..41 Thimbleberry Outage
2
Final Report Thimbleberry – Heart Lake 69kV Transmission Line Outage Summary: On October 12, 2010, at approximately 2:30 p.m., a system‐wide outage was initiated by the loss of the 69kV transmission line between Blue Lake Powerhouse and Jarvis Street substation during high wind conditions. Subsequent investigation revealed the line was severely damaged by a tree falling into the line approximately midway along the overland section of the Thimbleberry – Heart Lake line, or approximately 0.8 miles from the Blue Lake road. At least two poles were broken and all 3 primary 69kV conductors were also knocked down. This line is the sole transmission path from the Blue Lake and Green Lake Hydroelectric facilities therefore all hydroelectric generation was curtailed. Figure 1. Fallen Conductors, Thimbleberry - Heart Lake 69kV Transmission Line Thimbleberry Outage
3
Recognizing that the Jarvis Street Diesel plant can only generate 12.0 megawatts (MW) and the system load when the fault1 occurred was 15.6 MW, the line outage instantly created a generation deficiency and community wide emergency. As noted in Figure (2), system load almost always exceeds diesel generation capability. Authority to conduct energy curtailments is provided by Sitka General Code Section 15.01.040(G) – Curtailment, which states; “Should a serious power shortage develop, and should it become mandatory that the utility place into effect a curtailment program, the utility reserves the right to limit the use of electrical energy during such hours as may become necessary.” Electric
Generation
Oct 2009 - Oct 2010
22
20
18
MW Average/month
16
Max Diesel Generation
MW
Peak MW/month
14
12
10
8
Oct
Nov
Dec
Jan
Feb
Mar
Apr
May
Jun
Jul
Aug
Sep
Oct
Figure 2. System Average & Peak Loads
Simultaneous with the transmission line outage, there were numerous distribution level outages due to tree contacts and other problems associated with the high winds. Staffing and resource allocations were a critical issue in determining restoration priorities as well as providing public notification of the crises. Electric Department staff met immediately to plan restoration procedures including; assessment of additional damage and repairs required, system load reduction, placing all available diesel generation on line, developing work schedules for crews, public notifications, and notification to contract line crew for assistance. Due to the remote location of the line failure, logistics planning and staging of material for repair was a challenge. 1
For the purposes of this report the term “fault” will represent the entire 3 day period of the outage,
including the causation event, which was the tree contact in the 69kV line, and the subsequent generation
shortage which required rolling blackouts to be in effect until line repairs were completed.
Thimbleberry Outage
4
Initial public notifications were made via local media outlets including radio and newspaper. Unfortunately, the Electric Department headquarters building was subject to the rolling blackouts and this greatly hampered efforts to provide timely public updates. The Police Department, Fire Department, Public Works, and Customer Service were not on the initial list of notifications. This resulted in a bit of confusion for the public as these departments did not receive updates until the next day and in the meantime handled many phone calls requesting information without benefit of current information. The loss of the transmission line is also a reportable event to the Federal Energy Regulatory Commission (FERC) and this report was completed on December 8, 2010, Appendix (A). The existing plan for dealing with generation deficiencies is to keep the city core energized to the extent possible and conduct rolling blackouts on the outlying residential distribution feeders, namely Halibut Point and Sawmill Creek Roads, see Appendix (B). There were several problems that developed utilizing this strategy. The City’s fuel oil contractor was subject to rolling blackouts and could not load fuel trucks during off hours, despite the need for fuel for Jarvis Diesels and other Public Works emergency generators. There are several critical home‐care patients that faced serious consequences due to lack of power for home medical equipment. The main Post Office was subject to the blackouts and curtailed mail service due to concerns about their equipment due to frequent switching. Several businesses were adversely impacted by the outages and either operated on reduced hours or closed altogether. The community dealt with many issues during the fault including; misunderstanding or lack of communication about what was happening, difficulties with the rolling blackouts, and inconvenience of being without power. However, it is apparent many members of the public did not understand the critical nature of this emergency and recognize it impacted all of Sitka not just the outlying areas subject to rolling blackouts. In many ways we were fortunate, as there were several circumstances that helped mitigate this fault; the foul weather abated for much of the restoration period, relatively warm weather helped keep system loads down, heavy equipment was available to access the site, and all materials necessary for repair were on hand. If this event had happened in the middle of a winter cold spell, the results could have been disastrous! Power was restored at 10:31 a.m. on October 15, 2010 after nearly 70 hours of rolling blackouts, consumption of 46,876 gallons of diesel fuel, 150 gallons of lube oil, and at a cost of about $170,000. This report will address the many issues related to this event; transmission and generation deficiencies, critical staffing, materials, equipment, public response, communications, preparedness, and overall assessment of future events and what can be done to eliminate or reduce these risks. Thimbleberry Outage
5
This transmission outage was a very serious event and is the second outage of the Thimbleberry section of the transmission line this year and clearly highlights the two distinct and fundamental utility deficiencies we face, namely; transmission and generation. System Event: System Description Fault
Location
Figure 3. Overview of 69kV Transmission Line The electric system is comprised of various equipment and facilities that provide three distinct functions; generation, transmission, and distribution. Through these systems electrical energy is safely generated via hydroelectric and diesel generators, transmitted from the power plants via a single 69,000 V (69kV) transmission line to three substations where the voltage is stepped down to 12.47kV and distributed to our customers via nine (9) distribution circuits. There are three main generation plants, one transmission line, and three distribution substations. Total annual available energy, assuming normal precipitation, is: Facility Demand Energy Blue Lake, hydro 6 megawatts (MW) 62,500 megawatt‐hours (MWh) Green Lake, hydro FVU/Campground Hydro Jarvis Street, diesel Total Thimbleberry Outage
18 megawatts (MW) 1.54 megawatts (MW) 12 megawatts (MW) ========== 25.54 MW (hydro) 12.00 MW (diesel) 37.54 MW (total) 6
60,000 megawatt‐hours (MWh) 6,000 megawatt‐hours (MWh) Supplemental/Stand‐by as required 128,500 megawatt‐hours hydro System Fault This outage was clearly initiated by inclement weather. During this period, Sitka was experiencing sustained high wind conditions due to a major low pressure system in the Gulf of Alaska as shown in the infrared image in Figure (4). The spiraling frontal band leads a low center a little northwest of Sitka with significant storm effects. Local wind speed was measured at over 64 knots at 12:35 p.m., with gusts exceeding these at higher elevations. Figure 4. Gulf of Alaska Weather October 12, 2010 The fault occurred at approximately 2:29 p.m. when the Blue Lake powerhouse operator received multiple alarms and breaker operations indicating a major system disturbance. Appendix (C) provides the sequential evolution of the early phases of the outage, including the steps to test and reenergize the line upon completion of repairs. System protective devices indicated a C‐Phase and Ground fault on the 69 kV line between Blue Lake and Jarvis Diesel Plant. The line crew immediately began patrolling the line and approximately 45 minutes later located the damage on the 69kV transmission line and reported a tree top had broken and fell across the line taking down all three primary conductors. Fortunately, the communication circuits that are installed on the transmission line remained in service even after being knocked to the ground. Supervisory Control and Data Acquisition Thimbleberry Outage
7
(SCADA) control of the substations and generation plants are provided via these communications circuits. As noted in Figures (5) and (6), a large tree top broke and fell across all 3 primary conductors taking them to the ground, including damage to the structure. At this point, it was evident the outage would be extensive and require major mobilization of department resources to complete repairs. The following section will address the technical challenges created by this fault and the associated City response. Figure 5. Tree Contact - Thimbleberry - Heart Lake Transmission Line Thimbleberry Outage
8
Figure 6. Damaged Structure - 69kV Transmission Line System Response From a system operator perspective this outage was very unusual. There are circuit breakers at both ends of the transmission line that provide system protection and look at fault conditions on the line and open the necessary breaker(s) in the event of a fault. In this case breaker #66 at Jarvis Street opened and the breakers #11 and #33 at Green Lake and Blue Lake did not open. Blue Lake generators subsequently tripped off line from a high voltage condition due to the loss of system load. Closer inspection of the downed transmission line indicated why this happened. The tree contact broke all the conductors at the insulators on the pole resulting in the Jarvis side of the line falling to the ground but the Blue Lake side of the line stayed essentially intact, although one phase was contacting the structure and was burning. Therefore, the relays for breakers #11 and #33 were seeing a high impedance line and did not trip. Thimbleberry Outage
9
This is a very dangerous condition and will be evaluated as we look at the entire system protective scheme dynamic response. Due to the loss of the transmission line and available hydro generation, the resulting generation shortage impacted the entire community. The remainder of this report will address the various problems that arose from this outage and address our future plans for dealing with potential system outages. City & Borough of Sitka Response The response by the Electric Department and the City in general was outstanding. All members of the department contributed to the expeditious repair and restoration of electric service to the community. Cooperation and support from other city departments was exceptional and contributed to the timely completion of line repairs and continuation of other essential city services without interruption. Electric Department There are several fundamental questions that should be asked about this system disturbance. Could this outage have been prevented? Was the department and city in general prepared for this event? Did the electrical system operate as designed? Were there any unexpected consequences from the outage? What were the economic impacts resulting from this outage? As noted from Figure (4), this was a significant storm event for the entire Southeast Alaska region. Other communities in Southeast were seriously impacted by this storm as well, particularly Ketchikan. However, this was not an unusual event for our community. Inclement weather is a factor that will always be a challenge to reliable electric service. A fundamental tenant of utility system reliability is ongoing line maintenance, in particular, tree trimming. The department has actively pursued a comprehensive line maintenance program and has obligated over $100,000 per year for the past 6 years. The greatest exposure the department faces, relative to transmission system reliability, is the section of transmission line that is not accessible with department line equipment and vehicles. As noted in Figure (7), this is the overland section of line from Blue Lake powerhouse to Whale Park on Sawmill Creek Road. This is not a new concern. Thimbleberry Outage
10
Figure 7. Transmission Line Corridor - Blue Lake Powerhouse to Sawmill Creek Rd.
This section of line has experienced damage in recent times including events that occurred in 1989, March 1992, August 2002, March 2010, and October 2010. Consequently, the department established capital project No. 90410 in the FY2008 budget to look at an alternative transmission line along Sawmill Creek road that would reduce the risk of a transmission line failure. This project was a low priority project for the past few years but was recently renewed interest due to the planned Alaska Department of Transportation & Public Facilities (ADOT&PF) project to rebuild the section of Sawmill Creek Road from Whale Park to the Sawmill Creek Bridge. With the opportunity to partner with ADOT&PF during road construction of Sawmill Creek Road the department completed an initial design for the proposed transmission line from Whale Park to Blue Lake powerhouse. We conducted a public meeting on April 6, 2010, to present the proposed project and garner public input. In general, the attendees at the meeting were opposed to an overhead transmission line but did express support for an underground transmission line. Appendix (D) addresses this issue in greater detail. Thimbleberry Outage
11
The Electric Department subsequently addressed this issue with the Assembly at the regular meeting of April 13, 2010. The Assembly approved $250,000 in funding for a short‐term approach to mitigate the transmission deficiency, which included; additional right‐of‐way clearing and danger tree removal, design of line breakaways for the conductors, and procurement of repair materials for potential line outages. Shortly thereafter, our contract line maintenance crew spent about three weeks on this section of line and removed approximately 15 danger trees. One critical point to remember regarding line maintenance is the fact the transmission line cannot be taken out of service for maintenance or upgrades except for very infrequent and unpredictable times. Otherwise, due to our limited diesel generation, we will require system outages to perform line maintenance. With few exceptions, the electrical system operated as designed. A significant amount of switching and coordination was required to conduct the rolling blackouts and the SCADA system, including communications links, performed very well. However, there are several technical and operational issues to be addressed. System Deficiencies ƒ Blue Lake SCADA control of field devices was effective but device status was slow to update on the display. ƒ There is no back up SCADA control terminal at Jarvis Street Diesel Plant to assume control in event of a Blue Lake Powerhouse or communications failure. If we lose SCADA control at Blue Lake all plants would have to be manned and operated in local control only. ƒ The #11 breaker did not respond to repeated commands to close on October 12 but functioned normally on October 15. ƒ The 69kV voltmeter for the line side of breaker #11 is actually reading the load side of the breaker. ƒ The 1410 and 1430 feeders at Sawmill Cove Substation would not close by SCADA control. ƒ Radios used by the line crew began to suffer failures after a few days in the rain, we will upgrade to weatherproof handheld radios. ƒ Time stamping and recording of human inputs and equipment status changes would be very helpful for system control purposes. ƒ VHF radio coverage was occasionally spotty depending on location of user. ƒ Diesel plant had a few minor oil leaks, muffler on unit 3 requires replacement, and fuel oil day tank transfer shut down on one occasion. ƒ Cell phone coverage was inconsistent due to loss of power at Jarvis Street communications site. Thimbleberry Outage
12
ƒ
ƒ
The department received a few calls after the system was restored that the frequent switching had damaged customer’s electronic equipment. The Electric Department was subject to the rolling blackouts and this seriously impeded ability to provide updates and monitor work in progress. Customer Service The customer service department is our front line contact with the public and must have updated information on the situation to be effective. In this case, the operational challenges for the Electric Department to keep customers in service were very unusual. We did not notify customer service immediately and that was a valuable lesson learned. Exacerbating the problem was the fact we have not had a system‐wide problem for many, many months. The downside of a reliable electric system is often complacency. The public sometimes forgets we are an islanded system and subject to a host of problems that could result in long term electric shortages. A serious complicating factor was the fact the Electric Department itself was subject to the rolling blackouts and suffered communications and computer failures. The IT department did a masterful job in restoring these services but it was very difficult to provide public updates and notifications when our systems were constantly compromised and interrupted. The further challenge was conveying to the public what a rolling blackout actually meant and what the expected consequences may be. For example, the department published an anticipated schedule of times and what circuits would be on or off. Customer feedback indicates many folks took this quite literally, thinking that if they were scheduled to be on during even hours then at exactly 8:00 for example, the power would be on, guaranteed. This clearly is not the case and highlights the challenge we have in communicating the true nature of what we were doing to restore or maintain service. During an outage or other system emergency, our system is not operating normally, electric supply is not guaranteed, and customers must be prepared for interruptions in service. What could have been done to better communicate the details of the emergency from the electric department to customer service? One positive action was a customer service briefing conducted by the Electric Department. Staff provided a system drawing to customer service representatives and explained the nature of the outage, operational plans, and answers to basic questions customers had or may be calling in with. This basic information and ongoing updates significantly helped customer service deal with the many phone calls from customers, which also provided a huge relief to the Blue Lake powerhouse operator dealing with these information requests. Thimbleberry Outage
13
Public Works During the outage the wastewater system utilized nine (9) emergency diesel generators to supply power for lift stations and two (2) emergency diesel generators for water plant operations. The wastewater/water department did an exceptional job in keeping public water and wastewater services in normal operation during this outage. As noted in Appendix (E), the Environmental Superintendent commented the waste water lift stations were at critical levels with only a one‐hour black out period. This is a significant factor to understand for future outages. A minor electric outage could quickly cascade into a major health and safety issue if lift stations are out of power and the effluent backs up into customer’s homes and businesses. The Wastewater plant operator will be included in emergency notifications to ensure they have adequate time to respond before sewage begins backing up into people’s homes. Community Impacts Communications/Media In times of emergency effective communications to the public is crucial! In this case we had mixed results. One fundamental problem was the outdated Electric Department notification procedures. In the haste to report to the media the situation for a public update, we did not notify important internal organizations, such as the Police & Fire Departments, Public Works, and Customer Service until the next day. Notification to these departments would have significantly improved the overall City response and initial public understanding of the fault. As the situation developed, daily updates were provided to the media and to all city staff, see Appendix (F). In addition, department staff visited the local media with system drawings to help explain the nature of the fault. The intent was to provide additional technical details to the media so they truly understood the needs of the utility in dealing with the fault. The community of Sitka depends upon public notifications from the local radio stations for emergency notifications and the stations themselves do a good job broadcasting public emergency updates. However, both radio stations currently do not have emergency generators installed. It is our understanding that both stations are presently working on installation of emergency generators to provide power during electrical outages. All community entities that have installed emergency backup diesel generators were contacted and requested to maintain these machines on line to reduce system load. Appendix (E) provides a detailed summary of these participants and their comments and suggestions for future events. These members were invaluable in reducing system load Thimbleberry Outage
14
and allowing more customers to receive electric service during the duration of the outage. Many unexpected community issues surfaced during these periods of generation deficiencies. Rolling Blackout Impacts ƒ There are several home health patients that were seriously impacted by the rolling blackouts. Many were on an oxygen system that produced oxygen with commercial power and only had a limited one hour back‐up supply. The one‐hour blackouts created a very real health issue for these patients. On October 26, we met with Sitka General Hospital staff to address this issue and agreed additional work and coordination through the Emergency Planning group will be required. The short term solution for future events would be the evacuation of these patients to the hospital or other health care facility until normal electric service is restored. ƒ Some business’s that were subject to the rolling blackouts simply closed down for 3 days due to the interruption in service. ƒ The Sitka Sound Science Center has critical water supply pumps in the headwaters of Indian River and had to trek upstream to reset these pumps after each power shift. This was especially difficult after normal working hours when staff was normally not present. ƒ There were at least two reports from the Fire Department that customers had been in the middle of an operation, such as ironing or cooking when the power went out and they forgot to turn off the appliances resulting in near disasters when power was restored. ƒ The exhaust for the Pioneer Home emergency diesel discharges onto a side street and this exhaust generated several complaints from local business owners due to noise and exhaust fumes. ƒ Several key community agencies/functions were subject to rolling blackouts that interrupted important functions for the greater Sitka community: Main Post Office was closed for significant periods of time, GCI’s main communication transmitter site was impacted by rolling blackouts that disrupted communications services, the contract fuel vendor for the city was only able to fuel his truck a few Thimbleberry Outage
15
hours a day when several loads of fuel were required for city operations. ƒ
Daily schedules for rolling blackouts were not always according to the proposed schedule or time frame. The department must conduct a public information campaign or utilize flyers in monthly bills to clearly convey what happens during this type of situation and why customer flexibility will be required. Economic Impacts Direct Costs The loss of electric service results in economic losses and expenses for all citizens of the community. There are direct costs of replacing low cost hydroelectric generation with diesel, disruption of business for local vendors, loss of revenue for the utility and local business, cost of repair, and many other harmful effects. The following lists the direct City/Utility costs as a result of the fault. Direct Expense (actual) Diesel Fuel ‐ Jarvis Lube Oil ‐ Jarvis Diesel Fuel PW Equipment Rental Materials Contract Labor Labor Labor Total Cost 46,876 gal at $2.45/gal 145 gal at $11.81/gal 1,515 gal at $2.76/gal Track Hoe Line Repair Chatham Electric Department O/T PW Department O/T Direct $115,072
$1,712
$4,181
$5,995
$6,780
$14,509
$34,475
$3,800
$186,524
Indirect Costs There are many indirect costs resulting from this fault including; loss of productivity, disruption in normal activities, unscheduled business closures, loss of sales tax revenue, anxiety and stress for home‐health patients, and other impacts. Indirect Expense (estimated) Customer Generation Thimbleberry Outage
Customer Owned Diesel ‐ 1.6 MW 16
$22,800
Lost Revenue Lost Revenue Trail Repair Business Loss Electric Sales 262,336 kWh at $0.092/kWh Sales Tax Damage to Thimbleberry‐Heart Lake Trail Private $24,245
$7,500
$9,500
$150,000
Total Cost Indirect $214,045
Total Event Costs Direct Indirect Total $186,524
$214,045
$400,569
As noted, the total cost for this 70‐hour system outage is over $400,000. Should the event occur in winter conditions the costs and impact to the community will certainly be more severe. Recommendations o New emergency contact group email lists have been created that notify City departments, customers with emergency generators, and updated media contacts. o For future long‐term outages, the electric department will meet in person with appropriate city staff and emergency responders to update all on the situation. o The Electric Department will utilize the new Public Service Announcement (PSA) feature on the City’s website to provide updated information. The effectiveness of this may be impacted by customer’s lack of electric and/or communications services during an outage. o Department will plan and continue efforts to permit, procure, and install adequate standby/emergency diesel generation to meet system demand. A survey of Southeast Utilities indicates that with the exception of Sitka, all communities at present have back‐up diesel capacities that exceed normal system load. o Aggressive right‐of‐way clearance and danger tree removal will continue for the entire 69 kV transmission line. Thimbleberry Outage
17
o Evaluate alternative circuits and feeders to involve all customers in future rolling blackouts or curtailments. o Investigate options for redundant transmission line. This will require FERC license amendment. o Perform line maintenance on the Thimbleberry‐Heart Lake line segment as soon as possible. This should occur during warm weather when system load is at a minimum. So what’s the bottom line? In all likelihood, this will not be the last time we have an outage due to a failure of the transmission line. The Electric Department will be prepared for that eventuality ‐ however, the costs and consequences could be severe. The critical point highlighted by this event is the simple fact; we do not have adequate standby/emergency diesel generation! Fortunately, Sitka is a resilient and innovative community. Note the following photo which was inspired by this event – congratulations to the winners of the 2010 Sitka Stardust Ball costume contest, the Rolling Blackouts. Thimbleberry Outage
18
Thimbleberry Outage
19