Lessons Learned: Minor Power Outage Becomes

Lessons Learned: Minor Power Outage Becomes Catastrophic Situation
Steve Wilder, President & COO, Sorenson, Wilder & Associates
The Agility Story
Bob Boyd, President & CEO, Agility Recovery
**To download slides from today’s presentation,
please visit http://agil.me/hospitalpower
For audio, please call: 702‐489‐0005 then enter Access Code: 835‐525‐706,
or listen via your computer’s speakers
The Speaker: Steve Wilder
•
15 years as hospital Risk Manager and
Safety/Security Director
•
3 years Corporate Risk/Safety Officer for
15 long term care facilities
•
2 years as Corporate Director of
Safety/Security for a major healthcare
system, including 9 hospitals and 15 long
term care facilities
•
Current President and COO of consulting
firm, working with over 100 hospitals and
long term care facilities across the nation
on disaster preparedness, disaster
response and management, business
continuity and disaster recovery planning.
The Hospital
• Hospital located 50 miles south of Chicago, IL
• 250 bed Catholic hospital
• Only Trauma Center in area
• OB, Surgery, three ICU’s, Behavioral Health Unit, two
Med/Surgical units, a skilled nursing unit, and a private
infirmary for the nuns.
The Scenario
• 170 patients
• 4 ventilator patients in the
medical intensive care unit
• 1 surgery in progress
Local power outage
occurred at 2:15 PM on a
85 degree summer day.
The Event
Within 8 seconds, the
emergency generator kicked on
and was functioning properly.
The hospital had power.
At this time, normal hospital operations were not impeded.
The Event
But then, about 30 minutes after the outage……
• The speed governor on the generator failed.
• The fan belt tore in half and flew off the generator
and completely destroyed the fan guarding system.
In other words, the
generator exploded…..
The Event
The hospital lost all power.
* Generator failure in the field is very common.
The Response
Priority 1: The Patients
• Stabilize patients compromised by power loss
• Reassure patients who
were medically stable but
fearful and uncertain.
The Response
Priority 2: Get the hospital back online
• The generator was found to be completely out of
service and would take days to repair.
• In risk assessments, the risk of a complete catastrophic
generator failure scored low for probability.
• No backup systems were in place. No written backup
plan existed.
The Response
Priority 2: Get the hospital back online
• Began looking for gas powered generators and drop cords.
• A call was placed to the local fire department for portable
generators. They put out a mutual aid call for portable
generators from neighboring fire departments.
The Response
• Nursing staff continued to manually ventilate patients in
10 minute rotations.
• Within two hours, the hospital
was surrounded by fire trucks
with built-in generators. These
generators were used to supply
power to critical needs areas.
The Response
By now, the time was 6:30 PM.
The sun would set at approximately 8:00 PM.
The Response
Finding a Power Source
• Senior management
rushed to secure a similar
sized portable generator.
No one knew who to call.
• Hospital Board Member identified an option in St. Louis.
• A generator was identified. The time frame for it to arrive
onsite - 8 to 10 hours.
• It was after 7:00 PM.
The Response
Finding a Power Source
• Began to rotate generator equipped fire apparatus
every two hours throughout the night.
• An external staging area was established where the
original generator was located and protected.
• An electrician set up the site so generator could
immediately connected when it arrived.
• Generator arrived at approximately 4:00 AM the next
morning.
• Generator power was finally restored.
The Resolution
• The generator powered the hospital for 6 more hours
before the electric company was finally able to restore
electricity to the hospital.
• In total, the ordeal stretched over about 20 hours
• Crews worked around the clock to reconstruct the
damaged generator.
Lessons Learned: TACTICAL
• Utility Management plan
updated.
• Training updated to address
complete power failure, with
refreshers once a year.
• Redundant power systems established.
• Amount of emergency response equipment and
crash carts in the ICUs doubled
Summary
Lesson 1.
It can happen to you.
Summary
Lesson 2.
Plan for the unexpected.
Summary
Lesson 3.
Take an all hazards
approach to planning.
Summary
Lesson 4.
Make sure you know
who to call.
Summary
Lesson 5.
Have contingency plans
in place.
Summary
Lesson 6.
Test. Update. Train.
Then test again.
Summary
That was THEN.
Over 30 years, we’ve seen many types of “disaster”:
Outages
Natural
Disasters
Fires Floods
Even Impersonations
Shootings
TODAY the lessons learned still apply.
The Agility Story
Bob Boyd, President & CEO, Agility Recovery
Agility Recovery ‐ History
Started by General Electric 23 years ago.
Saw a need to recover at or near the organization’s normal location.
Photo taken by
Agility Recovery team
What We Do
We provide 4 key Elements of Disaster Recovery
1. Office Space: Everything needed for your employees & staff to work
2. Power for the office
3. Communications: Telephone and Internet access
4. Computer System: Computers, servers, printers, fax
Culture of Success
General Electric built an infrastructure to ensure success • 23 years
• Rescued 1000’s of members
• Never failed
Atlanta Distribution Center
New Vision
The industry focused on the needs of the Fortune 500. This model is too expensive for most organizations.
In 2004, Agility defined a new vision.
Agility will bring disaster recovery solutions to ALL organizations.
New Business Model ‐ ReadySuite
Agility created a solution that all organizations could afford.
For a small monthly fee, normally $495/month, you can protect your hospital.
When You Become a Member
A continuity planner will contact you and gather the info we need to recover your operations if you have a disaster. e.g.
Erin Mitchell Agility Rep
• How many employees need to be up and running?
• What are the power requirements of the facility?
• Where do you store your data?
• How do you want your phone calls handled during a disaster?
Ben Pritchard Member Services
When You Become a Member
All information is placed in a password protected site called myAgility.
myAgility is the foundation of your recovery plan.
When You Have A Disaster
• Agility’s operations team works with you to determine your needs.
• You only pay for Agility’s out‐of‐pocket expenses:  If we fly a technician to your office to set‐up computers, we charge you for the airfare, but not the time.
 If you need a generator, we'll deliver it and bill you our exact costs.
 If you need a server, we take one from our stock and ship it to you. You pay for the shipping. Agility Quickship Case
When You Have A Disaster
• Regardless if you have Agility or not, you would still have the same recovery needs. • But Agility will get it done faster, cheaper and more effectively.
• Most importantly, we get it done every time.
ReadySuite Mobile Office
Disasters Happen
Sometimes they’re small like a phone outage. Disasters Happen
Sometimes they’re big like a tornado.
Photo taken by
Agility Recovery team
Disasters Happen
If you don’t have a plan your organization will be forever altered.
Photo taken by
Agility Recovery team
Disasters Happen
• During a disaster there are more important things to focus on instead of trying to rebuild your infrastructure. • When you’re most vulnerable to being overcharged and underserved, you will have a partner you can trust:  23 years
 1000’s of recoveries
 Never failed
• Agility doesn’t profit from your disaster.
Bobbi Carruth Agility Member
Worthington Federal Bank
Ask Yourself…
• Do you believe a disaster could happen?
• Do you believe that without a plan, your operations will suffer?
• Do you believe Agility, after 23 years and 1000’s of recoveries will be there? DCH Credit Union Agility Members
Tornado Recovery
Ask Yourself – Do You Believe?
If you answered, “Yes,” then Agility is a fairly easy decision. We welcome you as a member.
Prepare to Survive.
QUESTIONS?
Bob Boyd
President & CEO, Agility Recovery
[email protected]
704-927-7922
**This presentation has been recorded and a link will be
sent out tomorrow to all registrants.
**To download slides from today’s presentation, please
visit http://agil.me/hospitalpower
Questions? Upcoming Agility Webinars
www.agilityrecovery.com/about/events
Title: Business Survival: Business Continuity and Disaster Recovery Basics
Date: July 31
Time: 2:00pm EDT
Facebook: http://www.facebook.com/AgilityRecovery
Twitter: http://twitter.com/AgilityRecovery
Newsletter: http://www2.agilityrecovery.com/about/52-tips
Website: www.agilityrecovery.com