How to Plan for, Respond and Recover from Calamity

Prepared for the Worst: How to Plan for, Respond and
Recover from Calamity !
Jim Hall, BC/DR Practice Manager, Bowne Management Systems!
February 17, 2014!
Agenda!
•  Disasters – types, costs, trends, phases!
•  Four keys to success!
•  Deeper dive re: technology !
•  Effective BC/DR planning!
•  Grant funding opportunities!
A Little About Me!
•  U.S military veteran !
•  IT implementer and consultant since 1989!
•  Have worked for NYPD, FDNY and NYC OEM!
•  NYC 9/11 - Emergency Mapping and Data Center
in City EOC through January 2002!
•  Other local disasters (e.g. Flight 587 crash,
Northeast Blackout, NYC Transit Strike) !
•  FEMA-certified Professional Continuity
Practitioner (PCP) !
A Little About Bowne!
•  IT company founded in 1982!
•  Headquartered in NYC area (in Mineola on
Long Island) !
•  Over 50 full-time staff!
•  Our primary focus is local government !
•  Part of the Bowne AE&T Group!
Definition: A Disaster Is …!
“A situation created by natural and or man-made
events … which demands total integration and coordination, by those responsible for administration of
the affected region including:
1. all rescue, relief and life support systems
required to meet the needs of the victims,
essential transportation and communication
systems.
2. repairs to the infrastructure.
3. post-disaster rehabilitation and recovery.”
(G. N. Ritchie, et al. 2001)
Disasters Come in Many Forms!
Source: Munich Re, “2011 Natural Catastrophe Year in Review,” January 4, 2012!
Source: Symantec’s 2012 SMB Disaster Preparedness Survey!
Source: Swiss Re!
Disasters Are:!
•  Increasing in frequency!
•  More varied that ever!
•  Increasing in financial impact!
•  More dangerous that ever!
The Burden for Municipalities!
•  Insurance only provides partial coverage for
most!
“By failing to prepare,!
you are preparing to fail.” !
- Benjamin Franklin!
Phases of a Disaster!
•  Mitigate – actions taken beforehand!
•  Respond – actions taken immediately !
•  Recover – activities to return to full capacity/
operations!
•  Restore – re-building over time !
Phases of Disaster!
Mitigate!
Disaster
strikes
Respond!
Recover!
Restore!
Timeframes!
Mitigate!
Disaster
strikes
Respond!
Minutes to!
days!
Recover!
Days to!
months!
Restore!
Weeks to!
years!
Operations Require Technology!
•  Municipal operations require access to
information!
•  Information is in digital form more than ever!
•  Technology provides access to the
information!
•  Therefore, operations require technology
more than ever before!
Disaster Timeline for Technology!
Disaster
strikes
Last!
backup!
System/database!
restored!
Last backup!
restored!
Data Perspective!
Data loss
Downtime
Potential
additional
downtime
Disaster
strikes
Last!
backup!
System/database!
restored!
Last backup!
restored!
Overall Systems Perspective!
Recovery point
objective (RPO)!
Data loss
Recovery time
objective (RTO)!
Downtime
Potential
additional
downtime
Disaster
strikes
Last!
backup!
System/database!
restored!
Last backup!
restored!
A Quick Disaster Lexicon!
“Disaster management” - NYS Archives!
“Response and recovery” - FEMA!
“Continuity of operations (COOP)” - FEMA,
public safety agencies!
“Hazard mitigation” - FEMA!
“Risk assessment” - FEMA!
“Disruption” - Gartner!
We don’t get hung-up on the words. !
Four Keys to Success!
1.  Analyze operations and resources!
2.  Prioritize!
3.  Decide and document!
4.  Mitigate!
1. Analyze Operations + Resources!
•  Focus on functions (instead of an org chart)!
•  Understanding how municipal operations/
functions are tied to their related resources is
critical to success!
Resources!
•  Evaluation of any municipal operation/
function should include assessment of: !
- 
Work flow including inputs, outputs & processing!
- 
Staffing/skills!
- 
Information/data/records!
- 
Technology!
- 
Constraints!
2. Prioritize!
•  Determine relative priorities for all municipal
operations and functions:!
-  Life and safety!
-  Transportation/infrastructure!
-  Financial and legal!
-  Time/attendance!
! and payroll!
-  Full operations!
3. Decide and Document!
•  Define responsibilities!
•  Make key decisions beforehand!
•  Plan for communication!
•  Enable action!
•  Document !
4. Mitigate!
•  Perform your action items that will reduce risk
and/or eliminate threats !
•  Use phasing!
•  Coordinate !
•  Test and fix!
•  “Advance to Go” … !
Examples of Mitigation Activities!
•  Develop mitigation standards, regulations, policies and
programs (e.g. updated land use/zoning policies,
building codes)
•  Optimize dam safety program, seawalls and levee
systems
•  Acquire flood prone and environmentally sensitive lands
•  Retrofit/harden/elevate structures and critical facilities
•  Relocate structures, infrastructure and facilities out of
vulnerable areas
•  Improve communications, warning and evacuation
systems
Results of Mitigation!
•  Get ahead of events
•  Maximize your ability to act effectively and
efficiently
•  Reduce risks to public safety and potentially
save lives
•  Prevent or reduce property damage
•  Reduce costs and economic losses
•  Protect public health
The Preparedness Cycle!
A FEMA version
“You can never protect yourself 100%.
What you do is protect yourself as
much as possible and mitigate risk to
an acceptable degree. You can never
remove all risk.” !
- Kevin Mitnick!
Deeper Dive re: Technology!
•  Information is the “life blood” of municipal
operations!
•  Municipal reliance on technology is
increasing !
•  Volume of data used is increasing fast!
•  Threats to access to information are growing
in number and potential negative impact!
Causes of Unplanned Downtime!
20%
Environmental
factors, Hardware,
Operating system,
Power, Disasters!
40%
Operator errors!
40%
Application
failure!
Source: Gartner!
Technology-based Solutions!
•  Backup and recovery!
•  Redundancy!
•  Security!
•  Replication !
•  Imaging of system components !
•  Virtualization !
•  Remote hosting!
!
Backup and Recovery!
•  Normally to tape or online storage!
•  Options - on network, removable media or to
remote host !
•  Pros - lower cost, generally reliable!
•  Cons – slow, often labor intensive, often
hardware/media dependencies!
Redundancy!
•  Adding additional systems and network
components to minimize or eliminate
SPOF(s)!
•  Options – “hot” or “cold”, available at any
infrastructure tier !
•  Pros – can be fast, generally reliable!
•  Cons – high cost, labor intensive,
configuration can be complex and is often colocated creating SPOF(s)!
Security!
•  Beyond firewalls
and anti-virus!
•  Look across the
OSI layers and
identify
vulnerabilities!
•  Assures
comprehensive
solution !
Replication!
•  Often used for data tier to protect information !
•  Options – real time or delayed !
•  Pros – fast, generally reliable!
•  Cons – higher cost, labor intensive,
configuration can be complex and sometimes
co-located to remain SPOF(s)!
Imaging of System Components !
•  Create snapshot of system component(s)!
•  Options – physical or virtual, full or partial !
•  Pros – recovery can be fast and accurate!
•  Cons – image is not real time, higher cost,
often technology dependencies!
Virtualization!
•  Creates IT assets that are platform
independent !
•  Options – can replace physical infrastructure
or compliment it !
•  Pros – fast, cheap, reliable, easy
administration!
•  Cons – some cost, takes time to convert
physical to virtual and re-direct users!
Remote Hosting!
•  External hosting of IT assets (“cloud”) !
•  Options – public or private, permanent or
elastic !
•  Pros – fast, cheap, reliable, excellent SLAs,
reduced labor and infrastructure costs, regain
space!
•  Cons – perception of risks, can be hard for
municipal governments to procure!
Requirements Must Drive the Solution!
•  Most municipalities have a hybrid IT
configuration!
•  There is no single off-the-shelf solution!
•  Your IT protection scheme should be based on
your RTOs and RPOs!
•  The balance between need and cost is critical!
Effective BC/DR Planning!
•  Use a team with relevant knowledge (muni, tech,
PE as appropriate)!
•  Collect the right information !
•  Analyze it to be prepared to respond, restore and
recover!
•  Understand and plan for resourcing!
•  Pre-define responsibilities!
•  Communicate with participants!
•  Exercise/test and adjust!
“Plans are nothing.!
Planning is everything.” !
- Dwight D. Eisenhower!
Effective BC/DR Planning (cont.)!
•  Document enough to enable effective
communication, ready access to information
and action!
•  Plan to update any documents developed!
•  Favor action over documentation !
Mitigation is Critical to Success!
•  Define mitigation action items by phases!
•  Complete them …!
Grant Funding Opportunities!
•  NYS Archives Local Government Records
Management Improvement Fund (LGRMIF)!
- 
- 
Competitive up to $75,000!
Applications accepted once a year (3/3/14) !
•  FEMA Hazard Mitigation Grant Program
(HMGP)!
- 
!
More funding available, but only in time windows!
FEMA Hazard Mitigation Plans!
•  Enable receipt of federal grant funds postdisaster!
•  Keep your HMP up-to-date!
•  Make sure it is accurate and complete – e.g.
address all viable natural, technological and
human threats!
•  FEMA prefers to fund projects defined in your
HMP!
Prepared for the Worst: How to Plan for,!
Respond and Recover from Calamity !
Questions?!
Thank you.!