Safety Metrics

Fuzzy Math
Question – What’s the single most
detrimental thing that ever happen to
organizational safety ?
Give it some thought –
We’ll come back to the question in a moment.
SMS Basics
SAFETY
POLICY
SAFETY
ASSURANCE
SMS
Safety
Management
System
SAFETY RISK
MANAGEMENT
SAFETY
PROMOTION
Safety Assurance - SMS
•
In a nutshell:

Measure of the safety activity and performance
Collecting data
Analyzing that data
Continuous monitoring
–
Assures meeting safety goals
–


Safety Performance Measurement
AC 120-92B
Safety Assurance
gives you confidence your
SMS is meeting safety objectives.
safety assurance requires
monitoring and measuring.
…..continuously improve the level
of safety performance and
correct where necessary…..
Question – What’s the single most detrimental
thing that ever happen to organizational safety ?
Answer – Accident Rate !
Why ?
It provides:
• A false sense of security for management
• Leaders with misleading & meaningless data
• No data regarding the health of your current safety effort
• A picture that minimalizes all the great things you do for
safety
Aviation Support
Quantitative
Hard Science
Mx, Training, Safety
Non-technical Skills
Technical Skills
Maintenance
Training
- Concrete Facts
- Following Written
Procedures
- Regulated and
Inspected by FAA
-Clearly Defined
Standards (PTS)
-Check Rides
Example
Annual Inspections
Example
Stalls, Go-Around
Simulator Schools
Metrics
61.58
Metrics
# of Inspections Completed
Qualitative
Soft Science
Safety
- Management Models
- Help Individuals
& Organizations Mitigate
-SMS
-CRM / ADM
Example
Risk Management
Swiss Cheese Model
Metrics
?
Safety-I: Avoiding Those
Things That Go Wrong
Safety
Management
is a
Science….
It Evolves !
Safety-II: Trying to make
sure that things go right
Safety Performance with 3 Std Dev.
Safety Focus:
Accidents & Disasters
Normal, Routine, Day-to-Day
Performance: Unknown and
Generally Ignored
Exceptional Performance
Gratefully Accepted
The Solution
Quantum Safety
Metrics
Safety Metrics
•
•
The term “Safety Metrics” refers to
the body of knowledge used to
quantify and measure safety
performance.
(Safety Performance, Christopher A. Janicak)
Quantum Safety
Metrics - QSM
Ashgate Publishing
July 2011
The Definition
Quantum Safety Metrics – A
methodology relating to measuring an
organizations degree of Accident
Prevention Effort (APE). Using a
numeric formula called the Sierra Scale
which quantifies organizational levels
of APE by assessing, projecting, and
improving APE thus reducing the
probability of an accident.
Provides Management
the ability to:
•Quick reference safety program health
•Quantify current status of safety goals
•Forecast impact of new safety initiatives
•Visualize investment vs. payoff
•Make data driven safety decisions
The Traditional way…
•Historical data
•Lagging indicator
•Accident Rates
•Incident Rates
•Damage to Aircraft during
towing / “hanger rash”
•Cost of preventable events
(i.e., work place injuries)
How ?
Introducing APE
Three
Assumptions
Related to Accident Prevention Effort……..
Three Assumptions
1- Some is better than none
2 - Trained is better than un-trained
3 - Resourced is better than un-resourced
The Formula
P (QE)
APE
The Formula
P (QE)
APE
Program or Initiative…
A program / Initiative can be any
formalized system, policy or process an
organization has in place to enhance
safety effort.
You determine the definition of a P/I for
your organization
Question……..
What are we doing that’s
good for aviation safety
right now?
Program Examples
•Aviation Safety Councils or Committees
•A FOQA Program
• Safety Risk Management Program
•An ASAP or Hazard Reporting Program
•Education and Awareness Campaign
•IEP, Audit, and Survey Program
•Safety Award Program
•Non Punitive Reporting Program
•Other Sub-elements of an SMS
The Formula
P (QE)
APE
Quantifiable Effect
•It’s the measurable yield of a P / I
•Example might be a number of hazards
identified then controlled or eliminated by a
given program.
For example……..
Hazard Reports
– 3
The Formula
Your Hazard Reporting Program
P (QE)
1 (3)
– 3
APE
3
Quantifiable Effect
Hazard Reporting – 3
Risk Assessment
Worksheets – 2
Internal Audit/Inspection – 1
3 Programs
6 QE
How can we increase APE ?
P (QE)
3 (6)
APE
18
Keep in mind your
APE is……
• A Barometer of your
organizations safety effort
(ROI)
• A means of trending your
safety effort
Three Assumptions
1- Some is better than none
2 - Trained is better than un-trained
3 - Resourced is better than unresourced
1 - Some
2 - Trained
3Resourced
P (QE)
APE
3
18
3
3
(6)
(6)
(6)
2
3
36
216
Flight Program
Safety Preformance Indicators (SPI) / Quatifiable Effects
2014
2015
(QE's)
Total
Total
Safety Committee / # of hazards addressed cubed
0
Safety Meetings / # of changes, updates, or training covered
squared
16
Monthly Safety Reports / # of individuals informed
25
Significant Safety Events (SSE's) / # of reports
4
Significant Safety Events (SSE's) / # of actions taken squared
0
IEP Internal Audits / # of safety actions taken squared
4
Safety Orientation Training / # conducted
0
Emergency Response Training / # of employees trained
0
Emergency Response Drills / # of drills (X 100)
0
Emergency Response Drills / # of changes resulting from drills
0
squared
Safety Survey Responses / # returned
0
Safety Survey Action Taken / # of actions taken based on
responses squared
0
Suggestions/Complaint Form inputs / # submitted
0
Suggestions/Complaint Form inputs / # of actions taken based
on submissions cubed
0
Special SMS Training / # of key personnel trained
0
External Audits / # completed cubed
External Audits / # safety actions taken cubed
0
SRM tool / # completed squared
0
SRM tool / # of safety actions taken cubed
0
SA Tool / # completed squared
0
SA Tool / # of safety actions taken cubed
0
Safety Policy Objectives met / # met on a monthly basis cubed
0
Pre-Flight Risk Matrix / # of safety actions taken as a result
cubed
0
Safety Awards issued / # awarded (X 100)
Records audit
1
FOM reviewed for revision / # of revisions or reviews cubed
8
Total APEScore
58
2016 - 2016 2016 - 2016 - 2016
1st Qtr. 2nd Qtr. 3rd Qtr. 4th Qtr. Total
125
125
1
729
113
17
16
25
6
3
1
441
25
0
0
0
4
0
0
64
13
2
4
0
1
8
0
25
9
0
0
0
0
64
0
0
0
0
0
0
6
8
216
169
512
0
0
0
0
0
0
0
0
0
0
0
0
0
8
0
0
25
125
4
64
8
0
1
18
27
0
0
0
0
8
0
6
1
2090
595
342
937
Pillar
Policy
Metric
Safety consult (number of hours)
WO Policy (number of contacts)
Policy conference calls (number calls with users)
Committee attendance SMS (number of days)
Committee outputs (number of pages)
APE
Risk Management RM Assessments Development (# of hazards)
RM Assessments Review (# of hazards)
Validation of RM (number)
Risk management conference calls (number)
APE
Assurance
Audit consult meetings (number of hours)
Audits performed (number of days)
Face to face assurance visits (number)
Airframe support (days)
ASAT Identified issues (# of issues)
Conference calls associated with assurance
(number)
APE
Promotion
Investigation reports generated (number)
SMS presentations (x number of students)
Other presentations (x number of students)
SSE Improvement Training (# of hours)
SSE Certification (# of credentials)
APE
January
10
3
0
5
0
0
0
0
1
29
0
26
0
0
1
2
20
0
131
6
18
1
56
159
Safety Metric
Accident Prevention Effort
Measure of Safety Effort
Reporting
ASTARs Pending
ASTARs Closed
Onsite Inspections/Reviews Reports
Accident Response
Accident/Incident Investigations Pending
Accident/Incident Investigations Closed
Corrective Actions Implemented
Training
Days of Safety Training
Simulator Training Days
Av. Safety Coord. Training w/in 24 mos.
Flight Evaluations
Safety Promotion
Safety Meetings
Safety Information to Field
Safety Council Meetings last 3 Months
Safeflight published within last 3 months
Progress of Safety Initiatives
CRM Implementation Plan (final)
CRM Presentations Revised
Safety Article Written for Safeflight
CRM Stage Check Meeting with Field Rep.
Safety Coordinator Conference Prep
Accident Prevention Effort (APE)
2016
Jan
QE
APE
Score
5
12
0
1
3
5
5
36
0
0
1
7
5
10
1-5
0
5
14
36
17
0
4
1
1
25
2
36
17
0
8
4
8
1
1
3
2
5
10
12
16
5
10
1
3
3
4
2
Progress
1-5
Completed
10-20
5
9
9
12
10
209
Program Mgt
60%
Safety Communications
Through Metrics
Accident
National Standdowns
Quarterly Safety Meeting
Safety Effort
Training
Prevention
Safety Meeting
(APE) Quarterly
Hazard Reporting
300
253
245
250
Closed Acc. Investigations
New Positions in QC/QA
Exe. Mgt. FBI/NTSB
Vacancy PM Training
Recurrent Safety Training Expired
APE Value
197
200
150
New Fiscal Year
No Funding
158
113
100
87
93
50
0
2014
2015
159
140
182
154
Summary
•It’s a leading indicator
•Provides management the ability to see proactive
measures taken for safety
•Provides leaders with the ability to make data driven safety
decisions
•Quantifies current safety effort
•Provides management the ability to visualize cost benefit
analysis, (ROI)
•Demonstrates value added of your safety department
•Helps leaders focus on what’s important
Questions
(and possibly answers)
Or
Comments