Oh My, That Never Happens with My Data – Improving

Oh My, That Never Happens with My Data – Improving
Data Quality Using Custom Monitoring and Alerting with
Information Steward's Data Insight Module
Justin Breen
Chad Seward
Session #3068Summer 2015
about
One of the world’s largest
regional airlines
 73 destinations
across Canada and
the U.S.
 4,184 employees
 124 aircraft
 ~800 flights each
weekday
 30,000 passengers
carried daily
 9.9 million
passengers carried
annually (2014)
 $1.7 billion operating
revenue (2014)
about
about us
Chad Seward
Manager, Enterprise Architecture & Design
Jazz Aviation LP
As Manager, Enterprise Architecture & Design at Jazz, Chad oversees the areas of Enterprise
Architecture, Process Design and Procurement at Jazz. In this role, Chad provides ongoing
organizational consulting and solution development that meet the needs of the organization and are
aligned with corporate strategy. He monitors industry trends for strategic investment opportunities
that enable Jazz to execute its corporate plan.
Prior to Jazz, Chad was Product Manager for Clear Picture Corporation where he led the software
development process for applications that powered services for customers such as IBM, Accenture
and EMC. He has also worked in the area of Software Development for SolutionInc and xwave.
Chad holds a Master’s Degree in Computer Science from Dalhousie University and a Bachelor’s
Degree in Computer Science from the Technical University of Nova Scotia. He is certified in ITIL v3,
TOGAF v9 and is a member of the Open Group.
Chad Seward
Manager, Enterprise Architecture & Design
Jazz Aviation LP
310 Goudey Drive, Halifax International Airport
Enfield, Nova Scotia, B2T 1E4
(W) 902.873.5055 (C) 902.483.3296
[email protected] www.flyjazz.ca
4
about us
Justin Breen
Systems Analyst, Maintenance and SQE
Jazz Aviation LP
Justin joined Jazz 5 years ago and is currently working as a Systems Analyst for the Maintenance department
and the Safety, Quality and Environment business unit. Responsible for software application development,
report development and maintaining the software used by the business units.
Justin became part of the Business Intelligence movement at Jazz when the Maintenance department
engagement started 2 years ago.
Before joining Jazz, Justin has worked as a Software Developer for over 10 years.
Justin Breen
Systems Analyst, Maintenance and SQE
Jazz Aviation LP
310 Goudey Drive, Halifax International Airport
Enfield, Nova Scotia, B2T 1E4
(W) 902.873.5015
[email protected] www.flyjazz.ca
LEARNING POINTS
 Think you got clean data? Think again!
 Data Quality is part man / part machine, like the
Terminator
 Don’t be shy about taking charge of data quality and
innovating beyond the limits set by the product.
What we are going to talk about today
 What to do when good
business processes go
wrong
 How data quality helps
keep your ETL running
smoothly
 Humanizing your data
quality program
 Overview of Information
Steward’s Data Insight
Module
 Design & Setup of our
custom monitoring &
alerting solution
 Don’t give your users a
barrow full of monkeys
Where are you at?
8
Data Quality Challenges
 Dealing with source system data quality issues is a fact of life when
integrating into business intelligence solutions.
 Often business processes shortcomings leave data gaps that come to
light during the rigor introduced by ELT and data validation.
 For Jazz, integrating our Aircraft Maintenance and Resource
Management systems with SAP's Business Objects platform allowed
us to witness these gaps first hand.
9
10
Data Quality Success Story
11
Maintenance BI
 In 2013 Jazz launched a Business Intelligence project for the
Maintenance department
 During the course of the project we cataloged a list of data validation
checks
 We knew that SAP had a data quality tool and contacted SAP for a
demo of Information Steward
 Introduced the Data Insight module into our environment
 Moved from a tool for analyzing BI source data to an operational tool
used to monitor data quality in production systems.
12
The Tool: Information Steward’s Data Insight
 Create Projects & Scorecards for monitoring data quality
 Profile source system data
 Enable alerting
 Capture failed/erroneous records
13
A Barrow of Monkeys
Business Problem Run Simple
Scenario 1
 Users selecting incorrect overtime reason codes when filling
out timesheets.
Scenario 2
 Aircraft parts are required to be effective to a specific
aircraft type before it can be requested during a work event.
15
Let’s use Information Steward’s Data Insight
PROBLEM:
Users do not want to log into Information Steward to see errors.
But they want to know exactly what the error is and how to fix it.
SOLUTION:
Created a custom emailing application, on top of the Data
Insight backend, that sends the failed records to specific users
via email.
16
17
Our Recipe for Simple Data Quality Success
18
CMC Information Steward
19
Custom Emailer – Data Insight Rule
When setting up tasks in Data Insight we take advantage of the failed data option.
20
Custom Emailer – Data Sources
Custom database
for custom rule management
InfoSteward Failed Data database
to determine the rules and failed data
21
Custom Emailer – Web Admin Page
22
Custom Emailer – Windows Service
23
Custom Emailer – Email Example
24
Recap: Our Recipe for Simple Data Quality Success
25
BEST PRACTICES
 Don’t overlook sizing of the environment
 Victim of our own success.
 We overloaded our data services box and needed to scale
out.
 Size & resize & size again.
 Leverage BI as an operational best practice
 drove down manual work
 improved our business processes (e.g. OT Codes)
KEY LEARNINGS
 Trust but verify. Don’t ever assume data quality without
verifying first. Expect to have data quality challenges.
 Humanize data quality, think about the people you need
to reach, make them care & meet them where they are
at. Don’t give them a barrow of monkeys.
 Sometimes you have to roll up your sleeves and make
the tools work for you and your business.
Questions & Answers
28
STAY INFORMED
Follow the ASUGNews team:
Tom Wailgum: @twailgum
Chris Kanaracus: @chriskanaracus
Craig Powers: @Powers_ASUG
SESSION CODE
3068