of the data

April 6, 2009
Obtaining local transportation data and processing
this data in order to serve a current
Transportation Framework over the Internet
Michael Leierer
Assistant Project Manager and Technical Lead
Transportation Pooled Fund: Software Tools for Sharing and Integrating GIS Data
Presentation Outline
ƒ
ƒ
ƒ
ƒ
ƒ
ƒ
ƒ
ƒ
ƒ
Project background and architecture,
Data Provider Interface,
Upload data to WA-Trans database (translation),
Change Detection and Change Management,
QA/QC Reports,
Truth of the data,
Coincident segments,
Production Data and the Data User Portal,
How to get involved
WA-Trans Work / Data Flow
Basic Data Provider
Rules of the Road
ƒ Only the Data Provider really knows their own GIS data,
ƒ Data Providers will continue to maintain their data in their own
GIS systems,
ƒ Changes to provider data, their business processes, or their
systems will be kept to a minimum,
ƒ Data Providers are encouraged to maintain connected road
centerline data at their jurisdictional boundaries,
ƒ Providers will submit their current GIS data on a regularly
scheduled update cycle,
ƒ WA-Trans processes are designed to support and facilitate
continued improvement of data by the provider
WA-Trans Conceptual Architecture
Data Provider
Report
Data to
Requester
by FTP
Preprocessing
QA/QC
Initial QA/QC
Data Provider Portal
Transformation
Process
Integration
Change
Detection
Transform and
Load Provider
Data
WA-Trans
Staging
Database
WA-Trans
Loading
Database
Postprocessing
QA/QC
Change
Management
New Provider /
Changed Data
Initiation Process
Promote Staging
To
Production
WA-Trans Control Process
WA-Trans
Production
Database
Data User Portal
WA-Trans Work / Data Flow
Data Provider Portal
Barriers to Sharing Data
ƒ Data translation / interoperability
ƒ Connectivity between disparate GIS systems
ƒ Bandwidth / internet availability
ƒ Spatial reference and metadata capture
ƒ Schema mapping
ƒ Client ease of access / interface
ƒ Documentation
ƒ Relationships
ƒ Public Disclosure Laws
WA-Trans Work / Data Flow
Transformation Process
Read Source Data
Alter source data
as necessary to
meet standards of
centralized
database
Create attribution
necessary for a
centralized
database
Map all source and
created attributes
to appropriate
database
attributes
Write changes to
centralized SDE
SQL Database
QA/QC
Why Translate Data?
ƒ Standardize the data; format, known spatial
references etc.
ƒ Make data less dependent on vendor software.
ƒ Store data to be maintained and retrieved by a
broad user group.
ƒ Store data once to use again and again.
ƒ Be able to use the same data in different
implementations and for different needs.
You need a database
WA-Trans Data Model
WA-Trans Work / Data Flow
Where does Change Detection start?
Data Provider Portal
Retrieve
source data
Alter source data
as necessary to
meet standards of
centralized
database
Create attribution
necessary for a
centralized
database
Validate
source data
Map all source and
created attributes
to appropriate
database
attributes
Read Source Data
Write changes to
centralized SDE
SQL Database
QA/QC
Data Comparison
ƒ Compare the dataset just submitted by a provider
with the most recent dataset previously submitted by
the same provider.
– Store in the database a copy of the most recent
dataset submitted by a provider.
– Compare the newly submitted data to that copy,
• after translation into the database
• but prior to any other processing
Change Detection Conceptual Flow
The “Truth” of the data
ƒ What do we really know about providers data?
ƒ We need to be careful with our judgments of
providers data,
ƒ We will have to accept that we will have some bad
data in production,
ƒ With a reliable source, QA/QC reports and user
feedback providers will possibly improve their data
WA-Trans Work / Data Flow
Provider Data and QA/QC Reports
ƒ Information for comparing with previously provided
data (ex. Number of records, number of changed,
number of added records, number missing),
ƒ Segment length information (for short segments),
ƒ Changes to route and address data,
ƒ Isolated segments,
ƒ Duplicate route descriptions and address,
ƒ Route overlaps and gaps,
ƒ Zero length address ranges,
ƒ Address range overlaps.
WA-Trans Work / Data Flow
Coincident Segments at Boundaries
Coincident Segments within a
jurisdiction
WA-Trans Staging Environment (Data Editing)
WA-Trans Work / Data Flow
Data User Portal
WA-Trans Work / Data Flow
Data Federation
Spatial
Spatial ETL
ETL
Tool
Tool
Every user group requires a particular view
Defined Output Datasets for Users
WA-Trans is currently focusing on several defined
output datasets that include centerline data for:
1. Geocoding and E911 uses,
2. Planning and event location to include all available
LRS attribution,
3. A combination of 1 and 2,
4. Data for a Collision Location system being
implemented by WSDOT and the Washington State
Patrol,
5. HPMS to meet 2010 reporting requirements,
How To Get Involved
ƒ TPF 5(108) – Software Tools for Sharing and
Integrating GIS Data,
ƒ “http://www.pooledfund.org”,
ƒ Seeking 4 more partners (at least),
ƒ We can also partner with other levels of government
and private companies,
ƒ Our goal is to having working prototypes in an Open
GIS environment as a result as well as processes
and other useful templates and information.
Tami Griffin
[email protected]
360.596-8926
Michael Leierer
[email protected]
360.596-8927
www.wsdot.wa.gov/mapsdata/transframework/default.htm
Transportation Pooled Fund: Software Tools for Sharing and Integrating GIS Data
http://www.pooledfund.org/projectdetails.asp?id=340&status=4