DIB 4 and DDF

DIB 4 and DDF
Michael Menousek
Software Architect, DIB & DDF
[email protected]
UNCLASSIFIED
In the beginning…
 Goal: Share Information between independent DCGS
systems
 Approach:
– Joint Governance Board (MET)
– Create the DCGS Integration Backbone
– Standardize ISR Discovery and Retrieval via federated catalogs
 Adopt compatible Data Format (Schemas)
 Adopt compatible Protocols (Services)
Briefer: [email protected]
Addressing Timely ISR Information
Challenges
 Pre-DIB: Historically, data has been available in
separate databases accessible through distinct queries
of those databases.
IMINT
Predator
MASINT
SIGINT
DATA
DATA
DATA
DATA
 Significant time seeking information
 Less time spent on analysis & fusion
DATA
 With DIB: Exposing data allows analysts to retrieve
information with a single query.
IMINT
Predator
RA
RA
RA
 Query Service
 Retrieval Service
 Alert for New Data
• Geo Position
• Time Stamp
• Keywords
• Combination
SIGACTS
TACREPs
Exploitation Assets
HUMINT
MASINT
SIGINT
DATA
DATA
DATA
RA
DATA
RA
DATA
HUMINT
SIGACTS
TACREPs
Exploitation Assets
RA = Resource Adapter
Approved for Public Release (66ABW-2011-0541)
3
DIB - Providing Data Interoperability
DCGS Integration Backbone (DIB)
• A common set of GPR code & commercial Open Source Software services & standards
• Used by participants in DI2E to connect distributed ISR nodes into a cohesive data
sharing environment
Why DIB?
Visible
Key DIB Guiding Documents
• Help discover the right information
DCGS Acquisition Decision Memorandum [24 Oct 03]
• Directed Services to incorporate the DCGS Integration Backbone
Accessible
• Requested USD(I) lead an effort to develop a more robust
• Make data retrievable
Understandable
• Standardized Metadata
Trusted
(DIB) standards
governance process
JROCM 117-04 [9 Jul 04]
• Directed Service DCGS programs to incorporate the Air Force DCGS
Block 10.2 DIB and the DIB architecture
• Directed all future Service DCGS related capabilities documents to
include this requirement
• Known sources
• Clear security & access control
MET Charter [Est. 2005; Updated - Aug 10]
• Defines MET & DMO functions
• Identifies MET members and their responsibilities
DIB crosses operating systems, integration platforms & databases
to provide data interoperability
4
Today’s DIB
 Completely redesigned for flexibility via DDF
– Software choice (OSS/COTS)
– Deployment choice
– Modular architecture
 Community-oriented
– Contributed capabilities
– Developer flexibility
 Faster… always faster!
Briefer: [email protected]
Distributed Data Framework
 The “Engine” of the DIB
– Modular container – “App”-based
– Federated Metadata Catalog
– Fundamentally Service Bus + Search + Geospatial
 Open Source
– © Codice Foundation
– LGPL License
– Publicly available on github.com/codice
 Addition and Subtraction
– DIB - DoD Special Sauce = DDF
– DDF + DoD Special Sauce = DIB
 As much ESB as it is Catalog/Search
Briefer: [email protected]
ESB Overview
 Enterprise Service Bus
– Enterprise? Not necessarily.
– Service? Yes, but not limited to services.
– Bus? Absolutely.
 Key Concepts
– Normalized Formats & Interfaces
 DDF Catalog Uses OGC Filter
– Pluggable Components
 Endpoints, Transformers
– Internal or External Services
Briefer: [email protected]
Another Popular Bus: USB
 Ever-evolving Specification
– Backwards compatible
 Common Plugs & Sockets
– Connect Keyboard, Mouse, Memory Stick,
Disk, Microwave?
 Alternate Sizes have evolved
– Still interoperable
 Bottom line?
Power of a common interface point
Briefer: [email protected]
Pre-integrated Tools
Briefer: [email protected]
DDF in use:
GeoBus 2014 Simplified Architecture
DCGS DIBs >1.2
GeoBus Web UI &
OWF Widget
Publishers:
GVS
Image Stores:
iSIS, iXP, IPL, NVS,
ABI
WFS Federated Source
Local Catalog:
Apache Solr
Clients:
DIB
OpenSearch
KML
DIB SOAP
GeoBus
Built on latest DDF & DIB
Sources:
DIB
OGC WFS
Authorization &
Authentication
OGC CSW
New GeoBus Interface
Security
Original DIB Interface
GeoAxis
CAS
Future GeoBus Interface
DIBs >1.2
OMAR,
GRiD, Jagwire
iSIS, iXP,
CASi, STIL
SAML
11
DDF for Plugfest Plus
» Promote Standards over Implementations
– Messaging: AMQP, Web Sockets, Stomp
• Determine message formats and topic names
– Services: SOAP & REST
• Determine service interface descriptions
– Leverage ESB capabilities of Camel, CXF
– Bridge Legacy Applications
• File system, FTP, raw TCP/UDP socket support
– Supports “cloud” scale as usage grows
» Brings DCGS Enterprise-compliant security
– SAML-based federated Identity & Access Mgmt
» Over time, integrate DDF capabilities
– Catalog messages for search/auditing/analysis
– Visualize messages for SA in Web UI
– Automatic, secure sharing with DI2E enterprise
Feature Overview
DIB Enterprise Suite (v4.1.0)
Federation
 Legacy Federation
– Extensively tested by DTL
 New Sources
– OGC (NGA)
– CDR (DMO & DI2EF)
 Secure Federation
– WS-Security, SAML-based. Available to Plugins.
– Working with DTL, EFT and JITC for conformance
Briefer: [email protected]
Simple UI
 Great for testing
 Our first third-party contribution
Briefer: [email protected]
Search UI
 Reference Implementation – prove out services
 Use in part or in whole
Briefer: [email protected]
DIB ES v4.1.0 Metrics
 Active by default in DIB ES v4.1.0
 Two destinations for Metrics
– DDF Catalog-specific Metrics accessible via Admin Console
 Tells nature/number of query
 UI and External API both subject to change
– HTTP Logging into standard NCSA log format
 Usable by many log analysis tools (AWStats, etc.)
 Tells IP address, URL requested, etc.
 Wiki Documentation
 https://tools.codice.org/wiki/display/DDF/Metrics+Collection
 https://tools.codice.org/wiki/display/DDF/Metrics+Reporting+Application
 https://confluence.di2e.net/display/GDIB/Install+AWStats+for+User+Met
rics
Briefer: [email protected]
Example Report for Catalog Queries
Available as
PNG, PPT,
CSV, XLS
Briefer: [email protected]
Sample AWStats Report
Briefer: [email protected]
Preview
New DDF & DIB Capabilities
Briefer: [email protected]
2014 New Capabilities
 “Built for Apps”
– Supports Information Assurance Process




Web Service Security/Single-Sign-on
Admin UI: Setup & Configuration
User Interface Reference Implementation
Asynchronous Query
– Champion: NGA
 Asynchronous Product Retrieval
– Champion: DCGS-MC
 Enhanced Search
 Metadata Neutrality
 Documentation & Software Development Kit
Briefer: [email protected]
Web SSO & REST Security
 Recommended for all non-SOAP HTTP contexts
– Works with DIB JSON, Web UI, OpenSearch, etc.
 Complements SOAP Security
– SAML-based, uses STS as authority
– STS is pluggable to LDAP, CAS, etc.
 Standardized
– Working with EFT Technical Profiles (In Progress)
 Supports Secure Federation
– User SAML will be sent with outgoing Federated queries
 Configurable
– Various SSO providers: CAS, SAML
– Various AuthN/AuthZ providers via existing infrastructure
– Configurable on any HTTP context
 Activated via Configuration
– Moving toward “Secure out of the Box”
– HTTPS with only localhost access until configuration (ala OWF)
Briefer: [email protected]
Rough Security Architecture
Briefer: [email protected]
Installation
 Unzip & Run
 New Setup Wizard
– Profiles, Customization, Configuration
Briefer: [email protected]
Admin UI
 Goal: Simply DIB Administration
– App-based: Install, Activate, Upgrade, Remove, Administer
– ‘Old’ Admin UI still available for low-level tweaking
Briefer: [email protected]
Additional 2014 Items in Work
 Metadata Neutrality
– Ingest and Export different Metadata Formats
 DDMS 2, DDMS 4.1, DDMS 5/TDF
– Pluggable
 Taxonomies
 Transformers
 Enhanced Search
– Goal: Enhance user experience and capability
– ‘Hybrid’ Federation
 Caching for Performance
 Distributed Relevance
– Targeted Searches, Range Searches
– Schema-aware searching
– Facets: “drill down”
Briefer: [email protected]
Thanks!
Time Permitting:
Questions and Answers
UNCLASSIFIED