Presentation

Transcription

Presentation

IBM Change Data Capture and Replication
Solutions
© 2008 IBM Corporation
Information Management Software
Unlocking the Business Value of Information
Financial
Risk Insight
Customer & Product
Profitability
Workforce
Dynamic
Optimization Supply Chain
Multi-channel
Marketing
Flexible Architecture
2
The IBM InfoSphere Vision
An Industry Unique Information Platform
• Simplify the delivery of Trusted Information
• Accelerate client value
• Promote collaboration
• Mitigate risk
• Modular but Integrated
• Scalable – Project to Enterprise
3
InfoSphere Warehouse – Top Warehouse Solution
Best-of-breed ETL Tools
Best-of-breed Warehouse Engine
Best-of-breed Reporting/Ad-hoc Query/Dashboard
Data Assets
Data Mart
Structures
Relational Data
Management
Aggregation
InfoSphere
Warehouse
Report/Ad-hoc
Query
Transformation
ERP Data
Integration
DashBoard
Flat-file Data
4
Datastage
InfoSphere
Cognos
InfoSphere Information Server
Information Services Director
Publish SOA services for information
integration and access
Business Glossary
Enterprise Data Dictionary
QualityStage
Data Quality: Standardize,
Correct & Match Data
DataStage
Information Analyzer
Data Source Profiling &
Problem Diagnosis
Global Name Recognition
Extract, Transform, and
Load in Batch or Real-time
Recognize & Classify
Multi-cultural names
Metadata Server / Metadata Workbench / FastTrack
Manage and track consistent metadata across information integration tasks
and automate generation of data flow logic
Parallel Processing
Rich Connectivity to Applications, Data, and Content
5
Federation Server
Virtualize access to
disparate information
CDC & Replication
Deliver and replicate
changed data
GUI Interface
6
Data Quality platform
•Foundation to any success business projects
•Source System analysis
•Auditable Data Quality
•DC rules will be stored in MDR
•DC rules will be implement in ETL
•DC result will be record
Data Quality management
7
•Set up Data source view by federation
•Auto generate verify SP with MDR
•Verify result will be store with MDR
Metadata management
Data Integration Platform：
：Infosphere Warehouse
Ease of
Design and
Manage via
Tooling
Eclipse Design Studio
DBA
BI Designer
Web Administration Console
Data Mining
Unstructured
Analysis
In-Line
Analytics
Cubing
Services
Extend
Value via
Analytics
Simplify
Warehouse
Infrastructure
BI
Specialist
SQW:
Design, Deploy,
Refine
Data Warehouse
pSeries Server
cpu 1
cpu n-1
pSeries Server
cpu 1
cpu n-1
cpu 2
cpu 2
cpu n
I/O Channels
BCU 1
8
WLM:
Workload
Management
DB2 ESE/DPF/QP/Compression
cpu n
I/O Channels
BCU 2
…
pSeries Server
cpu 1
cpu n-1
cpu 2
cpu n
I/O Channels
BCU N
DBA
IT Specialist
Introducing the “IBM Smart Analytics Solution”
Overcoming The Obstacles To Business Transformation
An integrated, high-performance analytics
solution for accelerating delivery of insights
for faster, smarter action
Able to adjust and grow based on your
company’s ever changing business needs
broad analytic capabilities
powerful warehouse capabilities
scalable and fully-integrated IBM hardware
set-up services and single point of premium
support
Delivering results in days instead of months
9
IBM Smart Analytics Solution
What's in the system? Infosphere Warehouse and Cognos
Deeply optimized by IBM experts
Flexible growth to meet changing business needs
Analytics Software Options
Business Intelligence capabilities
Cubing Services
Text Analytics & Data Mining
. . . more to come
Powerful Data Warehouse
Warehousing Platform
Advanced Workload Management
System Automation
Hardware & Services
Server Platform
Storage capacity
Build, Deploy, Health Check & Premium Support Services
Delivering results in days instead of months!
10
IBM Smart Analytics Solution – Scalable
Warehouse
Seamless Growth
Add capacity & analytic capabilities as requirements evolve
Start right
11
Add more capacity
Add new analytic capability
11
InfoSphere Information Server
Information Services Director
Publish SOA services for information
integration and access
Business Glossary
Enterprise Data Dictionary
QualityStage
Data Quality: Standardize,
Correct & Match Data
DataStage
Information Analyzer
Data Source Profiling &
Problem Diagnosis
Global Name Recognition
Extract, Transform, and
Load in Batch or Real-time
Recognize & Classify
Multi-cultural names
Metadata Server / Metadata Workbench / FastTrack
Manage and track consistent metadata across information integration tasks
and automate generation of data flow logic
Parallel Processing
Rich Connectivity to Applications, Data, and Content
12
Federation Server
Virtualize access to
disparate information
CDC & Replication
Deliver and replicate
changed data
Business Challenges Driving Real-Time Data
Integration
Dynamic Warehousing &
Business Intelligence
and Reporting
•
Yesterday’s data inadequate for
inventory and purchasing
decisions
Data Synchronization
and Replication
•
We need up to date information
flowing between applications and
to ensure an up-to-date version is
always available
Real-time Event
Detection
•
Need to pro-actively monitor
and respond to business
changes
……Without Impacting the Performance of Production Systems
13
What is IBM InfoSphere Change Data Capture?
• Provides real-time change
data capture and delivery for
• Dynamic change data
integration
• Synchronization
Developers
Deliver
Architects
IBM InfoSphere Change Data Capture and
Replication Solutions
Delivers real time changed data to
Information Server, applications and
targets or message queues
• Replication
• Minimal impact on
production systems
• High scalability and end-toend performance
• Guaranteed data integrity
14
Without impacting performance of
production systems
Business Application
EDW
Production Systems
Enterprise Data
Warehouse
ETL
ERP
Billing
Finance
POS
CRM
BI
Appliance
Real-Time
CDC
Event-Driven
SOA
ESB
Web
Applications
Real-Time CDC
MDR
Real-Time CDC
15
Operational
Business
Intelligence
Metadata Repository
Up-to-Date
Web Portals
Master Data
Management
Expansive Source, Target, Platform Support
DATABASES
Source & Target
DATABASES
Source Only
TARGETS
MESSAGE
QUEUE
OPERATING
SYSTEMS
HARDWARE
PLATFORMS
DB2 z/OS
IMS
Teradata
JMS
i5/OS
IBM System i
Oracle
ADABAS
Netezza
MQ Series
z/OS
IBM System z
Sybase
CA-IDMS
Greenplum
AIX
IBM System p
MS SQL Server
CA-Datacom
Informix
WebMethods
HP-UX
HP-9000
MySQL
BEA
Solaris
Intel
Information Server
MS Windows
Sun
Cognos Now!
Linux
Itanium
DB2 UDB
DB2
i
DB2
UDB
VSAM
IAM
Flat File
16
TIBCO
17
Replication Modes: Refresh
“Replication Log”
Continuous
(Real Time)
Refresh
Push
Engine
Net Change
(Periodic)
Database Table
Refresh
(Full Copy)
18
Replication Modes: Net Change
Replication Log
Continuous
(Real Time)
Scrape
Push
Engine
Net Change
(Periodic)
Database Table
Refresh
(Full Copy)
19
Replication Modes: Continuous Mirroring
Replication Log
Continuous
(Real Time)
Scrape
Push
Engine
Net Change
(Periodic)
Database Table
Refresh
(Full Copy)
20
InfoSphere CDC General Architectural Overview
Oracle
DB2
SQL Server
:
InfoSphere CDC
Binaries
InfoSphere CDC
Binaries
InfoSphere CDC Binaries
includes Source and Target
replication engines and
configuration agent.
InfoSphere CDC
Binaries
GUI Connection
GUI Connection
GUI Connection
Subscriptions
( Replication Threads)
Subscriptions
(Replication Threads)
Win/ Unix/ Linux
Subscriptions
( Replication Threads)
Management
Console
Management Console
configuration interface
.
Access Manager controls access
to the product
. Can be run on a
separate server to allow for
centralized access
Access
Manager
GUI Connection
GUI Connection
GUI Connection
InfoSphere CDC
Binaries
InfoSphere CDC
Binaries
InfoSphere CDC
Binaries
Interface Connections. Only
connected when GUI being
used
Replication Connections.
. Only
connected when actively
Mirroring or Refreshing
Oracle
21
DB2
DB2
Building A Low Latency ODS – General Technical Architecture
Source Server(1)
Meta
Linux/Unix/Windows
Agent
Administration GUI
DB Log
Access Server/Management Console Install
Engine
DB
Target ODS
InfoSphere CDC Install
Agent
Agent
Source Server(2)
Meta
DB Log
Engine
DB
Agent
Meta
DB Log
InfoSphere CDC Binaries
Engine
DB
22
Meta
InfoSphere CDC Metadata
Can contain both source and target
metadata depending on whether server is
publisher, subscriber or both. There is no
single unified metadata instance.
Replicating
“To” An ESB – General Technical Architecture (a)
InfoSphere CDC Event Server On Different Server As InfoSphere CDC
Source Server(1)
Linux/Unix/Windows
Agent
Meta
Administration GUI
DB Log
Engine
DB
Message Queue Server
Agent
Agent
Meta
InfoSphere Event Server Metadata
Meta
Target/Subscriber metadata only..
InfoSphere CDC/Event Server Binaries
23
Meta
Engine
JMS
XML or
User Defined
InfoSphere CDC Event Server Install
JMS
XML or
User Defined
Replicating “To” An ESB – General Technical Architecture (b)
InfoSphere CDC Event Server On Same Server As InfoSphere CDC
Source Server(1)
Linux/Unix/Windows
Agent
Meta
Administration GUI
DB Log
Engine
DB
JMS
XML or
User Defined
Engine
JMS
r
XML o ed
in
f
e
D
User
Meta
Meta
Agent
Agent
InfoSphere CDC Event Server Install
Meta
InfoSphere CDC/Event Server Binaries
24
InfoSphere Event Server Metadata
Target/subscriber metadata only..
Use Cases
25
Use Cases
Real-time Integration
Need up-to-date information delivered to a data
warehouse
Need up-to-date information delivered for eCommerce applications (ie.
Downtown
gift registry)
Store
Midtown
Store
Inventory
Uptown
Store
26
Use Cases
Replication
Need to report on operational systems but cannot impact
production system – must synchronize data across 1 or more
databases
Less
expensive
DBs
Need to maintain business continuity during application
migrations, consolidations, upgrades
27
Live, Operational Reporting
• Distribute reporting workload over existing environments
• Real-time data for Business Intelligence
Production
Systems
28
Data Consolidation and ETL Integration
HP Superdome
New GSM
Billing - Oracle
Sun
New CRM
Public
Website
IBM System i
Billing
Data
Warehouse
IBM System i
Billing
DataStage
Other
Applications
(a.o. CRM)
ODS (Oracle)
29
Direct Connect with InfoSphere DataStage
• Enabling real-time response to data changes
and business events
• Low impact log-based changed data capture
• New palette stages on InfoSphere Information
Server
• Stream data changes into InfoSphere Information
Server
30
31
Features in IBM InfoSphere CDC
32
Filtering
CUST_NO
L_NAME
F_NAME
PHONE
REP_NO
58699
Smith
John
404-555-3874
45
37283
Duggan
Ira
613-555-8367
25
89863
Quinn
Fran
905-555-1296
11
89732
Muntz
Muntz
704-555-2738
25
Integrate entire systems or only a subset of data
ROW SELECT
REP_NO = 25
Table/row/column-level filtering options available
33
CUST_NO
L_NAME
F_NAME
REP_NO
37283
Duggan
Ira
25
89732
Muntz
Josie
25
Data Translations
• Convert data representations on the fly to integrate disparate systems and
database table structures
EMP
LAST
FIRST
HIRE_DATE
STAT
SALARY
MAX
1234
Moreiro
Nicole
01/05/97
A
$55,000
$60,000
2345
Ellison
Val
04/12/97
I
$40,000
$50,000
34
Transform
Fields
Derived
Fields
Concatenation
Century
Dates
EMP_ID
FULL_NAME
HIRE_DATE
STATUS
%SALARYMAX
001234
Nicole Moreiro
01/05/1997
Active
92%
002345
Val Ellison
04/12/1997
Inactive
80%
Increase
Field Size
Table Mapping Methods
• One-to-one
• Source and target tables have similar table structures
• Audit
• Generates audit trail of data transactions from source
• Adaptive Apply
• Automatically synchronizes data for dissimilar sources and targets
• Summarization
• Keeps a running total of numerical values at the target
• Consolidation: One-to-One
• Merges data from several tables into a single row
• Consolidation: One-to-Many
• Used to apply a source lookup table change to all affected target rows
35
Guaranteed Data Integrity
• Data transactions are applied at the target in the same
order as it was generated at the source
• Target acknowledges each apply operation to ensure
delivery
• Data capture continues even if communications link is
unavailable
• Data applied at target upon communication availability
36
Monitoring
• Graphical visualization of replication processes
• Event logs, alerts & alarms
• Exportable throughput & latency statistics
37
Real-Time Change Data Capture and Delivery
Sense and respond to critical Information in real time
Provide real-time synchronization between applications
Reduce burden on operational systems
Maintain business continuity
Capitalize on Opportunities
Quickly Adapt to Competitive Pressures
Gain Operational Efficiencies
Minimize Risk During Migration/Consolidation/Upgrades
Protect critical business information
38
39
Backup Slides
40
Expansive Source, Target, Platform Support
DATABASES
Source & Target
DATABASES
Source Only
TARGETS
MESSAGE
QUEUE
OPERATING
SYSTEMS
HARDWARE
PLATFORMS
DB2 z/OS
IMS
Teradata
JMS
i5/OS
IBM System i
Oracle
ADABAS
Netezza
MQ Series
z/OS
IBM System z
Sybase
CA-IDMS
Greenplum
AIX
IBM System p
MS SQL Server
CA-Datacom
Informix
WebMethods
HP-UX
HP-9000
MySQL
BEA
Solaris
Intel
Information Server
MS Windows
Sun
Cognos Now!
Linux
Itanium
DB2 UDB
DB2
i
DB2
UDB
VSAM
IAM
Flat File
41
TIBCO
Data Consolidation and ETL Integration
HP Superdome
New GSM
Billing - Oracle
Sun
New CRM
Public
Website
IBM System i
Billing
Data
Warehouse
IBM System i
Billing
DataStage
Other
Applications
(a.o. CRM)
ODS (Oracle)
42
Business Application
EDW
Production Systems
Enterprise Data
Warehouse
ETL
ERP
Billing
Finance
POS
CRM
BI
Appliance
Real-Time
CDC
Event-Driven
SOA
ESB
Web
Applications
Real-Time CDC
MDR
Real-Time CDC
43
Operational
Business
Intelligence
Metadata Repository
Up-to-Date
Web Portals
Master Data
Management
Bostik Ltd. - Real-time Analysis
Business Challenge:
Business Value:
• Corporate data consolidated on a nightly
basis from two ERP systems for analysis and
reporting to not impact daytime performance
•On-time delivery of orders up 10%
•Volume of data and reporting needs
exceeded nightly batch load capacity and
hindered business performance
•Late orders decreased by 80%
•Flexible reporting and analysis schedules
•Single view of corporate ERP data
•Batch window time slot open
Solution:
•IBM InfoSphere Change Data Capture
consolidated data from ERP systems into a
single data warehouse, transformed into a
single format
•Analyses and reporting can be conducted
with the most current data at any time
•Eliminated nightly batch window to free up
system resources for other activities
44
“While building an effective data warehouse
infrastructure and delivering good information
to users is critically important, it is just the
beginning. The real benefit comes from the
ability to make better business decisions.”
– Paul Hudson, IT Manager, Bostik Ltd.
Tiffany & Company – Data Synchronization
Business Challenge:
Business Value:
• Needed access to up-to-date inventory and
product information for online purchasing
• Increased customer satisfaction with more
accurate Online Gift Registry information
which is a significant portion of Tiffany’s
online business
•Purchases made online needed to be
accurate reflected in the inventory system,
particularly for online gift registry
•Improved inventory tracking results in more
effective inventory management
Solution:
• Implemented IBM InfoSphere Change Data
Capture
• Real-time data flows to e-Commerce
application for online customer purchasing
and back to the inventory systems after
purchases are made.
“DataMirror’s robust, real-time data
integration provides us with visibility into
inventory levels, sales reports and e-Business
channels, allowing us to better manage sales
cycles and deliver superior customer
experiences worldwide."
- Manager of Database Services, Tiffany & Co.
45
CitiStreet – Data Replication
Business Challenge:
Business Value:
• CitiStreet is one of the largest and most
experienced global benefits providers
servicing over 9 million plan participants
across all markets. CitiStreet was formed in
partnership between subsidiaries of State
Street Corporation and Citigroup
• Ensure application availability for plan
participants and sponsors.
• Business continuity - Ensure 24x7 portal
access for plan participants and sponsors
• Improve data integrity with a reduced level
of maintenance
• Maintain p2p synchronization of profile
updates (approx 175,000 updates daily) in
real time
Solution:
• Use IBM InfoSphere Replication Server to
support redundant, active single sign-on
applications for failover processing,
replicating profile changes between systems
in real time.
46
“Since nearly 10 million of CitiStreet customers
are offered 24-hour access to their retirement
accounts, the company can't afford downtime and
must be able to replicate data changes when they
happen. We fully replicate our database over
redundancy data lines, so to us the stability and
speed of that asynchronous replication is
strategic for us.”
Barry Strasnick , CIO CitiStreet
Large Telco Company - Event Detection
Business Challenge:
Business Value:
•Wanted to give their CRM system access to
current billing information.
• Increase revenue through proactively selling
more time when paid minutes gets low
•The billing system is their most critical
business system – no performance impact or
increased risk could be tolerated
• Increase customer satisfaction by
anticipating customer need
•No performance impact or increased risk to
billing system
Solution:
• IBM InfoSphere Change Data Capture is
used to detect customer transactions from
billing system and push the information to a
message queue
• Able to detect specific business events,
make that information available to their CRM
system, allowing them to automatically
respond when a customer runs low on
minutes.
47
Billing
System
Customer
Transactions
CDC
Customer
Transactions
Message
Queue
CRM
1 Box Scenario
Box 1
CLIENT
ACCESS
SERVER
source
target
Only one system required
Only Intrasystem replication possible
Only possible if box 1 is NT or Unix
48
2 Box Scenario
Box 1
CLIENT
ACCESS
SERVER
source
target
Box 2
Does not require separate access server
Set-up of replication via the client dependent on Box 1 being active
Only Intrasystem replication possible
49
3 Box Scenario A
Box 1
CLIENT
ACCESS
SERVER
Box 2
Box 3
source
target
Requires administration of Access Server on all installed systems. Good
for locations with few administrators.
Only possible if box 1 is Windows NT or UNIX
50
3 Box Scenario B
Box 1
Box 2
Box 3
CLIENT
ACCESS
SERVER
source
target
Many clients can access CDC, if authorized
Does not require separate box for Access Server
51
3 Box Scenario C
Box 1
Box 2
Box 3
CLIENT
ACCESS
SERVER
source
target
Many clients can access CDC, if authorized
Does not require separate box for Access Server
52
4 Box Scenario
Box 3
Box 1
CLIENT
ACCESS
SERVER
Box 2
Box 4
source
target
• Does not require separate access server
53

Presentation

Transcription

Similar documents

HAIRHOUSE WAREHOUSE

InfoSphere Warehousing 9.5

2015 Annual Report - CDC Federal Credit Union

Facility Rental Packet

IG-PowerLinx

project information

the above infographic as a PDF

Week 1 Computing Systems History of Computing 2015

Global Presence Why SAFIRA Partners

CS 102 Introduction to Computing with Robots

The Prevention and Public Health Fund