Data warehouse generation algorithm explained. - BI

Data warehouse generation
algorithm explained.
Lulzim Bilali
Me
Lulzim Bilali
Master in information system development
7 years working with model to model transformation
and code generation
With main focus on DWH automaton
Traditional DWH project approach
Src
t
Stg
Src
t
T
DWH
T
DM
T
DM
T
DM
Points to improve
http://www.graymatter.co.in/doc/Lowering_cost_BI.pdf
http://www.softwareresults.us/2011/07/where-do-unused-features-come-from.html
Automation DWH project approach
Business
Model
Src
Rep’’
Rep’
t
Stg
Src
Rep’’’
t
T
DWH
T
DM
T
DM
T
DM
Data vault generation algorithm
Analyse
• 
• 
• 
• 
Mark tables as satellites
Mark tables as peg-legged links
Mark tables as links
Mark tables as hubs
• 
• 
• 
• 
Create hubs based on tables marked as satellites
Create links based on relationships of tables marked as hubs
Create links based on tables marked as links
Create satellites based on tables marked as satellites
Generate
Source model
Mark tables as satellites
Mark as satellite every table which is not referenced by a foreign key
and has only one foreign key with all the referencing columns also
primary key column and no other columns are part of primary key
Mark tables as peg-legged links
Mark as link every table which is not referenced by a foreign key
and has only one foreign key with all the referencing columns also
primary key columns and primary key is wider than the foreign key.
Mark tables as links
Mark as link every table which is not referenced by a foreign key
and has more than one foreign key with all the referencing columns also
primary key columns
Mark tables as hubs
Mark remaining tables as hubs
Create hubs (1)
Create hubs (2)
Create links based on relationships
of tables marked as hubs (1)
Create links based on relationships
of tables marked as hubs (2)
Create links based on tables
marked as links (1)
Create links based on tables
marked as links (2)
Create satellites (1)
Create satellites (2)
Code generation = reports (1)
Date: {date}
<%For each {Details} where date = {date} Group by {store}%>
Store: {Store Name}
Tag
| Turnover
<%For each {Product} group by {tag}%>
{name}
| {turnover}
Date: 21 may 2014
Store: Utrecht branch
Tag
Electronics
Clothes
| Turnover
| 1000
| 900
Store: Amsterdam branch
Tag
Electronics
Clothes
Other
| Turnover
| 2000
| 1100
| 250
Code generation = reports (2)
<%For each {Table}%>
CREATE TABLE {Table} (
<%For each {Column} of {Table} order by pos%>
{name}
{dataType}
{nullable},
)
CREATE TABLE Client_h (
hub_id
ClientCode
)
INTIGER
VARCHAR(30)
NOT NULL,
NOT NULL
CREATE TABLE Clinet_h_s (
hub_id
Load_date
Load_date_end
voided
Name
eMail
)
INTIGER
DATE
DATE
BOOLEAN
VARCHAR(50)
VARCHAR(150)
NOT NULL,
NOT NULL,
NOT NULL,
NOT NULL,
NOT NULL,
NOT NULL
Thank you!
(Applause)
Questions?
Contact:
Blog:
linkedIn:
[email protected]
www.dwhautomation.org
nl.linkedin.com/in/lulzimbilali/