Rule Engine

Transcription

Rule Engine

iRODS
i Rule Oriented Data System
Marcus Lindner
Zuse Institute Berlin
April 2010
Kathrin Peter, Marcus Lindner
1
Outline
Distributed Data Management:
Challenges, Problems
iRODS Architecture: Client-Server, Rule Engine
iRODS Features: Metadata, Replication,
Zone Federation, Authorization
iRODS Clients: iCommands, APIs, Liferay
2
Distributed Data Management:
Challenges, Problems
3
Motivation
distributed computing
distributed storage
heterogeneity
availability
security
accounting
monitoring
How to use storage resources transparently?
4
Data Access: User Perspective
user with local input data
processing of jobs in the grid
How to transfer data to other computers?
5
Data Access: Application Perspective
execution of jobs at different resources
needs data access
transparent access to data stored in Grid
job
transfer
6
Requirements
Distributed Storage
data is distributed in the system but not local
Locality
data should be placed where the program
is running (performance and transfer cost)
Heterogeneity
resources are mostly heterogeneous
(identical absolute data paths hard to provide)
Usability
management interface (mostly similar to local file system)
7
Data Abstraction
system contains heterogeneous resources
question of distinct, consistent names
identical absolute paths hard to provide
/dataBig/home/
/data/home/
/gridData/home/
8
Data Abstraction
data management layer
globally unique identifier (ID) for data objects
abstraction from the physical location
metadata management
system/user metadata
replication
/dataBig/irods/XYZ/public
/data/irods/XYZ/ID5
ID5
/gridData/irods/XYZ/project
9
iRODS Architecture:
Client-Server, Rule Engine
10
Architecture: Client-Server
11
Architecture: Rule Engine
Rule Engine as central element
Rules defined in rule files
Micro Services encapsulate functionality
Actions are named chains of Micro Services
and actually are Rules
Rule Engine is triggered by client calls
searches for matching Rules and executes the first one matching
if the Rule fails, it is rolled back and the next one matching is
tried
12
Rule Oriented Programming
defined set of functionalities: Micro Services
define Rules which chain Micro Services (or Rules)
workflow of Micro Services
define condition for Rule application
define recoverability for failure management
Syntax of a Rule:
actionDef | condition | workflow-chain | recovery-chain
how to rollback?
what to do?
when appropriate?
name of the rule / action
13
iRODS Features:
Metadata, Replication,
Zone Federation, Authorization
14
Features
storage system with globally unique identifiers
logical namespace
virtualization: abstraction from the physical location
uniform access to data stored at heterogeneous resources
replica management
transparent replication across resources (consistency manually)
metadata management (iCAT database)
management of users, groups and access rights
variety of clients: graphical, command line and
programming libraries
15
Features
flexibility through Rules
administrators can define site Policies
users can encode their preferences
groups can set their process requirements
control actions on collection level, format level, etc.
example:
Admin Policy: permit only certain file types and size
user Rule: extract metadata
16
Features
Zone Federation
separate iRODS instances  multiple zones
(independently administered)
zones can be federated
users in multiple zones with given permission
 access data/metadata in other zones
paths begin with the zone name (unique logical name)
Authentication
password based authentication
GSI-based authentication (Globus)
17
iRODS Clients:
iCommands, APIs, Liferay
18
Available Clients
iRODS Explorer
iRODS command line tools iCommands
Fuse interface (Unix)
Java Client API, PHP, …
19
iRODS Explorer
20
iRODS command line tools
Unix like commands:
ils
icp
imkdir
list content
copy
create collection
put files to iRODS:
ilsresc
iput -R demoResc ../src/icd.c nvo
lists the resources
adds file to the col. nvo on a resource
invoke a rule:
irule -vF ruleInp3
invokes the rule named ruleInp3
add and query metadata
imeta add -d foo1 speed 100 "mph"
imeta qu -d speed ">=" 100
21
Liferay, Portlet, Jargon (Java API)
22
What can you do with all this?
extend, adapt the middleware with Rules and Micro
Services internally – interfaces stay the same
possible modifications
metadata extraction
automatic replication
run analysis steps on added data
information
iRODS web site: http://irods.sdsc.edu/
open source BSD like-license
23
Thanks!
Questions?
24
DB Redundancy
25
Rule Engine - Attributes
•
Attributes are iRODS system variables
three different types (identified by their prefix)
# – persistent attribute set (names of columns in the iCAT metadata catalog)
currently only usable inside Micro Services
$ – session attribute set
black-board shared between all Micro Services and the Rule Engine
* – input/output arguments for Micro Services
used to pass arguments between Micro Services
msiDataObjOpen(/x/y/z,*FD)##msiDataObjRead(*FD,10000,*BUF)
26
Rule Engine – Micro Services
•
small basic building blocks of iRODS
•
implemented in C
•
perform well-defined tasks
•
works on the iRODS Attributes (#, $, *)
•
Micro Services can modify
•
e.g. the actual data structures of the iCAT (iRODS catalog)
Sample Micro Services:
•
createCollection, createPhysicalFile, computeChecksum, or
replicateObject
Users can write their own Micro Services and register them with the Rule Engine.
(at compile time of iRODS)
27
Rule Engine – Create Rules/Actions
Syntax of a Rule:
actionDef | condition | workflow-chain | recovery-chain
how to rollback?
what to do?
when appropriate?
name of the rule / action
actionDef:
condition:
workflow-chain:
recovery-chain:
name can be used in other rules or external functions
the rule only applies if condition is satisfied
Attributes can be used to express conditions
$rescName == demoResc8
list of Micro Services / rules – separated by ´##´
list of Micro Services / rules to rollback if something fails
nop, if no rollback is needed
28
CLI
Unix like commands
iinit
iput
iget
imkdir
ichmod
icp
irm
ils
ipwd
icd
irepl
iexit
store scrambled session password
put file to iRODS
retrieve file from iRODS
create directory
allow/restrict access by others
copy
delete
list
working directory
change working directory
replicate data objects
finish (iexit full deletes
session pw)
Informational
ilsresc
iuserinfo
imiscsvrinfo
list resources
list users
get server infos
Rules and Delayed rule execution
irule
invoke rules
iqstat
list queue
iqdel
delete queue entries
iqmod
modify queue entries
Administration
iadmin add/del/mod users, resources, etc
Metadata
imeta manipulate metadata
29

Rule Engine

Transcription

Similar documents

Mary Lindner`s Big Challenge

Order Form - Jack L. Marcus Company Institutional Sales Catalog

Marcus Lindsey Brochure

Peter Ting (Hong Kong 1951 - ) Pair of Lotus Stand Candle Holders

Fourth of July - Family, Food, Fireworks and Fishing

Witch Bay Camp

Cast all your on God because he cares for you. Cast all your on God

here - This Time I Mean It

Muskego Point Resort on Lake Vermilion

Clinical Manifestations and Treatment of Catatonia