JuxMem - Irisa

Comments

Transcription

JuxMem - Irisa
JuxMem: An Adaptive
Supportive Platform for Data
Sharing on the Grid
Gabriel Antoniu, Luc Bougé, Mathieu Jan
IRISA / INRIA & ENS Cachan, France
Grid Data Service (GDS) meeting, Rennes, September 22th 2003
Context: Data Management on the Grid

Distributed numerical simulations (code coupling)


Needs data sharing
No many functional systems
Designing a satellite
Solid mechanics
Optics
Dynamics
Thermodynamics
2
Existing Data Management Systems

Non-transparent large scale data management





GridFTP (Globus) and MPI-IO
Internet Backplane Protocol (IBP)
Explicit transfert
No consistency ensured
Transparent small-scale data management




Distributed shared memory (DSM)
TreadMarksTM
Consistency models and protocols
Transparent access
Small-scale, static and homogeneous architecture
3
Another approache: peer-to-peer systems

Peer-to-peer systems (P2P)




Sharing of immutable data




Distributed (large-scale)
Volatile peers
Peers have the same capacities and responsabilities
Centralized (Napster)
Flooding (Gnutella, KaZaA)
Distributed hash table (CFS, PAST)
Sharing of mutable data


One writer per data + static architecture (OceanStore)
Conflicts have to be manually resolved (Ivy)
4
Design principles

Proposition: data sharing service for the grid


DSM systems: consistency and transparent access
P2P systems: scalability and high dynamicity
DSM
Service for the Grid
P2P
Scale
101-102
103
104-106
Dynamicity
Null
Medium
High
Resource
homogeneity
Homogeneous
(clusters)
Rather heterogeneous
(clusters of clusters)
Heterogeneous
(Internet)
Data type
Mutable
Mutable
Immutable
Typical
applications
Scientific
computation
Scientific computation
and data storage
File sharing and
storage
5
A Data Sharing Service for the Grid
Persistent storage
Transparency of localization
?
Internet
Data transfert
6
A Data Sharing Service for the Grid
Optimization of access
Data consistency
Internet
Internet
Data transfert
7
A Data Sharing Service for the Grid
Internet
Internet
Scalability of the
architecture
8
A Data Sharing Service for the Grid
Internet
Internet
Handling volatility
9
JXTA: a framework for P2P

Open-source platform for programming P2P applications


Specify a set of protocols
A peer



Uniquely identified (ID)
Address independent of the physical location
Several network access point (TCP, HTTP, etc)
Peer
ID
Peer
ID
Peer
ID
Peer
ID
Peer
ID
Peer TCP/IP
PeerPeer
Peer
Firewall
Peer
Peer
ID
Peer
ID
Peer
ID
Peer
Peer
Firewall
Peer
Peer
Peer
HTTP
Peer
10
JXTA: peer groups

Set of peers that share a common set of interests



Scope of communications
Specific policy of management
Peer group services
NetPeerGroup
PeerGroupA
Peer
ID
Peer
ID
Peer
ID
Peer
ID
Peer
ID
Peer
ID
Peer
ID
Peer
ID
PeerGroupB
11
JXTA: Advertisements

Every resource is described by an advertisement
Peers
 Peer groups
 Communication
channels
 Services
 …

PeerGroup Advertisement:
<?xml version="1.0"?>
<!DOCTYPE jxta:PGA>
<jxta:PGA>
<GID>
urn:jxta: uuidBCBCDEABDBBBABEABBBABA000000
</GID>
<MSID>
urn:jxta:uuidBFEFDEDFBABAFRUDBACE00000001
</MSID>
<Name>
My Group
</Name>
<Desc>
This group is to be used for my own testing
</Desc>
12
</jxta:PGA>
JuxMem: a prototype
Peer group juxmem
Peer group data
Peer group cluster A
Peer group cluster C
Peer group cluster B
Logical architecture
Physical architecture
13
API of JuxMem

Alloc (size, attribs)

Map (id, attribs)

Put (id, value)

Get (id)

Lock (id)

Unlock (id)
14
Managing Memory Resources

Advertisement of type provider: peer group cluster
Peer group cluster
Memory
provided

Advertisement of type cluster: peer group juxmem
Peer group juxmem
Size 10
Size 10
15
Managing Shared Data Blocks

Allocation of a memory area = creation of a peer
group data




Consistency




Data blocks identified by the ID of the peer group
Advertisement published in the peer group juxmem
Shared access for clients by knowing the ID
Data blocks are replicated on providers
Updated simultaneously (logical multicast)
Clients are not notified of updates
Synchronization

Lock for each data block
16
Handling the Volatility of Peers

A manager by peer group (cluster and data)




Dynamic monitoring of available peers
Data blocks are automatically replicated (data)
Updates advertisement of type cluster (cluster)
Volatility of managers


Periodic exchange of hearbeats
Dynamic replication of managers if needed on
other peers
17
Implementation of JuxMem

JuxMem



JXTA service
+ 5000 lines
Graphical tool
18
Preliminary Evaluations

Cluster




PentiumII: 450 Mhz and 256 Mb of RAM
Network used: FastEthernet 100 Mb/s
Number of nodes used: 20
Experiments

Overhead memory consumption


Low: @ 6 % with respect to the underlying JXTA peers used
Study of the volatility of providers
19
Study of the Volatility of Providers (1)
Data of one byte
Degree of redundancy = 3
1 client: 100 iterations lock-put-unlock
16 providers
Data manager is not killed
Peer group juxmem
Peer group data
Peer group cluster
20
Study of the Volatility of Providers (1)
Data of one byte
Degree of redundancy = 3
1 client: 100 iterations lock-put-unlock
16 providers
Data manager is not killed
Peer group juxmem
Peer group data
Peer group cluster
21
Study of the Volatility of Providers (2)

Internal lock when replicating


Ensure the consistency of the new replica
Client is blocked
Peer group juxmem
Peer group data
Peer group cluster
22
Relative overhead (%)
Study of the Volatility of Providers (3)
100
80
Reconfiguration time @
60
11 seconds
40
20
0
160 140 120 100 80
60
50
40
30
Targeted volatility is
weaker ( >> 80 seconds)
Seconds elapsed between provider
loosses



JXTA and Java
Time-out detection adapted for the WAN
Independent of the data size
23
Conclusion

Defines an hierarchical architecture for a data
sharing service for the grid



Storage and transparent access to data blocks




Specific policy for each cluster
Scalability
Application level: data block ID
Persistency
Shared consistency
Handling volatility of peers

Actively taken into account
24
Future Work

Transparent mobility of data blocks




Parameterizable consistency


Unavaibility of nodes or even clusters
Affinity data – computations
Affinity data – data
Specific for each client
Hierarchical synchronization
25
26
Allocation Request
6
4
3b
1
2
3a
5
3b
3a
27

Similar documents