koukis

VHPC’13
g
g
[email protected]
FLEXIBLE STORAGE FOR HPC CLOUDS WITH
ARCHIPELAGO AND CEPH.
CEPH VANGELIS KOUKIS
TECHNICAL LEAD, SYNNEFO
1
Outline
VHPC’13
g
g
[email protected]
Archipelago overview
g resources,, clones/snapshots
p
Storage
HPC workflow with Archipelago
Resource Composition
Archipelago Implementation
Flexible I/O pipelines
g
with Synnefo
y
Integration
Future directions
2
p g overview
Archipelago
VHPC’13
g
g
[email protected]
Distributed Storage System
g storage
g in clouds
- Powering
Decouples storage resources from storage backends
- Files / Images / Volumes / Snapshots
Unified way to provision, handle, and present resources
Decouples logic from actual physical storage
g
- Software-Defined Storage
3
p g logic
g
Archipelago
VHPC’13
g
g
[email protected]
Thin provisioning, with clones and snapshots
- Independent from the underlying storage technology
Hash-based data deduplication
Pluggable architecture
- Multiple endpoint (northbound) drivers
- Multiple
M lti l backend
b k d (southbound)
(
thb
d) drivers
d i
Multiple storage backends
- Unified management
- with storage migrations
4
Unified view of resources
VHPC’13
g
g
[email protected]
Files
- User files, with Dropbox-like syncing
Images
- Templates for VM creation
Volumes
- Live disks, as seen from VMs
Snapshots
- Point-in-time
P i t i ti
snapshots
h t off Volumes
V l
5
VHPC’13
g
g
[email protected]
Spawn
my own Ubuntu
Freeze
6
VHPC’13
g
g
[email protected]
Clone
Ubuntu + user data
Snapshot
7
The bigg ppicture
VHPC’13
g
g
[email protected]
Archipelago Core
Storage
g backend 1
(e.g., Ceph cluster 1)
Storage
g backend 2
(e.g., Ceph cluster 2)
Storage
g backend 3
(e.g., NFS over NAS)
8
VHPC’13
[email protected]
g
g
9
g
End-to-end workflow with unified storage
VHPC’13
g
g
[email protected]
10
Live demo!
VHPC’13
g
g
[email protected]
Login, view/upload files
Unified image store: Images as files
View/create/destroy
d
servers from
f
Images
…on multiple storage backends
…on Archipelago,
A hi l
for
f thin,
thi super-fast
f t creation
ti
…with per-server customization, e.g., file injection
Take a point
point-in-time
in time snapshot of a VM’s disk,
disk in seconds
Share it with collaborators, with fine-grained Access Control
Create a virtual cluster from this Snapshot
…from the command-line, and in Python scripts
11
VHPC’13
[email protected]
g
g
12
block I/O
Linux block
driver
VHPC’13
Northbound
interface
[email protected]
g
g
RADOS
Volume Composer
p
Mapper
Monitor nodes
S
Storage
Archipelago
Core
Ceph/RADOS
driver
Southbound
interface
object I/O
Object Storage nodes
13
Resource composition
p
VHPC’13
[email protected]
g
g
14
p g interfaces
Archipelago
VHPC’13
g
g
[email protected]
GLUSTER DRIVER
GLUSTER
15
p g
Runningg Archipelago
VHPC’13
g
g
[email protected]
16
p
Implementation
Internals
VHPC’13
g
g
[email protected]
17
p
Flexible I/O ppipeline
Linux block drv
VHPC’13
g
g
[email protected]
Linux block drv
Volume
Composer
Blocker
Mapper
Storage backend
19
p
Flexible I/O ppipeline
VHPC’13
g
g
[email protected]
Linux block drv
Linux block drv
Cacher
Volume
Composer
Blocker
Mapper
Storage backend
20
p
Flexible I/O ppipeline
Linux block drv
VHPC’13
g
g
[email protected]
Linux block drv
Volume
Composer
Cacher
Blocker
Mapper
Storage backend
21
p g overview
Archipelago
VHPC’13
g
g
[email protected]
Distributed Storage System
g storage
g in clouds
- Powering
Decouples storage resources from storage backends
- Files / Images / Volumes / Snapshots
Unified way to provision, handle, and present resources
Decouples logic from actual physical storage
g
- Software-Defined Storage
20
p g logic
g
Archipelago
VHPC’13
g
g
[email protected]
Thin provisioning, with clones and snapshots
- Independent from the underlying storage technology
Hash-based data deduplication
Pluggable architecture
- Multiple endpoint (northbound) drivers
- Multiple
M lti l backend
b k d (southbound)
(
thb
d) drivers
d i
Multiple storage backends
- Unified management
- with storage migrations
21
g
y
Integration
with Synnefo
VHPC’13
g
g
[email protected]
IaaS open source cloud software
y, Storage,
g , Compute/Network/Image/Volume
p
g
services
Identity,
Production since June 2011
- Powering the ~okeanos
okeanos public cloud
Written in Python/Django, exposes OpenStack APIs
Uses Google Ganeti at the backend
BSD licensed
22
Cluster vs Cloud
VHPC’13
[email protected]
g
g
OPENSTACK
SYNNEFO
UI
OPENSTACK
OPENSTACK
API
SYNNEFO
CLOUD
OPENSTACK
CLUSTER
GANETI
LIBVIRT
KVM
NODE
KVM
HYPERVISOR
23
g Ganeti
Google
VHPC’13
g
g
[email protected]
Mature, production-ready VM cluster management
- used for Google’s corporate infrastructure
Multiple storage backends out of the box
- LVM, DRBD
- Files on local or shared directory
- RBD (Ceph/RADOS)
External Storage Interface for SAN/NAS support
Ganeti cluster = masterd on master, noded on nodes
Easy to integrate into existing infrastructure
- Remote API over HTTP, pre/post hooks for every action!
24
Architecture
VHPC’13
g
g
[email protected]
25
g service: Pithos
Storage
VHPC’13
g
g
[email protected]
Exposes the OpenStack Object Storage (Swift) API
- plus extensions, for sharing and syncing
Rich sharing, with fine-grained Access Control Lists
Content-based addressing for blocks
Partial file transfers, deduplication, efficient syncing
Backed by Archipelago
- Provides a northbound endpoint for Archipelago
- Implements
l
the
h HTTP gateway
- Exposes the Swift API to end users
26
g
y
Integration
with Synnefo
VHPC’13
g
g
[email protected]
27
Future Directions
VHPC’13
g
g
[email protected]
I/O Flow identification
Per-user, per-volume policy enforcement for QoS
Archipelago XSEG over RDMA
- Archipelago I/O pipelines across physical nodes
- 1-sided
1 id d operations
ti
for
f remote
t communication
i ti
Expose the Archipelago data path inside VMs
- Direct VM userspace access to storage
ultra-low
low latency access to storage
- for ultra
Improved handling of multiple backends - Tiered Storage
- Volumes
V l
li
living
i across backends,
b k d automated
t
t d migrations
i
ti
28
Tryy it out!
VHPC’13
g
g
[email protected]
http://www.synnefo.org
Support: [email protected]
synnefo@googlegroups com
29