PRACE: World Class HPC Services for Science

PRACE: World Class HPC Services for Science
F. Berberich, Forschungszentrum Jülich, February 2012, PRACE Workshop
on HPC Approaches on Life Sciences and Chemistry, Sofia, Bulgaria
Overview
• PRACE AS A RESEARCH INFRASTRUCTURE
• PRACE AS A EUROPEAN PROJECT
• PEER REVIEW AND ACCESS
• USER SERVICES
• TRAINING
Why do we need PRACE?
• Money:
– No “one system fits all problems”
• Quality:
– Scientific Excellence: European Peer Review Tier-0: HPCsystems of the highest performance class
– Access: guarantee independent access for all computational
scientists and European industry
• Sustainability:
– EU and national governments are motivated through PRACE to
establish robust and persistent funding schemes
3
PRACE AS A RESEARCH
INFRASTRUCTURE
PRACE History – An Ongoing Success Story
HPC part of the
ESFRI Roadmap;
creation of a vision
involving 15
European countries
Creation of the
Scientific Case
HPCEUR
2004
2005
Signature of the MoU
HET
2006
PRACE Initiative
2007
Creation of the PRACE
Research Infrastructure
PRACE RI
2008 2009
2010
2011
2012
PRACE Preparatory
PRACE-3IP
PRACE-1IP
Phase Project
PRACE-2IP
5
Realizing the ESFRI Vision for a HPC RI
– Tier-0: European Centres for Petaflop/s
– Tier-1: National Centres
– Tier-2: Regional/University Centres
• Creation of a European HPC
ecosystem
–
–
–
–
HPC service providers on all tiers
Scientific and industrial user communities
The European HPC hard- and software industry
Other e-Infrastructures
Tier-0
European
centres
Tier-1
National centres
Tier-2
Regional/University centres
# of systems
capability
• European HPC-facilities at the top of
an HPC provisioning pyramid
April, 23rd 2010 creation of
the legal entity (AISBL)
PRACE
with seat location in
Brussels, Belgium
24 PRACE
Members
68+ Million € from EC FP7
for preparatory and
implementation
Grants INFSO-RI-211528,
261557, and 283493
Complemented by ~ 50
Million € from PRACE
members
Interest by
Denmark,
Israel, …
Three pillars of the PRACE Mission
HPC
for Industry
HPC
for Science
HPC
by Vendors
Guaranteeing in
Europe the
independent
access to HPCcompetence for
industry
Implement the
ESFRI vision of
a European
HPC-service at
the top of an
HPC
provisioning
pyramid
Helping
European (hard& software)
vendors to
foster their
technology and
HPCcompetence
PRACE Tier-0 Capability and Support
• Accumulated
Tier-0
performance >
15 Pflop/s in 2013/14
• PRACE includes 18 Tier-1
systems with accumulated
capability of > 2 PF (building
on DEISA / DECI)
• PRACE provides capability
support competence centres
over several sites
9
Provision of Capacity and Access
• Binding commitments: France, Germany, Italy, Spain
− GENCI – GCS – CINECA – BSC
– Each: 100 Mio € over 5 years in terms of cycles
– Contribution accounted as TCO
• Access strictly by peer review at a European level
– Calls: 5/2010, 9/2010, 2/2011 ... incl. DECI call
– Start of provision: 8/2010
10
Tier-0
PRACE is building
the top of the pyramid...
Tier-1
Tier-2
First production system available:
1 Petaflop/s IBM BlueGene/P
(JUGENE) at GCS (Gauss Centre
for Supercomputing) partner FZJ
(Forschungszentrum Jülich)
Italy and Spain expect to deploy
their own Tier-0 systems from 2012.
Second production system available:
Bull Bullx CURIE at GENCI partner
CEA. Full capacity of 1.8 Petaflop/s
reached by late 2011.
Fourth production system available by mid 2012:
3 Petaflop/s IBM (SuperMUC) at GCS partner
LRZ (Leibniz-Rechenzentrum).
Third production system available by the
end of 2011: 1 Petaflop/s Cray (HERMIT)
at GCS partner HLRS (High Performance
Computing Center Stuttgart). Upgrade to 45 Petaflop/s planned in 2013.
Organization
Council Chair:
Achim Bachem
Scientific Communities in the Driver Seat
• Scientific Steering Committee
− Gives opinions on all matters of a
scientific and technical nature
− Proposes the members of the
Access Committee
− Members (Feb. 2012): Richard Kenway (Chair),
José M. Baldasano, Kurt Binder, Paolo Carloni
Giovanni Ciccotti, Sylvie Joussaume, Ben
Moore, Gernot Muenster, Risto Nieminen,
Modesto Orozco, Maurizio Ottaviani, Michele
Parrinello, Olivier Pironneau, Thierry Poinsot,
Simon Portegies Zwart, Kenneth Ruud,
Wolfgang Schroeder, Christof Schütte, Luis
Silva, Alfonso Valencia.
Scientific Communities in the Driver Seat
• Access committee
− Gives advice on the scientific use of
the Tier-0 Infrastructure
− Provides recommendations on the
allocation of PRACE resources based
on the Peer Review process
− Members (Feb. 2012): Kenneth Ruud (Chair),
Roberto Capuzzo Dolcetta, Peter Nielaba,
Manuel Peitsch, Andreas Schaefer, JeanClaude Andre and Hester Bijl.
14
Scientific Communities in the Driver Seat
• User Forum
– Open to all scientific and industrial user communities
– Main communication channel between HPC users and
PRACE AISBL
– Interaction with members of the PRACE AISBL
– Discussion and issuing recommendations to PRACE
AISBL
– Promoting HPC usage
– Fostering collaborations between user communities
Turlough Downes
PRACE User Forum - Dublin at the Bedford Hall in Dublin Castle March 14, 2012
PRACE AS A EUROPEAN
PROJECT
Preparatory Project
• 16 Partners (14 countries), funding 10 Million €
• Coordination: FZJ/JSC
• 1.1.2008 – 30.6.2010, successfully finished
• Review October 1st, 2010 in Brussels
17
Work Packages
•
•
•
•
•
•
•
•
WP1 Management
WP2 Organizational concept  Statutes
WP3 Dissemination, outreach and training
WP4 Distributed computing
WP5 Deployment of prototype systems
WP6 Software enabling for prototype systems
WP7 Petaflop/s systems for 2009/2010
WP8 Future Peta to Exaflop/s technologies
Example: Categorisation of Applications (2009)
Other 5.8
Plasma Physics 3.3
• Surveys of PRACE
Computational
partners’ HPC systems Engineering 3.7
and major applications Life Sciences 5.3
Astronomy &
• 24 systems and 69
Cosmology 5.8
applications
Earth & Climate 7.8
• Quantitative basis for
selecting representative
CFD 8.6
applications
Particle Physics 23.5
Computational
Chemistry 22.1
Condensed Matter
Physics 14.2
19
1st Implementation Project
• 21 partners (20 countries), funding 20 Million € by EC
• Coordination: FZJ/JSC
• 1.7.2010 – 30.6.2012, running
• Strong focus on application scaling in cooperation
with user groups
20
2nd Implementation Project
• 22 partners (21 countries), funding 18 Million €
• Preparation/Coordination: FZJ/JSC/PRACE PMO
• 1.9.2011 – 31.8.2013, running
• Scaling of important user codes
• Tier-1 Integration (DEISA  PRACE)
21
3nd Implementation Project
• Funding 20 Million €
• Under Preparation: FZJ/JSC/PRACE PMO
• Mid 2012 – Mid 2014
• Planned: Pre-commercial procurement exercise
• Planned: Industry application focus
22
Installed Petaflop Prototypes in Preparatory Project
IBM BlueGene/P (FZJ)
01-2008 (MPP)
IBM Power6 (SARA)
07-2008
Cray XT5 (CSC)
11-2008
IBM Cell/Power (BSC
)
12-2008 (CEA/FZJ):
Intel Nehalem/Xeon
installation date April 2009
IBM Cell/Power (BSC
)
12-2008
23
Prototyping is Mandatory for PRACE
•
•
Identification of
User
Requirements
Assessment of
emerging
Technologies
Recommendations for:
- Procurements
- Deployment of mature
Software Technologies
Technology
Watch
Joint
Developments
with Vendors
Recommendations for:
further Developments
Prototyping is a mandatory step in the selection and deployment of new technologies
Prototyping is a vehicle for cooperation with technology providers
24
Prototypes: Final Selection
Nr. Title
Partners
1
Exascale I/O
2
Novel MPP-Exascale system I/O
concepts
Interconnect Virtualization for
Scalable Heterogeneous Platforms
CEA, CINES, BSC, STFC,
JUELICH, HLRS
JUELICH, ETHZ, (IBM)
3
ETHZ, CaSToRC,
JUELICH, EPSRC, HLRS,
UYBHM, CINECA
4
NUMA-CiC: Numascale Cachecoherent Inter-Connect for Exascale
Clusters
5
Benchmarking Energy-to-Solution on LRZ, JKU, BSC, PSNC,
future hardware platforms
KTH
Total
UiO, GRNET, CSC,
JUELICH, CaSToRC,
PSNC
Total Costs
700 000 €
472 000 €
(+ 528 000 €)
60 000 €
400 000 €
(+ 250 000€)
800 550 €
2 432 550 €
25
PEER REVIEW AND ACCESS
PRINCIPLES
• Transparency
• Prioritisation
• Expert assessment
• Managing interests
• Confidentiality
• No parallel assessment
• Right to reply
with national organizations
• Ensure fairness to the
science proposed
PRACE - Peer Review
28
Types of Access
• Preparatory access
− only technical peer review
− prepare for project access, optionally with PRACE support
• Project access
− both technical and scientific peer review
− 12 month allocations, 2 calls per year: watch www.prace-ri.eu
• Programme access (to come)
− both technical and scientific peer review
Types of Access
Preparatory
Project
Programme
Technical Assessment
Yes
Yes
Yes
Scientific Assessment
No
Yes
Yes
Additional Assessment
No
No
Potentially
Mid-term Review
No
No
Potentially
Duration
6 months
12 months
24 months
Final Report
Technical
General
General
30
Preparatory Access – Types
• Type A
–
–
–
–
Code scalability testing to obtain scalability data
Assessment of applications using a light-weight application procedure
Maximum allocation for type A e.g. 100,000 core hours on JUGENE and 50,000 core hours on CURIE
Maximum allocation time: 2 months
• Type B
–
–
–
–
Code development and optimisation by the applicant using their own personnel resources
Applicants need to describe the planning for development in detail
Maximum allocation e.g. 250,000 core hours on JUGENE and 200,000 core hours on CURIE
Maximum allocation time: 6 months
• Type C
–
–
–
Code development with support from experts from PRACE
Maximum allocation e.g. 250,000 core hours on JUGENE and 200,000 core hours on CURIE
Maximum allocation time: 6 months
31
Received Proposals
• Early Access Call,1st-3rd regular call: 223 proposals
32
60 Awarded Proposals
33
Success Ratio
34
Peer Review: Next Calls
• 5th regular call
– April 17, 2012 Call Opened
– May 30, 2012 Call closed
– November 1, 2012 Tier-0 access for successful applicants
• Preparatory Access
Ongoing process. It is continuously open, with a cut-off every three
months.
35
USER SERVICES
User Services – a Project activity
• Evolution of PRACE Common Production
Environment (PCPE)
• Helpdesk
• User Documentation
• Applications Monitoring
• Advanced Application Production Assistance
37
PRACE Services
• PRACE Common Production Environment
• Common module environment deployed at all
PRACE sites
• First iteration based on DEISA DCPE
• Standard module environment for all PRACE
users regardless of architecture
38
PRACE Services - Helpdesk
• Central PRACE Helpdesk instance
• Central helpdesk for all issues
• Based on very successful DEISA model and tool
• Primary user interface via web-interface
• Secondary interface via email
• [email protected]
• Site specific email also available e.g. [email protected]
• Queues configured for each Tier-0 and Tier-1 site
• Additional internal queues for PRACE RI services
• WP6 provides resources to staff the Helpdesk
• Helpdesk on duty rotates through partners
• Similar process as used for DEISA
39
40
PRACE Services
• Applications Monitoring
• Central service for monitoring status of services
– Status of services on both Tier-0 and Tier-1 sites
– Includes monitoring of PCPE
– Based on very successful DEISA INCA model
• Single instance for both Tier-1 and Tier-0
– Concentrate development effort on one excellent central
monitoring service
– Central monitoring infrastructure hosted at LRZ
– Local components at host sites
41
PRACE Services
• User Documentation
• Each subtask leader owns User Docs for that area
– Network, Monitoring, Data, Compute, AAA, User
Services, Generic
• Advanced Application Production Assistance
• Support for new Tier-1 Users
• Similar concept as was used for DEISA DECI
− Site visits to new users
− Direct assistance to
42
TRAINING
Training in PRACE
44
PRACE-2IP
T4.1
Carry out Training
local infrastructure.
Survey to identify requirements AND existing
T4.2
Establishment of PRACE
(PATCs).
T4.3
Continue the series of EU/US
(Dublin, June 2012).
T4.4
Organise training events
(4 seasonal schools, 2 community-targeted workshops).
T4.5
Additional material & build on the Training
Advanced Training Centres
Summer Schools
Portal.
45
Training Surveys (users & trainers)
• 416 users (330 complete).
• 205 trainers (166 complete).
• Identified areas of HPC training to be prioritised (e.g.
performance analysis, optimisation, debugging techniques).
• PRACE is in an ideal position to complement local training
activities (through PATCs).
• Established database of HPC trainers who may be available
to implement PRACE training.
46
PRACE Advanced Training Centres
Establishment of 6 PRACE Advanced Training Centres
(PATCs) in 2012.
•Barcelona Supercomputing Center (Spain)
•CINECA - Consorzio Interuniversitario (Italy)
•CSC - IT Center for Science Ltd (Finland)
•EPCC at the University of Edinburgh (UK)
•Gauss Centre for Supercomputing (Germany)
•Maison de la Simulation (France)
47
PRACE Advanced Training Centres
• All centres responsible for developing and delivering a
coordinated programme of training events (designed
annually).
• Available to researchers and scientists from all over
Europe free of charge.
• Some events targeted at specific communities; some
targeted at developing HPC trainers.
• Build up a repository of material for the PRACE
Training Portal.
48
PRACE Seasonal Schools
Feb 6-10, Bologna, Italy
May 16-18, Cracow, Poland
http://www.prace-ri.eu/PRACE-Training-Events
49
EU-US Summer School, Dublin 2012
Date: 24-28 June
Application deadline: 18 March
Open to all European-based scientists, acceptance to school is
competitive.
PRACE covers accommodation & meal expenses. European-based
applicants to source travel funding from home institution.
Collaboration between PRACE & XSEDE
(35 EU students, 25 US students).
https://www.xsede.org/web/summerschool12/home
50
PRACE Training Portal
www.training.prae-ri.eu
51
PRACE Training Portal
52
30.8.2011
Thomas Lippert - FZJ
53
54