Barcelona Supercomputing Center Centro Nacional del - BSC-CNS

www.bsc.es
Barcelona Supercomputing Center
Prof. Mateo Valero, Director
BSC at a glance
One of only 13 centres recognised
with prestigious Severo Ochoa
award for excellence
National & European
HPC Infrastructure
Bridge to Latin
America
Spanish national HPC lab
PRACE Tier-0 Hosting Partner
- Extensive network of contacts
- Coordinator, RISC and OpenBio projects
Computer Sciences
- Architecture, programming
models, tools, Cloud and Grid
Life Sciences
Training Capability
- UPC, PATC, BMW, Pumps
Established 2005
Over 400 staff from 40 + countries
Key European
Research partner
- Participation in 63 FP6/7 projects, 12 as
coordinator
- Molecular and protein modelling,
computational genomics etc.
Earth Sciences
Atmospheric processes and
climate change modelling
Solid Partnerships
with Industry
Computer
Applications
- Repsol, Microsoft, IBM, Intel, Iberdrola, NVIDIA
- Leading role in ETP4HPC
- Simulation of complex problems
on High Performance Computers
Some Strategic Projects
Severo Ochoa
Air quality climate
modeling
A multidisciplinary research program to address the
complex challenges in the path towards Exascale. A set of
key strategic scientific projects and improvements in HR
management, training, mobility and communication.
Global models for climate change and
air quality prediction
Mont-Blanc
Developing an European Exascale
approach Based on embedded powerefficient technology
Human Brain Project
Riding on Moore’s
Law
10-year FET Flagship research project
to simulate human brain and design
computers based on its workings
Optimizing performance, energy consumption
and reliability of parallel computer architectures
through higher level abstraction
Personalised
Medicine
Combining Genomics, proteomics and
transcriptomics analysis with simulation
Alya Red
Computational mechanics simulation tools
designed for biomedical research. Winner 2012
Science-NSF visualisation challenge
Joint Research Centres
BSC-IBM Technology Center
for Supercomputing
Repsol-BSC Research
Center
Research into future challenges for
supercomputers including power efficiency and
scalability, new programming models, and tools
for analysis and optimization of applications
Research into advanced technologies
for the exploration of hydrocarbons,
subterranean and subsea reserve
modelling and fluid flows
BSC-Microsoft
Research Centre
Research into the design and interaction of
future microprocessors and software for the
mobile and desktop market segments
BSC-Iberdrola Research
Collaboration
Intel-BSC
Exascale Lab
Mathematical models to improve the design
of wind farms, including simulation of wind
flows for optimal turbine placement
BSC-NVIDIA CUDA
Research Center
Multi-year agreement focussing on
optimising efficiency through research into:
- Programming Models
- Performance Tools
- Applications
Training in Parallel Programming using CUDA
and StarSs
Optimising management of execution resources
in multi-GPU environments with GMAC
Navigating the Mare Nostrum
Are we planning to upgrade?.. Negotiating our next site ;)
Thank you!
BSC Team Annual Meeting
Navigating the Mare Nostrum
México, Marzo, 2012
MareNostrum
9
MareNostrum
10
Barcelona Supercomputing Center
The BSC-CNS objectives:
–
–
R&D in Computer Sciences, Life Sciences and Earth Sciences
Supercomputing support to external research
BSC-CNS is a consortium that includes :
–
–
–
the Spanish Government
– 51%
the Catalonian Government – 37%
the Technical University of Catalonia (UPC) – 12%
+400 people
Strategic Issues
Influence the way machines are built, programmed and used (Programming
models, Performance tools, Computer Architecture, Low Power).
Understand living organisms by means of theoretical and computational
methods (Molecular Modeling, Genomics, Proteomics).
Develop and implement global and regional state-of-the-art models for shortterm air quality forecast and long term climate applications.
Develop relevant scientific and engineering software for exploiting efficiently
supercomputing capabilities (Biomedical, Geophysics, Atmospheric, Energy,
Social and Economic simulations).
12
Ordinary
Funding
Competitive Funding
BSC Staff and BSC Funding
13
Talent attraction
Algeria
Argentina
Austria
Belgium
Brazil
Bulgaria
Canada
Chile
China
Colombia
Denmark
Ecuador
France
Germany
Greece
Hungary
India
Iran
Ireland
Israel
Italy
Japan
Mexico
Montenegro
Pakistan
Peru
Poland
Portugal
Russia
Serbia
Slovakia
South Africa
Syria
Sweden
Thailand
The Netherlands
Turkey
UK
Ukraine
USA
40% staff for 40 foreign countries
….. and Spain
14
BSC in Europe – Current Projects (Computer Sciences)
Computer Sciences
20 active projects
6 coordinated by BSC (*)
AXLE
*
Storage
Systems
*
Computer
Architecture
*
*
*
(3 groups)
Extreme
Computing
Grid Computing
and Clusters
Programming
Models
Autonomic
Systems and
e-Business
Platforms
OPTIMIS
*
Performance
Tools
HOPSA-EU
15
BSC in Europe – Current Projects
Operations, Applications, Life and Earth
Operations
5 projects
icordi
with Computer
Sciences and
Applications
with
Computer Science
with Computer
Sciences, Life
Sciences &
Earth Sciences
Appraisal
Earth
Sciences
4 projects
with
Computer Science
with Grid
Computing
and Clusters
with
Performance
Tools
Life
Sciences
7 projects
Applications
4 projects
PELE
Training
Education and Training – Future vision (Horizon 2020)
Centre of Excellence providing Training and Skills development
•
Based on cutting-edge
science
•
Training as meeting place for
scientists of different
disciplines
Training
curriculum
Sustain
industry
Sustain
research
community
•
Work to develop appropriate
HPC curricula across Europe
•
Building on the PATC s
•
Provide access to leading
edge systems and projects
through training
•Support universities to deliver HPC Master courses
•Facilitate staff exchanges
•Address long term career paths and career development
•Facilitate talent capture and retention
RISC partnership geographical
coverage:
Partnership:
Coppetec Fundaçao do Río de Janeiro
Universidad de Buenos Aires
Universidad Veracruzana
Universidad Autónoma de Manizales
Universidad de Chile
SUPERCOMPUTACiÓN
Barcelona Supercomputing Center
CINECA
Universidad Politécnica de Madrid
Menon
Universidade de Coimbra
RISC Objectives
Deepen strategic R&D cooperation between
Europe and Latino America in the field of High
Performance Computing (HPC)
Identify common needs, research issues and
opportunities for cooperative R&D on HPC
between EU and Latin America
Preparing the way for collaboration in the
framework of Horizon 2020
SUPERCOMPUTACIÓN @ Manizales, 2013
PROJECT OBJECTIVES AND STRATEGY
Consortium
European Partners
• Barcelona Supercomputing
Center – Centro Nacional de
Supercomputación (BSC).
• Consiglio Nazionale delle
Ricerche (CNR).
• Trust IT Services Ltd (TRUSIT).
• Universitat Politècnica de
València (UPVLC).
• Species 2000 (Sp2000).
Brazilian Partners
• Reference Center on
Enviromental Information
(CRIA).
• Recife Center for Advanced
Studies and Systems
(CESAR).
• Fluminense Federal
University (UFF).
• National Education and
Research Network (RNP).
EU-Brazil OpenBio
• Combining Biodiversity Science and the Open
Access Movement to deploy a joint European
and Brazilian
Two biodiversity use cases
e-Infrastructure of open access resources
supporting
the needs of the biodiversity
Computing resources
& SW community
platforms
scientific
Further EU-Brazil
collaboration in support
of the biodiversity area &
infrastructures
Joint EU-Brazil funded project,
with partners from each side
BSC leader on the European
side
COMPSs & VENUS-C used in
real production
23
Objectives
Objective 1: Interoperation of existing Brazilian and European e-Infrastructures in
the distributed computing, scientific data and portals & platform layers
Objectives
• Objective 2: Provide greater focus to the
integration of data software platforms running
through all of infrastructures
• Objective 3: Identification of further future
EU-Brazil collaboration in support to the
biodiversity area in all types of infrastructures
to demonstrate the efficiency of our approach
through two Use Cases.
Human Brain Project
10-year 1000M€ FET flagship
project
Goal: to pull together all
existing knowledge about the
human brain and to reconstruct
the brain in supercomputer
based models and simulations.
Expected outcomes: new treatments for brain disease
and new brain-like computing technologies
BSC role: Provision and optimisation of programming
models to allow simulations to be developed efficiently
MareNostrum part of the HPC platform for simulations
26
BSC in Europe
• 25 countries in the association, 4 of them “Hosting members”
• Spain is a “Hosting Member” and BSC is the Spanish representative
27
BSC and Industry: long-term collaborations
● R&D of High-Tech firms
● Spanish firms
28
BSC and Industry: other collaborations
29
Severo Ochoa: programme
The BSC is one of only eight Spanish research centres awarded with the
prestigious Severo Ochoa grant.
– The aim of the Severo Ochoa programme is to strengthen the very best
Spanish research centres, who are internationally amongst the most
competitive in their field.
– With the Severo Ochoa grant, the BSC-CNS will strengthen its strategic
research capacities, human resources, international collaboration and the
dissemination of its results to society.
30
Severo Ochoa scientific programme
- Cardiac Computational Model (CCM)
- Implementation and validation of advanced
physiological models
- Improved code usability and robustness
- Alya Red: A computational heart, winner, video
category 2012 NSF Visualization Challenge
Energyefficiency for
Exascale and
Bigdata
- Definition of user needs and
specification of use cases
- Evaluation of the use of the
algorithms in practical cases (chronic
lymphatic leucimia, type II diabetes,
melanomas and HIV-1
- Porting of applications to COMPSs
- Analysis (workflow and data
requirements) of representative apps
- Analysis of COMPSs integration with
the new storage platform
- First prototype of persistent objects
(as opposed to files) storage platform
- Platform for the design of Domain
Specific Languages (for acceleratorbased architectures)
- Energy-efficiency in OmpSs
- Alignment with Mont-Blanc
prototypes and requirements
- Definition NMMB/BSC Chemical Transport
Model
- Implementation of marine aerosol model
- Roadmap to optimize code for target architecture
Intel/BSC Exascale Laboratory
Objective: address the challenges on the way to Exascale: efficiency,
variability, memory, faults, scale (concurrency, strong scaling) and
complexity (hierarchy /heterogeneity)
Topics:
– OmpSs as programming model to express algorithms and runtime system
responsible for mapping them to resources (dynamic autotuning, resilience,
reductions and load balance)
– Avoid flying blind with the appropriate analysis tools: power monitoring and
modeling added to the powerful BSC’s analysis environment and prediction of
taskification strategies
– Applications and algorithms, the final target. Large applications and algorithmic
developments having in mind asynchrony and complexity (hybrid MPI/OmpSs)
Activities started on November 2011
CUDA Center of Excellence by NVIDIA
Recognition of BSC’s broad-based research in leveraging the
NVIDIA CUDA technology and GPU computing
Topics:
– Programming models: A cluster-aware Global Memory for Accelerators
(GMAC) runtime environment and the OmpSs programming model for
clusters of hybrid CPU/GPU nodes and auto-vectorization
– Applications: Reverse Time Migration, seismic imaging facility, Protein
Energy Landscape Exploration, Finite Differences Time Domain and
Network Analysis Open Source Toolkits
– Build an education program on parallel programming
using CUDA, OpenCL and StarSs and summer school
Awarded on November 2011, CUDA Research Center
in 2010
Mare Incognito: project structure
4 relevant apps:
Materials: SIESTA
Geophysics imaging: RTM
Comp. Mechanics: ALYA
Plasma: TORB
General kernels
Applications
Programming
models
StarSs: CellSs, SMPSs
OpenMP@Cell
OpenMP++
MPI + OpenMP/StarSs
Coordinated scheduling:
Run time, Process, Job
Power efficiency
Automatic analysis
Coarse/fine grain prediction
Sampling
Clustering
Integration with Peekperf
Performance
analysis tools
Models and
prototype
Interconnect
Contention
Collectives
Computation/communication
overlap
Load
balancing
Processor
and node
Contribution to new Cell design
Support for programming model
Support for load balancing
Support for performance tools
Issues for future processors
BSCMSR Centre: research topics
Architecture
Initial research topic
Transactional Memory
– Transactional memory
–
–
–
–
Functional
HTM
MSRC
Imperative
New research topics
Programming model
STM
Applications
• Software (STM), hardware (HTM)
and hardware-assisted STM
• Programming model, runtime and
OS/architecture support
• Benchmark suite with realistic
applications
BSC
Vector Support for Emerging Processors
Accelerating Databases with HW support
Big Data for Bioinformatics
Cloud Computing
35
REPSOL-BSC Research Center
2011-2015: REPSOL-BSC Research Center
– Framework contract to define BSC-REPSOL partnership
• Seismic Imaging with elastic waves (FM,RTM,FWI)
• Controlled Source Electromagnetic Methods
Pflops
Seismic Imaging RoadMap
FWI Viscoelastic
Petro-elastic Inv.
Vsalt = 14800 ft/s
100
θd
10
Vsed = 10500 ft/s
1
RTM elastic
0,1
RTM acoustic
0,01
Beams
Rays
1995
2000
2005
2010 2015 2020
Barcelona Seismic Imaging Tools (BSIT)
Kaleidoscope
Using acoustic waves max.
imaging dip at target
θd = 32°
FWI elastic
REPSOL-BSC
Reseach Center
1000
36
CASE: Medical & Pharmaceutical Industry
Heart Simulator : ALYA
– Simulation of the complete cycle of human heart
• Electrical impulse propagation
• Muscle strain + valves movement
• Blood Movement
37
BSC: A Hub for Personalized Medicine
Biomedical Research &
Experimental Facilities
National and European
infrastructures
Hight-throughput
sequencing
Patients &
Physicians
• Life Sciences Research Program
• HPC-facilities
• Data Storage
• Programming Models
• Code Optimization & Parallelization
• Large Data Management
• Green Computing
• Trainning
Chronic Cancer Leukemia
We set up a Sequencing pipeline for the Chronic Lymphocytic
Leukemia (CLL) Genome Project, that aims to generate a
comprehensive catalogue of genomic alterations involved in the
development and progression of the disease.
Genome
Sequencing
500 Patients
1 Patient ~ 1day
BSC
Data Management
expected 1.5 Pb
HPC Computing
Sustained 10-15%
BSC Supercomputer
Experimental
Validation
We discovered mutations that contribute to the clinical evolution of the patient
39
Life Sciences and Health: personalized medicine
Population
Organ
Tissue
Cell
Macromolecule
Small Molecule
Atom
Supercomputers, theory and experimentation
Fusion reactor
Accelerator design
Material sciences
Astrophysics
Experimentation
Aircraft
Automobile design
Theory
Simulation
Climate and weather
modeling
Life sciences
Top10
Rmax
Rpeak
Power
GFlops/
Watt
560640
261632
17,59
27,11
8,21
2,14
Titan
1572864
16,32
20,13
7,89
2,07
Sequoia
Fujitsu, K computer, SPARC64
VIIIfx 2.0GHz, Tofu interconnect
705024
10,51
11,28
12,65
0,83
K
DOE/SC/Argonne
National Laboratory
BlueGene/Q, Power BQC 16C
1.60GHz, Custom
786432
8,16
10,06
3,94
2,07
Mira
5
Forschungszentrum
Juelich (FZJ)
BlueGene/Q, Power BQC 16C
1.60GHz, Custom
393216
4,14
5,03
1,97
2,10
JUQUEEN
6
Leibniz
Rechenzentrum
iDataPlex DX360M4, Xeon E52680 8C 2.70GHz, Infiniband
FDR
147456
2,89
3,18
3,42
0,85
SuperMUC
7
Texas Advanced
Computing Center
PowerEdge C8220, Xeon E52680 8C 2.700GHz, Infiniband
FDR, Intel Xeon Phi
204900
112500
2,66
3,95
8
Tianjin, China
NUDT YH MPP, Xeon X5670 6C
2.93 GHz, NVIDIA 2050
186368
100352
2,56
4,70
4,04
0,63
Tianhe-1A
9
CINECA
BlueGene/Q, Power BQC 16C
1.60GHz, Custom
111104
1,72
2,09
0,82
2,10
FERMI
IBM Development
Power 775, POWER7 8C
Rank
Site
Computer
1
DOE/SC/OAK Ridge
National Lab
CRAY XK7, Opeteron 6274 16C,
2.20 GHz, Cray Gemini
interconnect, NVIDIA K20x
2
DOE/NNSA/LLNL
BlueGene/Q, Power BQC 16C
1.60 GHz, Custom
3
RIKEN Advanced
Institute for
Computational
Science (AICS)
4
Procs
Name
Stampede
DARPA
Strategic Issues
Influence the way machines are built, programmed and used (Programming
models, Performance tools, Computer Architecture, Low Power).
Understand living organisms by means of theoretical and computational
methods (Molecular Modeling, Genomics, Proteomics).
Develop and implement global and regional state-of-the-art models for shortterm air quality forecast and long term climate applications.
Develop relevant scientific and engineering software for exploiting efficiently
supercomputing capabilities (Biomedical, Geophysics, Atmospheric, Energy,
Social and Economic simulations).
43
Zaragoza, Marzo, 2011
44
Zaragoza, Marzo, 2011
45
Current Resources
● MareNostrum 2006
● 10240 PowerPC970 cores
● 94 TFlop/s
● MinoTauro 2011
● 128 compute nodes
● 182 TFlops
●
HPC Storage and Backup:
● 2.5 PB disk
● 6.0 PB tapes Robot
46
MareNostrum 3, phase 1
25x IBM iDataPlex Compute racks
– 84x IBM dx360 M4 compute nodes
• 2x SandyBridge-EP E5-2670 2.6GHz/1600 20M 8-core 115W
• 8x 4G DDR3-1600 DIMMs (2GB/core)
• 500GB 7200 rpm SATA II local HDD
4x IBM dx460 M4 compute nodes on a Management Rack
2,104 compute nodes
– 33,664 Intel cores
Memory 65.75 TB
– 32GB/node
Peak performance: 700.2 Tflop/s
– Node performance: 332.8 Gflops
– Rack Performance: 27.95 Tflops
– Rack Consumption: 28.04 kW/rack (nominal under HPL)
Estimated power consumption: 755.8 KW
Infiniband FDR10 non-blocking Fat Tree network topology
47
BSC in Spain
Altamira
Universidad de
Cantabria
MareNostrum
BSC
CaesarAugusta
Universidad de
Zaragoza
Magerit
Universidad Politécnica Madrid
Picasso
Universidad de
Málaga
Tirant
Universidad de Valencia
La Palma
IAC
Atlante
ITC
48
Energy-efficient prototype series @ BSC
200 PFLOPS
10 MWatt
20 GF/W
50 PFLOPS
7 MWatt
7 GF/W
1.5 GF/W
120 TFLOPS
80 Kwatt
256 nodes
250 GFLOPS
1.7 Kwatt
0.2 GF/W
2011
2012
2013
2014
2015
2016
2017
Proof of concept built with COTS components (ARM, ARM+GPU)
Properly integrated system with embedded components (Mont-Blanc)
49
BSC & Industry. Other collaborations
50
BSC & Industry. Long term collaborations
● R&D of High-Tech firms
● Spanish firms
51
BSC in Europe
• 24 countries in the association, 4 of them “Hosting members”
• Spain is a “Hosting Member” and BSC is the Spanish representative
52
Funding split
Total amount (2005/11): 57,35M€
53
www.bsc.es
Industrial Activity at BSC departments
Life Science: Strategic Research …
Deep insight into the origin, evolution and function of living
organisms using theory and computation
Target & Drug Discovery
BioPhysics
BioSupercomputing
Genomics & System Biology
– Rational drug design using computer-aided tool
– Simulation of macromolecular systems (proteins and nucleic acids)
with high biomedical and biotechnological impact
– Computation for Omics revolution: identification of targets for
complex disease (Cancer, Alzheimer, etc.)
55
Life Sciences future
Future strategic research line: PERSONALIZED MEDICINE
– develops specific treatments for segments
of population characterized by genetic factors
Life Sciences Future : Chronic Cancer Leukemia
We set up a Sequencing pipeline for the Chronic Lymphocytic
Leukemia (CLL) Genome Project, that aims to generate a
comprehensive catalogue of genomic alterations involved in the
development and progression of the disease.
Genome
Sequencing
500 Patients
1 Patient ~ 1day
BSC
Data Management
expected 1.5 Pb
HPC Computing
Sustained 10-15%
BSC Supercomputer
Experimental
Validation
We discovered mutations that contribute to the clinical evolution of the patient
57
Drug Discovery
We study how protein flexibility can improve the design of
new drugs, to develop better tools for pharmaceutical
industry.
Largest European database of
protein simulations (20 Tb)
33% of therapeutic targets
Rational Drug Design: New
algorithms for flexible 3D
models of proteins.
58
Industry: definition and execution of R+D projects
59
LS Future : Life Sciences and Health
Population
Organ
Tissue
Cell
Macromolecule
Small Molecule
Atom
Genomics and Personalized Medicine
Data is the main problem
Treat patients NOT pathologies
1
Estimated market 450B$ in 2015
12009 PricewaterhouseCoopers
FET: Human Brain Project
F Schürmann, H Markram (Blue Brain Project, EPFL)
Bucaramanga, Abril 2012
“Killer microprocessors”
10.000
MFLOPS
Cray-1, Cray-C90
NEC SX4, SX5
1000
Alpha AV4, EV5
Intel Pentium
100
10
1974
IBM P2SC
HP PA8200
1979
1984
1989
1994
1999
Microprocessors killed the Vector supercomputers
– They were not faster ...
– ... but they were significantly cheaper and greener
10 microprocessors approx. 1 Vector CPU
– SIMD vs. MIMD programming paradigms
M. Valero. “Vector Architectures: Past, Present and Future”. Keynote talk. ICS-11.
IEEE-ACM. Melbourne, 1998
63
The Killer Mobile processorsTM
1.000.000
Alpha
100.000
MFLOPS
Intel
AMD
10.000
Nvidia Tegra
Samsung Exynos
1.000
4-core ARMv8 1.5 GHz
100
1990
1995
2000
Microprocessors killed the
Vector supercomputers
They were not faster ...
... but they were significantly
cheaper and greener
2005
2010
2015
History may be about to
repeat itself …
Mobile processor are not
faster …
… but they are significantly
cheaper and greener
Can we build a supercomputer with mobile chips?
HPC
Servers
Desktop
Total cores in Nov‘12 Top500
– 14.9M Cores
Tablets sold in Q4 2011
Mobile
– > 106M Tablets
Smartphones sold 2012
– > 712M Phones
65
How about power? And cost?
2-socket Intel Sandy Bridge
8-socket ARM Cortex A-15
– > $3.000 (Intel sockets only)
– > 350W (CPU + DRAM)
– < $200 (ARM sockets only)
– < 100W (CPU + DRAM)
Strawman design, not the actual Mont-Blanc architecture
66
A big challenge, and a huge opportunity for Europe
GFLOPS / W
Built with the best
of the market
200 PFLOPS
~10 MWatt
Built with the best
that is coming
120 TFLOPS
80 Kwatt
256 nodes
250 GFLOPS
1.7 Kwatt
2011
What is the best
that we could do?
Integrated
ARM + GPU
2012
2013
2014
2015
2016
2017
Extend current mobile chips with the needed HPC features
–
–
Explore the use vector architectures in mobile accelerators (vector processor ARM-based, 15+ Teraflops chip, 150
watts)… unique opportunity for Europe
One design for all market segments: mobile, data centers, supercomputers
67
Press Impacts
www.bsc.es
Computer Sciences
Scientific mission
To influence the way high-performance computing systems
are built, programmed and used
– Proposal and development of novel processor, memory and
interconnect architectures, programming, performance analysis and
execution environments, resource management layers, ..., bridging
what computer technology offers and application requirements, usually
in collaboration with manufacturers
Origins in UPC Computer Architecture Department, inherited
know-how
1991-2000
2000-04
70
Scientific mission
Hardware and software components for
high-performance computing
Mobile multimedia devices
multicore
accelerators
Real-time systems
+
Personal computers
Performance, power/energy,
reliability and usability
Computing data centers
and Cloud
71
Strategic research and development lines
Strategic line I: Towards Exascale systems
Strategic line II: Cloud and BigData
Strategic line III: Architectures for application domains
Strategic line
PM
PT
aHPC
GC
Exascale
x
x
x
x
Cloud-BD
Architectures
x
x
eB
SS
ExtC
HetA
CAOS
ASPP
NA
x
x
x
x
x
x
x
x
72
Strategic line I: Towards Exascale systems
OmpSs programming model
–
–
–
–
Single source any target
Asynchronous data-flow
Intelligence to the runtime
Deployed and used !!!
BSC tools
– Extrae, Paraver, Dimemas.
– Performance analytics: Intelligence, insight
Low power HPC architectures and
systems
– Leveraging low end commodity
components
– Key advantage: tools, OmpSs,
architecture background
73
StarSs: data-flow execution of sequential programs
void Cholesky( float *A ) {
int i, j, k;
for (k=0; k<NT; k++) {
spotrf (A[k*NT+k]) ;
for (i=k+1; i<NT; i++)
strsm (A[k*NT+k], A[k*NT+i]);
// update trailing submatrix
for (i=k+1; i<NT; i++) {
for (j=k+1; j<i; j++)
sgemm( A[k*NT+i], A[k*NT+j], A[j*NT+i]);
ssyrk (A[k*NT+i], A[i*NT+i]);
}
#pragma
omp task inout ([TS][TS]A)
}
void spotrf (float *A);
#pragma omp task input ([TS][TS]A) inout ([TS][TS]C)
void ssyrk (float *A, float *C);
#pragma omp task input ([TS][TS]A,[TS][TS]B) inout ([TS][TS]C)
void sgemm (float *A, float *B, float *C);
#pragma omp task input ([TS][TS]T) inout ([TS][TS]B)
void strsm (float *T, float *B);
Decouple
how we write
form
how it is executed
Write
Execute
Used in projects and applications …
Undertaken significant efforts to port real large scale
applications:
–
• JSC, UJI, UoManchester, HLRS, IBM-ZRL, CNRS
–
• JSC, LRZ, GENCI, CINECA, BSC
–
DEEP
• EPFL, KULeuven, CYI, CERFACS, CINECA, CGGVS
– G8_ECS
• NCAR, Titech, …
– Consolider project (Spanish ministry)
• UoValencia
– BSC initiatives and collaborations:
• KTH, UoHeidelberg,…
Why Tools ?
Measurement techniques as
enablers of science
Are becoming vital for
program development at
exascale
Are important for “Lawyers”
Are vital for system architects
Performance analyst:
– A specialist understanding
displays
Leveraging techniques from many areas…
Spectral analysis techniques
Wavelet
High
frequency
Spectral density
Autocorrelation
T
CPI Stack model
Software Counters
Simulation
Correlate
tracing and
sampling
Histograms
Sequence alignment algorithms
Models
Clustering
Strategic line II: Cloud and BigData
Intersection between Cloud Computing and large
scale data analytics/management
Vertical approach integrating previous
technologies
– Programming environments and runtime systems
– Resource management in heterogenous systems
and workloads
– Storage architecture and management
To be demonstrated with “in-house” scientific
challenges
78
Strategic line III: architectures for application domains
Architectural support for critical real-time embedded
systems
–
–
–
predictability (WCET) and composability (workload
independent)
Multicores
System software, multiple criticalities ?
Vector architectures
–
Multimedia and data management systems
Architectural support to parallel programming paradigms
–
–
Accelerating synchronization, TLB shutdown, managed
runtimes, …
Transactional memory: optimistic synchronization,
speculation, support to dataflow and resilience
FPGA prototyping and development tool chain
–
–
TMBox (1st implementation of Hybrid TM with16 cores)
Application acceleration on FPGA (including OmpSs task
offloading) and development tool chain
79
Advanced ERC - Riding on Moore’s Law
5-year ERC Advanced Grant
Idea: a radically new conception of parallel
architectures, built using a higher level of
abstraction
Objective: ensure continued performance
improvements by riding on Moore's Law.
Holistic approach with parallel architecture partially implemented as a
software runtime management layer
Multicore architecture with vector accelerators exploiting both thread
and data level parallelism to optimize data movement
Handling parallelism, the memory wall and the power wall, in application
domains from mobile to supercomputers .
80
www.bsc.es
Computer Applications and
Engineering : CASE
CASE: Strategic Topics …
Optimized applications for science and engineering
– Biomechanics (Alya Red): coupled electrical/mechanical/fluid in very
large grids (108 elements)
– Geophysics (BSIT): pioneers in acoustic RTM, elastic FWI with real
data, electromagnetic methods
– Coupled Multiphysics (ALYA): Coupling fluids, solids, electromagnetics,
particles, chemistry on grids with 109 elements. Several CFD
modelization: free surface, compressible/incompressible,
subsonic/supersonic, RANS/LES/DNS, …
The heart simulation: Multiphysics and Multiscale
Purkinje Fibers
2012 NSF & Science
Journal Award to the best
scientific video
Arterial Network
Cell model + Electrical model + Solid
Mechanics model + CFD
O’Hara- Rudy Human Cell Model
83
CASE: Medical & Pharmaceutical Industry
Respiratory System simulator: ALYA
– Simulation of complete breathing cycle
Saint Mary’s
Hospital
84
REPSOL-BSC Research Center
2011-2015: REPSOL-BSC Research Center
– Framework contract to define BSC-REPSOL partnership
• Seismic Imaging with elastic waves (FM,RTM,FWI)
• Controlled Source Electromagnetic Methods
Pflops
Seismic Imaging RoadMap
FWI Viscoelastic
Petro-elastic Inv.
Vsalt = 14800 ft/s
100
θd
10
Vsed = 10500 ft/s
1
RTM elastic
0,1
RTM acoustic
0,01
Beams
Rays
1995
2000
2005
2010 2015 2020
Barcelona Seismic Imaging Tools (BSIT)
Kaleidoscope
Using acoustic waves max.
imaging dip at target
θd = 32°
FWI elastic
REPSOL-BSC
Reseach Center
1000
85
Kaleidoscope’s Business Impact (REPSOL)
Industrial Partners
Energy Industry
– Repsol
– Iberdrola
Construction Industry
– Holcim
– Heidelberg cement
– Strukton
Turbine Industry
–
–
–
–
Rolls Royce
Siemens
Turbomeca
ITP
Naval Industry
– JYD
– Puerto de Barcelona
Aeronautic Industry
–
–
–
–
Airbus / EADS
Dassault aviation
Aernova
Cerfacs
Automotive Industry
– Seat
– Saab
IT Industry
– IBM
– T-Systems
87
CASE: Proyecto Caleidoscopio
2007-2010
– Imagen Sísmica (RTM con ondas acústicas) usando aceleradores
(Cell/GPUs)
– Proyecto premiado internacionalmente en la industria de O&G
• IEEE Special Technology Winner 2008
• 2009 Platt’s Awards
Platform
Gflops
Power (W)
JS21(PPC)
8,3
267
0,03
QS22(Cell)
116,6
370
0,32
RTM vs. WEM
Gflops/W
x14 Cell
x40 GPU
M. Araya, et al.
Assessing Accelerator-based HPC Reverse Time Migration,
IEEE Transactions on Parallel and Distributed Systems,
22(1):147-162, January 2011
88
REPSOL-BSC Research Center
2011-2015: REPSOL-BSC Research Center
– Framework contract to define BSC-REPSOL partnership
• Seismic Imaging with elastic waves (FM,RTM,FWI)
• Controlled Source Electromagnetic Methods
Pflops
Seismic Imaging RoadMap
FWI Viscoelastic
Petro-elastic Inv.
Vsalt = 14800 ft/s
100
θd
10
Vsed = 10500 ft/s
1
RTM elastic
0,1
RTM acoustic
0,01
Beams
Rays
1995
2000
2005
2010 2015 2020
Barcelona Seismic Imaging Tools (BSIT)
Kaleidoscope
Using acoustic waves max.
imaging dip at target
θd = 32°
FWI elastic
REPSOL-BSC
Reseach Center
1000
89
CASE: CFD Atmospheric
WARIS: Win farm design
–
–
–
–
–
High resolution wind model in complex terrain
Modelling rotor wakes
Overset meshes (Chimera)
Wind farm modelling and Optimization
Post-processing on Google Earth
90
CASE: Atmospheric Transport
FALL3D
–
–
–
–
Volcanic ash cloud modelling
Volcanic ash cloud forecast (civil aviation)
Impact and crisis management
Future GPU based implementation
S. Scollo, et al.,Three-dimensional volcanic aerosol dispersal: A comparison between Multiangle
Imaging Spectroradiometer (MISR) data and numerical simulations. Journal of Geophysical
Research, 115(D24210), 2010.
91
CASE: Recycling industry
Separation of plastic waste (ALYA code)
– Design of a crashed plastic separating machine
– Simulating the fluid and particles
92
CASE: Recycling industry
Cement production from recycled concrete
– Clinkerisation oven: Multiphysics problem(ALYA code)
•
•
•
•
•
Radiation
Combustion
Chemistry
Fluids
Particles
93
CASE: Turbine Design
Project Copa-GT (ALYA code)
– Simulation of turbine design optimization
– Simulation of the whole turbine behaviour
94
CASE: Sailboat hydrodynamics
Racing sailboat design (JYD)
– Hull and appendages hydrodynamics (ALYA code)
95
CASE: Free surface problems
Optimizing the amount of water flow in a WC
96
CASE: Medical & Pharmaceutical Industry
Heart Simulator : ALYA
– Simulation of the complete cycle of human heart
• Electrical impulse propagation
• Muscle strain + valves movement
• Blood Movement
97
www.bsc.es
Earth Science Department
Objectives
Research in the Earth Sciences area is devoted to the development
and implementation of regional and global state-of-the-art models for
short-term air quality forecast and long-term climate applications.
Issues related to atmospheric dynamics, natural and anthropogenic
emissions, improvement of AQF, the transport and dispersion of
pollutants in complex terrain, urban air quality, mineral dust transport,
aerosol radiative effects and the feedback between meteorology and
air pollution. Together with the advances in the parallelization of air
quality and climate model codes, have allowed such high-resolution
simulations.
ES maintains two daily operational systems: AQF CALIOPE and MD
forecasts: BSC-DREAM8b and NMMB/BSC-CTM.
Origins in UPC Environmental Modelling Laboratory, inherited knowhow
99
BSC: Leader worldwide in …
Simulation and understanding of earth climate and air quality
– Development and application of “Mineral Dust Transport Models“:
BSC-DREAM8b (regional) and NMMB/BSC-Dust (global)
 Regional Center of World Meteorological Organization SDS-WAS
Nothern Africa-Midle East-Europe (NA-ME-E), by consortium between
BSC and AEMET.
– World reference in Air Quality Forecast Systems, by the operational
system CALIOPE
100
Earth Sciences Future
Development of unified Atmospheric – Chemistry – Climate
Models
– From short-term to climate projections
– Towards global high-resolution systems
NMMB/BSC-CTM
- Integrated approach
- Global/regional
- Chemistry
- Aerosols
- Climate
… 2020
101
Strategic alliance
Centro de Investigación Atmosférica de Izaña, AEMET
National Oceanic and Atmospheric Administration’s (NOAA)
National Centers for Environmental Predictions (NCEP)
The Trustees of Columbia University in the City of New York for
the International Research Institute for Climate and Society
(IRI)
ENES: Service for Climate Modeling in Europe
COST Action:
102
WMO Sand and Dust Storm Warning and Assessment System (SDS WAS)
in cooperation with World Meteorological Organization (WMO)
 To enhance the ability of participating countries to establish and improve
systems for forecasting and warning to suppress the impact of Sand and
Dust Storm
by
 Establishing a coordinated global network of Sand and Dust Storm
forecasting centers delivering products useful to a wide range of users in
understanding and reducing the impacts of SDS
North Africa, Middle East and Europe
BSC-CNS
AEMET, Spain
Asia
China Meteorological
Administration (CMA)
WMO REGIONAL CENTRES
103
Earth Sciences research lines
Air Quality Forecast
Mineral dust transport:
BSC-DREAM8b
Climate change modelling
New on-line mineral dust and CTM
model: NMMB/BSC-CTM
Transfer technology
(EIA and AQ studies)
WMO SDS WAS [AEMET-BSC]
ES-BSC air quality operational systems
AQF CALIOPE system: daily forecast and evaluation
–
Daily forecasts for meteorology and air quality (12 km for Europe and 4 km for
the Iberian Peninsula)  http://www.bsc.es/caliope
BSC-DREAM8b daily forecast and evaluation
North Africa/Mediterranean - 1/3 x 1/3 degree resolution
Asia domain - 1/2 x 1/2 degree resolution
 http://www.bsc.es/projects/earthscience/DREAM
CALIOPE: Air Quality Forecasting System
HERMES 2004: High-Elective Resolution Modelling Emission System (1km2-1h)
Database information management
Biogenic emissions
•
Land use map
•
Emission factors and
biomass
•
Meteorological
information (air temperature,
solar radiation)
On road, ships and
aircrafts traffic
•
Roads (urban streets, roads,
etc.), LTO cycles and ships routes
•
Hourly, weekly and monthly
profiles
•
Distribution of the vehicles
•
Velocity by road type
•
Meteorological information
(air temperature)
•
Type of combustibles
Industry and electric
generation
•
Location (point/area)
•
Production / electric
generation
•
Emission factors – energetic
intensity
•
Chimney height / diameter
Residential and
commercial
•
•
•
•
•
Density population map
Combustibles consume
Emission factors
Solvent use
Waste generation
Emission computation and speciation
BVOCs
NOx, VOCs, CO, SO2, particulate
matter, CO2, CH4 and N2O
NOx, VOCs, CO, SO2, particulate
matter, CO2, CH4 and N2O
NOx, VOCs, CO, SO2, particulate
matter, CO2, CH4 and N2O
Speciation
Management of the emission information
•
•
Visualization and analysis of the results
Management information tool
Management of emission data for air quality
modelling
•
•
•
•
Hourly aggregation of archives (4x4 km, 1x1 km grid size)
Height desaggregation (vertical layers)
Database resolution (1 - 4 km grid size)
Generation of netCDF emission files
Resolution improved to
12 km for all Europe, 4
km for the Iberian
peninsula, and 1 km for
hot spot regions within
MareNostrum
Supercomputer
Surface tropospheric ozone, 18-21 Jun 2007
Mineral dust transport: BSC-DREAM8b
109
Annual simulation
SDS-WAS RC NAMEE: NRT evaluation-AERONET and
intercomparison models (http://sds-was.aemet.es/models)
http://sds-was.aemet.es/models
Surface concentrations
Climate Modelling
Better understanding of the interactions among present climate change,
biogeochemical and human processes is a topic of increasing interest for
research and policy.
The main objectives are to investigate the causes and effects of climate
change and provide accurate and reliable information about its future
impact.
–
Participation in the Coupled Model Intercomparison
Project Phase 5 (CMIP-5), within the EC-EARTH project
(EU FP7), to provide future climate change information
that addresses outstanding scientific questions raised at
the IPCC-AR4 and that will be considered in the future
IPPC-AR5.
–
Improvement of the efficient use of high-performance
computers, model evaluation tool sets and access to
model results for the community, within the IS-ENES
project (EU FP7)
BSC Microsoft Research Centre Overview
More on: www.bscmsrc.eu
Barcelona Supercomputing
Centre: computer architecture,
parallel programming models
MSRC expertise: programming
language and operating system
design & implementation
Research at the intersection of computer architecture, language implementation,
and systems software
•Established in January 2008, building on a
previous joint project started in 2006
•BSC Effort: 2 Senior BSC researchers, 3
postdocs, 16 PhD students
•Support and part-time involvement of senior
researchers and faculty from UPC
BSCMSRC: Private/Public Partnership in Research
Two main models for company/university lablets
University Company
Closed
Generated IP is private
Students highly motivated
?
University Company
Open
Requires more generosity/effort
from company
?
Higher visibility for researchers
Finding the appropriate balance is key for success
closed
open
115
BSCMSR Centre: Modus Operandi
Open model
– No patents, public IP, papers and open source main focus
• Making positive impact in the community
• Contributing to standard setting bodies
– Participation of Microsoft Cambridge and Redmond senior
researchers
– Similar research agenda with parallel computing centers in US
•
•
•
•
Berkeley
Illinois
Rice
Stanford
Combining efforts is absolutely necessary
116
BSCMSRC Collaboration
Internships/stays:
– 5 BSC students interned at Microsoft Research
• Enables access to Microsoft proprietary technology
– 2 BSC students at long term stays at Microsoft Research
• Facilitates project progress through f2f interaction
– 6 External interns at BSCMSR in Cambridge
• Unlocks new collaboration opportunities with other centres
BSCMSRC - other Microsoft institute collaboration:
– Cosbi in Trento, Italy: study of complex human diseases in the
context of an FP7 EC project
EU Velox: BSC coordinating 9 partner, 5M€ project
– Microsoft Cambridge catalyst in introducing potential partners
– Microsoft observers to project (as is Intel and SUN)
117
BSCMSR Centre: visibility
Joint publications in conferences and journals
– 85 publications, 4 Best paper awards
Barcelona Multicore Workshops (joint BSCMSRC and HiPEAC events)
– In 2008, 2010, 2011
– More than 250 top academic experts participating in each
All applications and tools open source
– Used by more than 100 research groups worldwide
118
BSCMSR Centre: research topics
Initial research topic
– Transactional memory
Vector Support for Emerging Processors
Accelerating Databases with HW support
Big Data for Bioinformatics
Cloud Computing
Functional
HTM
MSRC
Imperative
–
–
–
–
STM
Programming model
New research topics
Transactional Memory
Applications
• Software (STM), hardware (HTM)
and hardware-assisted STM
• Programming model, runtime and
OS/architecture support
• Benchmark suite with realistic
applications
Architecture
BSC
119
BSCMSRC Website Jan. 2009
BSC role in joint participation in EU projects
BSC leading role in strategic projects that enable fundamental changes in
scientific discovery & innovation
Virtual multidisciplinary EnviroNments
USing Cloud Infrastructures
• Designed and deployed a general and generic
Cloud infrastructure to serve the wider eScience
community (long tail)
• 4.5 M € in 2010-2012 grant from the EU
• 14 partners (from industry and research)
• 27 different user communities (from the long
tail…)
• Spin-off of VENUS-C to Brazil
• A joint European and Brazilian e-Infrastructure
of open access resources to support the specific
needs of the biodiversity scientific community
• 1.2 M € in 2011-2013 grant from the EU
• 10 partners (from industry and research)
VENUS-C: Supporting the Long Tail of eScience
A user centric approach
Structural
Analysis for
Civil
Engineering
Building
Information
Management
Data for
Science AquaMaps
Civil
Protection
and
Emergencies
Bioinformatics
The initial 7 pilot applications
System
Biology
Drug
Discovery
The VENUS-C Platform
VENUS-C Enactment Services
Generic
Worker
Enactment
Service
VENUS-C API
COMPSs
Enactment
Service
CDMI
OCCI
OCCI
OCCI
OpenNebula
CDMI
FTP
COMPSs and Generic Worker compatible approach demonstrated interoperability, portability
and scalable performances
COMPSs Framework interoperability with Azure supports hybrid clouds
EU-Brazil OpenBio
•
Combining Biodiversity Science and the Open Access
Movement to deploy a joint European and Brazilian
e-Infrastructure of open access resources supporting the
needs of the biodiversity scientific community
Two biodiversity use cases
Computing resources
& SW platforms
Further EU-Brazil
collaboration in support
of the biodiversity area &
infrastructures
Joint EU-Brazil funded project,
with partners from each side
BSC leader on the European
side
COMPSs & VENUS-C used in
real production
124
Lessons learned
•
•
•
•
•
•
•
•
•
Thank you!
Fourth Champions Leage (2011)
3 – Barça / Manchester – 1
Cartagena, Colombia, May 18-20
Navigating the Mare Nostrum
México, Marzo, 2012
Are we planning to upgrade?
Negotiating our next site ;)