Discovering novel materials and sharing materials data

Transcription

http://www.aiida.net
Discovering novel materials and sharing materials data: Benefits and challenges
Giovanni Pizzi Theory and Simulation of Materials, EPFL, Switzerland
Atomistic simulations of molecules and crystals using quantum-mechanical first-principles methods (DFT, …) to predict and discover new materials and materials properties
How is open data important in our field?
What are the obstacles to open data?
What are we doing to go encourage open data?
Simulation codes
Accuracy and predictive power of quantum engines
Exponential increase in computational power in the past
years (“Moore’s law”)
Sumofthetop-500supercomputers
Number1
Number500
150,000xincrease inthepast20years 1month(1993) Computingpower1993-2013
10seconds(2015)
Result: materials design and discovery via high-throughput computations
...but did we think at how to manage simulations and share the data?
...but did we think at how to manage simulations and share the data?
What happens to the data?
0000000
0000010
0000020
0000030
0000040
0000050
0000060
0000070
0000080
0000090
00000a0
00000b0
00000c0
00000d0
00000e0
00000f0
0000100
0000110
0000120
0000130
0000140
0000150
0000160
0000170
0000180
0000190
00001a0
00001b0
00001c0
00001d0
00001e0
00001f0
0000200
0000210
0000220
0000230
0000240
0000250
0000260
0000270
0000280
0000290
00002a0
00002b0
00002c0
00002d0
00002e0
00002f0
89
00
73
66
61
60
e0
2f
20
40
64
20
b0
bd
b4
3c
fd
c0
03
c9
4a
2e
44
a0
d3
53
d8
d2
d4
73
af
80
00
02
04
10
40
80
00
02
00
01
04
00
01
04
00
02
50
00
00
69
12
7f
18
eb
ce
ec
6f
67
05
86
73
f3
39
e0
03
a5
7d
23
41
6d
48
1a
3a
21
dc
92
bd
00
02
08
10
40
00
00
02
08
01
04
10
01
04
10
02
08
4e
08
00
6c
60
c0
10
82
4e
1e
01
20
40
a1
7e
92
10
bf
ff
45
e7
00
54
3a
bc
3a
90
d8
9a
73
f7
02
08
20
40
00
01
02
08
20
04
10
40
04
10
88
08
20
47
9b
01
65
60
c0
e0
cc
2e
20
b1
fb
6c
a5
41
f5
1c
94
54
65
76
00
78
c6
e4
fc
5e
49
fe
f2
39
08
20
80
00
01
04
08
20
80
10
40
00
10
40
5b
20
80
0d
00
19
00
c8
c6
c3
c2
28
5b
d3
03
a8
a5
65
74
9e
81
84
50
61
2e
01
5e
5f
1b
5e
a7
1b
70
e9
20
80
00
01
04
10
20
80
00
40
00
01
40
00
80
80
00
0a
00
69
00
cd
c0
00
94
2a
24
21
90
bd
85
51
14
8c
81
98
63
f4
23
ec
ec
09
e8
92
a1
2b
f2
fa
80
00
02
04
10
40
80
00
02
00
01
04
00
01
60
00
02
1a
0c
43
28
2b
cb
04
c7
61
29
ec
cd
20
26
66
8c
62
e5
9a
18
00
01
dd
1c
8a
58
d2
69
e7
39
9d
00
02
08
10
40
00
00
02
71
01
04
08
01
04
93
02
08
0a
1f
43
91
29
c0
30
0b
60
1b
13
97
20
89
7a
0c
67
0f
21
99
00
78
0d
c7
86
43
a1
ec
f5
af
b5
02
08
c4
40
00
01
02
08
0b
04
10
36
04
10
b8
08
20
00
08
50
63
0a
cf
1a
b8
60
cc
60
04
e8
7e
46
0c
10
42
03
8c
00
a5
7c
46
44
7d
c6
10
9a
17
de
08
20
21
00
01
af
08
20
10
10
40
e1
10
40
a9
20
80
00
06
49
60
72
a0
15
52
cc
5e
35
66
98
20
89
cd
62
cc
83
19
09
3f
55
1b
6e
48
c2
85
ff
3d
f6
20
80
f0
01
04
00
20
80
6c
40
00
1a
40
00
48
80
00
00
00
43
60
77
c7
7c
52
00
00
10
33
92
08
82
18
08
a4
80
18
70
76
57
8a
a6
a5
94
60
3a
fb
5f
80
00
f6
04
10
c1
80
00
12
00
01
40
00
01
88
00
02
0d
00
43
52
52
60
bb
8b
b2
62
f6
81
9f
4a
23
18
80
97
3e
08
48
00
9d
b4
85
46
80
1b
cf
61
e6
00
02
38
10
40
26
00
02
37
01
04
00
01
04
00
02
08
49
00
20
48
88
96
c6
93
95
17
1d
ec
94
52
30
40
10
81
03
f1
59
00
ef
06
b4
b9
de
69
7b
ed
25
02
08
d2
40
00
5c
02
08
15
04
10
01
04
10
02
08
20
48
8d
50
2c
88
98
c0
81
cb
01
b0
e2
aa
2b
a4
e1
9b
61
c3
01
73
40
ff
70
43
cd
dc
a3
ef
b5
5e
08
20
90
00
01
08
08
20
09
10
40
04
10
40
08
20
80
44
23
72
28
8c
5c
08
f4
4b
1d
9a
4b
00
4a
52
0e
23
81
be
69
00
00
af
c2
34
f4
c8
46
f3
d7
08
20
80
04
01
04
08
20
80
11
40
00
10
40
00
20
80
00
52
34
6f
c8
52
5c
a2
1f
0a
08
90
87
f2
40
15
51
c1
0e
39
f3
00
49
be
4b
03
81
83
6d
90
7a
20
80
00
01
04
10
20
80
00
40
00
01
40
00
01
80
00
02
5-star open data (5stardata.info)
0000000
0000010
0000020
0000030
0000040
0000050
0000060
0000070
0000080
0000090
00000a0
00000b0
00000c0
00000d0
00000e0
00000f0
0000100
0000110
0000120
0000130
0000140
0000150
0000160
0000170
89
00
73
66
61
60
e0
2f
20
40
64
20
b0
bd
b4
3c
fd
c0
03
c9
4a
2e
44
a0
50
00
00
69
12
7f
18
eb
ce
ec
6f
67
05
86
73
f3
39
e0
03
a5
7d
23
41
6d
4e
08
00
6c
60
c0
10
82
4e
1e
01
20
40
a1
7e
92
10
bf
ff
45
e7
00
54
3a
47
9b
01
65
60
c0
e0
cc
2e
20
b1
fb
6c
a5
41
f5
1c
94
54
65
76
00
78
c6
0d
00
19
00
c8
c6
c3
c2
28
5b
d3
03
a8
a5
65
74
9e
81
84
50
61
2e
01
5e
0a
00
69
00
cd
c0
00
94
2a
24
21
90
bd
85
51
14
8c
81
98
63
f4
23
ec
ec
1a
0c
43
28
2b
cb
04
c7
61
29
ec
cd
20
26
66
8c
62
e5
9a
18
00
01
dd
1c
0a
1f
43
91
29
c0
30
0b
60
1b
13
97
20
89
7a
0c
67
0f
21
99
00
78
0d
c7
00
08
50
63
0a
cf
1a
b8
60
cc
60
04
e8
7e
46
0c
10
42
03
8c
00
a5
7c
46
00
06
49
60
72
a0
15
52
cc
5e
35
66
98
20
89
cd
62
cc
83
19
09
3f
55
1b
00
00
43
60
77
c7
7c
52
00
00
10
33
92
08
82
18
08
a4
80
18
70
76
57
8a
0d
00
43
52
52
60
bb
8b
b2
62
f6
81
9f
4a
23
18
80
97
3e
08
48
00
9d
b4
49
00
20
48
88
96
c6
93
95
17
1d
ec
94
52
30
40
10
81
03
f1
59
00
ef
06
48
8d
50
2c
88
98
c0
81
cb
01
b0
e2
aa
2b
a4
e1
9b
61
c3
01
73
40
ff
70
44
23
72
28
8c
5c
08
f4
4b
1d
9a
4b
00
4a
52
0e
23
81
be
69
00
00
af
c2
52
34
6f
c8
52
5c
a2
1f
0a
08
90
87
f2
40
15
51
c1
0e
39
f3
00
49
be
4b
How to make this process easy?
Why is it not so easy?
• Researchers’ main interests are
publications (and citations) Spending time to “make order” is not
perceived as giving direct benefit
• I don’t want to share my data
before publication (or patent) I want to publish as soon as possible;
After I’ll spend time to “prepare” the
data only if it won’t take too much time
• There is no easy tool to help me
organise my computational
research data
• Even if I organise my data, how
do I share it so that it is useful for
others?
Our (open-source) tool: AiiDA
We run computer laboratories: we should manage simulations and data with a dedicated platform to make the data share-ready already during its generation
http://www.aiida.net
MIT License
Approved License
What are our driving requirements for computational science? It should be…
reproducible
searchable
Often not possible from the data
reported in papers
Find and reuse existing calculations
reliable
shareable
Results in repositories, automated
verification procedures
towards “open data”, but private before publication
The ADES model
We have encoded these requirements in the four pillars for a computational science infrastructure
Low-level pillars
User-level pillars
G. Pizzi et al., Comp. Mat. Sci 111, 218-230 (2016)
Automation and Data in AiiDA
AD
What is AiiDA?
Automation: coupling to data
• Coupling automation to data:
– uniformity of the input data, usage of codes and computers
– full reproducibility of calculations (data is stored first)
Storage and provenance
• How do we store simulations preserving the connected
structure between them?
in1
• Inspiration from the open provenance model
in2
data
data
calc
F
• Any calculation: a function, converting inputs to outputs:
out1, out2 = F(in1, in2)
data
data
out1
out2
Data provenance: Directed Acyclic Graphs
G. Pizzi et al., Comp. Mat. Sci. 111, 218-230 (2016)
Need of advanced databases to store and efficiently query
Value of the data limited without knowing its provenance!
Environment and Sharing in AiiDA
ES
Keep data private...
...with seamless sharing capabilities
Sharing in AiiDA
•
•
•
Private AiiDA instances
UUIDs used to uniquely identify
all data/calculation objects
Data can be pushed to other
repositories or the outside world
...
Materials Cloud www.materialscloud.org
Access curated sets of
project results including
structures and their
properties
Discover
Learn
Learn using materials
including lectures and
video tutorials related to
materials science
Find documentation
and the AiiDA AppStore
of plugins and
workflows
Work
Explore
Directly browse and
visualise the AiiDA
database
Single calculations
Sub-workflows
Complex workflows in Materials Science
Main-Workflow
Relaxation #1
Structure
Relaxation
Relaxation #2
Restart
management
PW vc-relax
Restart (wall-time
exceeded, …)
PW vc-relax
Relaxation #n
Dynamical
matrices
Interatomic
force constants
PW vc-relax
Structure cell converged
several failure cases
handled automatically
Initialize PH
PH on q1
PH on q-grid
Parallelization
PH on q2
Phonon
dispersion
Collect phonons
PH on qn
Encoding the scientists’ knowledge in workflows
'pw_parameters': {
'SYSTEM': {'ecutwfc': 30.},
'ELECTRONS': {'conv_thr': 1.e-10}}
'ph_input':{'distance_kpoints_in_dispersion': 0.005,
StructureData (214814)
Code (139993)
UpfData (1658)
BaO3Ti
'pw-5.1.2-module'
'diagonalization': 'cg'}
}
structure
code
wf = asyncd(PhBandsWorkflow, **params)
params = {'input': {'kpoints_density': 0.2,
'convergence': 'tight'},
'structure': structure,
'pseudo_family': pseudo_family,
'machinename': 'mycluster',
'pw_input':{'volume_conv_threshold': 5e-2},
P
re
BaO3Ti
Code (139993)
'pw-5.1.2-module'
Code (209959)
'ph-5.1.2-module'
ParameterData (215223)
KpointsData (214808)
4x4x4 (+0.0,0.0,0.0)
parameters
settings qpoints
UpfData (1658)
structure
code
pseudo_O pseudo_Ti
parameters
6x6x6 (+0.0,0.0,0.0)
settings
kpoints
UpfData (1905)
pseudo_Ba
RemoteData (214841)
PwCalculation (214830)
scf FINISHED
RemoteData (214841)
parent_calc_folder
parent_calc_folder
code
remote_folder
Code (209959)
'ph-5.1.2-module'
PhCalculation (215224)
FINISHED
UpfData (1660)
structure
structure
retrieved
FolderData (215252)
code
code
code
parent_calc_folder
code
parent_calc_folder
retrieved
parent_calc_folder
code
parent_calc_folder
code
parent_calc_folder
code
code
parent_calc_folder
code
parent_calc_folder
parent_calc_folder
parent_calc_folder
code
parent_calc_folder
InlineCalculation (215309)
distribute_qpoints_inline()
qpoint_6
initial_folder
(1 kpts)
settings
parameters
qpoints
settings
FINISHED
(1 kpts)
parameters
qpoints
parameters
FINISHED
qpoint_4
(1 kpts)
settings
qpoints
settings
FINISHED
retrieved
parameters
qpoint_2
qpoints
parameters
qpoint_1
(1 kpts)
qpoints
retrieved
FolderData (215797)
retrieved_6
qpoint_0
qpoints
parameters
retrieved_2
settings
retrieved_0
settings
parameters
qpoints
FINISHED
retrieved_8
parameters
(1 kpts)
settings
qpoints
FINISHED
FolderData (215895)
parameters
qpoints
settings
parameters
FINISHED
retrieved
FolderData (215829)
retrieved_9
settings
(1 kpts)
FINISHED
retrieved
FolderData (215912)
retrieved_1
qpoint_7
(1 kpts)
retrieved
FolderData (215831)
retrieved_3
qpoints
retrieved
FolderData (215924)
retrieved_5
qpoint_9
(1 kpts)
FINISHED
retrieved
FolderData (215848)
retrieved_4
qpoint_8
settings
FINISHED
retrieved
FolderData (215978)
qpoint_3
(1 kpts)
FINISHED
retrieved
FolderData (215920)
qpoint_5
(1 kpts)
retrieved
FolderData (215976)
retrieved_7
recollect_qpoints_inline()
code
parent_calc_folder
retrieved
parent_calc_folder
code
parent_calc_folder
retrieved
Code (209960)
'q2r-5.1.2-module'
FolderData (215981)
force_constants
qpoint_6
qpoint_4
qpoint_5
kpoints
ForceconstantsData (216273)
code
code
parameters
Q2rCalculation (215983)
FINISHED
Code (209961)
'matdyn-5.1.2-module'
parent_calc_folder
parent_calc_folder code
(372 kpts)
code
parent_calc_folder
parameters
settings
qpoint_2
qpoint_3
qpoint_0
qpoint_1
MatdynCalculation (216285)
FINISHED
(1 kpts)
qpoints
(1 kpts)
settings
parameters
output_phonon_bands
BandsData (216385)
'Phonon bands'
retrieved
remote_folder
output_parameters
FolderData (216383)
RemoteData (216320)
(1 kpts)
qpoints
parameters
FINISHED
retrieved
retrieved_6
qpoints
parameters
retrieved_5
retrieved_3
FolderData (215831)
retrieved_0
retrieved_1
Folder
retrieved_8
retrieved
FolderData (215981)
Code (209960)
'q2r-5.1.2-module'
parameters
FINISHED
force_constants
Code (209961)
settings
retrieved
FolderData (215924)
retrieved_2
Paramete
FINISHED
retrieved
FolderData (215848)
retrieved_4
settings
FINISHED
retrieved
FolderData (215797)
qpoints
(1 kpts)
retrieved_9
Encoding the scientists’ knowledge in workflows
'pw_parameters': {
'SYSTEM': {'ecutwfc': 30.},
'ELECTRONS': {'conv_thr': 1.e-10}}
'ph_input':{'distance_kpoints_in_dispersion': 0.005,
Code (139993)
UpfData (1658)
BaO3Ti
'pw-5.1.2-module'
'diagonalization': 'cg'}
}
structure
code
wf = asyncd(PhBandsWorkflow, **params)
params = {'input': {'kpoints_density': 0.2,
'convergence': 'tight'},
'structure': structure,
'pseudo_family': pseudo_family,
'machinename': 'mycluster',
'pw_input':{'volume_conv_threshold': 5e-2},
P
re
BaO3Ti
Code (139993)
'pw-5.1.2-module'
Code (209959)
'ph-5.1.2-module'
4x4x4 (+0.0,0.0,0.0)
parameters
settings qpoints
UpfData (1658)
structure
code
pseudo_O pseudo_Ti
parameters
6x6x6 (+0.0,0.0,0.0)
settings
kpoints
UpfData (1905)
pseudo_Ba
RemoteData (214841)
PwCalculation (214830)
scf FINISHED
RemoteData (214841)
parent_calc_folder
parent_calc_folder
code
remote_folder
Code (209959)
'ph-5.1.2-module'
FINISHED
UpfData (1660)
structure
structure
retrieved
FolderData (215252)
code
code
code
parent_calc_folder
code
parent_calc_folder
retrieved
parent_calc_folder
code
parent_calc_folder
code
parent_calc_folder
code
code
parent_calc_folder
code
parent_calc_folder
parent_calc_folder
parent_calc_folder
code
parent_calc_folder
qpoint_6
initial_folder
(1 kpts)
settings
parameters
qpoints
settings
FINISHED
(1 kpts)
parameters
qpoints
parameters
FINISHED
qpoint_4
(1 kpts)
settings
qpoints
settings
FINISHED
retrieved
parameters
qpoint_2
qpoints
parameters
qpoint_0
qpoints
qpoint_1
(1 kpts)
retrieved
FolderData (215797)
qpoints
parameters
qpoint_9
(1 kpts)
settings
settings
qpoint_7
(1 kpts)
parameters
qpoints
FINISHED
retrieved
FolderData (215924)
qpoints
FINISHED
retrieved
FolderData (215848)
qpoint_8
settings
FINISHED
retrieved
FolderData (215978)
qpoint_3
(1 kpts)
FINISHED
retrieved
FolderData (215920)
qpoint_5
(1 kpts)
settings
qpoints
FINISHED
retrieved
FolderData (215831)
parameters
(1 kpts)
retrieved_4
retrieved_5
retrieved_2
retrieved_3
retrieved_0
FolderData (215912)
retrieved_1
retrieved_8
FolderData (215895)
qpoints
settings
parameters
FINISHED
retrieved
FolderData (215829)
retrieved_9
parameters
FINISHED
retrieved
retrieved
FolderData (215976)
Simple and reliable “turn-key” workflows can be a tool to validate results
contributed to a repository
retrieved_6
settings
(1 kpts)
retrieved_7
code
parent_calc_folder
retrieved
parent_calc_folder
code
parent_calc_folder
retrieved
Code (209960)
'q2r-5.1.2-module'
FolderData (215981)
force_constants
qpoint_6
qpoint_4
qpoint_5
kpoints
code
code
parameters
FINISHED
Code (209961)
'matdyn-5.1.2-module'
parent_calc_folder
(372 kpts)
code
parent_calc_folder
parameters
settings
qpoint_2
qpoint_3
qpoint_0
qpoint_1
MatdynCalculation (216285)
FINISHED
(1 kpts)
qpoints
(1 kpts)
settings
parameters
output_phonon_bands
BandsData (216385)
'Phonon bands'
retrieved
remote_folder
output_parameters
FolderData (216383)
RemoteData (216320)
(1 kpts)
qpoints
parameters
FINISHED
retrieved
retrieved_6
retrieved
FolderData (215848)
retrieved_4
settings
qpoints
parameters
FINISHED
retrieved
FolderData (215797)
qpoints
(1 kpts)
retrieved_5
FolderData (215924)
retrieved_2
retrieved_3
retrieved
FolderData (215831)
retrieved_0
retrieved_1
Folder
retrieved_8
retrieved
Code (209960)
'q2r-5.1.2-module'
parameters
FINISHED
force_constants
Code (209961)
settings
FINISHED
FolderData (215981)
Paramete
retrieved_9
App-store model: open tools for open data
“App-store”-like model for
plugins & workflows, e.g.
• Computers: automatically
setup a new cluster or
supercomputer
• DB importers: load structures
and data from COD, ICSD, …
The MaterialsCloud AiiDA app-store
Simulation codes
Quantum ESPRESSO
Yambo
CP2K
Fleur
BSC (ES)
Jülich (DE)
Optical properties
Transport
Computing clusters
• Calculations: plugin for your
simulation software
• Turn-key solutions: workflows to compute a
desired property
CSCS (CH)
CINECA (IT)
Turn-key workflows
• …
Electronic density
Phonons
Some final thoughts
• Publishers and funding agencies: How much should they
support/enforce publication of ‘raw’ data? (Things are moving: e.g.
Scientific Data@NPG, …)
• Quality of metadata: not only raw data, but structure:
provenance, reproducibility, and open/documented format
• Ontologies and interoperability: Domain-specific
ontologies (e.g. in RDF) for crystal structures, energy, forces, bulk
moduli, …; not easy to “convince” researchers… But how much
should we really invest in this?
• Open repository: curated or not? Limited submission or
accepts anything? How to verify the quality of data?
• Access to data: Fully open with no barriers, or via registration
+terms of use? How to know the actual data usage (e.g. to justify
project with funding agencies)? What about potential ethical
implications?
0000000 89 50 4e 47 0d 0a
1a 0a 00 00 00 0d 49 48 44
52
0000010 00 00 08 9b 00 00
0c 1f 08 06 00 00 00 8d 23
The AiiDA team
Acknowledgements: people
Giovanni
Pizzi
(EPFL)
Andrea
Cepellotti
(EPFL)
Martin Uhrin
(EPFL)
Leonid Kahle
(EPFL)
Nicolas
Mounet
(EPFL)
Riccardo
Sabatini
(EPFL)
Boris
Kozinsky
(BOSCH)
Nicola
Marzari
(EPFL)
Spyros Zoupanos
(EPFL)
Snehal Waychal
(EPFL)
Fernando Gargiulo
(EPFL)
Andrius Merkys
(Vilnius)
Plugin contributors — Quantum ESPRESSO NEB: Marco Gibertini (EPFL); Quantum ESPRESSO DOS, PDOS;
Wannier90: Daniel Marchand (EPFL); CP2K: Aliaksandr Yakutovich (EMPA), Uli Schauer (ETHZ), Tiziano Müller, Andreas
Gloess, Patrick Seewald (UZH); FLEUR: Jens Broeder, Gregor Michalicek, Daniel Wortmann (Jülich); Exciting: Anton
Kozhevnikov (CSCS); YAMBO: Andrea Ferretti, Giovanni Borghi, Daniele Varsano (CNR-NANO), Gianluca Prandini
(EPFL); SIESTA: Victor Garcia Suarez (Uni Oviedo); i-PI: Venkat Kali (EPFL); VASP: Mario Zic (Trinity College Dublin).
Contributors — Prof. Christoph Koch (EPFL); Valentin Bersier, Philippe Schwaller (THEOS EPFL); Marco Dorigo (ICAMS - Bochum); Eric Hontz (MIT & Bosch RTC)
Early beta testers — Giovanni Borghi, Ivano Castelli, Marco Gibertini (THEOS EPFL); Prateek Mehta (Bosch RTC)
Acknowledgements: funding
• MARVELNCCR http://marvel-nccr.ch SwissNationalCentreofCompetence
onComputationalDesignand
DiscoveryofNovelMaterials StartedMay2014,funded
2014-2026. 33professorsfrom11Swiss
Institutions.
• MaX http://max-center.eu “MAterialsdesignattheeXascale”:
EUH2020e-infrastructureproject FromSep2015for30months. GroupsfromModena,Trieste,
Lausanne,Barcelona,Julich +5supercomputingcenters:
CINECA,CSCS,BCS,Julich,KTH
Shift the scientist’s focus from the management of simulations...
…to the discovery of new materials…
…making it seamless to share research data
Contacts and info:
Website: http://www.aiida.net
Docs: http://aiida-core.readthedocs.io
Git repo: https://bitbucket.org/aiida_team/aiida_core/
https://www.facebook.com/aiidateam
@aiidateam

Discovering novel materials and sharing materials data

Transcription

Similar documents

prostep

Collector`s Guide 2016

262774 remote control