M. J. Ferrarotti, S. Decherchi and W. Rocchia CONCEPT Lab, Istituto

Category: Computational Chemistry - CC01
Poster
P6178
contact Name
Marco Jacopo Ferrarotti: [email protected]
A HIGHLY SCALABLE KERNEL BASED CLUSTERING ALGORITHM FOR MOLECULAR DYNAMICS
M. J. Ferrarotti, S. Decherchi and W. Rocchia
CONCEPT Lab, Istituto Italiano di Tecnologia, Via Morego 30, 16163 Genova, Italy
CLUSTERING MD TRAJECTORIES
A NOVEL ALGORITHM SUITED FOR MD: MINIBATCH DISTRIBUTED KERNEL K-MEANS
A BIG DATASET IN HIGHLY DIMENSIONAL SPACE
DESIGN GUIDELINES
DISTRIBUTION STRATEGY
• MD trajectory: discretized time evolution of the
atomic positions for an entire molecular system.
• Avoid featurization procedures: they are expensive and inaccurate, our
algorithm is based on Kernel K-means which does not require an explicit
feature space. Indeed the whole algorithm requires just a distance metric
being expressed in terms of a kernel matrix with the following form,
• K, f matrices as well as labels y are fully distributed.
• Up to 𝟏𝟏𝟏𝟏𝟏𝟏𝟏𝟏 frames (10μs simulation, 2fs timestep )
𝟓𝟓
• Each frame: ~𝟏𝟏𝟏𝟏 atom coordinates in 3D space
GETTING VALUE OUT OF DATA
−γ d2 xi ,xj
K i,j = e
2
2
; d = aligned RMSD xi , xj =
1
Natoms
Natoms
k=1
xi,k − xj,k
2
• Biomolecular processes of interest evolve through
a series of metastable states. How do we define
those states in a rigorous unbiased way? Can we
build reliable kinetic models?
• Target HPC facilities: being inspired by the work of Zhang, Rudnicky [3] we
devised a fully distributed algorithm based on the following quantities,
• Clustering is the core tool in probabilistic coarse
grained model such as Markov State Models [1].
• Think big: to scale on large datasets we divide the data into mini-batches
which are presented to the algorithm one after the other [4].
• Clustering by itself can be used to extract humaninterpretable mesostates to describe a process [2].
f xi , C = −
e.g. 10^6 frames
2
C
xj ∈C K
xi , xj ; g C =
16 nodes
32 GB /node
1
C2
xi ∈C
xj ∈C K
3 minibatches (single precision)
• Kernel matrix computation is carried out entirely on GPU with
a 3-stage pipeline in order to hide the latency of PCIe
communications.
• To the best of our knowledge the only publicly available GPU
implementation of such algorithm is the one introduced by Gil and
Guallar [6] in the python package pyRMSD. Such an implementation
shows good speedups but still there is room for improvements.
• It’s our plan to work on the data layout in memory to improve GPU
performances (in particular AoS to SoA transition will be explored
to allow coalesced memory access).
Nproc
all-to-all communications per step.
ALGORITHM PSEUDOCODE
D: dataset, B: batch, N: batch size, K: kernel matrix,
m: medoids, y: labels
• One CPU thread is dedicated to data fetching from disk and
Host-Device coordination, the other CPU threads iterate the
DKK algorithm on the available batch.
• It is fast and stable (can be implemented successfully in mixed
single/double precision fashion).
• 2Nc +
N
kernel elements.
4 minibatches (dobule precision)
• CPU and GPU work in a producer-consumer pattern: the GPU is
computing the kernel matrix for the (i + 1)-th batch while the
CPU is consuming the i-th one to perform the DKK iterations.
• Among the possible algorithms to compute best-aligned RMSD we
picked the quaternion based method proposed by Theobald [5].
• Each node deals with
N2
Nproc
xi , xj
GPU ACCELERATION
RMSD CUDA KERNEL
• Each node holds a partial copy of g and |C| .
CONCLUSIONS AND OUTLOOKS
• We present a distributed kernel k-means based algorithm to perform
large scale clustering on MD trajectories. The algorithm is fully
implemented in C++, distributed over MPI, modular and extensible.
• Single node performances are not sufficient for real-world scenarios
without GPU acceleration. The offload strategy presented here is ideally
paired with a self-tuning procedure in order to maximally squeeze the
hardware of modern HPC GPU-endowed facilities.
• The exploitation of approximate techniques where sparsity on the kernel
matrix is induced by randomly selecting a subset of samples as
representation basis [7], aiming at unprecedented scales (10^9 - 10^10
frames), which are compatible with the huge amount of simulative data
produced in the foreseeable future.
for i=1 to |D|/N do
B ← N frames uniformly sampled from D
K ← compute distributed kernel matrix
if (i==1) then mi ← kernelized k-means++ init
y ← label according to nearest neighbor medoid
while not converged do
compute local f, g, |C| based on K, y
allreduce g, |C|
for every sample: yj ← argminC ( f(xj,C) + g(C) )
allgather updated y on each node
end do
mi+1 ← find mini-batch medoids
mi+1 ← a ∙ mi + (1-a) ∙ mi+1 ; a=|C|i/(|C|i+|C|i+1)
end for
REFERENCES
[1] Voelz, Vincent A., et al. Journal of the American Chemical Society 132.5: 1526-1528 (2010).
[2] Decherchi, Sergio, et al. Nature communications 6 (2015).
[3] Zhang, Rong, et al. Pattern Recognition, 16th International Conference on. Vol. 4. IEEE (2002).
[4] Sculley, David. Proceedings of the 19th international conference on WWW. ACM,(2010).
[5] Theobald, Douglas L. Acta Crystallographica Section A 61.4: 478-480 (2005): .
[6] Gil, Víctor A., and Víctor Guallar. Bioinformatics: btt402 (2013).
[7] Chitta, Radha, et al. Proceedings of the 17th ACM SIGKDD. ACM (2011).
ACKNOWLEDGEMENTS
•
•
•
•
CompuNet – Computational unit for Drug Discovery and Development at IIT – for the support
CINECA – Italian computing center – for the HPC facilities
PRACE – Partnership for advanced computing in Europe – for the HPC facilities
AIRC – Italian association for cancer research – for funding (MFAG project 11899)