An Open-Source Vaa3D Plugin for Real-Time 3D

AN OPEN-SOURCE VAA3D PLUGIN FOR REAL-TIME 3D VISUALIZATION OF
TERABYTE-SIZED VOLUMETRIC IMAGES
Alessandro Bria1,2
1
2
Giulio Iannello1
Hanchuan Peng3,4
Integrated Research Centre, University Campus Bio-Medico of Rome, 00128 Rome, Italy
Department of Electrical and Information Engineering, University of Cassino and L.M., 03043 Cassino, Italy
3
4
Janelia Farm Research Campus, Howard Hughes Medical Institute, Ashburn, Virginia 20147, USA
Allen Institute for Brain Science, 551 North 34th Street, Suite 200, Seattle, Washington 98103, USA
ABSTRACT
Modern high-throughput bioimaging techniques pose the unprecedented challenge of exploring and analyzing the produced Terabyte-scale volumetric images directly in their 3D
space. Without expensive virtual reality devices and/or parallel computing infrastructures, this becomes even more demanding and calls for new, more scalable tools that help exploring these very large 3D data also on common laptops and
graphic hardware. To this end, we developed a plugin for the
open-source, cross-platform Vaa3D system to extend its powerful 3D visualization and analysis capabilities to images of
potentially unlimited size. When used with large volumetric images up to 2.5 Terabyte in size, Vaa3D-TeraFly exhibited real-time (subsecond) performance that scaled constantly
on image size. The tool has been implemented in C++ with
Qt and OpenGL and it is freely and publicly available both
as open-source and as binary package along with the main
Vaa3D distribution.
Terabyte-size [14, 15], or are limited owing to high expense of
licenses (e.g. Amira(VSG), Imaris (Bitplane)) and infrastructures [16]. This presents an obstacle for the unbiased, highthroughput and quantitative analysis of bioimage data and
creates tremendous need for the development of new techniques that help explore very large 3D data directly and efficiently without expensive virtual reality devices and/or parallel computing infrastructures.
The open-source Vaa3D system [17] was specifically designed for bridging such gaps, particularly for the tasks of
visualization, interaction, analysis and management of large
(Gigabyte-size), multidimensional (including 3D, 4D and 5D
and so on) microscopic images [18]. The underlying idea is to
map users’ inputs in the 2D plane of a computer screen to the
3D locations of biological entities (for example, cells, neurons or microtubules) in the volumetric space of a 3D image
stack. This is achieved through a family of 3D-WYSIWYG
(what you see in 2D is what you get in 3D) computer algorithms named Virtual Finger [19].
We showed in [19] how a single finger operation is efficiently used by our Vaa3D plugin TeraFly to generate a 3D
region of interest (ROI) for subsequent fetching and rendering of image data at a higher resolution, thus enabling fast
(subsecond) visualization of Terabyte-size images. In this article, we focus on the technological aspects of the tool and
demonstrate its scalability to images of potentially unlimited
size. We also propose a method to generate a 3D ROI for
mouse scroll input, which provides a user’s experience similar to how Google Earth allows viewing satellite imageries.
Finally, we show that Vaa3D-TeraFly exhibits real-time performance regardless on image size when used on both highand medium-end laptops.
Index Terms— Terabyte, visualization, Vaa3D
1. INTRODUCTION
Modern high-throughput bioimaging techniques enable rapid
three-dimensional imaging of large specimens at submicrometric resolution [1, 2, 3]. This led to the generation of images that easily reach or exceed the Terabyte size and, as
consequence, to the subsequent effort of biomage informaticians to develop more scalable techniques for typical bioimage informatics tasks such as image stitching and registration
[4, 5, 6], visualization [5, 7, 8] and automated analysis [9, 10].
In addition to scalability, another bottleneck is the inability
to efficiently explore the complicated 3D image content and
thus to input user-specified information of the observed image patterns directly in the 3D space. Indeed, a number of
bioimage-oriented software tools for Terabyte-scale [7, 8, 11]
and Gigabyte-scale [12, 13] images are based on 2D crosssectional views or a combined display with an arbitrary-angle
cutting plane. On the other hand, the few tools supporting
3D visualization and analysis either do not scale well to the
978-1-4799-2374-8/15/$31.00 ©2015 IEEE
2. APPROACH
Vaa3D-TeraFly is designed around three basic functionalities enabling the instant 3D Google Earth-like visualization
of Terabytes, and virtually infinite amount, of high-resolution
image data: (i) volume import, i.e. the generation of an ad
520
TMITREE
X
[…]
Y
Layer 2 𝐼
2
Layer 1 𝐼
1
Layer 0 𝐼
0
[…]
Z
column 0
(a)
prev:
Explorer
rend:
Renderer
(i)
next:
Explorer
rend:
Renderer
volume:
Volume
new explorer
set ROI
load ROI
get interpolated ROI
hide
show
new renderer
(ii)
column 3
row (3,0)
loading ended
row (3,3)
update image
(iii)
block (3,3,0) block (3,3,1) block (3,3,2)
Fig. 1. An example of TMITREE composed by three layers. Each layer is composed by a number of nonoverlapping
blocks of image data that are stored into separate files. These
files are organized according to a two-level hierarchy of folders where the first level subdivides the volume into columns
(or rows) and the second one into rows (or columns).
an object instantiated in the GPU thread
total time
an object instantiated in the IO thread
IO time saved
(b)
hoc multiresolution tree-structured representation of the volumetric image; (ii) zoom-in to a ROI, i.e. the generation of a
ROI when the user zooms-in into the displayed image with the
mouse scroll; and (iii) fetch and display of image data, i.e. the
loading of the ROI’s content from the storage and subsequent
3D display. These functionalities are detailed in the following
subsections.
(i)
(ii)
(iii)
(c)2. ROI’s loading and display. (a) UML Sequence DiaFig.
gram. (b) The effect of actions (i), (ii) and (iii) of (a) on the
viewport when zooming-in with the mouse scroll. The actions
taken by the tool to preview/display and to load the data are
performed in parallel on two different threads, named GPU
and IO, respectively. While the preview is calculated and dis(i) the GPU thread, image
(ii) data are loaded
(iii)by the IO
played by
thread. In this way, the first part of the IO time is transparent
to the user, i.e., it is not perceived as a delay.
2.1. Volume import
Vaa3D-TeraFly is built on the top of an image tiled multiresolution representation, hereafter referred to as Tiled Multiresolution Image Tree (TMITREE), which is generated as follows. Let be I the original, highest-resolution, very largesized image to be imported. Starting from I, we produce another k images {I (1) , I (2) , . . . , I (k) } obtained by iteratively
downsampling by two I (j) from I (j−1) ∀j = 1, . . . , k, where
I (0) = I. This process is iterated until the lowest-resolution
image I (k) fits within the 3D viewer box of size Bx (x) ×
By (y) × Bz (z). These define the maximum size in pixels of
displayable image data in the 3D renderer and can be set by
the user according to the graphic hardware capabilities (default is 256(x)×256(y)×256(z) pixels). At all resolutions,
image data are stored according to a tile-wise organization of
nonoverlapping three-dimensional blocks (see Fig. 1).
A converter tool, named TeraConverter, comes along
with Vaa3D-TeraFly for converting a given bunch of image
files into the TMITREE representation. With TeraConverter,
the user can select the number of layers to be produced, the
block dimensions and the format of the image files. Once the
TMITREE has been produced, it can be opened with Vaa3D-
TeraFly by selecting any of its layers I (j) , j = 0, . . . , k.
The 3D exploration then automatically starts by loading and
displaying the entire image content of the highest layer in
the TMITREE, i.e., the lowest resolution image I (k) . This
usually takes < 5 seconds as I (k) is conveniently stored in
one binary uncompressed file for fast volume opening.
2.2. Zoom-in to a ROI
The underlying idea is that zoom-in with the mouse wheel
is usually performed in the nonempty regions of an image
and, thus, without losing in generality, we can assume the
presence of one or more biological entities displayed on the
foreground of the viewport. Since the mouse-wheel zoom-in
in the Vaa3D renderer is viewport-centered, these entities are
likely to occur around the center of the viewport and, thus,
521
cache
Fig. 3. 3D exploration though different layers of the TMITREE with mouse-wheel scrolls up (zoom-in) and down (zoom-out)
and caching of ROIs for quick navigation
we can focus on this region to calculate the ROI’s center. To
this end we define a 2D point in the center of the viewport
and apply the point-pinpointing algorithm (PPA) of Virtual
Finger [19] to determine the correspondent 3D location in the
volumetric image. PPA repeatedly applies the mean-shift algorithm to each channel intensity along the shooting ray orthogonal to the screen until convergence and finally selects
the best point among the points found for all the channels.
The ROI is finally calculated as the Bx (x) × By (y) × Bz (z)
sized box centered on the 3D point.
Vaa3D-TeraFly loads the image content from the blocks of
the TMITREE layer L that intersect with ROI L and updates
the viewed ROI L with the higher resolution data. The detailed sequence of actions taken by the tool is shown in Fig. 2.
Throughout 3D exploration, the ROIs that have been already
viewed are cached in the graphic card memory (if possible) or in the system memory following a web-browser-like
strategy in which the user can instantly go back and forward
(see Fig. 3). This requires storing in the graphic card memory (k + 1) × Bx × By × Bz × c bytes at most, where c
is the number of channels and we assumed 8 bits per pixel
for display. Based on the criterion to choose k described in
Section 2.1, and after simple algebraic manipulations, the
maximum
memory requirement for a cubic image of N pixels is (log(N 1/3 ) − log(B)) log2 (e)) + 1 × B 3 × c bytes
where we chose Bx = By = Bz = B. Remarkably, this
corresponds to only 336 Megabytes for 1 Teravoxel threechannels image (i.e. 10, 0003 voxels) and to 480 Megabytes
for 1 Petavoxel three-channels image (i.e. 100, 0003 voxels).
2.3. Fetch and display of image data
The proposed mouse scroll zoom-in to a ROI requires the
image content of the requested ROI to be retrieved and displayed as quickly as possible for fast and smooth exploration
of the image. This is accomplished by performing, in parallel, two different operations: (i) reusing part of the currently
viewed content to display instantly a preview of the ROI; and
(ii) loading from the TMITREE only the blocks that contain
the required image content. More formally, let be ROI ` the
ROI currently viewed that belongs to the TMITREE layer
` (ROI ` ≡ I (k) when the exploration starts) and ROI L
the ROI to be retrieved and displayed from the higher resolution layer L, with L ≤ ` (usually L = ` − 1). Then,
Vaa3D-TeraFly interpolates to the resolution of L the portion of ROI ` that intersects with ROI L and displays the
result. This gives the clue that the system has correctly and
instantly responded to the request, and also makes more acceptable to wait for the full image to be loaded. Meanwhile,
3. RESULTS
We tested visualization of 3D image volumes of rat neurons
and whole mouse brains with sizes ranging from 22.8 Gb to
2.5 Tb. The statistics for each test case were produced by on
at least 20 trials of randomly selected target ROIs in arbitrarily
determined scales of the respective images. The total time for
generation, loading and displaying of a 3D ROI is reported in
Fig. 4 for three different laptops with different CPUs, GPUs
522
[4] A Bria and G Iannello, “Terastitcher - a tool for fast automatic 3D-stitching of teravoxel-sized microscopy images,”
BMC Bioinformatics, vol. 13, no. 1, pp. 316, 2012.
[5] AG Arivis,
“Vision 4D,” http://vision.arivis.
com/, 2014, [accessed 23-October-2014].
[6] L Cooper, K Huang, and M Ujaldon, “Parallel Automatic Registration of Large Scale Microscopic Images on Multiprocessor CPUs and GPUs,” in Parallel and Distributed Processing Workshops and Phd Forum (IPDPSW), 2011 IEEE International Symposium on, May 2011, pp. 1367–1376.
[7] T Pietzsch,
“Bigdataviewer,” http://fiji.sc/
BigDataViewer, 2014, [accessed 23-October-2014].
[8] W-K Jeong, J Beyer, M Hadwiger, R Blue, C Law, A V´azquezReina, RC Reid, J Lichtman, and H Pfister, “SSECRET
and neurotrace: Interactive visualization and analysis tools for
large-scale neuroscience data sets,” IEEE Comput. Graph.
Appl., vol. 30, no. 3, pp. 58–70, May 2010.
[9] P Frasconi, L Silvestri, P Soda, R Cortini, FS Pavone, and
G Iannello, “Large-scale automated identification of mouse
brain cells in confocal light sheet microscopy images,” Bioinformatics, vol. 30, no. 17, pp. i587–i593, 2014.
[10] J Stegmaier, JC Otte, A Kobitski, A Bartschat, A Garcia,
G Nienhaus, U Strhle, and R Mikut, “Fast segmentation of
stained nuclei in terabyte-scale, time resolved 3D microscopy
image stacks,” PLoS ONE, vol. 9, no. 2, pp. e90036, 02 2014.
[11] Z Husz, N Burton, B Hill, N Milyaev, and R Baldock, “Web
tools for large-scale 3d biological images and atlases,” BMC
Bioinformatics, vol. 13, no. 1, pp. 122, 2012.
[12] MD Abrmoff, PJ Magalhes, and SJ Ram, “Image Processing
with ImageJ,” Biophotonics International, vol. 11, no. 7, pp.
36–42, July 2004.
[13] F Long, J Zhou, and H Peng, “Visualization and analysis of
3d microscopic images,” PLoS Comput Biol, vol. 8, no. 6, pp.
e1002519, 2012.
[14] T Pietzsch, S Preibisch, C Rueden, S Saalfeld, B Schmid, JY Tinevez, DJ White, V Hartenstein, K Eliceiri, P Tomancak,
and A Cardona, “Fiji: an open-source platform for biologicalimage analysis,” Nat. Methods, vol. 9, no. 7, pp. 676–682,
2012.
[15] JL Clendenon, CL Phillips, RM Sandoval, S Fang, and
KW Dunn, “Voxx: a pc-based, near real-time volume rendering system for biological microscopy,” American Journal of
Physiology - Cell Physiology, vol. 282, no. 1, pp. C213–C218,
2002.
[16] D N Williams et al., “Ultrascale visualization of climate data,”
Computer, vol. 46, no. 9, pp. 68–76, 2013.
[17] H Peng, Z Ruan, F Long, JH Simpson, and EW Myers, “V3d
enables real-time 3d visualization and quantitative analysis of
large-scale biological image data sets,” Nat Biotech, vol. 8, no.
4, pp. 246–249, 2010.
[18] H Peng, A Bria, Z Zhou, G Iannello, and F Long, “Extensible
visualization and analysis for multidimensional images using
Vaa3D,” Nat. Protocols, vol. 9, no. 1, pp. 193–208, 2014.
[19] H Peng et al., “Virtual finger boosts three-dimensional imaging
and microsurgery as well as terabyte volume image visualization and analysis,” Nat. Communications, vol. 5, 2014.
3D-viewer total generation time for a ROI (ms)
2500
2000
Asus M70vn + external hard drive 2.5'' WD 1Tb USB 2.0
CPU Intel Core 2 Duo P8600, RAM 4 Gb, GPU NVIDIA GeForce 9650M GT 1Gb
1500
Sony Vaio VPCF21Z1E + external hard drive 2.5'' WD 1Tb USB 3.0
1000
CPU Intel Core 2 i7-2630QM, RAM 8 Gb, GPU NVIDIA GeForce GT 540M 1Gb
500
MacBook Pro Retina + external hard drive Seagate 4Tb USB 3.0
CPU Intel Core i7, RAM 16 Gb, NVIDIA GeForce GT 650M 1Gb
0
10
100
1.000
2.520
Image size (Gigabytes)
Fig. 4. Bench-test of ROI-generation time (mean ± s.d.) for
different large images on different laptops. Each point in the
graph corresponds to one test case run on one dataset of the
indicated size.
and external storages. Remarkably, the time scaled constantly
on image size regardless of the hardware employed. In particular, subsecond performance was achieved with a high-end
laptop, whereas on medium-level laptops it ranged from 1.3
to 2.2 seconds, which is still acceptable for effective 3D visualization. RAM usage was also monitored in all the experiments and resulted below 1 Gb.
4. CONCLUSIONS
3D visualization and analysis of very large bioimages are in
strong need of powerful and more scalable 3D exploration
techniques for high-throughput and quantitative analysis of
bioimage data. We have discussed here a software tool that
extends the Vaa3D powerful 3D visualization and analysis capabilities to images of potentially unlimited size. The results
provided demonstrate that the tool can visualize in real-time
Terabytes of 3D images and can scale well to even larger images without the need of expensive hardware or parallel infrastructures.
5. REFERENCES
[1] R Tomer, L Ye, B Hsueh, and K Deisseroth, “Advanced clarity
for rapid and high-resolution imaging of intact tissues,” Nat.
Protocols, vol. 7, no. 9, pp. 1682–1697, 2014.
[2] K Chung and K Deisseroth, “CLARITY for mapping the nervous system,” Nat. Methods, vol. 10, no. 6, pp. 508–513, 2013.
[3] L Silvestri, A Bria, L Sacconi, G Iannello, and FS Pavone,
“Confocal light sheet microscopy: micron-scale neuroanatomy
of the entire mouse brain,” Optic Express, vol. 20, no. 18, pp.
20582–20598, 2012.
523