View details

Proposal for
DICOM Multi-spectral presentation states
contributions from:
Max Derhak, Masahiro Yamaguchi, Phil Green,
Bas Hulsken
Bas Hulsken
Digital Pathology Solutions
March 19, 2015
What are multi-spectral
presentation states, and why
do we need them?
2
Ma rch 19, 2015
Di gital Pathology Solutions
Benefits of a DICOM multi-spectral presentation
state:
• Improved true color (wide gamut) rendering with 4 or more channels
• Standardized way to store and exchange multi-spectral image data
• Reproducible pseudo color images (e.g. for fluorescent imaging)
• Standardize channel un-mixing while preserving raw data
• Color reproduction under different illuminant
Current limitations in DICOM
• Only 3 input channel ICC v4 input profiles are available in DICOM
• 3 channels is too limited for most fluorescent & multi-spectral use cases
• ICCv4 profiles describe transformations to 3 channel PCS, which will lead
to information loss for all multi-spectral uses cases where true color
rendering is not the sole objective.
3
Ma rch 19, 2015
Di gital Pathology Solutions
True color with more than 3 color channels
• Most Whole Slide Imaging devices are using LED illumination, this allows:
– narrow band illumination with >3 channels for wide gamut imaging
– narrow band illumination for spectral analysis of images, e.g., for
“fluorescence-like” segmentation (for display& algorithms)
Multi-spectral bright field segmentation
Source:
http://www.rms.org.uk/Resources/Royal%20Microscopic
al%20Society/infocus/infocus%20June%2009%20%20Mansfield%20_%20Levenson%20Article.pdf
4
Ma rch 19, 2015
Di gital Pathology Solutions
5 channel HDTV, Source: Journal of Imaging
Science and Technology, Volume 49, Number 6,
November/December 2005 , pp. 594-604(11)
Multi spectral fluorescence
•
•
•
5
Remove raw capture channel crosstalk to get one channel per biomarker and autofluorescence suppression, for display and algorithms.
Define standardized pseudo color display for each bio-marker
Allow capture device calibration to get quantitative bio-marker concentrations
Ma rch 19, 2015
Di gital Pathology Solutions
How to extend DICOM for
these use cases
6
Ma rch 19, 2015
Di gital Pathology Solutions
DICOM extensions required for the different use
cases
1. Ability to define how to display multi-spectral images as true color visible light
images.
2. Ability to define how to un-mix multispectral input channels for the purpose of
deriving quantitative representations of individual biomarker intensities, said
markers can be fluorescent or chromogenic.
3. Ability to define how to display (un-mixed) multi-spectral images (fluorescent,
chromogenic) as pseudo color images. It should be possible to use the un-mixed
output from 2) as input for this mode.
7
Ma rch 19, 2015
Di gital Pathology Solutions
DICOM extensions required for the different use
cases
1. Ability to define how to display multi-spectral images as true color visible light
images.
Can be done within an ICCv4 input profile
2. Ability to define how to un-mix multispectral input channels for the purpose of
deriving quantitative representations of individual biomarker intensities, said markers
can be fluorescent or chromogenic.
Can NOT be done within an ICCv4 input profile
- option 1: use chained ICCv4 device link profiles
- option 2: use iccMAX material connection space
3. Ability to define how to display (un-mixed) multi-spectral images (fluorescent,
chromogenic) as pseudo color images. It should be possible to use the un-mixed
output from 2) as input for this mode.
Can be done within an ICCv4 input profile
8
Ma rch 19, 2015
Di gital Pathology Solutions
Extending DICOM for multi-spectral with ICCv4
Intermediate outputs:
• Calibrated channels
• Unmixed
biomarkers
Input
channels
9
Ma rch 19, 2015
Di gital Pathology Solutions
Device calibration &
Un-mixing (not in PCS!)
pseudo color
output
Extending DICOM for multi-spectral with iccMAX
(slide from Max Derhak)
• iccMAX (reference implementation in RefICCLabs)
– Spectral profile
– Calc element
 Need for complex routing of profiles in DICOM is greatly diminished
• Proposal of “Material Connection Space” Profiles
– MCS connection allowed between source biomarker Material Identification (MID)
and destination biomarker Material Visualization (MVIS) profiles
MultiSpectral
Channels
MID/Input
Profile
MVIS
Profile
MCS
Material Type
Array Tag
Channel 0: “mRGB-R”
Channel 0: “mRGB-R”
Channel 1: “mRGB-G”
Channel 1: “mRGB-G”
Channel 2: “mRGB-B”
Channel 2: “mRGB-B”
Channel 3: “BM2”
Channel 4: “BM3”
Channel 5: “BM5”
10
Channel 5: “BM7”
Ma rch 19, 2015
Material Type
Array Tag
Di gital Pathology Solutions
⌀
⌀
⌀
⌀
Channel 3: “BM1”
Channel 4: “BM2”
PCS
Output
Profile
Display
Device
Channels
Benefits of a DICOM multi-spectral presentation
state:
• Improved true color (wide gamut) rendering with 4 or more channels
• Standardized way to store and exchange multi-spectral image data
• Reproducible pseudo color images (e.g. for fluorescent imaging)
• Standardize channel un-mixing while preserving raw data
• Color reproduction under different illuminant
Current limitations in DICOM
• Only 3 input channel ICC v4 input profiles are available in DICOM
Ok!
• 3 channels is too limited for most fluorescent & multi-spectral use cases
• ICCv4 profiles describe transformations to 3 channel PCS, which will lead
Ok!
to information loss for all multi-spectral uses cases where true color
rendering is not the sole objective.
11
Ma rch 19, 2015
Di gital Pathology Solutions
Getting it in DICOM
12
Ma rch 19, 2015
Di gital Pathology Solutions
Steps to take to get multi-spectral presentation
states in DICOM
1)
2)
3)
4)
13
Discuss high-level DICOM implementation in WG26
WG26 to agree on high-level implementation
(scheduled for March 22nd 2015)
Send to WG6 for discussion
(tentatively scheduled for June 2015)
Write the full DICOM implementation
Ma rch 19, 2015
Di gital Pathology Solutions
Required DICOM extensions
(only for ICCv4, only for iccMAX)
Simple removal of limitations:
1) Allow multiple ICC profiles in one DICOM object (1 now)
2) Allow devicelink ICC profiles (only input profile now)
3) Allow ICC profiles for n channels (3 channels now)
4) Allow iccMAX profiles
More complex additional functionality
5) Add module describing multi ICC profile render pipeline
6) Define way to store multi-channel image data
7) Add module describing characteristics of channels (spectral,
biomarker, etc.)
14
Ma rch 19, 2015
Di gital Pathology Solutions
5) module describing multi ICC profile render
pipeline
(only needed for ICCv4, not for iccMAX, see example profiles slides 19& 20)
• Add module to describe chaining of ICC profiles in render pipeline
– Can use softcopy presentation state either for inspiration, or by
extending
– Need to combine multiple images and/or channels
 Option A: enhance presentation states to blend more than 2 images
• Currently the advanced blending and display pipeline is both too
complex and too limited. Too complex because ICC profiles contain
all required functionality, too limited because maximum 3 data
frames are supported(C7.6.23-1 of Part3).
 Option B: add new module describing chaining, use (chained) ICC
profiles for rendering pipeline
15
Ma rch 19, 2015
Di gital Pathology Solutions
6) Define way to store multi-channel image data
Option A, pack all channels in existing DICOM image IOD elements& modules
For n channel data: specify samples per pixel (0028,2000) to n
(currently allowed, but meaning undefined)
Define Photometric Interpretation(0028,0004) to multi-spectral
(currently allowed, but meaning undefined)
Advantages: easy, no DICOM enhancements required
Disadvantages: legacy DICOM tools can not do anything with these images, no easy way to define
subsampling, different bit depths (only via Photometric Interp.)
Option B, channels in separate DICOM IOD’s and combine with presentation states
For n channel data: use n monochrome images, or RGB + monochrome images
(currently allowed)
Enhance or make new softcopy presentation state: allow n referenced images
Advantages: legacy DICOM tools can handle the separate images (channels)
Disadvantages: requires new/enhanced presentation states. Cannot define correct rendering in Image IOD
itself.
Option C, channels in raw data, not image data, define all from scratch
Advantages: full flexibility
Disadvantages: lose all existing image handling tools/functionality
Option D, channels in multi-frame image, additional fields to define the colors of the frames
Advantages: easy, no DICOM enhancements required
Disadvantages: legacy DICOM wont know how to display these images
16
Ma rch 19, 2015
Di gital Pathology Solutions
7) Add module describing characteristics of
channels (spectral, biomarker, etc.)
• Add module that describes for the image the channel characteristics
– Can use multi-spectral MR for inspiration
– Multiple modes:
 1) Spectral characterization per channel (Illumination spectrum, detection
spectrum, excitation spectrum)
 2) Biomarker concentration representation
• Module should describe derived channels?
– ICC profile pipeline has intermediate results, which can have meaning (biomarker
concentration)
17
Ma rch 19, 2015
Di gital Pathology Solutions
Additional iccMAX
information (examples &
resources)
18
Ma rch 19, 2015
Di gital Pathology Solutions
MID/Input
Profile
Example iccMAX MCS Input Profile
(slide from Max Derhak)
Material Type
Array Tag
Profile Header
Profile Class: ‘scnr’
Channel 0: “mRGB-R”
Flags : MCS Sub-Set Requirements=false
Other
Metadata
Tags
Channel 1: “mRGB-G”
Device: “nc0006”
Channel 2: “mRGB-B”
PCS: XYZ
Channel 3: “BM2”
MCS: “mc0007”
Channel 4: “BM3”
Profile SubClass: ‘BMSL’
Channel 5: “BM5”
Channel 5: “BM7”
AtoM0Tag (MPE Calc)
1





 M
c0
c1
c2
c3
c4
c5
 
N 
 
 
 

 
 
  NxM 
 
AtoB1Tag (MPE)
mRGB-R
mRGB-G
mRGB-B
BM2
BM3
BM5
BM7
For connection to MCS visualization profile
⌀
19
Ma rch 19, 2015
Di gital Pathology Solutions
c0
c1
c2
c3
c4
c5
1 0 0 0 0 0
0 1 0 0 0 0 


0 0 1 0 0 0
1





 M
 
N 
 
 
 

 
 
  NxM 
 
For direct connection to display profile
X
Y
Z
Example iccMAX MCS Visualization Profile
(slide from Max Derhak)
Profile Header
Material Type
Array Tag
Profile Class: ‘mvis’
Other
Metadata
Tags
Flags : MCS Sub-Set Requirements=false
MCS: “mc0005”
PCS: XYZ
Channel 0: “mRGB-R”
Channel 1: “mRGB-G”
Channel 2: “mRGB-B”
Profile SubClass: ‘BMSL’
Channel 3: “BM1”
Channel 4: “BM2”
Material Default
Values Tag
MtoB0Tag (MPE)
mRGB-R
mRGB-G
mRGB-B
BM1
BM2
Channel 0: 0
1 0 0 r1 r2 
0 1 0 g g 
1
2

0 0 1 b1 b2 
1





 M
 
 
 
 
 
N 
 
 

 
NxM 
Ma rch 19, 2015
Channel 1: 0
Channel 2: 0
Channel 3: 0
Channel 4: 0
Provides link between MCS Input profile and display profile
20
X
Y
Z
Di gital Pathology Solutions
MVIS
Profile
Resources for iccMAX implementation
• iccMAX preliminary specification (currently Stage 0 in ISO TC130)
• RefIccMax reference implementation (open source code, Windows
executables)
– iccXml tool: generates XML from ICC and ICC from XML, enables simple
creation of iccMAX profiles without writing complete profile writing
library
– iccMAX CMM: handles processing required to connect iccMAX profiles
– Test tools: include tools to select and apply profiles to images, and a
range of example profiles
– ProfileDump: tests compliance with iccMAX preliminary specification
• User forum to discuss issues with implementing iccMAX
21
Ma rch 19, 2015
Di gital Pathology Solutions