heinrich reisenhofer

T HE C OMPLEX S HEARLET T RANSFORM AND A PPLICATIONS TO I MAGE Q UALITY
A SSESSMENT AND E DGE D ETECTION
Rafael Reisenhofer1, Emily King1, Gitta Kutyniok2, and Sebastian Bosse3
(1) AG Computational Data Analysis, Universität Bremen, (2) FG Angewandte Funktionalanalysis, TU Berlin (3) Fraunhofer Heinrich Hertz Institut
M AIN I DEA
Solve image processing tasks via a mathematically optimal transform which mimicks responses of V1 simple and complex cells.
N EUROPHYSIOLOGICAL P ERSPECTIVE
M ATHEMATICAL P ERSPECTIVE
A PPLICATION
Complex-valued Gabor filters can be used to describe the
receptive fields of adjacent V1 simple cells and to model the
responses of V1 complex cells.
From a certain mathematical point of view, transforms based on
anisotropic scaling (e.g. shearlet transforms) can provide
optimal descriptions of natural images.
Use coefficients of a complex-valued shearlet transform to solve
image processing tasks, namely:
◮ Image Quality Assessment
◮ Edge and Line Detection
S HEARLET T RANSFORMS ( CONT.)
G ABOR F ILTERS AS M ODELS OF V1 N EURONS
◮
In 1985, Daugman fitted Gabor filters of the form
2 2
2 2
−
π
(
x
−
x
)
α
+(
y
−
y
)
β e−2πi (u0 (x−x0 )+v0 (y−y0 )
0
0
g (x) = e
O PTIMALLY S PARSE A PPROXIMATIONS
(1)
to match experimentally obtained receptive fields of V1 simple
cells, where the parameters (x0, y0 ) determine the location, the
parameters (α, β ) the stretch in x- and y-direction and the
parameters (u0, v0 ) the spatial frequency and the orientation of
the filter g [1].
◮ The real (cosine) part of g models a symmetric simple cell
receptive field, while the imaginary (sine) part of g models an
odd-symmetric receptive field.
◮ Receptive fields of adjacent simple cells in V1 were found to be in
quadrature [3]. That is, they differ only by a 90◦ phase shift.
◮ Responses of locally shift invariant V1 complex cells can be
modeled by considering the sum of the squared responses of two
simple cells whose receptive fields are in quadrature [2].
2
◮ That is, for an image f ∈ L (R ) and a complex-valued Gabor
filter g (see (1)), |hf , g i| models the response of the complex cell
associated with the symmetric receptive field Re(g ) and the
odd-symmetric receptive field Im(g ), when viewing f .
S HEARLET T RANSFORMS
S HEARLETS [4, 5]
◮
Shearlets are constructed by anisotropically (i.e. in a directionally
dependent fashion) scaling, shearing and translating a shearlet
generator ψ ∈ L2 (R2 ), i.e.
ψa,s,t = ψ(SsAa (· − t )),
(2)
1 s
a 0
,
, a shear matrix Ss =
given a scaling matrix Aa =
0 1
0 aα
and parameters a ∈ R + , s ∈ R, t ∈ R2.
◮ The parameter α ∈ [0, 1] determines the degree of anisotropy,
where α = 1 yields a perfectly isotropic and α = 0 an extremely
anisotropic scaling matrix.
◮ There exist multiple ways to construct shearlet generators. A
simple separable shearlet generator can be defined by
ψ(x1, x2 ) = ψ1D (x1 )φ1D(x2 ),
Given a shearlet generator ψ ∈ L2 (R2 ) and a set of parameters
Γ ⊂ R + × R × R2, a shearlet-based transform decomposing
signals f ∈ L2 (R2 ) can be defined via the L2-inner product, i.e.
(4)
For specific choices of ψ and Γ, any signal f ∈ L2 (R2 ) can be
reconstructed from its shearlet-based decomposition using the
simple formula
f =
∑
hf , ψa,s,t iS−1 ψa,s,t ,
(5)
(a,s,t)∈Γ
where S−1 is an easy to compute bounded, self-adjoint and
positive operator [4, 9]. In some cases, S−1 reduces to the identity.
F IGURE : The lower row depicts the coefficients associated with all translates of
three differently scaled and sheared shearlets (computed with [6]).
Interdisciplinary College 2015
Günne at Lake Möhne, 06 Mar to 13 Mar 2015
FSIM
0.8805
0.6946
0.9634
0.8337
0.8566
0.9472
0.9279
0.9773
0.8708
MSSIM
0.8528
0.6543
0.9445
0.7922
0.8094
0.9607
0.9348
0.9736
0.8736
SSIM
0.7749
0.5768
0.9479
0.7963
0.8107
0.9544
0.9252
0.9625
0.8678
VIF
0.7496
0.5863
0.9631
0.8270
0.8799
0.9546
0.9170
0.9713
0.8582
IFC
0.5692
0.4261
0.9234
0.7540
0.5817
0.8766
0.8181
0.9445
0.7966
VSNR
0.7046
0.5340
0.9274
0.7616
0.7728
0.9330
0.9174
0.9515
0.8056
PSNR
0.5245
0.3696
0.8755
0.6864
0.9114
0.8682
0.9011
0.8300
0.7665
E DGE D ETECTION
Similar to the Gabor filter model, we consider complex-valued
shearlets
ψc = ψ + i H ψ,
(6)
where ψ is a (typically symmetric) real-valued shearlet generator
and H denotes the Hilbert transform.
◮ The Hilbert transform is a pseudo-differential operator that
interchanges the sine and cosine components of a function.
◮ Similar to V1 complex cells, the magnitude response of a complex
shearlet-based transform exhibits a certain extent of shift
invariance.
◮ For a symmetric real-valued shearlet ψ, the associated
complex-valued shearlet ψc = ψ + i H ψ bears many similarities
to the complex exponential eiϕ· = cos(ϕ·) + i sin(ϕ·) used in the
Fourier transform. Indeed, the magnitude response of the Fourier
transform is perfectly shift invariant.
◮
F IGURE : Differently scaled symmetric and odd-symmetric wavelets located at a
jump discontinuity. All wavelets have the same L1-norm, namely 0.25.
At the location of a jump discontinuity, the coefficients associated
with a symmetric wavelet remain approximately 0 on almost
every scale. However, assuming L1-normalization, the
coefficients associated with an odd-symmetric wavelet are
non-zero and approximately constant over different scales.
◮ We exploit this behavior to test for the presence of an edge with a
simple measure based on the real (symmetric) and imaginary
(odd-symmetric) parts of a complex-valued shearlet transform.
For an image f ∈ L2 (R2 ), a location x ∈ R2 and a shear
parameter s ∈ R, an edge measure is given by
c
c
∑ Im(hf , ψa,s,xi) − ∑ Re(hf , ψa,s,xi)
a∈A
a∈A
, (8)
Eψ (f , x, s) =
c
|A| max Im(hf , ψa,s,xi) + ǫ
◮
a∈A
F IGURE : The upper row shows a symmetric shearlet filter, its odd-symmetric
Hilbert transform and the absolute value of their associated complex shearlet.
Below, the respective coefficients - obtained from a complex shearlet transform
of the image shown on the left - are plotted.
I MAGE Q UALITY A SSESSMENT
where A ⊂ R + is a set of scaling parameters, ψ is a real-valued
symmetric shearlet and ǫ prevents division by zero [10].
◮ The just defined edge measure is by construction contrast
invariant and implicitly requires the complex valued coefficients
c
hf , ψa,s,x
i to possess a specific phase on all considered scales.
From this point of view, formula (8) is related to the phase
congruency measure introduced by Kovesi [12].
◮ The measure (8) provides estimates of the tangential directions of
an edge.
◮ It is possible to detect lines instead of edges by simply switching
the roles of the real and the imaginary part of the complex
shearlet transform in (8).
Full-reference image quality assessment is the task of numerically
determining the loss of image quality as subjectively perceived by
a human observer when given a pair of images, where one is a
somehow distorted version of the other.
2
2
◮ Given two images f1 , f2 ∈ L (R ), a complex shearlet-based
similarity measure at a location x ∈ R2 and a scale a ∈ R + is
given by
c
c
2 hf1, ψa,s,x i hf2, ψa,s,x i + C
Simψ (f1, f2, a, x) = ∏ , (7)
2
2
c
c
s∈Sa hf1, ψa,s,x i + hf2, ψa,s,x i + C
◮
S HEARLET T RANSFORMS
◮
Shear
SROCC
0.8566
TID 2008 [15]
KROCC
0.6633
SROCC
0.9324
LIVE [16]
KROCC
0.7683
gaussian noise 0.9176
blur
0.9655
jpg-comp
TID 2008
0.9608
jpg2000-comp 0.9784
jpg-trans-error 0.8827
F IGURE : While the mean squared error is approximately the same for both
distorted images, the perturbance on the right has a much more severe impact on
the subjectively perceived image quality.
F IGURE : The effects of scaling and shearing a shearlet generator in the time
domain (upper row) and frequency domain (magnitude response, lower row).
SHψ,Γ (f ) = (hf , ψa,s,t i)(a,s,t)∈Γ .
TABLE : Spearman and Kendall rank order correlation coefficients of image
quality metrics for two test databases.
C OMPLEX S HEARLET T RANSFORMS [10, 11]
(3)
where ψ1D is a wavelet and and φ1D a bump-like function (e.g.
the Gaussian).
◮ Shears are used instead of rotations for changing the orientation.
Note that for k ∈ Z, the integer grid is invariant under the shear
matrix Sk. In particular, this permits faithful digital
implementations of shearlet-based transforms (e.g. [6]).
◮
For members of a certain class of natural images, so-called
cartoon-like images (i.e. images consisting of smooth areas
separated by piecewise smooth boundary curves), shearlet-based
transforms were shown to provide optimally sparse
approximations [9, 8].
◮ That is, for every cartoon-like image, the error of the
reconstruction (see (5)) associated with the N largest coefficients
of its shearlet transform decays with O(N −1) (up to a negligible
log-factor).
◮ From a mathematical perspective, this decay rate is optimal [7].
◮
I MAGE Q UALITY A SSESSMENT ( CONT.)
where ψ is a real-valued shearlet, Sa ⊂ R a finite set of shear
parameters and C > 0 [10].
◮ The above similarity measure can be extended over an arbitrary
number of locations and scales by simply computing geometric
means.
◮ The measure defined in (7) solely considers the magnitude
response of a complex shearlet transform and hence only complex
cell responses.
◮ To test the validity of a computational image quality metric, its
measurements are compared to so-called mean opinion scores
experimentally collected for large databases of differently
distorted images by combining thousands of image quality
assessments provided by human test subjects [15, 16].
◮ State of the art methods include the Structural Similarity Index
(SSIM) [13] and the Feature Similarity Index (FSIM) [14].
F IGURE : Left: The original images. Center: Grayscale plot of the measure defined
in (8) with values ranging from 0 (black) to 1 (white). Right: Color-coded
approximations of the tangential directions.
R EFERENCES
[1] Daugman: Uncertainty relation for resolution in space, spatial frequency, and orientation optimized by two-dimensional visual cortical filters, 1985
[2] Adelson & Bergen: Spatiotemporal energy model for the perception of motion, 1985
[3] Pollen & Ronner: Phase relationships between adjacent simple cells in the visual cortex, 1981
[4] Guo, Kutyniok & Lim: Sparse Multidimensional Representations using Anisotropic Dilation and Shear Operators, 2005
[5] Grohs, Keiper, Kutyniok & Schäfer: α-Molecules, 2014
[6] Kutyniok, Lim & Reisenhofer: ShearLab 3D: Faithful Digital Shearlet Transforms Based on Compactly Supported Shearlets, 2015
[7] Donoho: Sparse components of images and optimal atomic decompositions, 2001
[8] Candès & Donoho: New Tight Frames of Curvelets and Optimal Representation of Objects with C2 Singularities, 2004
[9] Kutyniok & Lim: Compactly Supported Shearlets are Optimally Sparse, 2011
[10] Reisenhofer: The Complex Shearlet Transform and Applications to Image Quality Assessment, MSc Thesis, 2014
[11] Storath: Amplitude and sign decompositions by complex wavelets - Theory and applications to image analysis, 2013
[12] Kovesi: Phase congruency: A low-level image invariant, 2000
[13] Wang, Bovik, Sheikh & Simoncelli: Image Quality Assessment: From Error Visibility to Structural Similarity, 2004
[14] Zhang, Zhang, Mou & Zhang: FSIM: A Feature Similarity Index for Image Quality Assessment, 2011
[15] Ponomarenko, Lukin, Zelensky, Egiazarian, Carli & Battisti: TID2008 - A Database for Evaluation of Full-Reference Visual Quality Assessment
Metrics, 2009
[16] Sheikh, Wang, Cormack, & Bovik: LIVE Image Quality Assessment Database Release 2