Auguri Maurizio !

Different(ial) approach to
the Photometric Stereo
problem
Roberto Mecca
Istituto Italiano di Tecnologia
- Department of Pattern Analysis & computer VISion -
Auguri Maurizio !
Photometric Stereo as inverse problem
Photometric Stereo as inverse problem
Photometric Stereo as inverse problem
Photometric Stereo as inverse problem
Photometric Stereo as inverse problem
Photometric Stereo as inverse problem
Photometric Stereo as inverse problem
Photometric Stereo as inverse problem
Photometric Stereo as inverse problem
Photometric Stereo as inverse problem
Photometric Stereo as inverse problem
From traditional to realistic Photometric Stereo
traditional
realistic
traditional
schematic working setup
realistic
traditional
flectance)
schematic working setup
ystem)
realistic
s
n(x, y)
!j
(x, y, z)
θi
v
n
traditional
flectance)
schematic working setup
ystem)
realistic
s
n(x, y)
!j
(x, y, z)
viewing geometry
θi
n(x, y) = ( rz, 1)
v
n
traditional
flectance)
schematic working setup
ystem)
realistic
s
n(x, y)
!j
(x, y, z)
θi
viewing geometry
n(x, y) = ( rz, 1)
light propagation
! j 2 R3
v
n
traditional
flectance)
schematic working setup
ystem)
realistic
s
n(x, y)
!j
(x, y, z)
θi
viewing geometry
n(x, y) = ( rz, 1)
light propagation
! j 2 R3
irradiance equation
v
Ij (x, y) = ⇢(x, y)!j · n(x, y)
n
traditional
flectance)
schematic working setup
ystem)
realistic
s
n(x, y)
!j
(x, y, z)
θi
viewing geometry
n(x, y) = ( rz, 1)
light propagation
! j 2 R3
irradiance equation
v
n
Ij (x, y) = ⇢(x, y)!j · n(x, y)
Ii
differential formulation obtained by image ratio of pairs of images
Ij
⇢
b(x, y) · rz(x, y) = s(x, y) a.e.(x, y) 2 ⌦,
z(x, y) = g(x, y)
8(x, y) 2 @⌦,
where
(b(x, y), s(x, y)) = Ii !j
I j !i
traditional
flectance)
schematic working setup
ystem)
realistic
s
n(x, y)
!j
(x, y, z)
θi
viewing geometry
n(x, y) = ( rz, 1)
light propagation
! j 2 R3
irradiance equation
v
(⇠j , ⌘j , ⇣j )
n
lj (x, y, z)
n(x, y)
(⇠, ⌘, ⇣)
Ij (x, y) = ⇢(x, y)!j · n(x, y)
Ii
differential formulation obtained by image ratio of pairs of images
Ij
⇢
b(x, y) · rz(x, y) = s(x, y) a.e.(x, y) 2 ⌦,
z(x, y) = g(x, y)
8(x, y) 2 @⌦,
where
(b(x, y), s(x, y)) = Ii !j
I j !i
traditional
flectance)
schematic working setup
ystem)
realistic
s
n(x, y)
!j
(x, y, z)
θi
viewing geometry
n(x, y) = ( rz, 1)
light propagation
! j 2 R3
irradiance equation
v
(⇠j , ⌘j , ⇣j )
n
lj (x, y, z)
n(x, y)
(⇠, ⌘, ⇣)
Ij (x, y) = ⇢(x, y)!j · n(x, y)
Ii
differential formulation obtained by image ratio of pairs of images
Ij
⇢
b(x, y) · rz(x, y) = s(x, y) a.e.(x, y) 2 ⌦,
z(x, y) = g(x, y)
8(x, y) 2 @⌦,
where
(b(x, y), s(x, y)) = Ii !j
I j !i
traditional
flectance)
schematic working setup
ystem)
realistic
s
n(x, y)
!j
(x, y, z)
θi
viewing geometry
n(x, y) = ( rz, 1)
light propagation
! j 2 R3
irradiance equation
v
(⇠j , ⌘j , ⇣j )
n
lj (x, y, z)
n(x, y)
(⇠, ⌘, ⇣)
z
n(x, y) = 2 f rz, z + (x, y) · rz
f
Ij (x, y) = ⇢(x, y)!j · n(x, y)
Ii
differential formulation obtained by image ratio of pairs of images
Ij
⇢
b(x, y) · rz(x, y) = s(x, y) a.e.(x, y) 2 ⌦,
z(x, y) = g(x, y)
8(x, y) 2 @⌦,
where
(b(x, y), s(x, y)) = Ii !j
I j !i
traditional
flectance)
schematic working setup
ystem)
realistic
s
n(x, y)
!j
(x, y, z)
θi
viewing geometry
n(x, y) = ( rz, 1)
light propagation
! j 2 R3
irradiance equation
v
(⇠j , ⌘j , ⇣j )
n
lj (x, y, z)
n(x, y)
(⇠, ⌘, ⇣)
z
n(x, y) = 2 f rz, z + (x, y) · rz
f
✓
◆
z
z
lj (x, y, z) = ⇠j + x , ⌘j + y , ⇣j z
f
f
Ij (x, y) = ⇢(x, y)!j · n(x, y)
Ii
differential formulation obtained by image ratio of pairs of images
Ij
⇢
b(x, y) · rz(x, y) = s(x, y) a.e.(x, y) 2 ⌦,
z(x, y) = g(x, y)
8(x, y) 2 @⌦,
where
(b(x, y), s(x, y)) = Ii !j
I j !i
traditional
flectance)
schematic working setup
ystem)
realistic
s
n(x, y)
!j
(x, y, z)
θi
viewing geometry
n(x, y) = ( rz, 1)
light propagation
! j 2 R3
irradiance equation
v
Ij (x, y) = ⇢(x, y)!j · n(x, y)
(⇠j , ⌘j , ⇣j )
n
lj (x, y, z)
n(x, y)
(⇠, ⌘, ⇣)
z
n(x, y) = 2 f rz, z + (x, y) · rz
f
✓
◆
z
z
lj (x, y, z) = ⇠j + x , ⌘j + y , ⇣j z
f
f
Ij (x, y) = ⇢(x, y)lj (x, y) · n(x, y)
Ii
differential formulation obtained by image ratio of pairs of images
Ij
⇢
b(x, y) · rz(x, y) = s(x, y) a.e.(x, y) 2 ⌦,
z(x, y) = g(x, y)
8(x, y) 2 @⌦,
where
(b(x, y), s(x, y)) = Ii !j
I j !i
traditional
flectance)
schematic working setup
ystem)
realistic
s
n(x, y)
!j
(x, y, z)
θi
viewing geometry
n(x, y) = ( rz, 1)
light propagation
! j 2 R3
irradiance equation
v
Ij (x, y) = ⇢(x, y)!j · n(x, y)
(⇠j , ⌘j , ⇣j )
n
lj (x, y, z)
n(x, y)
(⇠, ⌘, ⇣)
z
n(x, y) = 2 f rz, z + (x, y) · rz
f
✓
◆
z
z
lj (x, y, z) = ⇠j + x , ⌘j + y , ⇣j z
f
f
Ij (x, y) = ⇢(x, y)lj (x, y) · n(x, y)
Ii
differential formulation obtained by image ratio of pairs of images
Ij
⇢
⇢
b(x, y) · rz(x, y) = s(x, y) a.e.(x, y) 2 ⌦,
r(x, y, z) · rz(x, y) = k(x, y, z), a.e.(x, y)
z(x, y) = g(x, y)
8(x, y) 2 @⌦,
z(x, y) = h(x, y)
(x, y) 2 @⌦
⇣
where
where
r(x, y, z) = Ii |li |(f ⇠j x⇣j ) Ij |lj |(f ⇠i x⇣i ),
(b(x, y), s(x, y)) = Ii !j Ij !i
⌘
Ii |li |(f ⌘j y⇣j ) Ij |lj |(f ⌘i y⇣i )
Synthetic case
Synthetic case
Synthetic case
Real case
Synthetic case
Real case
Modern approach to Shape-from-Shading
The dataset
Constancy, Intrinsic
Images, and
Shape Estimation
Color Constancy,
Intrinsic
Images, and Shape9 Estimation
TheColor
dataset
ab” Data
(b) “Lab”
“Lab” Data
Samples
(a)
(c)“Lab”
“Natural”
Data
(b)
Samples
(d) “Natural”
“Natural” Data
Samples(d)
(c)
9
“Natural” Samples
e use two Fig.
datasets:
the
“laboratory”-style
of the illuminations
MIT intrinsicof the MIT intrinsic
5. We
use
two datasets: theilluminations
“laboratory”-style
taset [2, 1]images
which dataset
are harsh,
mostly-white,
and well-approximated
point
[2, 1]
which are harsh,
mostly-white, and by
well-approximated
by point
nd a newsources,
dataset and
of “natural”
illuminations,
whichilluminations,
are softer and
much
a new dataset
of “natural”
which
are softer and much
rful. We model
a multivariate
on spherical
more illumination
colorful. We using
modeljust
illumination
using Gaussian
just a multivariate
Gaussian on spherical
illumination.
Shownillumination.
here are some
example
from our
datasets from our datasets
harmonic
Shown
hereilluminations
are some example
illuminations
les from our
allfrom
rendered
on Lambertian
spheres.
The samples
looks The samples looks
andmodels,
samples
our models,
all rendered
on Lambertian
spheres.
ly similar superficially
to the data, similar
suggesting
that
our suggesting
model is reasonable.
to the
data,
that our model is reasonable.
ion, we parametrize
log-shading
rather log-shading
than shading.
Thisthan
choice
makes This choice makes
illumination,
we parametrize
rather
shading.
tion easieroptimization
as we don’t easier
have toasdeal
with “clamping”
at 0, illumination at 0,
we don’t
have to dealillumination
with “clamping”
ows for easier
as the
space of log-shading
SH of
illuminaand itregularization
allows for easier
regularization
as the space
log-shading SH illuminaurprisingly
well-modeled
by awell-modeled
simple multivariate
Gaussian.
TrainingGaussian. Training
tions
is surprisingly
by a simple
multivariate
el is extremely
simple:
fit a multivariate
to the SH
illumi- to the SH illumiour model
is we
extremely
simple: we Gaussian
fit a multivariate
Gaussian
n our training
set.inDuring
inference,
cost inference,
we imposethe
is the
nations
our training
set. the
During
costnegative
we impose is the negative
hood under
that model: under that model:
log-likelihood
Constancy, Intrinsic
Images, and
Shape Estimation
Color Constancy,
Intrinsic
Images, and Shape9 Estimation
TheColor
dataset
ab” Data
(b) “Lab”
“Lab” Data
Samples
(a)
(c)“Lab”
“Natural”
Data
(b)
Samples
(d) “Natural”
“Natural” Data
Samples(d)
(c)
9
“Natural” Samples
e use two Fig.
datasets:
the
“laboratory”-style
of the illuminations
MIT intrinsicof the MIT intrinsic
5. We
use
two datasets: theilluminations
“laboratory”-style
taset [2, 1]images
which dataset
are harsh,
mostly-white,
and well-approximated
point
[2, 1]
which are harsh,
mostly-white, and by
well-approximated
by point
nd a newsources,
dataset and
of “natural”
illuminations,
whichilluminations,
are softer and
much
a new dataset
of “natural”
which
are softer and much
rful. We model
a multivariate
on spherical
more illumination
colorful. We using
modeljust
illumination
using Gaussian
just a multivariate
Gaussian on spherical
illumination.
Shownillumination.
here are some
example
from our
datasets from our datasets
harmonic
Shown
hereilluminations
are some example
illuminations
les from our
allfrom
rendered
on Lambertian
spheres.
The samples
looks The samples looks
andmodels,
samples
our models,
all rendered
on Lambertian
spheres.
ly similar superficially
to the data, similar
suggesting
that
our suggesting
model is reasonable.
to the
data,
that our model is reasonable.
ion, we parametrize
log-shading
rather log-shading
than shading.
Thisthan
choice
makes This choice makes
illumination,
we parametrize
rather
shading.
tion easieroptimization
as we don’t easier
have toasdeal
with “clamping”
at 0, illumination at 0,
we don’t
have to dealillumination
with “clamping”
ows for easier
as the
space of log-shading
SH of
illuminaand itregularization
allows for easier
regularization
as the space
log-shading SH illuminaurprisingly
well-modeled
by awell-modeled
simple multivariate
Gaussian.
TrainingGaussian. Training
tions
is surprisingly
by a simple
multivariate
el is extremely
simple:
fit a multivariate
to the SH
illumi- to the SH illumiour model
is we
extremely
simple: we Gaussian
fit a multivariate
Gaussian
n our training
set.inDuring
inference,
cost inference,
we imposethe
is the
nations
our training
set. the
During
costnegative
we impose is the negative
hood under
that model: under that model:
log-likelihood
Let’s test it !
Let’s test it !
Let’s test it !
Let’s test it !
Let’s test it !
Let’s test it !
Let’s test it !
Let’s test it !
Let’s test it !
Let’s test it !
Thank you !