slides - JP Lewis

Lifting Detail from Darkness
J.P.Lewis
[email protected]
Disney The Secret Lab
Lewis / Detail from Darkness – p.1/38
Brightness-Detail Decomposition
detail
image
image
intensity
• Separate detail by Wiener filter
• Represent brightness by membrane PDE
• “Unsharp masking 2.0”
Topics: Wiener filter, Laplace PDE, multigrid
Lewis / Detail from Darkness – p.2/38
Brightness-Detail Decomposition
detail
image
image
intensity
• Modify detail, keep brightness. Example: texture
replacement (subject to limitations of 2D technique)
• Modify brightness, keep detail. Example: 102
Dalmatians spot removal
Lewis / Detail from Darkness – p.3/38
Motivation: 102-Dalmatians
Lewis / Detail from Darkness – p.4/38
Motivation: 102-Dalmatians
..."It turned out to be a huge job, much
bigger than any of us had imagined."
..."any spot that procedural could get
rid of would help"...
— Jody Duncan, “Out, Out, Damned Spot,” Cinefex 84 (January 2001), p. 56.
Lewis / Detail from Darkness – p.5/38
No easy trick
Lewis / Detail from Darkness – p.6/38
No easy trick
Hypothetical spot luminance profile, blurred luminance
(heavy), and unsharp mask (bottom).
Lewis / Detail from Darkness – p.7/38
No easy trick
Spot transition width is quite different on opposite sides of
the same spot.
Lewis / Detail from Darkness – p.8/38
Wiener
•
optimal linear estimator; for Gaussian data is
optimal period.
•
same principles can be applied as
• filter
• interpolator
• predictor
•
apply in frequency domain or spatially,
recursive setting → Kalman
Lewis / Detail from Darkness – p.9/38
Wiener 1D
y =x+n
x̂ = ay
x: the unknown, y: observation
x̂ is estimate. Find best a
E
expectation operator
E[ax + by]
E is linear
= aE[x] + bE[y]
E[xn] = 0
noise not correlated with signal
Lewis / Detail from Darkness – p.10/38
Wiener 1D
mina E[(x̂ − x)2 ]
Find a that minimizes expected err2
mina E[a2 y 2 − 2ayx + x2 ]
expand square, x̂
mina E[a2 (x + n)2 − 2a(x + n)x + x2 ]
expand y
Expand products; E[xn] = 0:
mina E[a2 (x2 + 2xn + n2 ) − 2a(x2 + xn) + x2 ]
∂
∂a
= 0 = 2aE[x2 + n2 ] − 2E[x2 ]
a=
minimize
E[x2 ]
E[x2 +n2 ]
Lewis / Detail from Darkness – p.11/38
Wiener principle
E[x2 ]
a=
E[x2 + n2 ]
Signal variance divided by signal variance + noise
variance.
Lewis / Detail from Darkness – p.12/38
How to use Wiener
Interpret “noise” as detail, “signal” as brightness.
Lewis / Detail from Darkness – p.13/38
Membrane
∇2 u = 0
•
2
2
∂
∂
(∇2 = 2 + 2 )
∂x
∂y
scattered interp for when there are lots of
data (c.f. radial basis, other: invert N 2 matrix for N
data points)
•
minimizes integrated gradient-squared
(roughness)
•
minimizes small-deflection approx. to surface
area
Lewis / Detail from Darkness – p.14/38
Membrane
Left - impulses in ‘L’ pattern. Black means no data,
intepolate.
Right - membrane interpolation.
Lewis / Detail from Darkness – p.15/38
Membrane: minimize roughness
roughness
for a particular k:
R=
Z
X
|∇u| du ≈
(uk+1 − uk )2
2
dR
d
=
[(uk − uk−1 )2 + (uk+1 − uk )2 ]
duk
duk
= 2(uk − uk−1 ) − 2(uk+1 − uk ) = 0
uk+1 − 2uk + uk−1 = 0
or
→ ∇2 u = 0
Lewis / Detail from Darkness – p.16/38
Laplacian mask
1D:
1 -2
1
2D:
1
1 -4
1
1
Lewis / Detail from Darkness – p.17/38
Membrane as matrix eqn
∇2 u = 0
rewrite ∇2 as matrix
rows looks like:
Mu = 0
. . . 1, . . . . . . , 1, −4, 1, . . . . . . , 1, . . .
. . . 1, . . . . . . , 1, −4, 1, . . . . . . , 1, . . .
. . . 1, . . . . . . , 1, −4, 1, . . . . . . , 1, . . .
M is huge - number of pixels in the region, squared! But M
is sparse, five diagonal bands.
Lewis / Detail from Darkness – p.18/38
Membrane: relaxation
M u = 0 is too large to solve by conventional
matrix inverse, solve by relaxation.
uk+1 − 2uk + uk−1 = 0
uk ← 0.5(uk+1 + uk−1 )
Lewis / Detail from Darkness – p.19/38
Membrane: boundary conditions
For interpolation some ur,c are known/specified
rather than free. In setting up the linear system,
subtract these from both sides of the eq, so the
known quantities move to the rhs.
1
(u+0 + u−0 + u0+ + u0− − 4u00 ) = 0
2
h
Say u+0 is known/fixed, then
1
1
(u−0 + u0+ + u0− − 4u00 ) = − 2 u+0
2
h
h
Lewis / Detail from Darkness – p.20/38
Membrane artifact
Lewis / Detail from Darkness – p.21/38
Membrane vs. Thin Plate
Left - membrane interpolation, right - thin plate.
Lewis / Detail from Darkness – p.22/38
Multigrid
approximate solution
r: residual, e: error
but Ax = b so
Ax = b
x̂ = x + e
r = Ax̂ − b
r = Ax + Ae − b
r = Ae
Residual has lower frequencies, so e can be
solved at low res and subtracted from x̂ to give x.
Lewis / Detail from Darkness – p.23/38
Multigrid results
•
before: several minutes, > 1/2 gig of memory
•
after: several seconds, memory not noticed
Lewis / Detail from Darkness – p.24/38
Algorithm steps
Lewis / Detail from Darkness – p.25/38
Recovered fur
Lewis / Detail from Darkness – p.26/38
Recovered fur: detail
Lewis / Detail from Darkness – p.27/38
Versus hand cloning (manual)
Lewis / Detail from Darkness – p.28/38
Versus hand cloning (auto)
Lewis / Detail from Darkness – p.29/38
Versus hand cloning (manual), edge
Image has been sharpened
Lewis / Detail from Darkness – p.30/38
Versus hand cloning (auto), edge
Image has been sharpened
Lewis / Detail from Darkness – p.31/38
Other applications
Lewis / Detail from Darkness – p.32/38
Conclusions+Demo
•
image alterations produced quickly with no
artistic skill (cloning requires some paint skill)
•
produces consistent effects across frames
(cloning may ‘chatter’ unless done skillfully)
Lewis / Detail from Darkness – p.33/38
Demo
Lewis / Detail from Darkness – p.34/38
Title
Lewis / Detail from Darkness – p.35/38
Title
•
Lewis / Detail from Darkness – p.36/38
Title
•
Lewis / Detail from Darkness – p.37/38
Title
•
Lewis / Detail from Darkness – p.38/38
Title
•
Lewis / Detail from Darkness – p.39/38