A new model for image restoration

Transcription

Mohammed El Rhabi
Realeyes3d SA, Saint-Cloud.
in collaboration with
T.Z Boulmezaoud (LMV, UVSQ) and Gilles Rochefort (Realeyes3d)
http://www.realeyes3d.com
Paris, december 2009
Journee 40 ans du LJLL, Paris, December 2009.
1
Outline
1
Realeyes3D company overview
General Presentation
Our strategy : Camera Phone can do
more than taking pictures
2
General Presentation
Founded 2001, 15 staff.
Offices in Paris (HQ), Hong Kong, San Francisco,
Tokyo.
Privately held:
3M A-round 2003
7.5M B-round 2005
Publishing after patenting, Publishing when cannot
patent
38 international patents
3
our goal
We enable “visual intelligence at hand” and
enhance the value of camera phone
message personalization
self content creation
improved user experience
improved productivity
increased data traffic
4
Outline
2
Products & Realeyes3D
Technology, Products & services
5
Core Techs
Ink Extraction
Mobile Scanning : thru ink extraction technology
24 patents (worldwide)
c
Service solution : Qipit
Embedded Products :
c
Qipit White
c
Visual Cortex
c
Digitizer
c
w-Postcard
c
Clipper,
c
MagicWanda
6
Core Techs
Motion navigation
thru camera apparent motion estimation
9 patents
Products/Trademarks :
c
Motion Cortex,
c
MotionLens
4 mini-games in the Nintendo DSi, browser integration ...
http://www.higp.jp/katamuku/index.html
7
Core Techs
Image restoration
thru our unique deblurring technologies
3 patents
application to barcode restoration and decoding
Products/Trademarks : Xwalk
application to document restoration (documents, V card ...)
8
Outline
3
Image restoration
9
A starting problem : barcode Image Restoration
1D barcode is widely used, especially for manufactured
goods (billions of products) representation of data.
10
1D barcodes are used for check-our and inventory control
(UPC/EAN).
10
(UPC/EAN).
2D barcodes represent data in patterns of squares, dots,
etc. within images.
10
(UPC/EAN).
2D barcodes represent data in patterns of squares, dots,
etc. within images.
Recent mobile phones have built-in camera, making
virtually every cameraphone a barcode scanner
But shooting too far or too close ?
too far result in a lack of resolution
too close result in a blurred image
10
Blurred barcode
11
Blurred barcode
Blurred visit card
11
Problem statement
Problem statement : is it possible to help
cameraphones to read 1D barcodes? More precisely:
can we decode a blurred barcode image shot by a
cameraphone ?
12
Problem statement
cameraphone ?
Yes we can !
12
Problem statement
cameraphone ?
Yes we can !
by means of image restoration ...
12
Problem statement
cameraphone ?
Yes we can !
12
Problem statement
cameraphone ?
Yes we can !
12
Problem statement
cameraphone ?
Yes we can !
... and a standard barcode image reader solution
12
Problem statement
cameraphone ?
Yes we can !
... and a standard barcode image reader solution
We need some a priori knowledge about the
degradation.
This is an inverse problem
12
Blurring Model
We need a Mathematical description of how to blur an image:
The mathematical model
Ω ⊂ R2 : the domain of the image
u : Ω −→ R: the original image.
u0 : Ω −→ R: the observed image (a degradation
of u).
u0 = Ku + n,
n : is the noise (a random variable),
K is the blur operator (linear). In practice K = k ? .:
k = δ0 (dirac): no blur (denoising problem)
2
k = k0 e−θ|x| : a Gaussian blur.
k = k0 1 {|x|<R} : out-of-focus.
14
Energy methods
An approximation of u is given by solving the
minimization problem
Z
Z
min |Ku − u0 |2 dx + λ ϕ(|∇u|)dx.
u
Ω
Ω
Formally, u satisfies the equation
0
ϕ (|∇u|)
∇u = K ? u0 .
K ? Ku − λdiv
|∇u|
which writes
K ? Ku − λ
with N =
ϕ0 (|∇u|)
uTT + ϕ00 (|∇u|)uNN
|∇u|
= K ? u0 .
∇u
and T .N = 0.
|∇u|
15
Thus, the function ϕ could be chosen such that
Encourage smoothing (isotropic diffusion) in
regions of weak variations of u (∇u ≈ 0).
Direct the diffusion along the edges and not across
them (to preserve edges).
tϕ00 (t)
= 0.
t−→+∞ ϕ0 (t)
lim
√
Example 1: ϕ(t) = 1 + t 2
Example 2: ϕ(t) = t (Total variation model).
(the usual function ϕ(t) = t 2 does not work: edges are
not preserved).
16
There are two situations:
1
Restoring image u, with a known blurring K (as the
denoising problem)
2
Restoring both u and k , with some noise
Let us start with the first problem: restoration with a
known blurring K :
Z
Z
min E(u) =
|Ku − u0 |2 dx + λ ϕ(|∇u|)dx.
u
Ω
Ω
17
The continuous problem
Usualy, we look for a solution u living in the space
Z
BV (Ω) = {v ∈ L1 (Ω) |
|Du| < ∞},
Ω
where
Z
Z
|Du| = sup
u.divφdx; φ ∈ C01 (Ω)2 , |φ|∞ ≤ 1 ,
Ω
Ω
Theorem (ϕ(t) = t)
Suppose that K : L2 (Ω) 7→ L2 (Ω) is continuous and
K 1 6= 0. Then, the problem
Z
Z
min E(u) =
|Ku − u0 |2 dx + λ |Du|.
u∈BV (Ω)
Ω
Ω
admits one and only one solution.
This theorem can be extended with minor modification
to a larger class of functions ϕ
18
The real image model
Image: a collection of indexed pixels (i, j), i ≤ N, j ≤ M
and an intensity u ∈ X with X = RN × RM .
Define
The inner product:
N X
N
X
hu, v i =
ui,j vi,j , kuk = hu, ui1/2 .
i=1 j=1
The gradient ∇u = (ui+1,j − ui,j , ui,j+1 − ui,j )i≤N,j≤M .
The divergence: for p ∈ X 2 , divp ∈ X is such that
∀u ∈ X , hdivp, ui = −hp, ∇ui.
A few calculus gives
(1)
(1)
(2)
(1)
divp ≈ (pi,j − pi−1,j + pi,j − pi,j−1 ).
19
The discrete energy
The discrete energy writes
E(u) =
1
kKu − u0 k2 + λJ(u)
2
where
J(u) =
X
ϕ(|(∇u)i,j |).
i,j
In the sequel, we suppose that ϕ is strictly convexe.
20
A descent algorithm (without line-search subalgorithm)
√
Let φ(t) = ϕ( t). For each u ∈ X , set
w(u) = K ? (Ku − u0 ) − 2λdiv(φ0 (|∇u|2 )∇u).
Initial data: n = 0, u (0) = u0 .
Step 1: if w(u (n) ) = 0, then stop.
Step 2: choose d (n) such that |d (n) | = 1 and
hw(u (n) ), d (n) i ≥ τ |w(u (n) )| (τ > 0 is a constant).
Step 4: u (n+1) = u (n) − α(n) d (n) where
α(n) =
kKd (n) k2
hw(u (n) ), d (n) i
P 0
+ 2λ i,j φ (|(∇u (n) )i,j |2 )|(∇d (n) )i,j |2
n ← n + 1. Go to Step 1.
Proposition
If φ is concave, then the sequence (u (n) ) converges to
the minimizer of E(u) on X 2 .
21
Deblurring with an unknown blur
In practice, the blur function k is often unknown. The
problem we consider becomes
Z
Z
2
min E(k , u) =
|k ? u − u0 | dx + λ ϕ(|∇u|)dx.
k ,u
Ω
Ω
To be tractable, k is decribed by few parameters. For an
out of focus blur, k is of the form
k (r ) =
1
1 B (x),
πr 2 r
Here, r denotes the radius of the out of focus. The
problem writes
Z
Z
min E(r , u) =
|k ? u − u0 |2 dx + λ ϕ(|∇u|)dx.
r >0,u
Ω
Ω
22
Deblurring with an unknown blur
This problem is a non-convex problem. It can be solved
hierarchically as following
Initial data: u (0) = u0 , r (0) = r0 , n = 0.
Step 1: find u (n+1) minimizing E(r (n) , u) (see
previous algorithm)
Step 2: find r (n+1) minimizing E(r , u) (a 1D
problem)
Step 4: if (r (n+1) , u (n+1) ) is solution then stop. Else
n ← n + 1 and go Step 1.
On a significant bench
Nokia phone : N70 ; original
Nokia phone : N70 ; deblurred
Decoded
0%
85 %
False positive
0%
4%
Not decoded
100 %
11 %
23
A new model
If k is an unknown out-of-focus blur. Then we have:
k is radial k (x) = k (|x|),
Z
k (x)dx = 1,
Ω
k has a compact support suppk ⊂ [0, R].
Then, for each sufficiently smooth function u
k ?u ≈
N
X
R 2k Ak u,
k =0
for N sufficiently large. Here Ak denotes a linear
operator for each k ∈ {0, ..., N}.
Finally, we get the new model
Z
min E(R, u) =
R,u
|
N
X
Ω k =0
2k
2
Z
R Ak u −u0 | dx +λ
ϕ(|∇u|)dx.
Ω
24
A comparison: this new model and the classical blur
model
Example : original barcode
original barcode
300
250
200
150
100
50
0
0
50
100
150
200
250
300
350
400
450
500
25
A comparison: this new model and the classical blur
model
Example : blurring operation : convolutive model vs our
model ; r = 8, N = 495
Blurring operation : convolutive model vs our model, blur radius = 8, N = 495
300
convolutive model
our model1
250
200
150
100
50
0
0
50
100
150
200
250
300
350
400
450
500
26
A comparison: this model and the classical blur model
; zoom
Example : blurring operation : convolutive model vs our
model ; r = 8, N = 495
27
Example of barcode restoration in high level of blur and
noise:
Ideal
Blurred
+
Noisy
Restored
28
Example of V card restoration
Blurred Vcard shot by a cameraphone (Nokia N70)
29
Example
If we restore by part the previous image, we get:
Blurred image , OCR can’t see ant letter or number
30
Example
Blurred image , OCR can’t see ant letter or number
30
Example
Deblurred image, properly analysze by OCR
(ABBYY and CardIris)
30
Outline
4
Conclusion and perspectives
31
Conclusion and perspective
Image restoration : blind deblurring
By using the camera preview and the image streaming, we
can improve our actual embedded blind deblurring.
For barcode restoration, using a discrete optimization, work
in progress with Laurent Dumas (LJLL lab). Toward a joint
deblurring and decoding ?
Port the new model in its embedded version and try
(numerically) the anistropic case.
32
33

A new model for image restoration

Transcription

Similar documents

Farrel McWhirter Park

THE PIGHTLE, TOWN YARD

Porsche 1955 Speedster - Willhoit Auto Restoration

Thomas Quasthoff. - Philharmonie de Paris

VIEW FULL SIZE POSTER (pdf 6 MB)

Ecological Restoration of an altered area at the Majuy Mountain in

Nov 6 at 12:30-1:45pm Amasa Stone Chapel drinks, snacks, and

french leader in the information about higher education

Honda CB450 Restoration

Pargeting is sometimes characterized as a dying art, but