Scalar Quantization

Qua t at o
Quantization
• Quantization: main component in source coders
¾ Problem: Original image is often very large
⇒ Reduce the data
9 Remove redundancy
y ⇒ no loss ⇒ info content p
preserved
–
–
–
–
Prediction (DPCM, ADPCM)
Entropy coding
Transform coding (e.g., transforms used to decorrelate data)
subband coding
9 Remove unnecessary, “least important” info ⇒ loss
– scalar quantization
– vector
t quantization
ti ti
EEE 508 - Lecture 7
Scalar
Sca
a qua
quantization
t at o
• Example: Uniform and non-uniform scalar quantization
x
Uniform
quantization
0
t
¾ Better performance (in terms of quantization distribution) can be
achieved by designing quantizer that match the pdf of the signal x
EEE 508 - Lecture 7
Scalar
Sca
a qua
quantization
t at o
• Example: Uniform and non-uniform scalar quantization
x[n]
Non uniform
Non-uniform
quantization
0
n
¾ Might be better to have lots of steps at low levels and a few at high
levels
¾ Look at pdf of signal or image and match the quantizer to it
EEE 508 - Lecture 7
Scalar
Sca
a qua
quantization
t at o
• General Approach:
¾ Take histogram of image and normalize it (so that area = 1) to
obtain a pdf. Usually, it takes a form close to a Gaussian or a
Laplacian distribution
¾ Then design quantizer (choose {Vj} and {rj}) to match pdf and
minimize distortion
¾ To solve the obtained resulting equations is not impossible but
they are non-linear
non-linear. So,
So usually,
usually the equations have been solved
numerically for different distributions and number of quantization
levels and the parameters of optimal quantizers (Lloyd-Max) are
tabulated.
EEE 508 - Lecture 7
Scalar
Sca
a qua
quantization
t at o
• Simple example
Two level quantizer
δ
x input with pdf
p(x)
Q(x) = x
x
x
-δ
¾ What is the optimal reconstruction level δ ?
e = x − xˆ
( )
EQ2 = E e = σ =
2
2
e
∞
2
ˆ
(
)
x
−
x
p( x )dx
∫
−∞
EEE 508 - Lecture 7
Scalar
Sca
a qua
quantization
t at o
¾ Use symmetry of the problem:
∞
p( x ) = p(− x )
⎧
⎫
⎨
2
2⎬
⎩Q(x ) = −Q(− x ) ⇒ (e( x)) = (e(− x)) ⎭
∞
EQ2 = 2 ∫ ( x − xˆ ) p( x )dx = 2 ∫ ( x − δ ) p( x )dx
2
2
0
0
∞
∞
∞
⎤
dE
d ⎡
2
2
Take
=0⇒
p(x )dx + 2 ∫ δ p( x )dx ⎥ = 0
⎢2 x p (x )dx − 4δ ∫ xp
dδ
dδ ⎣ ∫0
0
0
⎦
∞
∞
⇒ 4δ ∫ p (x )dx − 4 ∫ xp( x )dx = 0
0
0
1
424
3
1
2
∞
∞
0
0
⇒ 2δ = 4 ∫ xp( x )dx ⇒ δ = 2 ∫ xp( x )dx
EEE 508 - Lecture 7
Scalar
Sca
a qua
quantization
t at o
⇒ Apply same idea for the more general M-level case
Note:
In previous example, the quantization levels are fixed and the
optimzation is performed with respect to the reconstruction levels
assuming
i a uniform-quantizer
if
ti
structure.
t
t
More generally, we would want to optimize with respect to both
quantization levels {Vj} and reconstruction levels {rj} ⇒ Lloyd-Max
Quantizer optimal in the mean square sense (mse distortion
minimized).
EEE 508 - Lecture 7
Scalar
Sca
a qua
quantization
t at o
• The Optimum Mean Square Quantizer (Lloyd-Max)
¾ The “quantization levels” of a scalar quantizer are defined in terms
of its decision levels, tj, as the open interval
Vj = [ tj, tj+1 )
¾
If M quantization levels ⇒ M+1 decision levels t1, …, tM+1,
i e input sample x is quantized to the j-th level rj if tj ≤ x ≤tj+1
i.e.,
⇒ resulting mapping is in the form of a staircase function
EEE 508 - Lecture 7
Scalar
Sca
a qua
quantization
t at o
x
rk
rk-1
t1
t2 t3
tk-1 tk
tM+1 x
r2
r1
f ( x ) = signal
i l pdf
df ⇒ p j = prob(
b( x quantized
ti d to
t level
l l j)=
t j +1
d
∫ f (x )dx
tj
EEE 508 - Lecture 7
Scalar
Sca
a qua
quantization
t at o
¾ Problem: For M fixed and given signal pdf f(x), find decision levels
{tk} and reconstruction levels {rk} of the M-level scalar quantizer
such that the mse error
EQ2
2
⎡⎛
⎞ ⎤ t M +1
⎟ ⎥ = (x − Q( x ))2 f x (x )dx
(
)
= E ⎢⎜ x − Q
x
{⎟ ⎥ ∫
⎢⎜⎝
xˆ
⎠ ⎦ t1
⎣
is minimized.
EEE 508 - Lecture 7
Scalar
Sca
a qua
quantization
t at o
⇒ Use same method as in previous example: set derivatives with
respectt tto {tk} and
d {r
{ k} equall to
t 0
M t j +1
EQ2 = ∑
∫ (x − r ) f (x )dx
2
j
j =1 t j
x
144
42444
3
e j = mse for jth level
∂EQ2
=
∂rj
∂e j
∂rj
t j +1
= 2 ∫ (x − rj ) f x ( x )dx = 0
tj
t j +1
t j +1
∫ xf (x )dx ∫ xf (x )dx
x
⇒ rj =
tj
x
=
t j +1
∫ f (x )dx
tj
pj
[
= E x | x ∈V j
]
x
tj
⇒ lie at center of mass of pdf between optimum decisions levels
EEE 508 - Lecture 7
Scalar
Sca
a qua
quantization
t at o
∂EQ2
∂t j
= (t j − r j −1 ) f x (t j ) − (t j − rj ) f x (t j ) = 0
2
⇒ tj =
(r
j
2
+ rj −1 )
2
⇒ lie halfway between optimum {rj}
– Note:
d
db(t )
da(t )
(
)
(
(
)
)
f
τ
d
τ
=
f
b
t
−
f (a(t ))
dt a∫(t )
dt
dt
b (t )
EEE 508 - Lecture 7
Scalar
Sca
a qua
quantization
t at o
¾ For most distributions, there is no direct solution for {tk} and {rk}
but equations are solved iteratively
¾ Note: Lloyd-Max quantizer optimal only under the constraint that
the number of quantization levels M is fixed ⇒ may not be optimal
under other constraints
EEE 508 - Lecture 7
Scalar
Sca
a qua
quantization
t at o
¾ Properties of the optimum Mean Square Quantizer
9 output xˆ = Q( x ) is an unbiased estimate of x
E [xˆ ] = E [x ]
9
e = x − Q( x ) = x − xˆ
xˆ
is orthogonal to
[
[ ]
] [ ] [ ]
2
E [( x − xˆ )xˆ ] = 0 ⇒ E [xxˆ ] = E xˆ 2 ⇒ E ( x − xˆ ) = E x 2 − E xˆ 2
– Sketch Proof of error orthogonality to
[
]
xˆ
[
]
[ ]
2
E ( xxˆ ) = E {E xxˆ | x ∈ V j } = ∑ p j rj E x | x ∈ V j = ∑ p j rj2 = E ( xˆ )
14243 j =1
j =1
L
rj
EEE 508 - Lecture 7
L
Scalar
Sca
a qua
quantization
t at o
¾ From properties, an optimal mse quantizer has
Distortion
EQ2 = σ − ∑ p r
2
2
j j
j
Assume
zero-mean
signal
The entropy of quantized signal, i.e. of {rj} is
Entropy in bits/sample
M
H = −∑ p j log 2 p j ;
j =1
t j +1
pj =
∫ f (x )dx
tj
The entropy H and the quantization error EQ2 give the rate-distortion
quantizer: as M increases,, D decreases but R increases
function of the q
EEE 508 - Lecture 7
Scalar
Sca
a qua
quantization
t at o
¾ Example of a quantizer R-D function
9 The entropy H and the quantization error EQ2 give the rate-distortion
function of the quantizer: as M increases, D decreases but R increases
RD
M
D
EEE 508 - Lecture 7
Scalar
Sca
a qua
quantization
t at o
• Uniform Optimal Quantizer
¾ If pdf f(x) is uniform ⇒ Max-Lloyd (MSE optimal scalar) quantizer
becomes a uniform (linear) quantizer.
¾ Lloyd-Max equations
t j +1
∫ xf (x )dx
x
rj =
tj
pj
and
tj
(
r
=
become linear.
¾ For M fixed and f(x) uniform,
⎧ 1
, t1 ≤ x ≤ t M +1
⎪
f (x ) = ⎨ t M +1 − t1
⎪⎩ 0,
otherwise
EEE 508 - Lecture 7
j
+ rj −1 )
2
Scalar
Sca
a qua
quantization
t at o
• Uniform Optimal Quantizer
¾ Setting f(x) to be uniform in Lloyd-Max equations, we obtain
rk =
t k +1 + t k
;
2
tk =
t k +1 + t k −1
2
⇒ tk-tk-1 = tk+1-tk = constant = q ← step size
⇒ rk-rk-1 = rk+1-rk = constant = q
⇒ q=
t M +1 − t1
;
M
Offset
t k = t k −1 + q = (k + O)q; rk = t k +
q
2
⇒ quantizer completely described by offset O and step size q
EEE 508 - Lecture 7
Scalar
Sca
a qua
quantization
t at o
¾ Quantization error
Let A = tM+1- t1 = range of x ;
x uniformly distributed
f(x)
1/A
t1
⇒
Let
tM+1
x
A
A2
2
σx =
12
e = x − Q( x ) = x − xˆ ⇒ e uniformly distributed on (-q/2,q/2)
q
2
2
q
1
2
⇒ σ =
e de =
= EQ2
∫
12
q q
2
e
−
EEE 508 - Lecture 7
2
Scalar
Sca
a qua
quantization
t at o
¾ Note: Entropy constrained design: fix bit-rate B, instead of M
9 If fixed bit-rate B ⇒ M=2B (since H = log2 M for uniform
distribution):
M
1
1
B = H = −∑ log 2
= log 2 M ⇒ M = 2 B
M
j =1 M
t j +1
⇒
∫
pj =
f ( x )dx =
tj
⇒
⇒
Dmse
2
2B
t j +1 − t j
A
=
1
q
A
A t −t
=
⇒q=
= B = M +1 B 1
2
A M
M 2
A2
σ2
=σ =
= 2 B = σ 2 2−2 B ; B ≥ 0
2B
12 2
2
2
e
σ2
( )
1
σ2
=
⇒ RD = B = logg 2
D
2
D
Shannon optimum ((same
as R-D of Gaussian R.V.)
Note: All other scalar memoryless quantizers have a worse
Rate-Distortion performance
EEE 508 - Lecture 7
Scalar
Sca
a qua
quantization
t at o
¾ We can also find the optimal uniform quantizer for non-uniform
distributions
⇒ Find q such that
∂E
=0
∂q
∂q
Constraints:
9 Uniform Quantizer
9 Fixed
Fi d number
b off levels,
l
l M
Performace: less than optimum
uniform without constraints.
EEE 508 - Lecture 7
Scalar
Sca
a qua
quantization
t at o
Results:
9 Optimal uniform for non-uniform distribution performs almost within
0.2546 bits of the Shannon optimum if combined with entropy coding
and better than plain Lloyd
Lloyd-Max
Max (optimum mse) if no entropy coding
used for Lloyd-Max.
9 With combined entropy coding, optimum quantizer would be the one
that minimizes the distortion for a specified output entropy (EntropyConstrained Design)
Design).
EEE 508 - Lecture 7
Scalar
Sca
a qua
quantization
t at o
Shannon optimum
Sh
ti
quantizer
ti
Uniform quantizer with uniform pdf
Lloyd-Max quantizer with uniform pdf
RD
Optimum uniform, non-uniform pdf
with entropy coding
Lloyd-Max, non-uniform pdf without
entropy coding
Optimum uniform, non-uniform pdf
without entropy coding
σ2 D
EEE 508 - Lecture 7
Scalar
Sca
a qua
quantization
t at o
¾ Optimal levels in case of a symmetric pdf
9 For an even symmetric pdf i.e. fx(x)=fx(-x), a possible optimal Lloyd-Max
L-level quantizer is obtained (assuming L is even) by having:
t ⎧⎛ L
= −t ⎧⎛ L
r⎧⎛ L
= −r⎧ L
⎞ ⎫
⎨ ⎜ +1 ⎟ + k ⎬
⎩⎝ 2 ⎠ ⎭
⎞ ⎫
⎨ ⎜ +1 ⎟ + k ⎬
⎩⎝ 2 ⎠ ⎭
; k = 0,1,,...,,
⎫
⎞
⎨ ⎜ +1 ⎟ − k ⎬
2
⎠ ⎭
⎩⎝
; k = 0,1,...,
⎫
⎨ −k ⎬
⎩2 ⎭
L
2
(optimal decision level )
L
− 1 (optimal reconstruction level)
2
For
k = 0, t ⎧⎛ L
⎞⎫
⎨⎜ +1 ⎟ ⎬
⎩⎝ 2 ⎠ ⎭
EEE 508 - Lecture 7
= −t ⎧⎛ L
⎞⎫
⎨ ⎜ +1 ⎟ ⎬
⎩⎝ 2 ⎠ ⎭
⇒ t⎛ L
⎞
⎜ +1 ⎟
⎝2 ⎠
=0
Scalar
Sca
a qua
quantization
t at o
9 In general, if pdf even-symmetric with respect to a point xc, i.e.
f( c-x)=f(x
f(x
) f( c+x),
) then
th it can be
b shown
h
(from
(f
Lloyd-Max
Ll d M equations
ti
for
f tk
and rk) that one possible set of optimal levels satisfy:
t ⎧⎛ L
= xc
t ⎧⎛ L
+ t ⎧⎛ L
⎫
r⎧⎛ L
+ r⎧ L
⎫
⎞⎫
⎨ ⎜ +1 ⎟ ⎬
⎩⎝ 2 ⎠ ⎭
⎞
⎨ ⎜ +1 ⎟ + k ⎬
2
⎠ ⎭
⎩⎝
⎞
⎨ ⎜ +1 ⎟ + k ⎬
⎩⎝ 2 ⎠ ⎭
EEE 508 - Lecture 7
⎞ ⎫
⎨ ⎜ +1 ⎟ − k ⎬
⎩⎝ 2 ⎠ ⎭
⎫
⎨ −k ⎬
⎩2 ⎭
= 2 xc ; k = 0,1,...,
= 2 xc ; k = 0,1,...,
L
2
L
−1
2
Scalar
Sca
a qua
quantization
t at o
9 Since a uniform quantizer is derived as a Lloyd-Max with uniform
di t ib ti
distribution
⇒ by
b convention,
ti
its
it levels
l
l are always
l
taken
t k to
t satisfy
ti f the
th
above symmetry relations when the pdf is even symmetric (even when
pdf is non-uniform)
9 Example: Design of an optimal L-level uniform quantizer for a nonuniform symmmetric pdf
x input with pdf
f(x)
-a
a
x
Lq
– Set q=2a/L and find optimal q (or equivalently a) such that the MSE EQ2 is
minimized.
EEE 508 - Lecture 7
Scalar
Sca
a qua
quantization
t at o
9 Example(continued): Design of an optimal L-level uniform quantizer for
a non-uniform
if
symmetric
t i pdf
df
– Let fx(x) = even function and L = even integer
– For a fixed L, the optimum uniform quantizer is determined completely by
the quantization step size q. Define 2a = Lq => q= 2a/L.
– Determine a to minimize mse
∞
L −1 t j +1
E=∑
∫ (x − r ) f (x )dx + 2 ∫ (x − r ) f (x )dx
j =2 t j
2
j
2
x
L
a −q
9 Since for uniform quantizer
L / 2 −1
E=2∑
j =1
EEE 508 - Lecture 7
x
rj =
2 j −1
q
2
(2 j − 1)q ⎞ f (x )dx + 2
(L − 1)q ⎞ f (x )dx
⎛
⎛
x
x
−
−
⎟ x
⎟ x
⎜
⎜
∫
∫
2
2
⎠
⎠
( j −1)q ⎝
( L / 2 −1)q ⎝
jq
2
∞
2
Scalar
Sca
a qua
quantization
t at o
To minimize E for L>2, we iteratively solve the nonlinear equation
dE
=0
dq
F L =2
For
2, we get:
t
∞
a = 2 ∫ xff x ( x )dx
0
EEE 508 - Lecture 7