Homework 3

EE376A: Homeworks #3
Due on Thursday, February 5, 2015
You can hand in the homework either after class or deposit it, before 5 PM, in the EE376A
drawer of the class file cabinet on the second floor of the Packard Building.
1. Shannon code.
Consider the following method for generating a code for a random variable X which
takes on m values {1, 2, . . . , m} with probabilities p1 , p2 , . . . , pm . Assume that the
probabilities are ordered so that p1 ≥ p2 ≥ · · · ≥ pm . Define
Fi =
i−1
X
pk ,
(1)
k=1
the sum of the probabilities of all symbols less than i. Then the codeword for i is the
number Fi ∈ [0, 1] rounded off to li bits, where li = dlog p1i e.
(a) Show that the code constructed by this process is prefix-free and the average
length satisfies
H(X) ≤ L < H(X) + 1.
(2)
(b) Construct the code for the probability distribution (0.5, 0.25, 0.125, 0.125).
2. Sequence length.
How much information does the length of a sequence give about the content of a
sequence? Suppose we consider a i.i.d. Bernoulli(1/2) process {Xi }.
Stop the process when the first 1 appears. Let N designate this stopping time.
Thus X N is an element of the set of all finite length binary sequences {0, 1}∗ =
{0, 1, 00, 01, 10, 11, 000, · · · }.
(a) Find I(N ; X N ).
(b) Find H(X N |N ).
(c) Find H(X N ).
Let’s now consider a different stopping time. For this part, again assume Xi ∼Bernoulli(1/2)
but stop at time N = 6, with probability 1/3 and stop at time N = 12 with probability
2/3. Let this stopping time be independent of the sequence X1 X2 · · · X12 .
(d) Find I(N ; X N ).
(e) Find H(X N |N ).
(f) Find H(X N ).
Homework 3
Page 1 of 3
3. Bad codes.
Which of these codes cannot be Huffman codes for any probability assignment?
(a) {0, 10, 11}.
(b) {00, 01, 10, 110}.
(c) {01, 10}.
4. Huffman Code.
Let X ∼ p(x), where
X = {1, 2, 3, 4, 5, 6, 7, 8}
1 1 1 1 1 1 1 1
p = ( , , , , , , , )
4 4 8 8 16 16 16 16
(a) Find the Huffman binary code for p.
(b) Find its expected length and the entropy H(X).
(c) Find the Huffman 3-ary code for p.
5. Fair to bent.
Let X ∼ p(x), where
X = {1, 2, 3, 4, 5, 6}
1 1 1 1 1 1
p = ( , , , , , )
4 4 4 8 16 16
(a) Describe how to use fair coin flips Z1 , Z2 , Z3 , . . . to generate X so as to minimize
the expected number of flips required.
(b) What is the expected number of flips required?
(c) What is H(X)?
6. Bad wine.
One is given 6 bottles of wine. It is known that precisely one bottle has gone bad
(tastes very bad). From inspection of the bottles it is determined that the probability
8 6 4 2 2 1
pi that the ith bottle is bad is given by (p1 , p2 , . . . , p6 ) = ( 23
, 23 , 23 , 23 , 23 , 23 ). Tasting
will determine the bad wine.
Suppose you taste the wines one at a time. Choose the order of tasting to minimize
the expected number of tastings required to determine the bad bottle. Remember, if
the first 5 wines pass the test you don’t have to taste the last.
Homework 3
Page 2 of 3
(a) What is the expected number of tastings required?
(b) Which bottle should be tasted first?
Now you get smart. For the first sample, you mix some of the wines in a fresh glass
and sample the mixture. You proceed, mixing and tasting, stopping when the bad
bottle has been determined.
(c) What is the minimum expected number of tastings required to determine the bad
wine?
(d) What mixture should be tasted first?
7. Optimal codeword lengths.
Although the codeword lengths of an optimal variable length code are complicated
functions of the message probabilities {p1 , p2 , . . . , pm }, it can be said that less probable
symbols are encoded into longer codewords. Suppose that the message probabilities
are given in decreasing order p1 > p2 ≥ · · · ≥ pm .
(a) Prove that for any binary Huffman code, if the most probable message symbol has
probability p1 > 2/5, then that symbol must be assigned a codeword of length 1.
(b) Prove that for any binary Huffman code, if the most probable message symbol has
probability p1 < 1/3, then that symbol must be assigned a codeword of length ≥ 2.
8. Prefix and Uniquely Decodable codes
Consider the following code and answer the following questions:
u Codeword
a
10
b
00
c
11
d
110
(a) Is this a Prefix code?
(b) Show that this code is uniquely decodable.
Homework 3
Page 3 of 3