Chapter 6: Multimedia Networking

Transcription

4/14/16
Ch. 5: Multimedia Networking
Multimedia Systems
Prof. Ben Lee
School of Electrical Engineering and Computer Science
Oregon State University
Outline

Introduction
Wireless LANs
◦ What is the Internet?
◦ 802.11
Physical layer
Medium Access Control (MAC) protocol
Mobility

Dealing with packet jitter and loss
◦ Issues
◦ Reducing Delay Jitter
Fixed vs. Adaptive Playout
◦ Recovering from Packet Loss
FEC
Interleaving

Case Study: Loss Resilience Features in H.264
2
1
4/14/16
Introduction

Explosive growth of network applications that
transmit audio and video over the Internet.
A new class continuous media applications:
◦ Entertainment video, IP telephony, Internet radio,
Multimedia WWW sites, teleconferencing, interactive
games, virtual worlds, distance learning ...

Service requirements are significantly different
from those of traditional data oriented
applications.
◦ A need for a network architecture (multimedia
networking) that can support these applications.
3
Multimedia Over Internet

Multimedia applications are highly sensitive to
end-to-end delay and delay variations (jitter) but
typically loss tolerant.
◦ In contrast to applications, such as Web text/image, email, FTP, and Telnet, where long delays are annoying
but not harmful.

IP protocol only provides a best-effort service:
◦ No guarantees of end-to-end delay.
◦ No guarantees of variations of packet delay within a
packet stream.
Will concentrate
on WLANs!
4
2
4/14/16
Multimedia Networking Issues

Application classes:
◦ Streaming stored audio/video
◦ Multicast streaming of real-time audio/video
◦ Real-time interactive audio/video

Application issues:
◦ Packet jitter
◦ Packet loss/recovery

Internet protocols for multimedia
◦ RTSP, RTP/RTCP

Network QoS and Resource Management
◦ Intserv/RSVP, Diffserv

Congestion avoidance
◦ Avoiding packet loss
5
Multimedia Networking

Need to understand how E-to-E communication is
achieved:
◦ Internet - TCP/UDP, IP, Link Layer
◦ Various technologies that comprise the Internet
◦ …and how they affect the end-to-end delay, jitters, packet losses
of media data being transmitted.

Need something beyond best effort:
◦ Making the best of best effort - buffering
◦ Recovering from packet loss - redundancy
◦ Protocols for Real-time Interactive applications
RTP, RTCP
◦ Congestion Avoidance
◦ QoS - end-to-end guarantees
6
3
4/14/16
Outline

Introduction
Wireless LANs
◦ 802.11
Physical layer
Mobility

Dealing with packet jitters and loss
◦ Issues
◦ Reducing Delay Jitters
FEC
Interleaving

7
What is the Internet?
Core Network
Edge Network
8
4
4/14/16
Internet Protocol Architecture
Host A
Application Layer
TCP identify applications
Using 16-bit port numbers
App X
App Y
Transport Layer
Logical TCP connection
TCP/UDP
Network Layer
Link Layer
Host B
Global network address
IP
Software
Network Access
Type A
Physical Layer
Physical
Type A
Hardware
Router
App X
App Y
Application Layer
TCP/UDP
Transport Layer
IP
Network Layer
Network Access
Type B
Physical
Type B
Link Layer
Physical Layer
IP
Network
Type A
NA
NA
Type A Type B
Physical Physical
Type A Type B
Network
Type B
9
Protocol Data Units
User Data
Application
TCP
Header
TCP Segment
IP
Header
Preamble
Frame
Header
IP Datagram
CRC
Frame Packet
Frame format depends on the specific type of network.
10
5
4/14/16
Wireless LANs

Wireless LANs: untethered (often mobile)
networking.
IEEE 802.11 standard:
◦ MAC protocol
◦ Unlicensed frequency spectrum: 900 MHz, 2.4 GHz, 5
GHz

Not meant to replace wired networks:
◦ A way to cover locations that are difficult to cover.
Manufacturing plants
Historical buildings, etc.

Provides mobility, relocation,Ad Hoc networking:
◦ Home networking
◦ PANs
◦ Sensor networks
11
WLAN Architecture

Basic Service Set (BSS)
contains:
◦ Mobile Stations (MSs)
◦ Access point (AP): base
station.
Defined by Service Set
Identifier (SSID)

Extended Service Set
(ESS)
◦ A set of infrastructured
BSSs.
Typical Range: 30~300 ft
12
6
4/14/16
Ad Hoc Networks

Ad hoc network: IEEE 802.11 stations can
dynamically form network without AP.
Applications:
◦
◦
◦
◦

“laptop” meeting in conference room, car.
Interconnection of “personal” devices.
Battlefield
Sensor networks
IETF MANET (Mobile Ad hoc Networks)
working group
13
What is IEEE 802.11?

Family of standards set forth by IEEE to define the
specifications for WLANs (adopted in 1997).
1 and 2 Mbps at distances of 150-2000 ft.
IEEE 802.11 standard:
◦ Medium Access Control (MAC) protocol
◦ Physical Layer (PHY) specifications

802.11b
802.11a
802.11g
802.11n
802.11ac
◦ Power Saving Mechanism
14
7
4/14/16
802.11x Comparison
802.11b
802.11a
802.11g
802.11n
Standard Ratified
Sept. 1999
Sept. 1999
June 2003
Sept. 2009
Frequency
2.4 GHz
5 GHz
2.4 GHz
2.4 or 5 GHz
Range (typical)
100-150 ft
100-150 ft
~165 ft
Data Rates
1, 2, 5.5, 11
6, 9, 12, 18, 24,
36, 48, 54
6, 9, 12, 18, 24,
36, 48, 54
540 (max.)
Realistic
Throughput
~4-6 Mbps
~27 Mbps
~20 Mbps
~200 Mbps
Modulation
CCK
OFDM
OFDM
OFDM
# of channels
Non Overlapping
11/3
12/8
11/3
11/3
# of users
32
64
64
96
25-35 ft
15
802.11ac

Gigabit Wi-Fi

5GHz band only

Wider BW
◦ 6.93 Gbps max. data rate
◦ Less interference and more non-overlapping channels.
◦ 80 MHz (433 Mbps) and 160 MHz (867 Mbps) channels
Compared to 20 MHz and 40 MHz of 802.11n

Higher order modulation
◦ OFDM 256 QAM, 8 bits per symbol
Higher order MIMO
◦ Up to 8 spatial streams, max. 4 streams per STA.
Multi-User MIMO (MU-MIMO)
◦ Allows AP to simultaneously service multiple STAs.
Beamforming
◦ Improves data rate for a given range.
16
8
4/14/16
IEEE 802.11 MAC Layer

Same MAC layer for all variants (802.11b, a, g, n,
ac)
◦ DCF (Distributed Coordination Function)
CSMA/CA (Carrier Sense Multiple Access with collision
avoidance)
All stations may not hear each other:
Hidden node problem
Exposed node problem
◦ PCF (Point Coordination Function)
Polling (optional)
17
Joining the Network

Scanning
◦ Need to find an AP to join.

Authentication request/response
◦ Need to authenticate users that are authorized to
use the network.

(Re-)Assocation request/response
◦ A MS needs to register, or associate, with an AP. The
server can then use the registration information to
determine which AP to use to communicate with MS.
18
9
4/14/16
Passive Scanning
MS waits for a beacon every 102.4 ms.
Must check all 11 channels.
Allows MS to power down if needed

Beacon Frame
2
2
Frame
Duration
Control
6
6
6
DA
SA
BSSID
2
Seq.
Control
0 - 23 12
4
payload
CRC
Ti mes ta mp ,
Beac on int er val,
Cap abili ty i nfo , e tc.
Na me assi gn ed to AP
Sou rce (AP ) MAC ad dr ess
Desti na tio n MAC ad dr ess (Br oa dc ast )
NAV (N etw or k Allo cati on Vec to r)
Fr a me Ty pe :
Ma na ge m ent , c ont rol ,
or da ta
19
Active Scanning (1)
MS probes for APs.
Must check all 11 channels! (i.e., perform a full scan)

Probe request/response Frame
2
2
Frame
Duration
Control
Ch. 1
6
6
6
DA
SA
BSSID
2
Seq.
Control
0 - 23 12
4
payload
CRC
SSID ( ne two rk na m e),
Sup po rte d rat es
Probe request
Probe response
Name assigned to AP
Sou rce ( MS) MA C a dd re ss
Probe response
Desti na tio n MAC ad dr ess (Br oa dc ast )
Probe request
NAV (N etw or k Allo cati on Vec to r)
Ch. 6
Fr a me Ty pe :
Ma na ge m ent , c ont rol ,
or da ta
20
10
4/14/16
Active Scanning (2)
Ava ila bl e
Ch an n els
Ho st
Ini ti at e
Sca n ni ng
Ch. Switch Time

MS has to wait if there is no
responses from APs
Channel # 1
No APs in Ch. #1
MinChann elTime
Ch. Switch Time
Channel # 2
MaxChann elTime
…
Multiple APs in
Ch. #2
…
MinChannelTime - Time
MaxChannelTime -Time
MS has to wait if a
response is heard
Selects AP with best
signal strength.

Ch. Switch Time
Channel #N
MinChann elTime
No AP in Ch. #N
Act iv e s ca n
com pl et e
21
Authentication

Open-system:
◦ Does not verify the identity of MS.

Username and passwd
Address filtering:
◦ Authorized MAC address list
◦ Kept in a centralized server (OSU uses “Maintain”)

Shared-key
◦ WEP
22
11
4/14/16
(Re-)Association

Allows the new AP to keep track of the location
of MS.
The Switch becomes aware of location MS, i.e.,
needs to forward frames to new AP.
◦ More on this later…

MS can now communicate with the world!
23
Comm. with Outside World
Switch Table
MAC ad dr ess : xx :xx: xx:x x:xx :xx
Router
Internet
Not aw ar e o f APs!
1
ARP Table
IP Address
MAC Address
128.193.143. 14 8
00:19:E3:0E:65: 18
…
…
…
…
4
Interface
3
xx:xx:xx:xx:xx:xx
4
Server
Switch
2
MAC Address
00:19:E3:0E:65: 18
3
0 - 23 12
4
Payload
CRC
MAC ad dr ess of
IP Datagram
ro ute r i nte rf ace to
whic h AP is
att ach ed
Only us ed in
MAC ad dr ess
ad ho c mo de
of MS or AP
tra ns mitti ng thi s f ra me
Atrium
APy
APz
APx
Data Frame
2
2
6
6
6
2
6
Frame
Address Address Address Seq. Address
Duration
Control
1
2
3
Control
4
MAC ad dr ess
of MS or AP to
rec eiv e t his f ra m e
MAC ad dr ess : 0 0: 19:E 3: 0E:6 5: 18
IP a dd res s: 12 8.1 93 .1 43. 14 8
Classrooms
Ethernet (802.3) frame Û 8 02.11 frame
24
12
4/14/16
Roaming
Example Roaming Path of 1 st floor of KEC
Switch has to
forward to APz
Switch Table
MAC Address
Interface
00:19:E3:0E:65: 18
2®3
4
1
Server
Switch
2
3
3. Update Switch Table
Atrium
2. Broadcast( 00:19:E 3:0E :6 5:1 8)
APy
APz
MS
1. Scan, authentication, reassociation
MAC ad dr ess : 0 0: 19:E 3: 0E:6 5: 18
IP a dd res s: 12 8.1 93 .1 43. 14 8
APx
Classrooms
SSID=“OSU_PUB”
25
Handoff Delay
Ava ila bl e
Ch an n els
Ho st
Ini ti at e
Ha nd o ff

Ch. Switch Time
Time MS has to wait
if there is no
responses from APs
Channel # 1
No APs in Ch. #1
MinChann elTime
Ch. Switch Time
Channel # 2
MaxChann elTime
Time MS has to wait if a
response is heard

…
◦ Channel Switch Time = 5~19 ms
◦ MinChannelTime = 1~7 ms
◦ MaxChannelTime = ~11 ms
Multiple APs in
Ch. #2
…

Resulting in scanning delay of
350~500 ms!

Represents > 90% of overall
handoff delay.
Ch. Switch Time
Channel #N
MinChann elTime
Act iv e s ca n
com pl et e
◦ Will loose packets!
No AP in Ch. #N
Channel # 2
No set standard for these
parameters.
A recent study of 802.11b based
on Intersil Prism2 and Atheros
5212 NICs:
◦ Authentication & Association each
take < 10 ms.
◦ Wired network update takes < 20
ms
Ha nd o ff c om p le te
26
13
4/14/16
Medium Access Control

How do you control the
access to wireless medium?
Signals attenuate as function
of distance:
◦ Carrier Sense detects signals
at the transmitter.
◦ But, collision occurs at the
receiver.

Cannot listen while you send:
◦ Transceiver is half duplex.
◦ Rx cannot tune to Tx
frequency.

All you hear is yourself!
◦ Your own signal strength at
your antenna is much stronger
than anybody else’s signal.
27
IEEE 802.11 Carrier Sensing

802.11 uses both physical (at PHY) and virtual
(at MAC) carrier sensing.
◦ Physical carrier sensing
Medium consider busy when transmitting
◦ Virtual carrier sensing
Maintained by each station using Network Allocation Vector
(NAV), i.e., 16-bit counter in µs increments.
Set based on Duration field in received frames (e.g., RTS,
CTS).
When nonzero, station thinks medium is busy.

Medium considered busy whenever either
indicates carrier.
28
14
4/14/16
IEEE 802.11 MAC Protocol:
CSMA/CA
Physical layer monitors the
802.11 CSMA: sender
If sense channel idle for DISF (Distributed
Inter Frame Space)
Then transmit entire frame (no collision
detection)
If sense channel busy
then binary backoff
If ACK not received
then retransmit
energy level on the radio
frequency
802.11 CSMA receiver:
If received OK
return ACK after SIFS (Short Inter
Frame Space)
Link-layer ACK
29
Inter Frame Spacing

Priorities are defined through different inter frame spacing (IFS)
◦ SIFS (Short Inter Frame Spacing)
Highest priority, for ACK, CTS, and polling response
◦ PIFS (PCF IFS)
Medium priority, for time-bounded service using PCF
◦ DIFS (DCF IFS)
Lowest priority, for best-effort data
◦ EIFS (Extended IFS)
Lowest priority interval used to report bad or unknown frame.
30
15
4/14/16
IEEE 802.11 MAC Protocol
DIFS
contention window
(randomized back-off
mechanism)
DIFS
medium busy
next frame
t
direct acc ess if
medium is free ³DIF S
slot time
Station ready to send starts sensing the medium.
If the medium is free for the duration of DIFS the station can
start sending.
If the medium is busy, the station has to wait for a DIFS, then
the station must additionally wait a random back-off time
(collision avoidance, multiple of slot-time).
If another station occupies the medium during the back-off
time of the station, the back-off timer stops (fairness).

31
Link Layer ACK
SIFS = 10 µs
short slot_time = 9 µs (802.11g only)
long sl ot_time = 20 µs (802.11b/g)
DIFS = 2 x sl ot_time + SIFS
DIFS
Sender
data
SIFS
Receiver
ACK
slot_time
DIFS
Other
stations
NAV
…
data
t
defer access
Back off delay
32
16
4/14/16
Random Backoff

Backoff delay uniformly chosen in the range [0,
CW-1], where CW is called Contention Window.
◦ Backoff timer is decreased as long as medium is sensed to
be idle for DIFS.
◦ Backoff timer is frozen medium is busy.
•
•
•
•
CW reaches 0, station transmits.
Initially CW =1. After each unsuccessful
transmission (no CTS or ACK) CW is doubled (up
to a max. value).
Once CW reach CW MAX, it stays at that value until
reset.
CW is reset after every successful transmission of a
data or RTS packet.
33
Hidden Terminal Effect

Hidden terminals:A, C cannot hear each other
◦ Obstacles, signal attenuation
◦ Collisions at B

Goal: avoid collisions at B
CSMA/CA: CSMA with Collision Avoidance
34
17
4/14/16
Hidden & Exposed Terminal
Problem
Hidden Terminal
Currently transmitting
Currently transmitting
A
C
Exposed Terminal
B
B cannot hear A
A
B
C
D
C hears B’s
Transmission:
Cannot transmit to D!
35
Collision Avoidance (1)

CSMA/CA - Explicit
channel reservation:
◦ Sender: send short (fixed
size) RTS: request to
send.
◦ Receiver: reply with short
(fixed size) CTS: clear to
send.

CTS reserves channel for
sender, notifying (possibly
hidden) stations
Avoid hidden station
collisions
36
18
4/14/16
Collision Avoidance (2)
NAV set
RTS
CTS
NAV set
NAV set
Data
ACK
NAV set
37
RTS-CTS Exchange
DIFS
sender
RTS
data
SIFS
receiver
other
stations
SIFS
SIFS
CTS
ACK
DIFS
NAV (RTS)
data
NAV (CTS)
t
defer access
contention
38
19
4/14/16
802.11e

QoS support in WLANs:
◦ Also referred to as WiFi MultiMedia (WMM).

Backward compatible with DCF and PCF.
Enhanced Distributed Channel Access (EDCA):
◦ Evolution of DCF
◦ Introduces traffic priority
◦ Favors VoIP applications
39
EDCA

Traffic differentiation:

New Inter-frame Spec for each AC:

4 Access Categories (ACs) with 8 Traffic Classes
(TCs).
Arbitration Inter Frame Space (AIFS)
Transmission Opportunity (TXOP)
40
20
4/14/16
Access Categories (ACs)

Media access is based on the AC of frames.
4 AC’s are defined:
◦ AC_BK (AC0) – Background
◦ AC_BE (AC1) – Best-effort
◦ AC_VI (AC2) – Video
◦ AC_VO (AC3) – Voice

The size of Contention Window (CW) and
Inter-frame space (IFS) is dependent on AC.
41
Priority
ACs map directly from Ethernet-level Class of
Service (CoS) priority levels
Priority
AC
Designation
1
AC_BK (0)
Background
2
AC_BK (0)
Background
0
AC_BE (1)
Best Effort
3
AC_BE (1)
Best Effort
4
AC_VI (2)
Video
5
AC_VI (2)
Video
6
AC_VO (3)
Voice
7
AC_VO (3)
Voice
42
21
4/14/16
Arbitration Interframe Space
(AIFS)

AIFS is used to defer the contention window or
transmission for each AC.
The value of AIFS differ corresponding to AC:
◦ AIFS[AC] = AIFSN[AC] x SlotTime + SIFS
where AIFSN[AC] is an AIFS number
◦ AIFSN for each AC is broadcast via beacon frame
containing “EDCA Parameter Set” element.

DIFS = 2 x SlotTime + SIFS
43
AC-based Backoff
44
22
4/14/16
CWmin Differentiation
CW is a function of
CW min[AC] and
CW max[AC].
Higher priority stations
have smaller CW min and
CW max.
Smaller CW min and
CW max correspond to
on average fewer backoff slots being traversed
per transmission.

AC
CW min
CW max
AC_BK
CW min
CW max
AC_BE
CW min
CW max
AC_VI
(CW min+1)/2 -1
CW min
AC_VO
(CW min+1)/4 -1
(CW min+1)/2 -1
45
46
Priority
23
4/14/16
Transmission Opportunity
(TXOP)
The interval of time when a particular STA has the right
to initiate transmissions.
Frames exchange sequences are separated by SIFS.
When will a STA get a TXOP ?

◦ Win a contention in EDCA, or
◦ Receive a CF-poll (“polled TXOP”)
47
Default EDCA Parameters
AC
CWmin
CWmax
AIFSN
Max TXOP
Background (AC_BK)
31
1023
7
0
Best Effort (AC_BE)
31
1023
3
0
Video (AC_VI)
15
31
2
3.008 ms
Voice (AC_VO)
7
15
2
1.504 ms
Legacy DCF
15
1023
2
0
48
24
4/14/16
Other Features of 802.11 MAC

Power management
◦ Informs AP it wants to sleep: AP buffers frames.
◦ Periodically wakes up to receive beacon signals:
Indicates whether frames have been buffered.
Requests awaiting frames from AP.
49
Outline

Introduction
Wireless LANs
◦ 802.11
Physical layer
Mobility

◦ Issues
FEC
Interleaving

50
25
4/14/16
Issues

Real-time interactive multimedia applications, e.g.,
Internet phone and real-time video conferencing are
very sensitive to
◦
◦
◦
◦
Delay
Loss
Delay-Jitter
Out-of-Order Arrivals
End-to-End Delay
Jitter
< 150 ms not perceived by user.
150 ms - 400 ms acceptable but not ideal.
> 400 ms seriously hinders interactivity.
> Jitter max
Send
X
X
Receive
Perfect
Delivery
Packet
Loss
Delay-Jitter
Out-of-order
Arrivals
51
Sources of Packet Loss, Jitter,
and Out-of-Order Arrival
Core Network
Edge Network
52
26
4/14/16
Packet Delay and Loss in Core
Network
End-to-end delays:
DE_to_E = DTx_proc + DQueuing + DPropagation + DRx_proc
Delay Jitter:

◦ Random queuing delays in the routers.

Packet Loss:
◦ Packet drop due to overflowed queues in the routers.

Out-of-Order Arrivals: No TCP
◦ Packets taking different paths.
◦ Random queue delays in the routers.
53
Packet Delay and Loss in Edge
Network
WLANs are pervasive.
Carrier Sense interference

◦ Access serialization to
medium => delay

Hidden Node interference
◦ Packet collision = > loss
◦ MAC retry (up to Retry Limit
= 7 times)

Home Entertainment
Increase in peer-to-peer
HD Video Streaming over
WLANs:
◦ e.g., Apple AirPlay, Google
Chromecast, Intel WiDi, etc.
N-Screen
54
27
4/14/16
Current Solutions

To mitigate the impact of “best-effort”
protocols, we can:
◦ Use UDP to avoid TCP’s slow-start phase and
retransmission.
◦ Buffer content at client and control playback to
remedy jitter.
◦ Timestamp packets at the sender (so receiver knows
when the packets should be played back).
◦ Prefetch data during playback (when client has extra
storage and bandwidth).
◦ Adapt compression level to available bandwidth.
55
Internet Phone Example
Voice sampling:
8 kHz @ 8 bits
Talkspurt
Chunk = 160 bytes
t
20 ms
Silent Periods
> 20 ms
UDP segment
Transmit
Header
Will need
• Sequence number
• Timestamp
…but wait...UDP and IP do not have this...
56
28
4/14/16
Reducing Delay-Jitter (1)
p = packaging period of the media at the sender
p
2p
2p
3p
Send
a
b
c
d
e
f
g
h
Receive
Playout
Display
Queue
a
d
c
b
e
d
e
f
g
h
Playing the samples as soon as they arrive ensures minimal
end-to-end latency but at the cost of an increased gap-rate.
57
Reducing Delay-Jitter (2)
3p
Send
a
b
c
d
e
f
g
h
Receive
Playout
Display
Queue
a
b
a
c
b
a
d
c
b
e
d
c
e
d
f
e
g
f
g
h

Enqueuing the first sample for some duration better ensures
continuous playout.

Which initial playout strategy is better? It depends…
◦ But at the cost of higher playout latency throughout the conference.
58
29
4/14/16
Fixed Playout Delay

When should it playout? p - r
or p’ - r?
Packets
However, playout delay should
not exceed some threshold,
e.g.,
Packets
generated
Missed
playout
◦ 250 ms for audio
◦ 10-15 frames/s for video (66.6
ms - 100 ms)

Tradeoff between playout
delay vs. packet loss.
For interactive services, such
as internet phone, long delays
can be bothersome if not
intolerable.
Packets
received
Playout schedule
p-r
Playout schedule
pʼ - r
Time
s
r
pʼ
p
59
Adaptive Playout Delay (1)

Often end-to-end delay distribution of packets
within a talk spurt is not known.
◦ …and can change over time.

Estimate the network delay and its variance, and
adjust the playout delay accordingly.
Determine play out delay on a per-talkspurt
basis.
60
30
4/14/16
qi
di
Send
Receive
Kvi
si
si+1
ri
Playout
r i+1
pi
pi+1
qi
si = timestamp of the i th packet generated by the sender
r i = time packet i is received by the receiver
pi= time packet i is played at the receiver
d i= average network delay
vi= average deviation of network delay
61

E-to-E delay for the ith packet will be ri - si . The average
network delay of the ith packet d i is (similar to how TCP
RTT is estimated)
d i = (1 - u)d i-1 + u(ri - s i),
where u is fixed constant (e.g., u = 0.01)

The average delay deviation v i is
vi = (1 - u)vi-1 + u|ri - s i - d i|,

The receiver then uses the following
pi = s i + d i + Kvi,
where K is called a congestion estimator (e.g., 4).
62
31
4/14/16
When should playout delay be changed?
For voice, transmission occurs in alternating talk spurts
and silence periods.
If no packet loss, receiver can determine that a packet is
the first packet of the talk spurt.

◦ e.g., s i - s i-i > 20 ms, then receiver knows ith packet start a new
talkspurt

What if there is a packet loss? Two successive packets
received may have timestamp that differ by more than
20 ms.
◦ Use sequence numbers.
63
Real-Time Protocol

Internet-standard protocol for transporting
real-time data, including audio and video.
A standardized packet structure that includes
fields for audio/video data, sequence number,
timestamp, an other useful fields.
◦ PCM or GSM for audio
◦ MPEG1 and MPEG2 for video

Runs on top of UDP.
64
32
4/14/16
RTP Packet Format
Version # (2 bits)
Padding Bit
Extension Bit
# of Contributing Sources
Marker bit
Payload type (7 bits)
V=2P X
3 words
CC
M
PT
Seque nce Num ber
Timestamp
Synchron izatio n Source (SSRC) Ide ntifier
0-15 words
Contrib uting So urce (CSRC) Ide ntifier
…
Extension H ead er
RTP Payload
65
Payload Type

Indicates the type of multimedia data carried by
the packet.
◦ Audio: PCM 𝜇-law, GSM, MPEG Audio, etc.
◦ Video: Motion JPEG, H.261, MPEG-1, MPEG-2, etc.

Defines sampling rate of timestamp.
◦ Audio
8 kHz sampling rate (every 125 𝜇s)
20 ms minimum inter-frame time.
Marker (M) bit set on the first packet after a silence period.
66
33
4/14/16
Sequence No.

Allows the receiver to detect missing and outof-ordered packets.
◦ 16 bits long.

Sender increments the sequence number by
one for each transmitted packet.
RTP does not do anything when a packet is
detected to be lost.
◦ Instead, it is left to the application to decide what
should be done.
67
Timestamp
Enables the receiver to playback samples at appropriate
intervals and to enable different media streams to be
synchronized.
32 bits long.
A counter of “ticks”, where time between ticks is
dependent on encoding in use.
For example,

◦ Audio application that samples data every 125 𝜇s uses this as its
clock resolution.
◦ Multiple audio samples can be transmitted together.
◦ If packet is generated every 10 ms (each containing 10 ms/125 𝜇
s = 80 samples), timestamp between successive packets would
have timestamp differ by 80.
68
34
4/14/16
Timestamp vs. Sequence No.

Timestamp relates packet to real-time:
◦ Timestamp values sampled from a media specific
clock.

Sequence number relates packet to other
packets.
Allows many packets to have the same
timestamp but different sequence numbers.
69
MPEG Example

How does the timestamp/sequence-number
mechanism support MPEG?
◦ Out-of-order transmission.
Sequence numbers increase monotonically.
Timestamps reflect reference relationships.
◦ Large frames.
One video frame likely to be split into parts and packed into
multiple RTP packets.
Timestamps associate packets together as part of the same
frame, while seq. no distinguish packets from each other.
70
35
4/14/16
Audio Silence example

Nothing is sent during silent periods.
Why might this cause problems?
◦ Other side needs to distinguish between loss and
silence.

How does the timestamp/seq. no mechanism
help?
◦ After receiving no packets for a while, next packet
received will reflect a big jump in timestamp, but have
the correct next seq. no. Thus, receiver knows what
happened.
71
Recovering from Packet Loss
Packet is considered lost if it never arrives or arrives
after its scheduled playout time.
Sending packets over TCP may not viable:

◦ Retransmission => unacceptable end-to-end delay.
◦ Congestion control => transmission rate is reduced
Packet loss rate of 1% - 20% is tolerable (depending
how voice is encoded and transmitted).
Forward Error Correction and Interleaving can help conceal
loss packets.
If packet loss exceeds 10 - 20%, due to congestion...

72
36
4/14/16
Forward Error Correction
Redundancy is
introduced into the
stream to enable the
receiver to recover from
errors due to loss.
Forms of redundancy

RTP Packet
RTP Header
header extension
RTP payload
◦ Simple replication and retransmission of original
data.
◦ k-way XOR
◦ Replication, recoding, and
re-transmission of original
data.
FEC payload
73
FEC Basic Concept
f, e
Send
g, f
h, g
i, h
j, i
k, j
X
l, k
m, l
X
X
Receive
Display e
Queue d
c

e
d
g
f
e
g
f
i
h
g
j
i
h
j
i
j
m
l
Key issue: If a sample is lost, how do we ensure that the redundant
information necessary for the repair arrives?
◦ How much bandwidth should we dedicate to FEC?
◦ Where should we place the redundant information in the stream?
74
37
4/14/16
FEC Method 1: n-way XOR
Send a redundant encoded chunk after every n chunks.
Transmit the word-by-word XOR of groups of n chunks.
If a single packet is lost, receiver can fully reconstruct the
lost packet.
Example: 3-way XOR

10100011 ⊕ 10100011 ⊕ 00000011 = 00000011
Sample 1
Sample 2
Lost!
Sample 3
Redundant Code
10100011 ⊕ 00000011 ⊕ 00000011 = 10100011
Sample 1

Redundant Code
Sample 3
Reconstructed
Sample 2
n can’t be too large, higher prob. of packet loss and delay. n
can’t be too small, higher transmission rate.
75
FEC Method 2: Sending Lowbit-Rate Encoded Chunks
Can cope with non-consecutive packet loss.
Nominal stream
(e.g., PCM @ 64 kbps)
Low-bit-rate stream
(e.g., GSM @ 13 kbps)
Occasional low-bit-rate chunks in between high quality
give overall audio quality.
76
38
4/14/16
Interleaving

Divide 20 ms chunks into
four 5 ms units and
interleave.
Upon loss have a set of
partially filled chunks.
Multiple small gaps as
opposed to a single large gap!
Low overhead, but can
increase latency.
Maybe not great for Internet
phone, but streaming stored
audio or video.
77
Outline

Introduction
Wireless LANs
◦ 802.11
Physical layer
Mobility

◦ Issues
FEC
Interleaving

78
39
4/14/16
Introduction

Two layers of H.264:
◦ Video Coding Layer (VCL) – translates video data into
bitstream (discussed in Chapter 4).
◦ Network Abstraction Layer (NAL) – maps and
packetizes VCL bitstream into units for storage or
transmission.

NAL - “Network friendly” features:
◦ Error resiliency (encoder) – improves the decoder’s
capability to recover/conceal errors.
◦ Error concealment (decoder) – recovers from errors.
NAL Units
Sequence
Parameter
Set
Picture
Parameter
Set
79
Picture
Parameter
Set
80
40
4/14/16
Loss Resilience

H.264 adds Baseline and Extended Profiles to
provide error resiliency:
◦ Add redundancy to the bitstream.
◦ Facilitates a better error concealment (i.e., recover
lost parts of the picture).

Baseline Profile:
◦ Arbitrary Slice Ordering (ASO)
◦ Flexible Macroblock Ordering (FMO)
◦ Redundant Pictures or Slices

Extended Profile:
◦ Data Partitioning
81
Arbitrary Slice Order

Slices can be arranged in any order.
Aids in error concealment at the decoder.
No ASO
Received Order: 0, 1, 2, 3
Slice 0
ASO
Received Order: 0, 2, 1, 3
Slice 0
Predicted from Slice 0 & 2 Lost!
Slice 1
Slice 1
Slice 2
Lost!
Slice 2
Slice 3
Lost!
Slice 3
Predicted from Slice 2
Lost!
Errors tend to occur in spurts
82
41
4/14/16
Flexible Macro Ordering

A picture can be subdivided into regions, called
slice groups:
◦ Each slice group can contain one or more slices.

Each slice is transmitted independently in
separate packets:
◦ Within a slice, MBs are usually processed in raster
scan order

Using FMO, a frame can be split into many MB
scanning patterns that results in good error
robustness.
FMO together with advanced resilience tools
can maintain visual quality even with packet loss
rate of 10%!
83
7 FMO Types
Each color represents a slice group
Run-length
Grows\shrinks in
circular pattern
Regions of interests
Scattered
Row-wise
Column-wise
Type 6 is the most general and allows full flexibility to the user.
84
42
4/14/16
FMO Example

Type 1 (scattered or dispersed) is especially useful in
concealing errors in networks with high packet loss
rate.
Easier to conceal
Slice 1 lost!
Slice 0
Slice 1
Slice 2
No FMO (3 slices)
0
1
2
3
0
1
2
3
0
1
2
2
3
0
1
2
3
0
1
2
3
0
0
1
2
3
0
1
2
3
0
1
2
2
3
0
1
2
3
0
1
2
3
0
0
1
2
3
0
1
2
3
0
1
2
2
3
0
1
2
3
0
1
2
3
0
0
1
2
3
0
1
2
3
0
1
2
2
3
0
1
2
3
0
1
2
3
0
0
1
2
3
0
1
2
3
0
1
2
FMO Type 1 (4 slice groups)
85
Redundant Slices

Allows an encoder to send an extra
representation of a picture region (typically at
lower fidelity) that can be used if the primary
representation is corrupted or lost:
◦ Also referred to as Forward Error Correction (FEC).

May improve performance, but at a cost of
reduced compression.
86
43
4/14/16
Data Partitioning (DP)
Separate more important and less important syntax
elements.
Each compressed slice is partitioned into:

◦ Partition A - contains the most important information, such as
slice header, MB types, quantization parameters, and MVs.
◦ Partition B (intra partition) - contains intra-coded block pattern
(CBP) and coefficients.
◦ Partition C (inter partition) - contains only inter CBPs and inter
coefficients.

Partitions can be treated differently by the delivery
layer:
◦ Unequal Error Protection in PHY and MAC
◦ Prioritized or Guaranteed Services in MAC (e.g., 802.11e)
87
DP Illustration
Partition A
If lost, slice cannot
be reconstructed!
If lost, error
concealment can
be used
Partition B
(I- & SI-slice)
Partition C
(P-, B-, & SP-slice)
88
44
4/14/16
Error Concealment

Error resilience features of H.264 allows for
error concealment:
◦ Reconstruct lost information (within manageable
complexity).

Types of error concealment:
◦
◦
◦
◦

Spatial – Weighted Pixel Averaging, MV recovery
Temporal – Copy-Paste, MV recovery
Hybrid
Others…
Can be applied to both lost MBs and frames.
89
Error Characteristics

Depends on the severity and type of loss, error
resilience features used, and characteristics of video:
◦ I-slice – Intra prediction mode and residual
◦ P-slice – Motion vector and residual
Slices lost due
to packet loss
Sent frame
Received frame
Figure 9: Slices 2 and 4 lost.
Error Concealment
Error propagation 5 frames later
Figure 11: Chapter
Error propagation
three frames
later due to poor EC.
6: Multimedia
Networking
used
to conceal
lostfor
slices
and 4. work [10].
dia Player.Figure
It was 10:
alsoFrame
the thecopy
media
player
of choice
our 2previous
FFmpeg is a collection of libraries including libavcodec, which contains an H.264
decoder. The H.264 decoder contains an EC module, where different MV EC
instance, consider a car moving across a screen. Putting that single frame in
techniques can be found. The FFmpeg H.264 decoder applies spatial mean MV,
the context of neighboring frames adds a time dimension thus revealing whether
spatial median MV, last MV, and zero MV for P-frames [11]. Spatial mean MV
the car is stationary or if it is indeed moving in a particular direction. Such
simply takes the average of surrounding MVs to recover a lost MV while spatial
conclusions can be drawn from correlations witnessed within a single frame or in
median MV involves using the median of surrounding MVs instead. Using the
the context of multiple frames. The underlying concept of these correlations is
last MV for MV recovery involves copying the MV from the same spatial locausually motion. By exploiting the spatial and temporal relationships that exist
tion of the previous inter-predicted reference frame (usually a P-frame) and then
among MVs, it is possible to perform EC via MV recovery.
pasting in place of the lost MV. Zero MV simply uses a zero MV to estimate a
lost MV, and is also referred to as frame copy.
2.3Corrupted
ErrorB-frames
Concealment
Techniques
are handled
differently. Since B-MBs are bi-directionally
predicted,
each lost
MB in the
a B-frame
two MV used
estimates.
The decoders.
two MVs
This subsection
discusses
defaultrequires
EC techniques
by H.264
are
temporally
weighting
the
MVs
of a future
P-frame
as shown
Ourgenerated
discussionby
is based
on FFmpeg,
which
is the
underlying
software
of VLC
Mein Fig. 12. The first set of estimated MVs refers to a past reference frame. This
is generated by multiplying the MVs from the next P-frame by tt12 . The second
set of estimated MVs, which points backwards, is generated by multiplying the
1 t2
same MVs from the next P-frame by t11
t2 .
90
45
4/14/16
DP – Perceptual Quality
Original
Only residual
(Partition C)
is lost!
I-frame
w/o residual
P-frame
w/o residual
91
Copy-Paste

Temporal technique
Replaces the missing image part with the
spatially corresponding part inside a previously
decoded frame.
Very effective when there is very little
movement.
Can be applied to lost MBs or frames (called
frame copy).
Can be applied to I- or P-frames.
92
46
4/14/16
Copy-Paste - Perceptual
Quality
Original
Error
Copy-Paste
93
Weighted Pixel Averaging

Based on assumption of
continuity of natural scene
content in space
Spatial concealment technique. p
L
Used for I-frames:
pT
dL
dT
dR
p̂
pR
◦ Can also be used for P-frames

dB
Use pixel values of
surrounding available MBs
Disadvantage – blurred
reconstruction
pB
p̂ =
pT d B + pB dT + pL d R + pR d L
d B + dT + d R + d L
94
47
4/14/16
WPA - Perceptual Quality
Original
Error
Weighted
Pixel
Averaging
95
MV Recovery

For P- and B-frames:
◦ MV can be lost
◦ Residual (error) can be lost
◦ Or both can be lost

FFmpeg – decoder for VLC Media
Player:
Spatial technique
◦ Spatial - Left MV, Right MV, Top
MV, Bottom MV, Mean MV, Median
MV
◦ Temporal - Last MV, Zero MV
(frame copy)

Each estimated MV generates an
estimated MB, which is compared
against surrounding MBs using
Boundary Matching Algorithm
(BMA).
◦ The best one is used as the final
MB!
Substitute
MB
Reference
Frame
Missing
MB
Current
Frame
Temporal technique
96
48
4/14/16
Boundary Matching Algorithm
Calculates the total
absolute pixel value
difference between the
edge pixels of the
estimated MB and the
adjacent pixels of
surrounding MBs.
Choose the one with
minimum error.
MVs neighboring blocks

Median/Average MV
from neighboring blocks
Es timate
d
MB
Es timate
d
MB
Boundary Pixels
MV from previous frame
Es timate
d
MB
No MV
Es timate
d MB
97
MV Recovery - Perceptual
Quality
Original
Error
MV Recovery
98
49
4/14/16
Iterative Process of EC

Each MB in a lost region is concealed in the
order shown below.
Good slice
Lost slice
Good slice
Lost MB
Consider a simplified lost slice with good slices above and below it.
99
100
Questions?
50