Lecture 1 - Stanford University

ee376a:
Information Theory
winter 2015
Tsachy Weissman
today
•
what is information theory (high level)?
•
main topics
•
course goals and expectations
•
EE376a in the curriculum
•
course info
•
what is information theory, really?
what is information theory?
science of !
compression, !
storage, and !
transmission of information
main problems and course structure
• lossless compression • communication
• lossy compression • joint source-channel coding
problem i: lossless compression
problem ii: (reliable) communication
problem iii: lossy compression
?
Decoder (estimator)
Ed() ⇤ D
d : X ⇥ Xˆ ⇧ R+
⌃
problem iv: lossy communication (aka joint source channel coding)
mathematical objects and tools
• entropy • relative entropy • mutual information • chain rules • method of types
goals and expectations
acquaintance with:
• information measures: entropy, relative
entropy, mutual information • compression, storage, communication • “fundamental limits” • schemes for compression • “random coding” • “typical sequences” and interplay between
information, probability, and statistics
advanced information
theory courses:
audio, image, video
compression
- network information theory
- universal schemes
communication:
information theory
376A
statistical signal processing, denoising,
compressed sensing -digital -optical
-wireless
coding:
-algebraic -codes on graphs
course info
•
what you’ll be held “accountable” for
•
references
•
midterm + final
•
course staff
“accountability”
•
lectures: Tue&Thu 11:00am-12:15pm, Art 4
•
weekly homework sets:
Handed out every Thursday, and due the following
Thursday at 5pm (at 2nd floor of Packard dropbox).
First set will be handed out on Jan 15th.
main textbook
supporting texts (for further reading)
• R. B. Ash, “Information Theory”, Dover, 1990!
• T. Berger, “Rate Distortion Theory: A Mathematical
Basis for Data Compression”, Prentice Hall, 1971.!
• I. Csiszar and J. Korner, “Information Theory: Coding
Theorems for Discrete Memoryless Systems”,
Academic Press, 1981.!
• A. El Gamal and Y.H. Kim, “Network Information
Theory”, Cambridge University Press, 2011.!
• R.G. Gallager, “Information Theory and Reliable
Communication”, Wiley, 1968.
exams
•
midterm: Tuesday, Feb 10, 7pm-9pm
•
final: Monday, March 16, 3:30pm-6:30pm
•
material: see “accountability”
•
final grade: hw 20%, midterm 35%, final 45%
prerequisite
•
solid first course in probability !
•
maturity and motivation to cope with a few abstract
concepts
staff
•
Instructor: Tsachy Weissman, Packard 256
office hours: Thursdays 2:00 to 4:00pm or by appointment
[email protected]
TAs
Idoia Ochoa
[email protected]
Jiantao Jiao
[email protected]
supporting role:
Albert No
[email protected]
office hours: to be given shortly
Kartik Venkat
[email protected]
!
•
Admin: Doug Chaffee, Packard 258
[email protected]
•
Grader: Youngsuk Park
[email protected]
website
http://web.stanford.edu/class/ee376a/index.html
(please give us a couple more days before perusing)
questions?
what is information theory, really?
1948
C. E. Shannon,
“A Mathematical Theory of Communication,”
Bell Syst. Tech. J., vol. 27, pp. 379–423, 623–656, Jul.–Oct. 1948.
characterization of the “fundamental limits”
focus is on the “what” more than the “how”
2 types of results:
•
“converse theorems”
•
“direct theorems”
example I:
lossless compression of a ternary source
example II:
binary source + channel
example III:
lossy compression of a Gaussian
example IV:
communication over the AWGN channel