Entropy and Information Theory

Entropy and Information TheoryEntropy and Information Theory

ii

Entropy and Information Theory
Robert M. Gray Information Systems Laboratory Electrical Engineering Department Stanford University

Springer-Verlag New York

iv
A TEX and reproduced by Springer-Verlag This book was prepared with L from camera-ready copy supplied by the author.

c 1990 by Springer Verlag

v

to Tim, Lori, Julia, Peter, Gus, Amy Elizabeth, and Alice and in memory of Tino

vi

Contents
Prologue 1 Information Sources 1.1 Introduction . . . . . . . . . . . . . . . . . . 1.2 Probability Spaces and Random Variables . 1.3 Random Processes and Dynamical Systems 1.4 Distributions . . . . . . . . . . . . . . . . . 1.5 Standard Alphabets . . . . . . . . . . . . . 1.6 Expectation . . . . . . . . . . . . . . . . . . 1.7 Asymptotic Mean Stationar