Information Theory

Printed English carries between 0.6 and 1.3 bits of information per letter, although an alphabet of 26 letters and a space could carry 4.75. Claude Shannon measured that in 1951 by asking people to predict English text one letter at a time and recording how many guesses they needed. The gap is redundancy: a reader who knows the language can predict most of what comes next, and whatever can be predicted does not need to be transmitted.