Shannon's Experiment to Calculate the Entropy of English

Claude Shannon, the inventor of information theory, devised an experiment aimed at determining the entropy of an English letter (the amount of information in bits that we obtain on the average when we learn one letter of English). The experiment is carried out as follows. The user faces a sequence of dashes, each representing one of the 26 English letters or a space. The user is to guess the successive letters of the sentence using only the knowledge of the letters previously guessed. At each step we record the total number of guesses to reach the correct letter for that position. Shannon views the sequence of numbers of guesses as an encoding of the orginal sentence. He uses the entropy of the resulting random variable as an estimate for the entropy of an English letter.

The following applet allows you to simulate the Shannon experiment. Start typing on the keyboard your guesses for the first letter of the sentence. When you type the correct letter, the letter will appear in that position and the number of guesses will be displayed underneath it. Go on to guess the next letter in the same manner. You should press the space bar when you believe that a word has ended. Note that at the bottom of the Applet the letters of the alphabet and space appear in blue. When you guess a letter incorrectly, the computer helps you keep track of your guesses by erasing those letters from the alphabet.

When the sentence is complete or when you tire of guessing letters and click on Entropy, the computer will output the estimate of the entropy that results from this particular experiment. Perhaps we should mention that large scale experiments of this kind yield that the amount of information we gain when we learn a letter of English, within a sentence, is aproximately 1.1 bits. We should also mention that in a classroom of about 60 students, with everybody venturing guesses for each next letter, we consistently obtained a value of about 1.6 bits for the estimate of the entropy.



[Back| Home| Programs| Documentation| Internet| People]