Shannon Index

Shannon's entropy index, as the name suggests, is a measure of the entropy that applies to any numerical data. It measures the frequencies of appearance of the characters, and the more they are different, the more difficult it will be to predict the content (thus a greater uncertainty, and thus a greater entropy).

For a string of characters with \( n \) distinct characters, each character having a frequency of \( p \). The entropy of Shannon \( H \) is calculated according to the formula $$ H = -\sum_{i=1}^n p_i \log_2 p_i \\ \approx $$

Example: DCODE has 5 characters including 4 distinct characters, the letter D appears 2 times (frequency: 2/5), and the 3 letters C, O and E each appear 1 time (frequency: 1/5), the calculation is: \( H = -\left( \frac{2}{5} \log_2{\frac{2}{5}} + 3 \times \frac{1}{5} \log_2{\frac{1}{5}} \right) \approx 1.921928 \)

From the Shannon index, the optimal encoding of a string can be deduced. If the Shannon index of a string is 3.5, then it will take 4 bits (rounded up) by characters to encode it optimally.