In information theory, entropy is a measure of the uncertainty associated with a random variable. The term by itself in this context usually refers to the Shannon entropy, which quantifies, in the sense of an expected value, the information contained in a message, usually in units such as bits. Equivalently, the Shannon entropy is a measure of the average information content one is missing when one does not know the value of the random variable. The concept was introduced by Claude E. Shannon in his 1948 paper "A Mathematical Theory of Communication".

Shannon's entropy represents an absolute limit on the best possible lossless compression of any communication, under certain constraints: treating messages to be encoded as a sequence of independent and identically-distributed random variables, Shannon's source coding theorem shows that, in the limit, the average length of the shortest possible representation to encode the messages in a given alphabet is their entropy divided by the logarithm of the number of symbols in the target alphabet.

A fair coin has an entropy of one bit. However, if the coin is not fair, then the uncertainty is lower (if asked to bet on the next outcome, we would bet preferentially on the most frequent result), and thus the Shannon entropy is lower. Mathematically, a coin flip is an example of a Bernoulli trial, and its entropy is given by the binary entropy function. A long string of repeating characters has an entropy rate of 0, since every character is predictable. The entropy rate of English text is between 1.0 and 1.5 bits per letter,^[1] or as low as 0.6 to 1.3 bits per letter, according to estimates by Shannon based on human experiments.^[2]

• 1 Definition
• 2 Example
• 3 Rationale
• 4 Aspects
  • 4.1 Relationship to thermodynamic entropy
  • 4.2 Entropy as information content
  • 4.3 Data compression
  • 4.4 Limitations of entropy as information content
  • 4.5 Data as a Markov process
  • 4.6 b-ary entropy
• 5 Efficiency
• 6 Characterization
  • 6.1 Continuity
  • 6.2 Symmetry
  • 6.3 Maximum
  • 6.4 Additivity
• 7 Further properties
• 8 Extending discrete entropy to the continuous case: differential entropy
• 9 References
• 10 See also
• 11 External links

 

转载本文请联系原作者获取授权，同时请注明本文来自黄富强科学网博客。
链接地址：https://blog.sciencenet.cn/blog-89075-242441.html

上一篇：Trygve M. H. Reenskaug coordinates
下一篇：How to Design Programs