Person Award 36431

abbas3
Abbas
El Gamal
Hitachi America Chair in the School of Engineering and Fortinet Founders Chair, Department of Electrical Engineering
Stanford University

Entropy

Definitions

Let \(X\) be a discrete random variable defined on a finite alphabet \(\mathcal{X}\) and with probability mass function \(p_X\). The entropy of \(X\) is the random variable \(H(X)\) defined by

\[ H(X)=\log\frac{1}{p_X(X)}.\] 

The base of the logarithm defines the unit of entropy. If the logarithm is to the base 2, the unit of entropy is the bit. If the if the logarithm is to the base \(e\), the unit of entropy is the nat.