Inequalities in information theory

Inequalities are very important in the study of information theory. There are a number of different contexts in which these inequalities appear.

Entropic inequalities

Main article: Entropic vector

Consider a tuple ${\displaystyle X_{1},X_{2},\dots ,X_{n}}$ of ${\displaystyle n}$ finitely (or at most countably) supported random variables on the same probability space. There are 2ⁿ subsets, for which (joint) entropies can be computed. For example, when n = 2, we may consider the entropies ${\displaystyle H(X_{1}),}$ ${\displaystyle H(X_{2}),}$ and ${\displaystyle H(X_{1},X_{2}),}$. They satisfy the following inequalities (which together characterize the range of the marginal and joint entropies of two random variables):

• ${\displaystyle H(X_{1})\geq 0}$
• ${\displaystyle H(X_{2})\geq 0}$
• ${\displaystyle H(X_{1})\leq H(X_{1},X_{2})}$
• ${\displaystyle H(X_{2})\leq H(X_{1},X_{2})}$
• ${\displaystyle H(X_{1},X_{2})\leq H(X_{1})+H(X_{2}).}$

In fact, these can all be expressed as special cases of a single inequality involving the conditional mutual information, namely

${\displaystyle I(A;B|C)\geq 0,}$

where ${\displaystyle A}$, ${\displaystyle B}$, and ${\displaystyle C}$ each denote the joint distribution of some arbitrary (possibly empty) subset of our collection of random variables. Inequalities that can be derived as linear combinations of this are known as Shannon-type inequalities.

For larger ${\displaystyle n}$ there are further restrictions on possible values of entropy. To make this precise, a vector ${\displaystyle h}$ in ${\displaystyle \mathbb {R} ^{2^{n}}}$ indexed by subsets of ${\displaystyle \{1,\dots ,n\}}$ is said to be entropic if there is a joint, discrete distribution of n random variables ${\displaystyle X_{1},\dots ,X_{n}}$ such that ${\displaystyle h_{I}=H(X_{i}\colon i\in I)}$ is their joint entropy, for each subset ${\displaystyle I}$. The set of entropic vectors is denoted ${\displaystyle \Gamma _{n}^{*}}$, following the notation of Yeung [1]. It is not closed nor convex for ${\displaystyle n\geq 3}$, but it's topological closure ${\displaystyle {\overline {\Gamma _{n}^{*}}}}$ is known to be convex and hence it can be characterized by the (infinitely many) linear inequalities satisfied by all entropic vectors, called entropic inequalities.

The set of all vectors that satisfy Shannon-type inequalities (but not necessarily other entropic inequalities) contains ${\displaystyle {\overline {\Gamma _{n}^{*}}}}$. This containment is strict for ${\displaystyle n\geq 4}$ and further inequalities are known as non-Shannon type inequalities. Zhang and Yeung reported the first non-Shannon-type inequality.[2] Matus[3] proved that no finite set of inequalities can characterize (by linear combinations) all entropic inequalities. In other words, the region ${\displaystyle {\overline {\Gamma _{n}^{*}}}}$ is not a polytope.

Lower bounds for the Kullback–Leibler divergence

A great many important inequalities in information theory are actually lower bounds for the Kullback–Leibler divergence. Even the Shannon-type inequalities can be considered part of this category, since the bivariate mutual information can be expressed as the Kullback–Leibler divergence of the joint distribution with respect to the product of the marginals, and thus these inequalities can be seen as a special case of Gibbs' inequality.

On the other hand, it seems to be much more difficult to derive useful upper bounds for the Kullback–Leibler divergence. This is because the Kullback–Leibler divergence D_KL(P||Q) depends very sensitively on events that are very rare in the reference distribution Q. D_KL(P||Q) increases without bound as an event of finite non-zero probability in the distribution P becomes exceedingly rare in the reference distribution Q, and in fact D_KL(P||Q) is not even defined if an event of non-zero probability in P has zero probability in Q. (Hence the requirement that P be absolutely continuous with respect to Q.)

Gibbs' inequality

Main article: Gibbs' inequality

This fundamental inequality states that the Kullback–Leibler divergence is non-negative.

Kullback's inequality

Main article: Kullback's inequality

Another inequality concerning the Kullback–Leibler divergence is known as Kullback's inequality.[4] If P and Q are probability distributions on the real line with P absolutely continuous with respect to Q, and whose first moments exist, then

${\displaystyle D_{KL}(P\|Q)\geq \Psi _{Q}^{*}(\mu '_{1}(P)),}$

where ${\displaystyle \Psi _{Q}^{*}}$ is the large deviations rate function, i.e. the convex conjugate of the cumulant-generating function, of Q, and ${\displaystyle \mu '_{1}(P)}$ is the first moment of P.

The Cramér–Rao bound is a corollary of this result.

Pinsker's inequality

Main article: Pinsker's inequality

Pinsker's inequality relates Kullback–Leibler divergence and total variation distance. It states that if P, Q are two probability distributions, then

${\displaystyle {\sqrt {{\frac {1}{2}}D_{KL}^{(e)}(P\|Q)}}\geq \sup\{|P(A)-Q(A)|:A{\text{ is an event to which probabilities are assigned.}}\}.}$

where

${\displaystyle D_{KL}^{(e)}(P||Q)}$

is the Kullback–Leibler divergence in nats and

${\displaystyle \sup _{A}|P(A)-Q(A)|}$

is the total variation distance.

Other inequalities

Hirschman uncertainty

In 1957,[5] Hirschman showed that for a (reasonably well-behaved) function ${\displaystyle f:\mathbb {R} \rightarrow \mathbb {C} }$ such that ${\displaystyle \int _{-\infty }^{\infty }|f(x)|^{2}\,dx=1,}$ and its Fourier transform ${\displaystyle g(y)=\int _{-\infty }^{\infty }f(x)e^{-2\pi ixy}\,dx,}$ the sum of the differential entropies of ${\displaystyle |f|^{2}}$ and ${\displaystyle |g|^{2}}$ is non-negative, i.e.

${\displaystyle -\int _{-\infty }^{\infty }|f(x)|^{2}\log |f(x)|^{2}\,dx-\int _{-\infty }^{\infty }|g(y)|^{2}\log |g(y)|^{2}\,dy\geq 0.}$

Hirschman conjectured, and it was later proved,[6] that a sharper bound of ${\displaystyle \log(e/2),}$ which is attained in the case of a Gaussian distribution, could replace the right-hand side of this inequality. This is especially significant since it implies, and is stronger than, Weyl's formulation of Heisenberg's uncertainty principle.

Tao's inequality

Given discrete random variables ${\displaystyle X}$, ${\displaystyle Y}$, and ${\displaystyle Y'}$, such that ${\displaystyle X}$ takes values only in the interval [−1, 1] and ${\displaystyle Y'}$ is determined by ${\displaystyle Y}$ (such that ${\displaystyle H(Y'|Y)=0}$), we have[7][8]

${\displaystyle \mathbb {E} {\big (}{\big |}\mathbb {E} (X|Y')-\mathbb {E} (X|Y){\big |}{\big )}\leq {\sqrt {I(X;Y|Y')\,2\log 2}},}$

relating the conditional expectation to the conditional mutual information. This is a simple consequence of Pinsker's inequality. (Note: the correction factor log 2 inside the radical arises because we are measuring the conditional mutual information in bits rather than nats.)

See also

• Cramér–Rao bound
• Entropy power inequality
• Entropic vector
• Fano's inequality
• Jensen's inequality
• Kraft inequality
• Pinsker's inequality
• Multivariate mutual information

References

1. Yeung, R.W. (1997). "A framework for linear information inequalities". IEEE Transactions on Information Theory. 43 (6): 1924–1934. doi:10.1109/18.641556.)
2. Zhang, Z.; Yeung, R. W. (1998). "On characterization of entropy function via information inequalities". IEEE Transactions on Information Theory. 44 (4): 1440–1452. doi:10.1109/18.681320.
3. Matus, F. (2007). Infinitely many information inequalities. 2007 IEEE International Symposium on Information Theory.
4. Fuchs, Aimé; Letta, Giorgio (1970). L'inégalité de Kullback. Application à la théorie de l'estimation. Séminaire de Probabilités. Lecture Notes in Mathematics. 4. Strasbourg. pp. 108–131. doi:10.1007/bfb0059338. ISBN 978-3-540-04913-5. MR 0267669.
5. Hirschman, I. I. (1957). "A Note on Entropy". American Journal of Mathematics. 79 (1): 152–156. doi:10.2307/2372390. JSTOR 2372390.
6. Beckner, W. (1975). "Inequalities in Fourier Analysis". Annals of Mathematics. 102 (6): 159–182. doi:10.2307/1970980. JSTOR 1970980.
7. Tao, T. (2006). "Szemerédi's regularity lemma revisited". Contrib. Discrete Math. 1: 8–28. arXiv:math/0504472. Bibcode:2005math......4472T.
8. Ahlswede, Rudolf (2007). "The final form of Tao's inequality relating conditional expectation and conditional mutual information". Advances in Mathematics of Communications. 1 (2): 239–242. doi:10.3934/amc.2007.1.239.

External links

• Thomas M. Cover, Joy A. Thomas. Elements of Information Theory, Chapter 16, "Inequalities in Information Theory" John Wiley & Sons, Inc. 1991 Print ISBN 0-471-06259-6 Online ISBN 0-471-20061-1 pdf
• Amir Dembo, Thomas M. Cover, Joy A. Thomas. Information Theoretic Inequalities. IEEE Transactions on Information Theory, Vol. 37, No. 6, November 1991. pdf