Fisher information

In mathematical statistics, the Fisher information (sometimes simply called information[1]) is a way of measuring the amount of information that an observable random variable X carries about an unknown parameter θ of a distribution that models X. Formally, it is the variance of the score, or the expected value of the observed information. In Bayesian statistics, the asymptotic distribution of the posterior mode depends on the Fisher information and not on the prior (according to the Bernstein–von Mises theorem, which was anticipated by Laplace for exponential families).[2] The role of the Fisher information in the asymptotic theory of maximum-likelihood estimation was emphasized by the statistician Ronald Fisher (following some initial results by Francis Ysidro Edgeworth). The Fisher information is also used in the calculation of the Jeffreys prior, which is used in Bayesian statistics.

The Fisher-information matrix is used to calculate the covariance matrices associated with maximum-likelihood estimates. It can also be used in the formulation of test statistics, such as the Wald test.

Statistical systems of a scientific nature (physical, biological, etc.) whose likelihood functions obey shift invariance have been shown to obey maximum Fisher information.[3] The level of the maximum depends upon the nature of the system constraints.

Definition

The Fisher information is a way of measuring the amount of information that an observable random variable X carries about an unknown parameter θ upon which the probability of X depends. Let f(X; θ) be the probability density function (or probability mass function) for X conditional on the value of θ. It describes the probability that we observe a given outcome of X, given a known value of θ. If f is sharply peaked with respect to changes in θ, it is easy to indicate the “correct” value of θ from the data, or equivalently, that the data X provides a lot of information about the parameter θ. If the likelihood f is flat and spread-out, then it would take many samples of X to estimate the actual “true” value of θ that would be obtained using the entire population being sampled. This suggests studying some kind of variance with respect to θ.

Formally, the partial derivative with respect to θ of the natural logarithm of the likelihood function is called the score. Under certain regularity conditions, if θ is the true parameter (i.e. X is actually distributed as f(X; θ)), it can be shown that the expected value (the first moment) of the score is 0:[4]

${\displaystyle {\begin{aligned}&\operatorname {E} \left[\left.{\frac {\partial }{\partial \theta }}\log f(X;\theta )\right|\theta \right]\\[3pt]={}&\int {\frac {{\frac {\partial }{\partial \theta }}f(x;\theta )}{f(x;\theta )}}f(x;\theta )\,dx\\[3pt]={}&{\frac {\partial }{\partial \theta }}\int f(x;\theta )\,dx\\[3pt]={}&{\frac {\partial }{\partial \theta }}1=0.\end{aligned}}}$

The variance of the score is defined to be the Fisher information:[5]

${\displaystyle {\mathcal {I}}(\theta )=\operatorname {E} \left[\left.\left({\frac {\partial }{\partial \theta }}\log f(X;\theta )\right)^{2}\right|\theta \right]=\int \left({\frac {\partial }{\partial \theta }}\log f(x;\theta )\right)^{2}f(x;\theta )\,dx,}$

Note that ${\displaystyle 0\leq {\mathcal {I}}(\theta )}$. A random variable carrying high Fisher information implies that the absolute value of the score is often high. The Fisher information is not a function of a particular observation, as the random variable X has been averaged out.

If log f(x; θ) is twice differentiable with respect to θ, and under certain regularity conditions,[4] then the Fisher information may also be written as[6]

${\displaystyle {\mathcal {I}}(\theta )=-\operatorname {E} \left[\left.{\frac {\partial ^{2}}{\partial \theta ^{2}}}\log f(X;\theta )\right|\theta \right],}$

since

${\displaystyle {\frac {\partial ^{2}}{\partial \theta ^{2}}}\log f(X;\theta )={\frac {{\frac {\partial ^{2}}{\partial \theta ^{2}}}f(X;\theta )}{f(X;\theta )}}-\left({\frac {{\frac {\partial }{\partial \theta }}f(X;\theta )}{f(X;\theta )}}\right)^{2}={\frac {{\frac {\partial ^{2}}{\partial \theta ^{2}}}f(X;\theta )}{f(X;\theta )}}-\left({\frac {\partial }{\partial \theta }}\log f(X;\theta )\right)^{2}}$

and

${\displaystyle \operatorname {E} \left[\left.{\frac {{\frac {\partial ^{2}}{\partial \theta ^{2}}}f(X;\theta )}{f(X;\theta )}}\right|\theta \right]={\frac {\partial ^{2}}{\partial \theta ^{2}}}\int f(x;\theta )\,dx=0.}$

Thus, the Fisher information may be seen as the curvature of the support curve (the graph of the log-likelihood). Near the maximum likelihood estimate, low Fisher information therefore indicates that the maximum appears "blunt", that is, the maximum is shallow and there are many nearby values with a similar log-likelihood. Conversely, high Fisher information indicates that the maximum is sharp.

Discrepancy in definition

There exist two versions of the definition of Fisher information. Some books and notes define

${\displaystyle {\cal {I}}(\theta ):=\operatorname {E} \left[-{\frac {\partial ^{2}}{\partial \theta ^{2}}}\log f(X\mid \theta )\right]}$

where ${\displaystyle \log f(X\mid \theta )}$ is the log-likelihood for one observation, whereas others define

${\displaystyle {\cal {I}}(\theta ):=\operatorname {E} \left[-{\frac {\partial ^{2}}{\partial \theta ^{2}}}\ell (X\mid \theta )\right]}$ where ${\displaystyle \ell }$ is the log-likelihood function for all observations.

Some textbooks may even use the same symbol ${\displaystyle {\cal {I}}(\theta )}$ to denote both versions under different topics (for example, a book that defines ${\displaystyle {\cal {I}}(\theta )}$ to be the all-observation version when discussing Cramer–Rao lower bound and may still let the same symbol refer to the one-observation version when presenting the asymptotic normal distribution of maximum likelihood estimator). One should be careful with the meaning of ${\displaystyle {\cal {I}}(\theta )}$ in a specific context; however, if the data are i.i.d. the difference between two versions is simply a factor of ${\displaystyle n}$, the number of data points in the sample.

Informal derivation of the Cramér–Rao bound

The Cramér–Rao bound[7][8] states that the inverse of the Fisher information is a lower bound on the variance of any unbiased estimator of θ. H.L. Van Trees (1968) and B. Roy Frieden (2004) provide the following method of deriving the Cramér–Rao bound, a result which describes use of the Fisher information.

Informally, we begin by considering an unbiased estimator ${\displaystyle {\hat {\theta }}(X)}$. Mathematically, "unbiased" means that

${\displaystyle \operatorname {E} \left[\left.{\hat {\theta }}(X)-\theta \right|\theta \right]=\int \left({\hat {\theta }}(x)-\theta \right)\,f(x;\theta )\,dx=0{\text{ regardless of the value of }}\theta .}$

This expression is zero independent of θ, so its partial derivative with respect to θ must also be zero. By the product rule, this partial derivative is also equal to

${\displaystyle 0={\frac {\partial }{\partial \theta }}\int \left({\hat {\theta }}(x)-\theta \right)\,f(x;\theta )\,dx=\int \left({\hat {\theta }}(x)-\theta \right){\frac {\partial f}{\partial \theta }}\,dx-\int f\,dx.}$

For each θ, the likelihood function is a probability density function, and therefore ${\displaystyle \int f\,dx=1}$. A basic computation implies that

${\displaystyle {\frac {\partial f}{\partial \theta }}=f\,{\frac {\partial \log f}{\partial \theta }}.}$

Using these two facts in the above, we get

${\displaystyle \int \left({\hat {\theta }}-\theta \right)f\,{\frac {\partial \log f}{\partial \theta }}\,dx=1.}$

Factoring the integrand gives

${\displaystyle \int \left(\left({\hat {\theta }}-\theta \right){\sqrt {f}}\right)\left({\sqrt {f}}\,{\frac {\partial \log f}{\partial \theta }}\right)\,dx=1.}$

Squaring the expression in the integral, the Cauchy–Schwarz inequality yields

${\displaystyle 1={\biggl (}\int \left[\left({\hat {\theta }}-\theta \right){\sqrt {f}}\right]\cdot \left[{\sqrt {f}}\,{\frac {\partial \log f}{\partial \theta }}\right]\,dx{\biggr )}^{2}\leq \left[\int \left({\hat {\theta }}-\theta \right)^{2}f\,dx\right]\cdot \left[\int \left({\frac {\partial \log f}{\partial \theta }}\right)^{2}f\,dx\right].}$

The second bracketed factor is defined to be the Fisher Information, while the first bracketed factor is the expected mean-squared error of the estimator ${\displaystyle {\hat {\theta }}}$. By rearranging, the inequality tells us that

${\displaystyle \operatorname {Var} \left({\hat {\theta }}\right)\geq {\frac {1}{{\mathcal {I}}\left(\theta \right)}}.}$

In other words, the precision to which we can estimate θ is fundamentally limited by the Fisher information of the likelihood function.

Single-parameter Bernoulli experiment

A Bernoulli trial is a random variable with two possible outcomes, "success" and "failure", with success having a probability of θ. The outcome can be thought of as determined by a coin toss, with the probability of heads being θ and the probability of tails being 1 − θ.

Let X be a Bernoulli trial. The Fisher information contained in X may be calculated to be

${\displaystyle {\begin{aligned}{\mathcal {I}}(\theta )&=-\operatorname {E} \left[\left.{\frac {\partial ^{2}}{\partial \theta ^{2}}}\log \left(\theta ^{X}(1-\theta )^{1-X}\right)\right|\theta \right]\\[5pt]&=-\operatorname {E} \left[\left.{\frac {\partial ^{2}}{\partial \theta ^{2}}}\left(X\log \theta +(1-X)\log(1-\theta )\right)\right|\theta \right]\\[5pt]&=\operatorname {E} \left[\left.{\frac {X}{\theta ^{2}}}+{\frac {1-X}{(1-\theta )^{2}}}\right|\theta \right]\\[5pt]&={\frac {\theta }{\theta ^{2}}}+{\frac {1-\theta }{(1-\theta )^{2}}}\\[5pt]&={\frac {1}{\theta (1-\theta )}}.\end{aligned}}}$

Because Fisher information is additive, the Fisher information contained in n independent Bernoulli trials is therefore

${\displaystyle {\mathcal {I}}(\theta )={\frac {n}{\theta (1-\theta )}}.}$

This is the reciprocal of the variance of the mean number of successes in n Bernoulli trials, so in this case, the Cramér–Rao bound is an equality.

Matrix form

When there are N parameters, so that θ is an N × 1 vector ${\displaystyle \theta ={\begin{bmatrix}\theta _{1}&\theta _{2}&\dots &\theta _{N}\end{bmatrix}}^{\textsf {T}},}$ then the Fisher information takes the form of an N × N matrix. This matrix is called the Fisher information matrix (FIM) and has typical element

${\displaystyle {\bigl [}{\mathcal {I}}(\theta ){\bigr ]}_{i,j}=\operatorname {E} \left[\left.\left({\frac {\partial }{\partial \theta _{i}}}\log f(X;\theta )\right)\left({\frac {\partial }{\partial \theta _{j}}}\log f(X;\theta )\right)\right|\theta \right].}$

The FIM is a N × N positive semidefinite matrix. If it is positive definite, then it defines a Riemannian metric on the N-dimensional parameter space. The topic information geometry uses this to connect Fisher information to differential geometry, and in that context, this metric is known as the Fisher information metric.

Under certain regularity conditions, the Fisher information matrix may also be written as

${\displaystyle {\bigl [}{\mathcal {I}}(\theta ){\bigr ]}_{i,j}=-\operatorname {E} \left[\left.{\frac {\partial ^{2}}{\partial \theta _{i}\,\partial \theta _{j}}}\log f(X;\theta )\right|\theta \right]\,.}$

The result is interesting in several ways:

• It can be derived as the Hessian of the relative entropy.
• It can be understood as a metric induced from the Euclidean metric, after appropriate change of variable.
• In its complex-valued form, it is the Fubini–Study metric.
• It is the key part of the proof of Wilks’ theorem, which allows confidence region estimates for maximum likelihood estimation (for those conditions for which it applies) without needing the Likelihood Principle.
• In cases where the analytical calculations of the FIM above are difficult, it is possible to form an average of easy Monte Carlo estimates of the Hessian of the negative log-likelihood function as an estimate of the FIM.[9][10][11] The estimates may be based on values of the negative log-likelihood function or the gradient of the negative log-likelihood function; no analytical calculation of the Hessian of the negative log-likelihood function is needed.

Orthogonal parameters

We say that two parameters θ_i and θ_j are orthogonal if the element of the ith row and jth column of the Fisher information matrix is zero. Orthogonal parameters are easy to deal with in the sense that their maximum likelihood estimates are independent and can be calculated separately. When dealing with research problems, it is very common for the researcher to invest some time searching for an orthogonal parametrization of the densities involved in the problem.

Singular statistical model

If the Fisher information matrix is positive definite for all θ, then the corresponding statistical model is said to be regular; otherwise, the statistical model is said to be singular.[12] Examples of singular statistical models include the following: normal mixtures, binomial mixtures, multinomial mixtures, Bayesian networks, neural networks, radial basis functions, hidden Markov models, stochastic context-free grammars, reduced rank regressions, Boltzmann machines.

In machine learning, if a statistical model is devised so that it extracts hidden structure from a random phenomenon, then it naturally becomes singular.[13]

Multivariate normal distribution

The FIM for a N-variate multivariate normal distribution, ${\displaystyle \,X\sim N\left(\mu (\theta ),\,\Sigma (\theta )\right)}$ has a special form. Let the K-dimensional vector of parameters be ${\displaystyle \theta ={\begin{bmatrix}\theta _{1}&\dots &\theta _{K}\end{bmatrix}}^{\textsf {T}}}$ and the vector of random normal variables be ${\displaystyle X={\begin{bmatrix}X_{1}&\dots &X_{N}\end{bmatrix}}^{\textsf {T}}}$. Assume that the mean values of these random variables are ${\displaystyle \,\mu (\theta )={\begin{bmatrix}\mu _{1}(\theta )&\dots &\mu _{N}(\theta )\end{bmatrix}}^{\textsf {T}}}$, and let ${\displaystyle \,\Sigma (\theta )}$ be the covariance matrix. Then, for ${\displaystyle 1\leq m,\,n\leq K}$, the (m, n) entry of the FIM is:[14]

${\displaystyle {\mathcal {I}}_{m,n}={\frac {\partial \mu ^{\textsf {T}}}{\partial \theta _{m}}}\Sigma ^{-1}{\frac {\partial \mu }{\partial \theta _{n}}}+{\frac {1}{2}}\operatorname {tr} \left(\Sigma ^{-1}{\frac {\partial \Sigma }{\partial \theta _{m}}}\Sigma ^{-1}{\frac {\partial \Sigma }{\partial \theta _{n}}}\right),}$

where ${\displaystyle (\cdot )^{\textsf {T}}}$ denotes the transpose of a vector, tr(·) denotes the trace of a square matrix, and:

${\displaystyle {\begin{aligned}{\frac {\partial \mu }{\partial \theta _{m}}}&={\begin{bmatrix}{\frac {\partial \mu _{1}}{\partial \theta _{m}}}&{\frac {\partial \mu _{2}}{\partial \theta _{m}}}&\cdots &{\frac {\partial \mu _{N}}{\partial \theta _{m}}}\end{bmatrix}}^{\textsf {T}};\\{\frac {\partial \Sigma }{\partial \theta _{m}}}&={\begin{bmatrix}{\frac {\partial \Sigma _{1,1}}{\partial \theta _{m}}}&{\frac {\partial \Sigma _{1,2}}{\partial \theta _{m}}}&\cdots &{\frac {\partial \Sigma _{1,N}}{\partial \theta _{m}}}\\[5pt]{\frac {\partial \Sigma _{2,1}}{\partial \theta _{m}}}&{\frac {\partial \Sigma _{2,2}}{\partial \theta _{m}}}&\cdots &{\frac {\partial \Sigma _{2,N}}{\partial \theta _{m}}}\\\vdots &\vdots &\ddots &\vdots \\{\frac {\partial \Sigma _{N,1}}{\partial \theta _{m}}}&{\frac {\partial \Sigma _{N,2}}{\partial \theta _{m}}}&\cdots &{\frac {\partial \Sigma _{N,N}}{\partial \theta _{m}}}\end{bmatrix}}.\end{aligned}}}$

Note that a special, but very common, case is the one where ${\displaystyle \Sigma (\theta )=\Sigma }$, a constant. Then

${\displaystyle {\mathcal {I}}_{m,n}={\frac {\partial \mu ^{\textsf {T}}}{\partial \theta _{m}}}\Sigma ^{-1}{\frac {\partial \mu }{\partial \theta _{n}}}.\ }$

In this case the Fisher information matrix may be identified with the coefficient matrix of the normal equations of least squares estimation theory.

Another special case occurs when the mean and covariance depend on two different vector parameters, say, β and θ. This is especially popular in the analysis of spatial data, which often uses a linear model with correlated residuals. In this case,[15]

${\displaystyle {\mathcal {I}}(\beta ,\theta )=\operatorname {diag} \left({\mathcal {I}}(\beta ),{\mathcal {I}}(\theta )\right)}$

where

${\displaystyle {\begin{aligned}{\mathcal {I}}{(\beta )_{m,n}}&={\frac {\partial \mu ^{\textsf {T}}}{\partial \beta _{m}}}\Sigma ^{-1}{\frac {\partial \mu }{\partial \beta _{n}}},\\[5pt]{\mathcal {I}}{(\theta )_{m,n}}&={\frac {1}{2}}\operatorname {tr} \left(\Sigma ^{-1}{\frac {\partial \Sigma }{\partial \theta _{m}}}{\Sigma ^{-1}}{\frac {\partial \Sigma }{\partial \theta _{n}}}\right)\end{aligned}}}$

Properties

Chain rule

Similar to the entropy or mutual information, the Fisher information also possesses a chain rule decomposition. In particular, if X and Y are jointly distributed random variables, it follows that:[16]

${\displaystyle {\mathcal {I}}_{X,Y}(\theta )={\mathcal {I}}_{X}(\theta )+{\mathcal {I}}_{Y\mid X}(\theta ),}$

where ${\displaystyle {\mathcal {I}}_{Y\mid X}(\theta )}$ is the Fisher information of Y relative to ${\displaystyle \theta }$ calculated with respect to the conditional density of Y given a specific value X = x.

As a special case, if the two random variables are independent, the information yielded by the two random variables is the sum of the information from each random variable separately:

${\displaystyle {\mathcal {I}}_{X,Y}(\theta )={\mathcal {I}}_{X}(\theta )+{\mathcal {I}}_{Y}(\theta ).}$

Consequently, the information in a random sample of n independent and identically distributed observations is n times the information in a sample of size 1.

Sufficient statistic

The information provided by a sufficient statistic is the same as that of the sample X. This may be seen by using Neyman's factorization criterion for a sufficient statistic. If T(X) is sufficient for θ, then

${\displaystyle f(X;\theta )=g(T(X),\theta )h(X)}$

for some functions g and h. The independence of h(X) from θ implies

${\displaystyle {\frac {\partial }{\partial \theta }}\log \left[f(X;\theta )\right]={\frac {\partial }{\partial \theta }}\log \left[g(T(X);\theta )\right],}$

and the equality of information then follows from the definition of Fisher information. More generally, if T = t(X) is a statistic, then

${\displaystyle {\mathcal {I}}_{T}(\theta )\leq {\mathcal {I}}_{X}(\theta )}$

with equality if and only if T is a sufficient statistic.

Reparametrization

The Fisher information depends on the parametrization of the problem. If θ and η are two scalar parametrizations of an estimation problem, and θ is a continuously differentiable function of η, then

${\displaystyle {\mathcal {I}}_{\eta }(\eta )={\mathcal {I}}_{\theta }(\theta (\eta ))\left({\frac {d\theta }{d\eta }}\right)^{2}}$

where ${\displaystyle {\mathcal {I}}_{\eta }}$ and ${\displaystyle {\mathcal {I}}_{\theta }}$ are the Fisher information measures of η and θ, respectively.[17]

In the vector case, suppose ${\displaystyle {\boldsymbol {\theta }}}$ and ${\displaystyle {\boldsymbol {\eta }}}$ are k-vectors which parametrize an estimation problem, and suppose that ${\displaystyle {\boldsymbol {\theta }}}$ is a continuously differentiable function of ${\displaystyle {\boldsymbol {\eta }}}$, then,[18]

${\displaystyle {\mathcal {I}}_{\boldsymbol {\eta }}({\boldsymbol {\eta }})={\boldsymbol {J}}^{\textsf {T}}{\mathcal {I}}_{\boldsymbol {\theta }}({\boldsymbol {\theta }}({\boldsymbol {\eta }})){\boldsymbol {J}}}$

where the (i, j)th element of the k × k Jacobian matrix ${\displaystyle {\boldsymbol {J}}}$ is defined by

${\displaystyle J_{ij}={\frac {\partial \theta _{i}}{\partial \eta _{j}}},}$

and where ${\displaystyle {\boldsymbol {J}}^{\textsf {T}}}$ is the matrix transpose of ${\displaystyle {\boldsymbol {J}}.}$

In information geometry, this is seen as a change of coordinates on a Riemannian manifold, and the intrinsic properties of curvature are unchanged under different parametrization. In general, the Fisher information matrix provides a Riemannian metric (more precisely, the Fisher–Rao metric) for the manifold of thermodynamic states, and can be used as an information-geometric complexity measure for a classification of phase transitions, e.g., the scalar curvature of the thermodynamic metric tensor diverges at (and only at) a phase transition point.[19]

In the thermodynamic context, the Fisher information matrix is directly related to the rate of change in the corresponding order parameters.[20] In particular, such relations identify second-order phase transitions via divergences of individual elements of the Fisher information matrix.

Applications

Optimal design of experiments

Fisher information is widely used in optimal experimental design. Because of the reciprocity of estimator-variance and Fisher information, minimizing the variance corresponds to maximizing the information.

When the linear (or linearized) statistical model has several parameters, the mean of the parameter estimator is a vector and its variance is a matrix. The inverse of the variance matrix is called the "information matrix". Because the variance of the estimator of a parameter vector is a matrix, the problem of "minimizing the variance" is complicated. Using statistical theory, statisticians compress the information-matrix using real-valued summary statistics; being real-valued functions, these "information criteria" can be maximized.

Traditionally, statisticians have evaluated estimators and designs by considering some summary statistic of the covariance matrix (of an unbiased estimator), usually with positive real values (like the determinant or matrix trace). Working with positive real numbers brings several advantages: If the estimator of a single parameter has a positive variance, then the variance and the Fisher information are both positive real numbers; hence they are members of the convex cone of nonnegative real numbers (whose nonzero members have reciprocals in this same cone).

For several parameters, the covariance matrices and information matrices are elements of the convex cone of nonnegative-definite symmetric matrices in a partially ordered vector space, under the Loewner (Löwner) order. This cone is closed under matrix addition and inversion, as well as under the multiplication of positive real numbers and matrices. An exposition of matrix theory and Loewner order appears in Pukelsheim.[21]

The traditional optimality criteria are the information matrix's invariants, in the sense of invariant theory; algebraically, the traditional optimality criteria are functionals of the eigenvalues of the (Fisher) information matrix (see optimal design).

Jeffreys prior in Bayesian statistics

In Bayesian statistics, the Fisher information is used to calculate the Jeffreys prior, which is a standard, non-informative prior for continuous distribution parameters.[22]

Computational neuroscience

The Fisher information has been used to find bounds on the accuracy of neural codes. In that case, X is typically the joint responses of many neurons representing a low dimensional variable θ (such as a stimulus parameter). In particular the role of correlations in the noise of the neural responses has been studied.[23]

Derivation of physical laws

Fisher information plays a central role in a controversial principle put forward by Frieden as the basis of physical laws, a claim that has been disputed.[24]

Machine learning

The Fisher information is used in machine learning techniques such as elastic weight consolidation,[25] which reduces catastrophic forgetting in artificial neural networks.

Relation to relative entropy

See also: Fisher information metric

Fisher information is related to relative entropy.[26] Consider a family of probability distributions ${\displaystyle f(x;\theta )}$ where ${\displaystyle \theta }$ is a parameter which lies in a range of values. Then the relative entropy, or Kullback–Leibler divergence, between two distributions in the family can be written as

${\displaystyle D(\theta \parallel \theta ')=\int f(x;\theta )\log {\frac {f(x;\theta )}{f(x;\theta ')}}dx=\int f(x;\theta )\left(\log f(x;\theta )-\log f(x;\theta ')\right)dx,}$

while the Fisher information matrix is:

${\displaystyle KL(p:q)=\int p(x)\log {\frac {p(x)}{q(x)}}d\mu (x).}$

If ${\displaystyle \theta }$ is fixed, then the relative entropy between two distributions of the same family is minimized at ${\displaystyle \theta '=\theta }$. For ${\displaystyle \theta '}$ close to ${\displaystyle \theta }$, one may expand the previous expression in a series up to second order:

${\displaystyle D(\theta \parallel \theta ')={\frac {1}{2}}(\theta '-\theta )^{\textsf {T}}\underbrace {\left({\frac {\partial ^{2}}{\partial \theta '_{i}\,\partial \theta '_{j}}}D(\theta \parallel \theta ')\right)_{\theta '=\theta }} _{\text{Fisher information}}(\theta '-\theta )+\cdots }$

Thus the Fisher information represents the curvature of the relative entropy.

${\displaystyle I_{ij}(\theta )=E_{p_{\theta }}\left[{\frac {\partial }{\partial \theta _{i}}}\log p_{\theta }(x){\frac {\partial }{\partial \theta _{j}}}\log p_{\theta }(x)\right].}$

Schervish (1995: §2.3) says the following.

One advantage Kullback-Leibler information has over Fisher information is that it is not affected by changes in parameterization. Another advantage is that Kullback-Leibler information can be used even if the distributions under consideration are not all members of a parametric family.

...

Another advantage to Kullback-Leibler information is that no smoothness conditions on the densities … are needed.

History

The Fisher information was discussed by several early statisticians, notably F. Y. Edgeworth.[27] For example, Savage[28] says: "In it [Fisher information], he [Fisher] was to some extent anticipated (Edgeworth 1908–9 esp. 502, 507–8, 662, 677–8, 82–5 and references he [Edgeworth] cites including Pearson and Filon 1898 [. . .])." There are a number of early historical sources[29] and a number of reviews of this early work.[30][31][32]

See also

• Efficiency (statistics)
• Observed information
• Fisher information metric
• Formation matrix
• Information geometry
• Jeffreys prior
• Cramér–Rao bound
• Minimum Fisher information

Other measures employed in information theory:

• Entropy (information theory)
• Kullback–Leibler divergence
• Self-information

Notes

1. Lehmann & Casella, p. 115
2. Lucien Le Cam (1986) Asymptotic Methods in Statistical Decision Theory: Pages 336 and 618–621 (von Mises and Bernstein).
3. Frieden & Gatenby (2013)
4. Suba Rao. "Lectures on statistical inference" (PDF).
5. Fisher (1922)
6. Lehmann & Casella, eq. (2.5.16), Lemma 5.3, p.116.
7. Cramer (1946)
8. Rao (1945)
9. Spall, J. C. (2005). "Monte Carlo Computation of the Fisher Information Matrix in Nonstandard Settings". Journal of Computational and Graphical Statistics. 14 (4): 889–909. doi:10.1198/106186005X78800.
10. Spall, J. C. (2008), "Improved Methods for Monte Carlo Estimation of the Fisher Information Matrix," Proceedings of the American Control Conference, Seattle, WA, 11–13 June 2008, pp. 2395–2400. https://doi.org/10.1109/ACC.2008.4586850
11. Das, S.; Spall, J. C.; Ghanem, R. (2010). "Efficient Monte Carlo Computation of Fisher Information Matrix Using Prior Information". Computational Statistics and Data Analysis. 54 (2): 272–289. doi:10.1016/j.csda.2009.09.018.
12. Watanabe, S. (2008), Accardi, L.; Freudenberg, W.; Ohya, M. (eds.), "Algebraic geometrical method in singular statistical estimation", Quantum Bio-Informatics, World Scientific: 325–336, Bibcode:2008qbi..conf..325W, doi:10.1142/9789812793171_0024, ISBN 978-981-279-316-4.
13. Watanabe, S (2013). "A Widely Applicable Bayesian Information Criterion". Journal of Machine Learning Research. 14: 867–897.
14. Malagò, Luigi; Pistone, Giovanni (2015). Information geometry of the Gaussian distribution in view of stochastic optimization. Proceedings of the 2015 ACM Conference on Foundations of Genetic Algorithms XIII. pp. 150–162. doi:10.1145/2725494.2725510. ISBN 9781450334341.
15. Mardia, K. V.; Marshall, R. J. (1984). "Maximum likelihood estimation of models for residual covariance in spatial regression". Biometrika. 71 (1): 135–46. doi:10.1093/biomet/71.1.135.
16. Zamir, R. (1998). "A proof of the Fisher information inequality via a data processing argument". IEEE Transactions on Information Theory. 44 (3): 1246–1250. CiteSeerX 10.1.1.49.6628. doi:10.1109/18.669301.
17. Lehmann & Casella, eq. (2.5.11).
18. Lehmann & Casella, eq. (2.6.16)
19. Janke, W.; Johnston, D. A.; Kenna, R. (2004). "Information Geometry and Phase Transitions". Physica A. 336 (1–2): 181. arXiv:cond-mat/0401092. Bibcode:2004PhyA..336..181J. doi:10.1016/j.physa.2004.01.023.
20. Prokopenko, M.; Lizier, Joseph T.; Lizier, J. T.; Obst, O.; Wang, X. R. (2011). "Relating Fisher information to order parameters". Physical Review E. 84 (4): 041116. Bibcode:2011PhRvE..84d1116P. doi:10.1103/PhysRevE.84.041116. PMID 22181096. S2CID 18366894.
21. Pukelsheim, Friedrick (1993). Optimal Design of Experiments. New York: Wiley. ISBN 978-0-471-61971-0.
22. Bernardo, Jose M.; Smith, Adrian F. M. (1994). Bayesian Theory. New York: John Wiley & Sons. ISBN 978-0-471-92416-6.
23. Abbott, Larry F.; Dayan, Peter (1999). "The effect of correlated variability on the accuracy of a population code". Neural Computation. 11 (1): 91–101. doi:10.1162/089976699300016827. PMID 9950724.
24. Streater, R. F. (2007). Lost Causes in and beyond Physics. Springer. p. 69. ISBN 978-3-540-36581-5.
25. Kirkpatrick, James; Pascanu, Razvan; Rabinowitz, Neil; Veness, Joel; Desjardins, Guillaume; Rusu, Andrei A.; Milan, Kieran; Quan, John; Ramalho, Tiago (2017-03-28). "Overcoming catastrophic forgetting in neural networks". Proceedings of the National Academy of Sciences. 114 (13): 3521–3526. doi:10.1073/pnas.1611835114. ISSN 0027-8424. PMC 5380101. PMID 28292907.
26. Gourieroux & Montfort (1995), page 87
27. Savage (1976)
28. Savage(1976), page 156
29. Edgeworth (September 1908, December 1908)
30. Pratt (1976)
31. Stigler (1978, 1986, 1999)
32. Hald (1998, 1999)

References

• Cramer, Harald (1946). Mathematical methods of statistics. Princeton mathematical series. Princeton: Princeton University Press. ISBN 0691080046.

• Edgeworth, F. Y. (Jun 1908). "On the Probable Errors of Frequency-Constants". Journal of the Royal Statistical Society. 71 (2): 381–397. doi:10.2307/2339461. JSTOR 2339461.
• Edgeworth, F. Y. (Sep 1908). "On the Probable Errors of Frequency-Constants (Contd.)". Journal of the Royal Statistical Society. 71 (3): 499–512. doi:10.2307/2339293. JSTOR 2339293.
• Edgeworth, F. Y. (Dec 1908). "On the Probable Errors of Frequency-Constants (Contd.)". Journal of the Royal Statistical Society. 71 (4): 651–678. doi:10.2307/2339378. JSTOR 2339378.

• Fisher, R. A. (1922-01-01). "On the mathematical foundations of theoretical statistics". Philosophical Transactions of the Royal Society of London. A. 222 (594–604): 309–368. doi:10.1098/rsta.1922.0009. Retrieved 2020-08-12.

• Frieden, B. R. (2004) Science from Fisher Information: A Unification. Cambridge Univ. Press. ISBN 0-521-00911-1.
• Frieden, B. Roy; Gatenby, Robert A. (2013). "Principle of maximum Fisher information from Hardy's axioms applied to statistical systems". Physical Review E. 88 (4): 042144. arXiv:1405.0007. Bibcode:2013PhRvE..88d2144F. doi:10.1103/PhysRevE.88.042144. PMC 4010149. PMID 24229152.
• Hald, A. (May 1999). "On the History of Maximum Likelihood in Relation to Inverse Probability and Least Squares". Statistical Science. 14 (2): 214–222. doi:10.1214/ss/1009212248. JSTOR 2676741.
• Hald, A. (1998). A History of Mathematical Statistics from 1750 to 1930. New York: Wiley. ISBN 978-0-471-17912-2.
• Lehmann, E. L.; Casella, G. (1998). Theory of Point Estimation (2nd ed.). Springer. ISBN 978-0-387-98502-2.
• Le Cam, Lucien (1986). Asymptotic Methods in Statistical Decision Theory. Springer-Verlag. ISBN 978-0-387-96307-5.
• Pratt, John W. (May 1976). "F. Y. Edgeworth and R. A. Fisher on the Efficiency of Maximum Likelihood Estimation". Annals of Statistics. 4 (3): 501–514. doi:10.1214/aos/1176343457. JSTOR 2958222.

• Rao, C. Radhakrishna (1945). "Information and accuracy attainable in the estimation of statistical parameters". Bulletin of the Calcutta Mathematical Society. 37: 81–91. doi:10.1007/978-1-4612-0919-5_16.

• Savage, L. J. (May 1976). "On Rereading R. A. Fisher". Annals of Statistics. 4 (3): 441–500. doi:10.1214/aos/1176343456. JSTOR 2958221.
• Schervish, Mark J. (1995). Theory of Statistics. New York: Springer. ISBN 978-0-387-94546-0.
• Stigler, S. M. (1986). The History of Statistics: The Measurement of Uncertainty before 1900. Harvard University Press. ISBN 978-0-674-40340-6.
• Stigler, S. M. (1978). "Francis Ysidro Edgeworth, Statistician". Journal of the Royal Statistical Society, Series A. 141 (3): 287–322. doi:10.2307/2344804. JSTOR 2344804.
• Stigler, S. M. (1999). Statistics on the Table: The History of Statistical Concepts and Methods. Harvard University Press. ISBN 978-0-674-83601-3.
• Van Trees, H. L. (1968). Detection, Estimation, and Modulation Theory, Part I. New York: Wiley. ISBN 978-0-471-09517-0.