Holm–Bonferroni method

In statistics, the Holm–Bonferroni method,[1] also called the Holm method or Bonferroni–Holm method, is used to counteract the problem of multiple comparisons. It is intended to control the family-wise error rate and offers a simple test uniformly more powerful than the Bonferroni correction. It is named after Sture Holm, who codified the method, and Carlo Emilio Bonferroni.

Motivation

When considering several hypotheses, the problem of multiplicity arises: the more hypotheses are checked, the higher the probability of obtaining Type I errors (false positives). The Holm–Bonferroni method is one of many approaches for controlling the family-wise error rate (probability that one or more Type I errors will occur) by adjusting the rejection criteria for each of the individual hypotheses.

Formulation

The method is as follows:

• Let ${\displaystyle H_{1},\ldots ,H_{m}}$ be a family of ${\displaystyle m}$ null hypotheses and ${\displaystyle P_{1},\ldots ,P_{m}}$ the corresponding p-values.
• Start by ordering the p-values (from lowest to highest) ${\displaystyle P_{(1)}\ldots P_{(m)}}$ and let the associated hypotheses be ${\displaystyle H_{(1)},\ldots ,H_{(m)}}$
• For a given significance level ${\displaystyle \alpha }$, let ${\displaystyle k}$ be the minimal index such that ${\displaystyle P_{(k)}>{\frac {\alpha }{m+1-k}}}$
• Reject the null hypotheses ${\displaystyle H_{(1)},\ldots ,H_{(k-1)}}$ and do not reject ${\displaystyle H_{(k)},\ldots ,H_{(m)}}$
• If ${\displaystyle k=1}$ then do not reject any of the null hypotheses and if no such ${\displaystyle k}$ exist then reject all of the null hypotheses.

This method ensures that ${\displaystyle {\text{FWER}}\leq \alpha }$, where FWER is the family-wise error rate.

Rationale

The simple Bonferroni correction rejects only null hypotheses with p-value less than ${\displaystyle {\frac {\alpha }{m}}}$, in order to ensure that the risk of rejecting one or more true null hypotheses (i.e., of committing one or more type I errors) is at most ${\displaystyle \alpha }$. The cost of this protection against type I errors is an increased risk of failing to reject one or more false null hypotheses (i.e., of committing one or more type II errors).

The Holm–Bonferroni method also controls the maximum family-wise error rate at ${\displaystyle \alpha }$, but with a lower increase of type II error risk than the classical Bonferroni method. The Holm–Bonferroni method sorts the p-values from lowest to highest and compares them to nominal alpha levels of ${\displaystyle {\frac {\alpha }{m}}}$ to ${\displaystyle \alpha }$ (respectively), namely the values ${\displaystyle {\frac {\alpha }{m}},{\frac {\alpha }{m-1}},\ldots ,{\frac {\alpha }{2}},{\frac {\alpha }{1}}}$.

• The index ${\displaystyle k}$ identifies the first p-value that is not low enough to validate rejection. Therefore, the null hypotheses ${\displaystyle H_{(1)},\ldots ,H_{(k-1)}}$ are rejected, while the null hypotheses ${\displaystyle H_{(k)},...,H_{(m)}}$ are accepted (not rejected).
• If ${\displaystyle k=1}$ then no p-values were low enough for rejection, therefore no null hypotheses are rejected (i.e., all null hypotheses are accepted).
• If no such index ${\displaystyle k}$ could be found then all p-values were low enough for rejection, therefore all null hypotheses are rejected (none are accepted).

Proof

Holm–Bonferroni controls the FWER as follows. Let ${\displaystyle H_{(1)}\ldots H_{(m)}}$ be a family of hypotheses, and ${\displaystyle P_{(1)}\leq P_{(2)}\leq \cdots \leq P_{(m)}}$ be the sorted p-values. Let ${\displaystyle I_{0}}$ be the set of indices corresponding to the (unknown) true null hypotheses, having ${\displaystyle m_{0}}$ members.

Let us assume that we wrongly reject a true hypothesis. We have to prove that the probability of this event is at most ${\displaystyle \alpha }$. Let ${\displaystyle h}$ be the first rejected true hypothesis (first in the ordering given by the Bonferroni–Holm test). Then ${\displaystyle H_{(1)},\ldots ,H_{(h-1)}}$are all rejected false hypotheses and ${\displaystyle h-1\leq m-m_{0}}$. From there, we get ${\displaystyle {\frac {1}{m-h+1}}\leq {\frac {1}{m_{0}}}}$ (1). Since ${\displaystyle h}$ is rejected we have ${\displaystyle P_{(h)}\leq {\frac {\alpha }{m-h+1}}}$ by definition of the test. Using (1), the right hand side is at most ${\displaystyle {\frac {\alpha }{m_{0}}}}$. Thus, if we wrongly reject a true hypothesis, there has to be a true hypothesis with P-value at most ${\displaystyle {\frac {\alpha }{m_{0}}}}$.

So let us define the random variable ${\displaystyle A=\left\{P_{i}\leq {\frac {\alpha }{m_{0}}}{\text{ for }}i\in I_{0}\right\}}$. Whatever the (unknown) set of true hypotheses ${\displaystyle I_{0}}$ is, we have ${\displaystyle \Pr(A)\leq \alpha }$ (by the Bonferroni inequalities). Therefore, the probability to reject a true hypothesis is at most ${\displaystyle \alpha }$.

Alternative proof

The Holm–Bonferroni method can be viewed as closed testing procedure,[2] with Bonferroni method applied locally on each of the intersections of null hypotheses. As such, it controls the family-wise error rate for all the k hypotheses at level α in the strong sense. Each intersection is tested using the simple Bonferroni test.

It is a shortcut procedure since practically the number of comparisons to be made equal to ${\displaystyle m}$ or less, while the number of all intersections of null hypotheses to be tested is of order ${\displaystyle 2^{m}}$.

The closure principle states that a hypothesis ${\displaystyle H_{i}}$ in a family of hypotheses ${\displaystyle H_{1},\ldots ,H_{m}}$ is rejected – while controlling the family-wise error rate of ${\displaystyle \alpha }$ – if and only if all the sub-families of the intersections with ${\displaystyle H_{i}}$ are controlled at level of family-wise error rate of ${\displaystyle \alpha }$.

In Holm–Bonferroni procedure, we first test ${\displaystyle H_{(1)}}$. If it is not rejected then the intersection of all null hypotheses ${\displaystyle \bigcap \nolimits _{i=1}^{m}H_{i}}$ is not rejected too, such that there exist at least one intersection hypothesis for each of elementary hypotheses ${\displaystyle H_{1},\ldots ,H_{m}}$ that is not rejected, thus we reject none of the elementary hypotheses.

If ${\displaystyle H_{(1)}}$ is rejected at level ${\displaystyle \alpha /m}$ then all the intersection sub-families that contain it are rejected too, thus ${\displaystyle H_{(1)}}$ is rejected. This is because ${\displaystyle P_{(1)}}$ is the smallest in each one of the intersection sub-families and the size of the sub-families is the most ${\displaystyle m}$, such that the Bonferroni threshold larger than ${\displaystyle \alpha /m}$.

The same rationale applies for ${\displaystyle H_{(2)}}$. However, since ${\displaystyle H_{(1)}}$ already rejected, it sufficient to reject all the intersection sub-families of ${\displaystyle H_{(2)}}$ without ${\displaystyle H_{(1)}}$. Once ${\displaystyle P_{(2)}\leq \alpha /(m-1)}$ holds all the intersections that contains ${\displaystyle H_{(2)}}$ are rejected.

The same applies for each ${\displaystyle 1\leq i\leq m}$.

Example

Consider four null hypotheses ${\displaystyle H_{1},\ldots ,H_{4}}$ with unadjusted p-values ${\displaystyle p_{1}=0.01}$, ${\displaystyle p_{2}=0.04}$, ${\displaystyle p_{3}=0.03}$ and ${\displaystyle p_{4}=0.005}$, to be tested at significance level ${\displaystyle \alpha =0.05}$. Since the procedure is step-down, we first test ${\displaystyle H_{4}=H_{(1)}}$, which has the smallest p-value ${\displaystyle p_{4}=p_{(1)}=0.005}$. The p-value is compared to ${\displaystyle \alpha /4=0.0125}$, the null hypothesis is rejected and we continue to the next one. Since ${\displaystyle p_{1}=p_{(2)}=0.01<0.0167=\alpha /3}$ we reject ${\displaystyle H_{1}=H_{(2)}}$ as well and continue. The next hypothesis ${\displaystyle H_{3}}$ is not rejected since ${\displaystyle p_{3}=p_{(3)}=0.03>0.025=\alpha /2}$. We stop testing and conclude that ${\displaystyle H_{1}}$ and ${\displaystyle H_{4}}$ are rejected and ${\displaystyle H_{2}}$ and ${\displaystyle H_{3}}$ are not rejected while controlling the family-wise error rate at level ${\displaystyle \alpha =0.05}$. Note that even though ${\displaystyle p_{2}=p_{(4)}=0.04<0.05=\alpha }$ applies, ${\displaystyle H_{2}}$ is not rejected. This is because the testing procedure stops once a failure to reject occurs.

Extensions

Holm–Šidák method

Further information: Šidák correction

When the hypothesis tests are not negatively dependent, it is possible to replace ${\displaystyle {\frac {\alpha }{m}},{\frac {\alpha }{m-1}},\ldots ,{\frac {\alpha }{1}}}$ with:

${\displaystyle 1-(1-\alpha )^{1/m},1-(1-\alpha )^{1/(m-1)},\ldots ,1-(1-\alpha )^{1}}$

resulting in a slightly more powerful test.

Weighted version

Let ${\displaystyle P_{(1)},\ldots ,P_{(m)}}$ be the ordered unadjusted p-values. Let ${\displaystyle H_{(i)}}$, ${\displaystyle 0\leq w_{(i)}}$ correspond to ${\displaystyle P_{(i)}}$. Reject ${\displaystyle H_{(i)}}$ as long as

${\displaystyle P_{(j)}\leq {\frac {w_{(j)}}{\sum _{k=j}^{m}w_{(k)}}}\alpha ,\quad j=1,\ldots ,i}$

Adjusted p-values

The adjusted p-values for Holm–Bonferroni method are:

${\displaystyle {\widetilde {p}}_{(i)}=\max _{j\leq i}\left\{(m-j+1)p_{(j)}\right\}_{1},{\text{ where }}\{x\}_{1}\equiv \min(x,1).}$

In the earlier example, the adjusted p-values are ${\displaystyle {\widetilde {p}}_{1}=0.03}$, ${\displaystyle {\widetilde {p}}_{2}=0.06}$, ${\displaystyle {\widetilde {p}}_{3}=0.06}$ and ${\displaystyle {\widetilde {p}}_{4}=0.02}$. Only hypotheses ${\displaystyle H_{1}}$ and ${\displaystyle H_{4}}$ are rejected at level ${\displaystyle \alpha =0.05}$.

The weighted adjusted p-values are:

${\displaystyle {\widetilde {p}}_{(i)}=\max _{j\leq i}\left\{{\frac {\sum _{k=j}^{m}{w_{(k)}}}{w_{(j)}}}p_{(j)}\right\}_{1},{\text{ where }}\{x\}_{1}\equiv \min(x,1).}$

A hypothesis is rejected at level α if and only if its adjusted p-value is less than α. In the earlier example using equal weights, the adjusted p-values are 0.03, 0.06, 0.06, and 0.02. This is another way to see that using α = 0.05, only hypotheses one and four are rejected by this procedure.

Alternatives and usage

Main article: Family-wise error rate § Controlling procedures

The Holm–Bonferroni method is "uniformly" more powerful than the classic Bonferroni correction, meaning that it is always at least as powerful.

There are other methods for controlling the family-wise error rate that are more powerful than Holm–Bonferroni. For instance, in the Hochberg procedure, rejection of ${\displaystyle H_{(1)}\ldots H_{(k)}}$ is made after finding the maximal index ${\displaystyle k}$ such that ${\displaystyle P_{(k)}\leq {\frac {\alpha }{m+1-k}}}$. Thus, The Hochberg procedure is uniformly more powerful than the Holm procedure. However, the Hochberg procedure requires the hypotheses to be independent or under certain forms of positive dependence, whereas Holm–Bonferroni can be applied without such assumptions. A similar step-up procedure is the Hommel procedure, which is uniformly more powerful than the Hochberg procedure.[3]

Naming

Carlo Emilio Bonferroni did not take part in inventing the method described here. Holm originally called the method the "sequentially rejective Bonferroni test", and it became known as Holm–Bonferroni only after some time. Holm's motives for naming his method after Bonferroni are explained in the original paper: "The use of the Boole inequality within multiple inference theory is usually called the Bonferroni technique, and for this reason we will call our test the sequentially rejective Bonferroni test."

References

1. Holm, S. (1979). "A simple sequentially rejective multiple test procedure". Scandinavian Journal of Statistics. 6 (2): 65–70. JSTOR 4615733. MR 0538597.
2. Marcus, R.; Peritz, E.; Gabriel, K. R. (1976). "On closed testing procedures with special reference to ordered analysis of variance". Biometrika. 63 (3): 655–660. doi:10.1093/biomet/63.3.655.
3. Hommel, G. (1988). "A stagewise rejective multiple test procedure based on a modified Bonferroni test". Biometrika. 75 (2): 383–386. doi:10.1093/biomet/75.2.383. ISSN 0006-3444.