Fleiss' kappa

Fleiss' kappa (named after Joseph L. Fleiss) is a statistical measure for assessing the reliability of agreement between a fixed number of raters when assigning categorical ratings to a number of items or classifying items. This contrasts with other kappas such as Cohen's kappa, which only work when assessing the agreement between not more than two raters or the intra-rater reliability (for one appraiser versus themself). The measure calculates the degree of agreement in classification over that which would be expected by chance.

Fleiss' kappa can be used with binary or nominal-scale. It can also be applied to Ordinal data (ranked data): the MiniTab online documentation gives an example. However, this document notes: "When you have ordinal ratings, such as defect severity ratings on a scale of 1–5, Kendall's coefficients, which account for ordering, are usually more appropriate statistics to determine association than kappa alone." Keep in mind however, that Kendall rank coefficients are only appropriate for rank data.

Introduction

Fleiss' kappa is a generalisation of Scott's pi statistic, a statistical measure of inter-rater reliability. It is also related to Cohen's kappa statistic and Youden's J statistic which may be more appropriate in certain instances. Whereas Scott's pi and Cohen's kappa work for only two raters, Fleiss' kappa works for any number of raters giving categorical ratings, to a fixed number of items. It can be interpreted as expressing the extent to which the observed amount of agreement among raters exceeds what would be expected if all raters made their ratings completely randomly. It is important to note that whereas Cohen's kappa assumes the same two raters have rated a set of items, Fleiss' kappa specifically allows that although there are a fixed number of raters (e.g., three), different items may be rated by different individuals (Fleiss, 1971, p. 378). That is, Item 1 is rated by Raters A, B, and C; but Item 2 could be rated by Raters D, E, and F.

Agreement can be thought of as follows, if a fixed number of people assign numerical ratings to a number of items then the kappa will give a measure for how consistent the ratings are. The kappa, ${\displaystyle \kappa \,}$, can be defined as,

(1)

${\displaystyle \kappa ={\frac {{\bar {P}}-{\bar {P_{e}}}}{1-{\bar {P_{e}}}}}}$

The factor ${\displaystyle 1-{\bar {P_{e}}}}$ gives the degree of agreement that is attainable above chance, and, ${\displaystyle {\bar {P}}-{\bar {P_{e}}}}$ gives the degree of agreement actually achieved above chance. If the raters are in complete agreement then ${\displaystyle \kappa =1~}$. If there is no agreement among the raters (other than what would be expected by chance) then ${\displaystyle \kappa \leq 0}$.

An example of the use of Fleiss' kappa may be the following: Consider fourteen psychiatrists are asked to look at ten patients. Each psychiatrist gives one of possibly five diagnoses to each patient. These are compiled into a matrix, and Fleiss' kappa can be computed from this matrix (see example below) to show the degree of agreement between the psychiatrists above the level of agreement expected by chance.

Definition

Let N be the total number of subjects, let n be the number of ratings per subject, and let k be the number of categories into which assignments are made. The subjects are indexed by i = 1, ... N and the categories are indexed by j = 1, ... k. Let n_ij represent the number of raters who assigned the i-th subject to the j-th category.

First calculate p_j, the proportion of all assignments which were to the j-th category:

(2)

${\displaystyle p_{j}={\frac {1}{Nn}}\sum _{i=1}^{N}n_{ij},\quad \quad 1=\sum _{j=1}^{k}p_{j}}$

Now calculate ${\displaystyle P_{i}\,}$, the extent to which raters agree for the i-th subject (i.e., compute how many rater--rater pairs are in agreement, relative to the number of all possible rater--rater pairs):

(3)

${\displaystyle P_{i}={\frac {1}{n(n-1)}}\sum _{j=1}^{k}n_{ij}(n_{ij}-1)}$

${\displaystyle ={\frac {1}{n(n-1)}}\sum _{j=1}^{k}(n_{ij}^{2}-n_{ij})}$

${\displaystyle ={\frac {1}{n(n-1)}}\left[\left(\sum _{j=1}^{k}n_{ij}^{2}\right)-(n)\right]}$

Now compute ${\displaystyle {\bar {P}}}$, the mean of the ${\displaystyle P_{i}\,}$'s, and ${\displaystyle {\bar {P_{e}}}}$ which go into the formula for ${\displaystyle \kappa \,}$:

(4)

${\displaystyle {\bar {P}}={\frac {1}{N}}\sum _{i=1}^{N}P_{i}}$

${\displaystyle ={\frac {1}{Nn(n-1)}}\left(\sum _{i=1}^{N}\sum _{j=1}^{k}n_{ij}^{2}-Nn\right)}$

(5)

${\displaystyle {\bar {P_{e}}}=\sum _{j=1}^{k}p_{j}^{2}}$

Worked example

Table of values for computing the worked example

${\displaystyle n_{ij}}$	1	2	3	4	5	${\displaystyle P_{i}\,}$
1	0	0	0	0	14	1.000
2	0	2	6	4	2	0.253
3	0	0	3	5	6	0.308
4	0	3	9	2	0	0.440
5	2	2	8	1	1	0.330
6	7	7	0	0	0	0.462
7	3	2	6	3	0	0.242
8	2	5	3	2	2	0.176
9	6	5	2	1	0	0.286
10	0	2	2	3	7	0.286
Total	20	28	39	21	32
${\displaystyle p_{j}\,}$	0.143	0.200	0.279	0.150	0.229

In the following example, fourteen raters (${\displaystyle n}$) assign ten "subjects" (${\displaystyle N}$) to a total of five categories (${\displaystyle k}$). The categories are presented in the columns, while the subjects are presented in the rows. Each cell lists the number of raters who assigned the indicated (row) subject to the indicated (column) category.

Data

See table to the right.

N = 10, n = 14, k = 5

Sum of all cells = 140
Sum of P_i = 3.780

Calculations

The value ${\displaystyle p_{j}}$ is the proportion of all assignments (${\displaystyle N\times n}$, here ${\displaystyle 10\times 14=140}$) that were made to the ${\displaystyle j}$th category. For example, taking the first column,

${\displaystyle p_{1}={\frac {0+0+0+0+2+7+3+2+6+0}{140}}=0.143}$

And taking the second row,

${\displaystyle P_{2}={\frac {1}{14(14-1)}}\left(0^{2}+2^{2}+6^{2}+4^{2}+2^{2}-14\right)=0.253}$

In order to calculate ${\displaystyle {\bar {P}}}$, we need to know the sum of ${\displaystyle P_{i}}$,

${\displaystyle \sum _{i=1}^{N}P_{i}=1.000+0.253+\cdots +0.286+0.286=3.780}$

Over the whole sheet,

${\displaystyle {\bar {P}}={\frac {1}{(10)}}(3.780)=0.378}$

${\displaystyle {\bar {P}}_{e}=0.143^{2}+0.200^{2}+0.279^{2}+0.150^{2}+0.229^{2}=0.213}$

${\displaystyle \kappa ={\frac {0.378-0.213}{1-0.213}}=0.210}$

Interpretation

Landis and Koch (1977) gave the following table for interpreting ${\displaystyle \kappa }$ values. This table is however by no means universally accepted. They supplied no evidence to support it, basing it instead on personal opinion. It has been noted that these guidelines may be more harmful than helpful, as the number of categories and subjects will affect the magnitude of the value. The kappa will be higher when there are fewer categories.

${\displaystyle \kappa }$	Interpretation
< 0	Poor agreement
0.01 – 0.20	Slight agreement
0.21 – 0.40	Fair agreement
0.41 – 0.60	Moderate agreement
0.61 – 0.80	Substantial agreement
0.81 – 1.00	Almost perfect agreement

The MiniTab documentation cited earlier states that Automotive Industry Action Group (AIAG) "suggests that a kappa value of at least 0.75 indicates good agreement. However, larger kappa values, such as 0.90, are preferred."

Tests of Significance

Statistical packages can calculate a standard score (Z-score) for Cohen's kappa or Fleiss's Kappa, which can be converted into a P-value. However, even when the P value reaches the threshold of statistical significance (typically less than 0.05), it only indicates that the agreement between raters is significantly better than would be expected by chance. The p value does not tell you, by itself, whether the agreement is good enough to have high predictive value.

See also

• Pearson product-moment correlation coefficient
• Matthews correlation coefficient
• Krippendorff's alpha

References

1. ^ MiniTab Inc. Kappa statistics for Attribute Agreement Analysis. https://support.minitab.com/en-us/minitab/18/help-and-how-to/quality-and-process-improvement/measurement-system-analysis/how-to/attribute-agreement-analysis/attribute-agreement-analysis/interpret-the-results/all-statistics-and-graphs/kappa-statistics/ Accessed Jan 22 2019.
2. ^ Fleiss, J. L. (1971) "Measuring nominal scale agreement among many raters." Psychological Bulletin, Vol. 76, No. 5 pp. 378–382
3. ^ Scott, W. (1955). "Reliability of content analysis: The case of nominal scale coding." Public Opinion Quarterly, Vol. 19, No. 3, pp. 321–325.
4. ^ Powers, D. M. W. (2011). "Evaluation: From Precision, Recall and F-Measure to ROC, Informedness, Markedness & Correlation". Journal of Machine Learning Technologies 2 (1): 37–63
5. ^ Powers, David M. W. (2012). "The Problem with Kappa". Conference of the European Chapter of the Association for Computational Linguistics (EACL2012) Joint ROBUS-UNSUP Workshop.
6. ^ Landis, J. R. and Koch, G. G. (1977) "The measurement of observer agreement for categorical data" in Biometrics. Vol. 33, pp. 159–174
7. ^ Gwet, K. L. (2014) Handbook of Inter-Rater Reliability (4th Edition), Chapter 6. (Gaithersburg : Advanced Analytics, LLC) ISBN 978-0970806284. http://www.agreestat.com/book4/9780970806284_chap2.pdf
8. ^ Sim, J. and Wright, C. C. (2005) "The Kappa Statistic in Reliability Studies: Use, Interpretation, and Sample Size Requirements" in Physical Therapy. Vol. 85, No. 3, pp. 257–268

Further reading

• Fleiss, J. L. and Cohen, J. (1973) "The equivalence of weighted kappa and the intraclass correlation coefficient as measures of reliability" in Educational and Psychological Measurement, Vol. 33 pp. 613–619
• Fleiss, J. L. (1981) Statistical methods for rates and proportions. 2nd ed. (New York: John Wiley) pp. 38–46
• Gwet, K. L. (2008) "Computing inter-rater reliability and its variance in the presence of high agreement", British Journal of Mathematical and Statistical Psychology, Vol. 61, pp29–48

External links

• AgreeStat 360: cloud-based inter-rater reliability analysis, Cohen's kappa, Gwet's AC1/AC2, Krippendorff's alpha, Brennan-Prediger, Fleiss generalized kappa, intraclass correlation coefficients
• Kappa: Pros and Cons contains a good bibliography of articles about the coefficient.
• Online Kappa Calculator calculates a variation of Fleiss' kappa.