Estimation statistics

Estimation statistics is a data analysis framework that uses a combination of effect sizes, confidence intervals, precision planning, and meta-analysis to plan experiments, analyze data and interpret results.[1] It is distinct from null hypothesis significance testing (NHST), which is considered to be less informative.[2][3] Estimation statistics, or simply estimation, is also known as the new statistics,[3] a distinction introduced in the fields of psychology, medical research, life sciences and a wide range of other experimental sciences where NHST still remains prevalent,[4] despite estimation statistics having been recommended as preferable for several decades.[5][6]

Not to be confused with Estimation theory.
For other uses, see Estimation (disambiguation).

The primary aim of estimation methods is to report an effect size (a point estimate) along with its confidence interval, the latter of which is related to the precision of the estimate.[7] The confidence interval summarizes a range of likely values of the underlying population effect. Proponents of estimation see reporting a P value as an unhelpful distraction from the important business of reporting an effect size with its confidence intervals,[8] and believe that estimation should replace significance testing for data analysis.[9]

History

Physics has for long employed a weighted averages method that is similar to meta-analysis.[10]

Estimation statistics in the modern era started with the development of the standardized effect size by Jacob Cohen in the 1960s. Research synthesis using estimation statistics was pioneered by Gene V. Glass with the development of the method of meta-analysis in the 1970s.[11] Estimation methods have been refined since by Larry Hedges, Michael Borenstein, Doug Altman, Martin Gardner, Geoff Cumming and others. The systematic review, in conjunction with meta-analysis, is a related technique with widespread use in medical research. There are now over 60,000 citations to "meta-analysis" in PubMed. Despite the widespread adoption of meta-analysis, the estimation framework is still not routinely used in primary biomedical research.[4]

In the 1990s, editor Kenneth Rothman banned the use of p-values from the journal Epidemiology; compliance was high among authors but this did not substantially change their analytical thinking.[12]

More recently, estimation methods are being adopted in fields such as neuroscience,[13] psychology education[14] and psychology.[15]

The Publication Manual of the American Psychological Association recommends estimation over hypothesis testing.[16] The Uniform Requirements for Manuscripts Submitted to Biomedical Journals document makes a similar recommendation: "Avoid relying solely on statistical hypothesis testing, such as P values, which fail to convey important information about effect size."[17]

In 2019, the Society for Neuroscience journal eNeuro instituted a policy recommending the use of estimation graphics as the preferred method for data presentation [18]

Methodology

Many significance tests have an estimation counterpart;[19] in almost every case, the test result (or its p-value) can be simply substituted with the effect size and a precision estimate. For example, instead of using Student's t-test, the analyst can compare two independent groups by calculating the mean difference and its 95% confidence interval. Corresponding methods can be used for a paired t-test and multiple comparisons. Similarly, for a regression analysis, an analyst would report the coefficient of determination (R2) and the model equation instead of the model's p-value.

However, proponents of estimation statistics warn against reporting only a few numbers. Rather, it is advised to analyze and present data using data visualization.[2][6][7] Examples of appropriate visualizations include the Scatter plot for regression, and Gardner-Altman plots for two independent groups.[20] While historical data-group plots (bar charts, box plots, and violin plots) do not display the comparison, estimation plots add a second axis to explicitly visualize the effect size.[21]

The Gardner–Altman plot. Left: A conventional bar chart, using asterisks to show that the difference is 'statistically significant.' Right: A Gardner–Altman plot that shows all data points, along with the mean difference and its confidence intervals.

Gardner–Altman plot

The Gardner–Altman mean difference plot was first described by Martin Gardner and Doug Altman in 1986;[20] it is a statistical graph designed to display data from two independent groups.[6] There is also a version suitable for paired data. The key instructions to make this chart are as follows: (1) display all observed values for both groups side-by-side; (2) place a second axis on the right, shifted to show the mean difference scale; and (3) plot the mean difference with its confidence interval as a marker with error bars.[3] Gardner-Altman plots can be generated with custom code using Ggplot2, seaborn, or DABEST; alternatively, the analyst can use user-friendly software like the Estimation Stats app.

The Cumming plot. All raw data is shown. The effect size and 95% CIs are plotted on a separate axes beneath the raw data. For each group, summary measurements (mean ± standard deviation) are shown as gapped lines.

Cumming plot

For multiple groups, Geoff Cumming introduced the use of a secondary panel to plot two or more mean differences and their confidence intervals, placed below the observed values panel;[3] this arrangement enables easy comparison of mean differences ('deltas') over several data groupings. Cumming plots can be generated with the ESCI package, DABEST, or the Estimation Stats app.

Other methodologies

In addition to the mean difference, there are numerous other effect size types, all with relative benefits. Major types include Cohen's d-type effect sizes, and the coefficient of determination (R2) for regression analysis. For non-normal distributions, there are a number of more robust effect sizes, including Cliff's delta and the Kolmogorov-Smirnov statistic.

Flaws in hypothesis testing
Main article: Statistical hypothesis testing § Criticism
See also: p-value § Criticism, and Misuse of p-values

In hypothesis testing, the primary objective of statistical calculations is to obtain a p-value, the probability of seeing an obtained result, or a more extreme result, when assuming the null hypothesis is true. If the p-value is low (usually < 0.05), the statistical practitioner is then encouraged to reject the null hypothesis. Proponents of estimation reject the validity of hypothesis testing[3][7] for the following reasons, among others:

  • P-values are easily and commonly misinterpreted. For example, the p-value is often mistakenly thought of as 'the probability that the null hypothesis is true.'
  • The null hypothesis is always wrong for every set of observations: there is always some effect, even if it is minuscule.[22]
  • Hypothesis testing produces arbitrarily dichotomous yes-no answers, while discarding important information about magnitude.[23]
  • Any particular p-value arises through the interaction of the effect size, the sample size (all things being equal a larger sample size produces a smaller p-value) and sampling error.[24]
  • At low power, simulation reveals that sampling error makes p-values extremely volatile.[25]

Benefits of estimation statistics

Advantages of confidence intervals

Confidence intervals behave in a predictable way. By definition, 95% confidence intervals have a 95% chance of capturing the underlying population mean (μ). This feature remains constant with increasing sample size; what changes is that the interval becomes smaller (more precise). In addition, 95% confidence intervals are also 83% prediction intervals: one experiment's confidence interval has an 83% chance of capturing any future experiment's mean.[3] As such, knowing a single experiment's 95% confidence intervals gives the analyst a plausible range for the population mean, and plausible outcomes of any subsequent replication experiments.

Evidence-based statistics

Psychological studies of the perception of statistics reveal that reporting interval estimates leaves a more accurate perception of the data than reporting p-values.[26]

Precision planning

The precision of an estimate is formally defined as 1/variance, and like power, increases (improves) with increasing sample size. Like power, a high level of precision is expensive; research grant applications would ideally include precision/cost analyses. Proponents of estimation believe precision planning should replace power since statistical power itself is conceptually linked to significance testing.[3]

gollark: ```pythonx = {0: None}for i in x: del x[i] x[i+1] = None print(i)```
gollark: The list is appended to, but the tuple can't be mutated.
gollark: It trickled into higher-level languages, which now arbitrarily BREAK EQUALITY.
gollark: Oh, you know what else bad? IEEE 754 NaN handling.
gollark: Oh, and more bees: IMPLICIT STRING CONCATENATION?`"a" "b" "c"` → `"abc"` (string literals only).

See also

References
  1. Ellis, Paul. "Effect size FAQ".
  2. Cohen, Jacob. "The earth is round (p<.05)" (PDF).
  3. Cumming, Geoff (2012). Understanding The New Statistics: Effect Sizes, Confidence Intervals, and Meta-Analysis. New York: Routledge.
  4. Button, Katherine; John P. A. Ioannidis; Claire Mokrysz; Brian A. Nosek; Jonathan Flint; Emma S. J. Robinson; Marcus R. Munafò (2013). "Power failure: why small sample size undermines the reliability of neuroscience". Nature Reviews Neuroscience. 14 (5): 365–76. doi:10.1038/nrn3475. PMID 23571845.
  5. Altman, Douglas (1991). Practical Statistics For Medical Research. London: Chapman and Hall.
  6. Douglas Altman, ed. (2000). Statistics with Confidence. London: Wiley-Blackwell.
  7. Cohen, Jacob (1990). "What I have Learned (So Far)". American Psychologist. 45 (12): 1304. doi:10.1037/0003-066x.45.12.1304.
  8. Ellis, Paul (2010-05-31). "Why can't I just judge my result by looking at the p value?". Retrieved 5 June 2013.
  9. Claridge-Chang, Adam; Assam, Pryseley N (2016). "Estimation statistics should replace significance testing" (PDF). Nature Methods. 13 (2): 108–109. doi:10.1038/nmeth.3729. PMID 26820542.
  10. Hedges, Larry (1987). "How hard is hard science, how soft is soft science". American Psychologist. 42 (5): 443. CiteSeerX 10.1.1.408.2317. doi:10.1037/0003-066x.42.5.443.
  11. Hunt, Morton (1997). How science takes stock: the story of meta-analysis. New York: The Russell Sage Foundation. ISBN 978-0-87154-398-1.
  12. Fidler, Fiona (2004). "Editors Can Lead Researchers to Confidence Intervals, but Can't Make Them Think". Psychological Science. 15 (2): 119–126. doi:10.1111/j.0963-7214.2004.01502008.x. PMID 14738519.
  13. Yildizoglu, Tugce; Weislogel, Jan-Marek; Mohammad, Farhan; Chan, Edwin S.-Y.; Assam, Pryseley N.; Claridge-Chang, Adam (2015-12-08). "Estimating Information Processing in a Memory System: The Utility of Meta-analytic Methods for Genetics". PLOS Genet. 11 (12): e1005718. doi:10.1371/journal.pgen.1005718. ISSN 1553-7404. PMC 4672901. PMID 26647168.
  14. Hentschke, Harald; Maik C. Stüttgen (December 2011). "Computation of measures of effect size for neuroscience data sets". European Journal of Neuroscience. 34 (12): 1887–1894. doi:10.1111/j.1460-9568.2011.07902.x. PMID 22082031.
  15. Cumming, Geoff. "ESCI (Exploratory Software for Confidence Intervals)".
  16. "Publication Manual of the American Psychological Association, Sixth Edition". Retrieved 17 May 2013.
  17. "Uniform Requirements for Manuscripts Submitted to Biomedical Journals". Archived from the original on 15 May 2013. Retrieved 17 May 2013.
  18. "Changing the Way We Report, Interpret, and Discuss Our Results to Rebuild Trust in Our Research".
  19. Cumming, Geoff; Calin-Jageman, Robert (2016). Introduction to the New Statistics: Estimation, Open Science, and Beyond. Routledge. ISBN 978-1138825529.
  20. Gardner, M. J.; Altman, D. G. (1986-03-15). "Confidence intervals rather than P values: estimation rather than hypothesis testing". British Medical Journal (Clinical Research Ed.). 292 (6522): 746–750. ISSN 0267-0623. PMC 1339793. PMID 3082422.
  21. Ho, Joses; Tumkaya; Aryal; Choi; Claridge-Chang (2018). "Moving beyond P values: Everyday data analysis with estimation plots". bioRxiv: 377978. doi:10.1101/377978.
  22. Cohen, Jacob (1994). "The earth is round (p < .05)". American Psychologist. 49 (12): 997–1003. doi:10.1037/0003-066X.49.12.997.
  23. Ellis, Paul (2010). The Essential Guide to Effect Sizes: Statistical Power, Meta-Analysis, and the Interpretation of Research Results. Cambridge: Cambridge University Press.
  24. Denton E. Morrison, Ramon E. Henkel, ed. (2006). The Significance Test Controversy: A Reader. Aldine Transaction. ISBN 978-0202308791.
  25. Cumming, Geoff. "Dance of the p values".
  26. Beyth-Marom, R; Fidler, F.; Cumming, G. (2008). "Statistical cognition: Towards evidence-based practice in statistics and statistics education". Statistics Education Research Journal. 7: 20–39.
This article is issued from Wikipedia. The text is licensed under Creative Commons - Attribution - Sharealike. Additional terms may apply for the media files.