Anecdotal evidence

Anecdotal evidence (also proof by selected instances, or, more pejoratively, anecdata) is use of one or more anecdotes (specific instances of an event; stories) to either support or refute a claim. The use of anecdotal evidence to draw a conclusion is like using the NBA all-star teams to estimate the average height of Americans.

Cogito ergo sum
Logic and rhetoric
Key articles
General logic
Bad logic
v - t - e
Also known as the meta-anecdote. (Source)
“”[Bloodletting] removes coma. Mr. Henry Clymer was suddenly relieved of this alarming symptom, in the fever of 1794, by the loss of twelve ounces of blood.
—Benjamin Rush, A Defense of Blood-letting[1]

Whereas anecdotal evidence is sometimes the starting point of a proper scientific investigation, it is all too often the ending point and every point of a pseudoscientific investigation. In the world of pseudoscience, an anecdote is the equivalent of a peer-reviewed, double-blind, repeatable scientific experiment with consistent results.

Anecdotal evidence is often used in politics, journalism, blogs and many other contexts to make or imply generalisations based on very limited and cherry-picked examples, rather than reliable statistical studies. A classic instance was Ronald Reagan's story of a "welfare queen" who was abusing the system, who Reagan attempted to portray as indicative of the average welfare recipient. It turned out she didn't even exist when some reporters finally decided to look for her.

Anecdotal evidence is especially vulnerable to confabulation or outright deceit.

Remember: the plural of "anecdote" is not "data".[note 1]

Form
P1: X happened once when Y.
P2: (unstated) Things that happen once will happen every time.
C: X happens every time when Y.

For example, "Billy-Bob ate 3 pounds of apples a day for a year and his cancer disappeared – so can you!"

It should be clear that P2 is invalid. Take the following syllogism:

P1: Abby choked once when she ate some goat.
P2: (unstated) Things that happen once will happen every time.
C: Everyone will choke when they eat goat.

Consume that goat, and face the wrath of the Goat Overlord!

Another form:

P1: Horatio started drinking orange juice after being diagnosed with cancer and then his cancer went into remission.
P2: (unstated) There is no other plausible explanation for Horatio's cancer going into remission than the OJ.
C: Orange juice can cure cancer.

It's clear from the above that fallacies like post hoc ergo propter hoc and argument from incredulity can play supporting roles in the perception of anecdotes as convincing.

Legitimate use

In two instances, it is possible to use anecdotes non-fallaciously:

  • If you use one or more anecdotes as an example of a general rule which is already supported by a broad, comprehensive investigation (i.e., your evidence/argument does not rely on the anecdotes, they are just used to illustrate the point).

These are, by far, not the most common uses of anecdotes.

Problems
“”If you are basing your claims on anecdotal experience, then any treatment will seem to work for anything and everything.
—Steven Novella, The CAM Worldview[2]

While anecdotes — when true, at least — are nice illustrative stories, they do not constitute evidence. This is because anecdotes only ever apply to individuals or individual experiences and are subject to the biases that this brings with it. It is impossible to say that an individual anecdote is representative and it is also impossible to actually detect the real cause of the anecdote.

For instance, with life-saving medical treatments (say, pills that reduce heart-disease and subsequently lower the death rate), there are some deaths that occur whether or not the medication was taken. Therefore, if someone who is on the medication dies, you cannot tell if they would have died anyway without it — you can't prove that the medical intervention worked, or not, from the one case study.

It is very rare for an intervention to be, by itself, a sufficient cause of something. Rather, they tend to change the probability of a given event occurring. This means, obviously, that one can cherry pick examples that show something does or does not work, regardless of what effect it actually has. For instance, if the municipal government of some city enacts a law to reduce crime, one can find anecdotes "demonstrating" the exact opposite of the law's actual effect, whatever it may be. If it's effective: "Before the law I never had any problems with crime, but since its enactment I've been mugged once and had my house burglarized! This law is useless!" If it's not: "Before the law, I was robbed twice, but ever since the law, I haven't been robbed once! It must be working!"

Ultimately, anecdotal evidence is very prone to false positives.

Regression to the mean
See the main article on this topic: Regression to the mean

If someone with a rash gets better after taking a homeopathic remedy, that doesn't mean the remedy caused the improvement, as it could simply have been an exception to the rule. Maybe it would have gotten better anyway — for example, it could have just been an allergic reaction, or a sweat-rash or anything along those effects. The anecdote doesn't tell you anything about all the people who did not get better; many other people with rashes could have taken homeopathic remedies to no effect.

Adulteration

Not to mention that these remedies can contain actual active substances, which are misleadingly listed under "non-medicinal ingredients". A "homeopathic" anti-acne cream may contain tea tree oil, which is an effective antifungal.

In addition, some people selling alternative medicines sometimes adulterate them with drugs in order to give them the desired medicinal effect and thus generate return customers. For instance, a "herbal" cold remedy may be adulterated with paracetamol and antihistamines.[3] In these sorts of cases, the improvements may very well be caused by the drugs rather than the herbs.

Improbability

There are also anecdotes in which a person suffered from a disease for many years, and no conventional treatment was able to cure them, but they improved after applying some alternative remedy for a few days/weeks/months. For instance:

I having ministered [tobacco] to many people as well men as women, in great number, and being grieved of ten, and of twenty years they have healed old rotten sores in legs, and other parts of the body, with only this remedy to the great admiration of all men.[4]

The implication is that the improvement cannot possibly have been a coincidence, because it was so unlikely. Of course, since millions of people use alternative medicines every day, such coincidences are certain to happen all the time to many people, even if the odds of it happening in one specific instance are low. The odds of a particular person winning the lottery are extremely low, but somebody always does, regardless. Not to mention that most people get sick many times in the course of their life, and the number of such events may number in the thousands or even tens of thousands, if one includes such mundane things as pain, cuts, nausea, or blemishes. Some improbable coincidence is bound to happen during at least one, if not several, of these many illnesses. It is worth also considering a practitioner's situation. Over the course of her career, she will see countless thousands of patients. Even if her treatments are effective, many of her patients' recoveries (and possible website testimonials) will be unrelated to her treatments, and the large number of cases almost certainly means that some of these coincidental recoveries will be extremely unlikely.

While "case studies" exist within medicine that are essentially anecdotal, these are treated mostly as curiosities or communications between doctors, rather than hard evidence on par with real trials.

Selection bias
See the main article on this topic: Selection bias

Selective reporting is the main factor that popularly disseminated anecdotes will be almost certainly unrepresentative. People tend to focus on the interesting, the unusual or the powerfully emotive events while ignoring the mundane. After all, no one ever sold a newspaper where the front-page headline read "girl receives new vaccine; suffers zero complications and is now alive and well". People who believe in alternative medicine may, due to confirmation bias, selectively remember stories that support their belief (which they will, of course, tell everybody about), and ignore those that don't.

Selection bias can have an impressive effect in terms of PR. Let's assume 1 million people (1,000,000) decide to take some ineffective remedy to cure their cancer. Let's further assume (for the sake of argument) that only 0.1% of this million will experience spontaneous remission (the actual remission rate, for breast cancer and basal cell carcinoma at least, is closer to 20%[5][6]), and that 0.3% were misdiagnosed and so do not actually have cancer. This makes for a total of 0.4%, or 4000 people. Now, 4000 people translates into a lot of testimonials, "CAM cured my cancer" blogs, Internet comments, newspaper articles, and real-life word-of-mouth, so this makes an extremely positive impression for the therapy. But the other 99.6% died and so are not around to leave any testimonials, positive or otherwise. Thus, even in a hypothetical scenario that assumes statistically very low false positive rates, the quantity of false positives is nevertheless numerically quite large. In reality, the number of past and present cancer patients relying on alternative medicine is much larger than one million, and the false positive rate is significantly higher than 0.4%, making the possibility of a given supposedly successful "cancer cure" anecdote being a coincidence even more likely.

Even if a given treatment were effective, anecdotal evidence would still be useless, because many of the cases would be the result of natural improvement, and it would be impossible to tell which anecdotes are cures and which ones are flukes. There is no way to separate the signal from the noise with only single data points suffering from selection bias.

Lack of follow up/incompleteness

Another issue is that anecdotes are usually not complete. They often cover only the period of the improvement, and anything happening after the publication of the anecdote is not included. In addition, important factors relating to the "cure" may be omitted (such as, for instance, the patient using conventional medicine together with the alternative remedy). In Mark Twain's essay A Majestic Literary Fossil, which describes various forms of old-fashioned quackery, there is this anecdote:

Then follows a long account of how a dying woman, who had suffered nine hours a day with an ague during eight weeks, and who had been bled dry some dozens of times meantime without apparent benefit, was at last forced to swallow several wads of "Spiders-web," whereupon she straightway mended, and promptly got well. So the sage is full of enthusiasm over the spider-webs, and mentions only in the most casual way the discontinuance of the daily bleedings, plainly never suspecting that this had anything to do with the cure.

If here, the fact that the bloodletting was discontinued was "mentioned only in the most casual way", it is easy to imagine that many anecdotes leave out important facts like this altogether.

A cancer patient named Kim Tinkham went on Oprah with a proponent of the alkaline diet and claimed the diet had cured her, but she died of the cancer a few years later. Now, the eventual fate of Tinkham was revealed in news sources because she was famous, but if an anonymous (or at least non-famous) author of an online testimonial died of a supposedly "cured" illness, it is not likely that this fact would be published on the website of the original testimonial, or possibly anywhere.

In a case study involving spontaneous remission, a woman's diabetes went away completely, with no treatment, shortly after she woke up from a coma. In a CAM anecdote, that would be that. But since this was a case study rather than a testimonial, the person's post-recovery situation was observed, and two years later, a mild form of diabetes reappeared.[7] Similarly, an anecdotal case of a "cured" disease may later turn out to be illusory.

Advantages of anecdotal evidence

My grandma heeded anecdotal evidence and she lived to 106.

“”When a cook in [Jean] Nicot's household nearly cut off his thumb with a chopping knife, the steward ran for the tobacco plant and bound the thumb back on; after five or six dressings of the same sort, the wound healed.
—Medicinal uses of tobacco in history[8]

Also, Jeanne Calment, the person with the longest confirmed lifespan (122 years) smoked for almost her entire adult life and nothing bad happened to her.[9][10] And besides Calment, there are many centenarians who are heavy smokers.[11] The composer Jean Sibelius was an alcoholic and smoker for most of his life, and he lived to be 91, even saying: "All the doctors who wanted to forbid me to smoke and to drink are dead."[12] By alternative medicine logic, this should be indisputable proof that tobacco is actually good for you. As noted by Dan King in his 1861 book Tobacco: What it is, and What it does:

“”Some are ready to contend that tobacco can have no tendency to shorten life because there are many old persons who use it. This is the drunkards' standing argument, and if it proved anything it would prove quite too much: if we are to conclude that tobacco is harmless because old men are found who use it, then by the same course of reasoning we must conclude that intoxicating liquors are harmless because drunkards sometimes live to old age; and by the same rule it might be shown that nobody had been slain or wounded in battle because some old soldiers still survive. Out of fifty persons bitten by a mad dog several may escape the hydrophobia, but such rare instances do not show that there is no danger from the bites of rabid animals. Peculiarity of temperament or idiosyncrasy may enable some constitutions to withstand for many years the pernicious effects of a virulent poison, but such exceptions to a general rule do not prove that poisons are harmless.

The power of anecdotes

Anecdotal evidence is not accepted as proper proof or scientific evidence for good reason, yet they still dominate people's thinking. This is because while facts and figures may be true, they are difficult for people to relate to. Anecdotes have convincing and strong narratives that resonate with people on a key individual level — someone telling an anecdote will lead the listener to think that it just could happen to them too. There are further biases at work. Anecdotes, even implausible ones, can contain a high level of detail. It is this detail that causes people to become involved in the story and to imagine it more clearly; a story about a young girl won't be nearly as widely spread and accepted as a story about a five year-old girl called Sally from Wisconsin who was cycling to school one day.

The biases at work in causing people to add disproportionate weight to anecdotes cause major problems during health scares. A major case in point is vaccine hysteria.

  • Hard, scientific and proven (yet dry and difficult to grasp) evidence about the value of vaccines exists on the one side.
  • Strong narrative stories told by concerned parents competes on the other side.

It is often advised in such circumstances that those on the side of science should also use their own anecdotes to take advantage of their persuasive power. For example instead of giving statistics of illnesses increasing when vaccination uptake dropped, tell — in great detail — stories about specific deaths that have occurred from vaccine preventable illnesses. Respectable scientists, however, may be against such manipulative tactics.

gollark: Or Brain[REDACTED].
gollark: I think the most portable interpreted language by now is *probably* WebAssembly or JS?
gollark: You can make a C interpreter if you want to.
gollark: It can be ish via WASM.
gollark: *Can* you cross-compile aßembly?

See also

Notes
  1. Unless, of course, you gather so many anecdotes that it's possible to construct a representative statistical model.

References
  1. A Defence of Blood-letting, as a Remedy for Certain Diseases
  2. The CAM Worldview, Science-Based Medicine, October 23, 2013.
  3. Media Releases for 2003, MedSafe.
  4. Of the Tabaco and of His Greate Vertues
  5. Spontaneous Regression of Melanoma May Offer Insight Into Cancer Immunology, Carrie Printz, JNCI J. Natl. Cancer Inst. (2001) 93 (14): 1047-1048.
  6. Study reveals surprising cancer remission rate, Dartmouth Medicine Magazine.
  7. Spontaneous Remission of Diabetes, R. B. Haining and R. G. Haining.
  8. http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1079499/
  9. http://extremelongevity.net/supercentenarians/worlds-top-ten-oldest-people-ever/oldest-person-ever-jeanne-calment/
  10. http://www.endlesshumanpotential.com/jeanne-calment.html
  11. Even the Long-Lived Smoke, Drink, and Don't Exercise
  12. The Rest Is Noise, Alex Ross.
This article is issued from Rationalwiki. The text is licensed under Creative Commons - Attribution - Sharealike. Additional terms may apply for the media files.