Selection bias

Selection bias is the bias that occurs in a survey or experimental data when the selection of data points isn't sufficiently random to draw a general conclusion. If selection bias isn't accounted for or acknowledged, and the results are claimed to be widely applicable, it is known as the spotlight fallacy.

Cogito ergo sum Logic and rhetoric

Key articles

Logical fallacy Syllogism Argument

General logic

Appeal to tradition Argumentum ad hominem List of media biases Media was wrong before Preaching to the choir Principle of explosion

Bad logic

Appeal to faith Argument from authority DARVO God of the gaps PIDOOMA Presentism v - t - e

Examples

Examples of selection bias are rife amongst online polls. These polls are completed only by visitors to the site, and so those with an interest in the website's mission or ones that agree with its POV are the only ones to be surveyed. While the informality of online polls is often clear and admitted to, it's important to not draw generalised conclusions from them.

A website devoted to preventing harassment of women unsurprisingly concluded that nearly all women were victims of harassment at some point,[1] meanwhile a poll on alternative medicine managed to receive more votes than people who read the associated article, a sure sign that something very odd was happening.[2] Polls by the right-wing news site WND are as famous for showing that their readership have right-wing views as for having ridiculous numbers of available options.[3] Similarly, Facebook's recent "Questions" feature spreads via friends in similar circles who are likely to be in a similar area – so asking "what town is the best?" will be easily slanted. Reviews, of restaurants, films, or medical services for example, may also be highly biased towards the negative because people are more likely to rant on the internet after having a bad time, than they are to rave about something after a good or adequate time.

In science

Scientific selection bias is a massive problem and often great pains are taken to avoid it. In Andrew Wakefield's heavily discredited paper on the MMR vaccine, he took 12 children who had behavioural disorders and attempted to link them to vaccines. No attempt was made, in the paper, to look at children without behavioural disorders to see if their exposure to the MMR vaccine was greater or less. This would also be an example of confirmation bias.

Selection bias doesn't just apply to people. With meta-analyses of existing research and scientific literature, the cited literature may be subjected to a selection bias based on several factors. Only certain types of studies might be included or only ones indexed in certain journals – certainly meta-analysis is restricted to data that is published, and publication bias is a huge problem in itself. This is why good meta-analyses clearly state their search terms and criteria and include all papers that fit these terms, so selection bias is either minimised or at least acknowledged and controlled objectively. Often in pseudoscience the literature is chosen specifically to support a point of view. In this case it is referred to as cherry picking.

Time scale is also a possible biasing factor. Surveys done only at certain times or for too short a period of time will bias a result so that it is not applicable to other times. This is often seen in political approval ratings, where approval during a specific time frame (such as during a crisis or scandal) is not representative of public approval over the course of several years. A clear bias can be made if sample data is restricted to a certain time frame that emphasises a change. RationalWiki's pageviews between March 15^th and March 21^st 2011 went down significantly according to Alexa.com,[4] but this isn't representative of a longer trend that actually shows an increase. Again, intentional massaging of data by cutting off certain time periods is cherry picking, and greatly frowned up on science and statistics but is often the way global warming denialism presents data.

See also

• Cherry picking
• Confirmation bias
• Dewey Defeats Truman, an example of how bad polling prompted a newspaper to call an election prematurely for the losing candidate
• Spotlight fallacy
• Push poll
• Survivorship bias – a form of selection bias focusing on what has survived to the present and ignoring what must have been lost.

References

1. Stop Street Harassment - Statistics
2. Vote on alternative medicine falls victim to dark arts of the internet
3. WND - Why do you think CBS News ratings are so low? As a random example.
4. Alexa.com - Rationalwiki.org, site info - Observing the noise, it's possible to clip the data at several points to show many different trends.

Articles on Cognitive biases

  Clustering illusion  -  Confabulation  -  Confirmation bias  -  Delusion  -  Demonization  -  Dunning–Kruger effect  -  Experimenter bias  -  Framing  -  Frequency illusion  -  Gambler's fallacy  -  Hindsight bias  -  Just world hypothesis  -  Moral licensing  -  Outcome bias  -  Overconfidence effect  -  Overgeneralization  -  Response bias  -  Survivorship bias  -  Ultimate attribution error  -  Walled garden