Improbable things happen

Creationists (e.g., William Lane Craig) and all manner of non-rationalists like to disparage their opponents or bolster their own arguments by pointing out the lack of probability of something happening. Out of all the possibilities, they say, this one is the one that occurred — how fantastically unlikely and amazingly miraculous! It is simply impossible to believe that it just happened by chance!

Cogito ergo sum
Logic and rhetoric
Key articles
General logic
Bad logic
v - t - e
“”That a particular specified event or coincidence will occur is very unlikely. That some astonishing unspecified events will occur is certain. That is why remarkable coincidences are noted in hindsight, not predicted with foresight.
—David G. Myers[1]

Improbable things happen all the time.

But improbable things happen all the time because "improbability" is an illusion based on our preconceptions. Often it has nothing to do with statistical truth. The trouble is that we can't grasp the difference between (a) "This particular improbable pattern of lottery numbers came up on this particular day in this particular lottery" and (b) "Some improbable pattern of lottery numbers came up sometime in the last five years somewhere in the world."

In short: "improbability" does not imply "impossibility".

Fat chance

The lottery

Possibly the simplest example is a lottery. These often have incredible odds that seem impossible to beat, but indeed someone (almost) always wins.[note 1] This is because of the sheer number of people playing. Even though an individual has a low chance of success, overall it's almost certain that it will be won by somebody[note 2]. Most people will refrain from submitting a ticket with six sequential numbers because of the rationalisation that such a draw is too improbable — despite the fact that all draws are equally likely.

The idea of this can also be expressed by looking at car licence plates. Imagine seeing one with the configuration HJB-546.File:Wikipedia's W.svg. That's one out of a combination of over 17 million, so it seems like a remarkably improbable feat if you treat it in the same way as the statistically illiterate. But any combination is equally improbable, and you're certain to see one of the combinations if you look for it. It would only become remarkable if you predicted the configuration in advance.

As Richard Feynman once quipped:

“”You know, the most amazing thing happened to me tonight. I was coming here, on the way to the lecture, and I came in through the parking lot. And you won't believe what happened. I saw a car with the license plate ARW 357. Can you imagine? Of all the millions of license plates in the state, what was the chance I would see that particular one tonight? Amazing!

The same birthday

Consider a party attended by thirty people: what are the chances that two of them have the same birthday (ignoring leap years, and assuming the attendees' birthdays are entirely random)? One in twelve, or, roughly 8% (30/365)? After all, it's a 1 in 365 chance someone will share your birthday, and by the lottery analogy above, there's 30 shots at winning.

No, the odds are significantly better than that. In fact, there is a 70% probability.[note 3]

This is known as the "birthday problem". The apparently miraculous breaking of odds is attributed to the fact that the question is "what is the chance that any two people have the same birthday?", whereas most people following common sense tend to translate the question as "what is the chance that someone has the same birthday as mine?". So while you get 30 shots at this 1 in 365 lottery, so does everyone else. More specifically, every possible pairings of two individuals in the group of 30 has a shot at this 1 in 365 chance. Regardless, the answer is very non-intuitive and is a good display of how people don't do well at guessing probabilities. Once the problem is known, however, calculating the real odds is just a simple case of exploiting the correct mathematics.[2]

Shuffling a deck of cards

Do you want to witness an "improbable" event right now in your very own home?

Take a standard deck of 52 cards, shuffle it well and spread the cards in a line. Look at them well. Assuming an ideally random shuffle, the probability of a card sequence in this exact order is…

1 in 80658175170943878571660636856403766975289505440883277824000000000000.

Really. And yet despite this very low probability, you just got that sequence. Which may be mindblowing if you haven't studied statistics or combinatorics. Of course, this is because the probability that is given to you is ex ante and when you are reading the sequence of the cards after you shuffle them, you are simply validating what you see. The ex post probability of getting that particular sequence is always 100%.[note 4]

Ourselves

It is wildly improbable that any person alive is — well, alive. To be alive, for instance yours truly sitting in front of a computer and writing, is so improbable a priori, that "Borel's Law" (poor Borel) rules out my existence. For countless generations a particular sperm cell had to encounter a particular egg and produce every single descendant — and that goes for all the other lines as well. The a priori probability is staggeringly low — and yet, here I am. According to "creationist probability" I shouldn't exist.

Beatrice, Nebraska

Beatrice, Nebraska, is probably not widely known, but a miracle took place there in the evening of March 1, 1950. The church choir was due to meet at 7:20 p.m. All 15 members were late due to 10 separate reasons. The church exploded at 7:25 p.m. The members obviously wondered about God's hand in this.[3][4]

Mathematics-filled analysis for people who inexplicably like math
See also Littlewood's law, Ramsey theory

One example in statistics is the phenomenon of "at least one". Imagine 6 cards are laid out face-down and the only certainty is that 2 cards are aces and 4 cards are not aces. What many people will assume from intuition is the chance of choosing at least one ace when flipping two cards over is 2 in 6 (~33%). This is only true of drawing an ace on the first attempt, however. The actual chance of picking up at least one ace is much better than that.

This works because the probability of at least one is equal to 1 minus the probability of none, and it's this calculation that needs to be made. It might seem backwards — because it is — but this is the easiest way to calculate an "at least one" probability as this also includes the chances of drawing more than one automatically. In this case the probability of not drawing any aces can be determined with the formula P(A)*P(B|A), which is read as "the probability of A multiplied by the probability of B assuming that event A already occurred". P(A) is the probability of not turning over an ace out of 6 cards, and P(B|A) is the probability of not turning over an ace out of five cards assuming you didn't the first time (since there is no replacement of the first card). This quite clearly gives us the odds of not turning over an ace across two attempts and is all we need to solve the problem. So, P(A)*P(B|A) would work out as (4/6)*(3/5), which equals 12/30, or 40%. Therefore, we can conclude that the odds of drawing "at least one" ace is actually 60%.

Working from a full deck of 52 illustrates why this backwards method for "at least one" works more efficiently. To do the calculation forward you would have to calculate and combine the individual odds of drawing one, two, three and four aces in different combinations. For example, drawing an ace on the second attempt is different as you're drawing from 51 cards, not 52 so you need to calculate (48/52)*(47/51) and add it to a stack of other possible combinations. This becomes increasingly complicated and only becomes more so if you start increasing the number of attempts. On the other hand, it's only a single calculation to work out the probability of drawing no aces. This is (48/52)*(47/51)*(46/50)*(45/49), about 0.72. So the probability of drawing at least one ace in four attempts is 0.28, about 3 in 10 — remarkably good odds for such a "rare" card.

This is similar to the case described above of many players playing the lottery. The 2 in 6 odds are true for any one selection. But if we were given a second chance to play again from scratch and at least one had to successfully draw an ace, these odds would additively combine to 4 in 6, or ~67%.

Actual uses

The System

The effect was exploited in Derren Brown's TV special "The System," where he presented a system for winning bets placed on multiple race horses. He began with several thousand volunteers and then subsequently only followed the winners; the final product that was televised only featured one individual, making his "system" seem miraculous. To demonstrate the system, he also performed the coin tossing trick, taking around 9 hours to film all of his attempts until he did come up with a successful combination.[5]

Paul the Octopus

A similar thing happened in the 2010 South Africa World Cup, when Paul the Octopus was thought to have predicted the outcome of eight matches. A great part of the real explanation is very simple: there was a 1 in 256 probability that Paul could predict the outcome of eight games, and Paul just casually happened to be that one in 256 that was reported in the media. (Magical thinking, of course, processed this fact as Paul being a psychic octopus.)

Large sporting events like the World Cup generate masses of interest and undoubtedly many people will try to predict the outcome — in fact, it would be unlikely that an event this size should attract less than the 256 people or processes required to statistically guess the 8 matches correctly. Similar to the Derren Brown example discussed above, this shall be self-selecting. Only a fraction shall guess the first game correctly, a fraction of those shall guess the second and so on. By the time it's whittled down to the last few games (not unlike a football tournament, of course) people might be gathering attention as being "on a lucky streak". Naturally, the ones who fall at the final hurdle lose their streak, while the winners emerge as skilled, or psychic.

The main difference between sports betting and the other examples above, however, is that the odds are not mathematically perfect. Teams have different levels of performance and ranking, and favourites are very likely to emerge. As a result, it's never really a 50:50 chance for any team entering a match — seriously, ask any bookies to give you evens on Brazil v England and they will laugh in your face. As a result, for most people keen on the sport it's actually a little under the 1-in-256 odds required to guess 8 games in a row. This only goes to convert an apparently improbable performance of prediction into a dead certainty.

Redskins Rule

The Washington Redskins moved to Washington, DC in 1937. Since then there have been 18 US Presidential elections and in 17 of those, the following rule has remained true:

If the Redskins win their last home game before the election, the party that won the previous election (the incumbent party) wins the next election. If the Redskins lose this last home game, the incumbent also loses and the challenging party's candidate wins.

Folklore had established this rule by the early 1990s, but only became widely known around 2000. Since it was brought to light in 2000, though, there have only been 3 elections, and two of them (2004 and 2012) didn't obey the rule at all — demonstrating past observations don't affect future probabilities. This is a firm example of post hoc reasoning via selection. There are dozens of teams in the NFL (add to that the NBA, NHL, and so on…) and so the odds of at least one of these teams' results syncing up with the election is more modest than you'd think. Certainly, if the rule didn't hold true, it shouldn't be reported. Much like in the System, above, the rule is self-selecting, as fewer teams — since the 1930s — would sync up with the election so well. For instance, if we start in the 1932 election between Herbert Hoover and Franklin D. Roosevelt, then about half of all teams playing in the 1932 season would have won their last home game, and obeyed the rule quite well. From there, it's a trivial case of letting random chance converge on a team that correlates quite well.

The Law of Large Numbers

According to the ergodic hypothesis, given an infinite universe, every event with non-zero probability, however small, shall eventually occur. Or put another way: given enough chances, even the most unlikely event is certain to happen.

When talking about the improbable, it's easy to ignore the cases where the event does not happen. People are naturally self-centered and think about their own experience first: from any one individual's point of view, the odds of winning the lottery are minuscule and the odds of finding someone with the same birthday are exactly as you'd expect.

But when considered in a more comprehensive and inclusive way, the true odds are revealed. For example, the probability of one particular mutation during evolution may be tiny, but there are billions of mutations happening continuously and being sorted by natural selection. Because of all these chances, that one minute possibility isn't really unlikely at all. It's a certainty.

We tend to pay attention to the improbable things that do happen and never to the improbable things that don't happen and don't defy the odds. This particular cognitive bias is an important aspect of the Black Swan theory of improbable events. We may be staggered by an event with a 1-in-a-million odds, but completely ignore that at least 999,999 other 1-in-a-million events just happened to have not occurred. This is often boosted by a form of post hoc fallacy that explains the event that happened but discounts the events that don't, analogous to rolling a die but only ever telling someone or acknowledging the roll when it's a 6; indeed the die may be invisible and no one knows it's being rolled until it shows a 6.

In short, one-in-a-million events happen all the time.

gollark: I do generally prefer flying to cars for reasonably long distances, but the whole "security" thing at airports really does a great job at making me reevaluate that. And is not actually improving security at all.
gollark: How does hyperloop compare to regular maglevs?
gollark: The Thought Police ~~would like to~~ know your location.
gollark: Yes.
gollark: It seems like score voting (or approval I guess, easiest change) would be the best system for voting. But there are a lot of annoying tradeoffs and weird issues. Also Arrow's theorem, but IIRC that only affects ranked ones.

See also

Notes
  1. There have even been dozens of instances where individuals or groups have won the jackpot of a lottery twice.
  2. For absolute certainty that there is a winner, try a raffle.
  3. Don't believe us? Here's the math. Take your group of people N and the group of people other than you K. Divide by two, because A paired with B is the same as B paired with A.
    (30 * 29) / 2 = 435
    Now, 364 out of 365 scenarios the person you meet shall not have the same birthday as you (because there are 365 days of the year, see?). So, the chances of someone not having the same birthday as you are 364/365, or 0.997260. Now, to figure out the chances of our group of people not having at least one birthday pair, we raise 0.997260 to the 435th power. Making sense?
    .997260435 = 0.303147
    We have now ascertained the probability that our group does not have a pair. To find the probability that our group does have a pair, we use the simple formula 1 - the probability that we don't have a pair.

    1 - 0.303147 = 0.696853

    If we round up, it's 70%. If we don't, it's a 69.6853% chance that we have at least one pair of birthdays in a group of thirty.
  4. Where ex ante and ex post just refer to before and after you shuffle the cards, respectively. The probability that something did occur, given that it occurred, is always 100%!

References
  1. The Power of Coincidence by David G. Myers (September 23, 2002) Hope College.
  2. Craig Andrea - The Birthday Problem Calculator
  3. Taken from Warren Weaver's, Lady Luck, 1963.
  4. "Church Explosion Spares Choir". Snopes.
  5. Derren Brown's The System on YouTube, 3 February 2018.
This article is issued from Rationalwiki. The text is licensed under Creative Commons - Attribution - Sharealike. Additional terms may apply for the media files.