The Scientific Method

You're doing it right.

There are two kinds of scientific progress: the methodical experimentation and categorization which gradually extend the boundaries of knowledge, and the revolutionary leap of genius which redefines and transcends those boundaries. Acknowledging our debt to the former, we yearn, nonetheless, for the latter.

—Academician Prokhor Zakharov, Sid Meier's Alpha Centauri

Ah, the scientific method, hated by The Fundamentalist, loved by anyone else. Not understanding (or using) it is a key part of Did Not Do the Research. Misrepresenting or outright vilifying it leads immediately into You Fail Indexes Forever territory.

If a Straw Vulcan tries to use it, expect the method to be rather unscientific.

The actual Scientific Method is rather simple. You'd think this would be hard to mess up, but it is done anyway. It comes in four basic steps:

1. Observe the conditions. (You have to lay down certain ground rules before the rest.)
2. Form a hypothesis. (Ask a question, or make a statement. "Since A behaves in X way, if Y is done Z should happen, because G." Or something of that sort.)
3. Make a prediction based on your hypothesis. (This gives you something to test. It also limits the power of your bias to interpret your results to be what you wanted. For example, if you're testing the effectiveness of a cleaning fluid, you can't just clean a mess with it, inspect the result, and decide afterward that what you see is "pretty clean"; making a specific prediction beforehand, with guidelines to determine how you measure things, forces you to interpret whatever really happened.)
4. Test it. (This being a rather important step, as it helps give you evidence to your hypothesis.)

It doesn't have to be exclusively used by scientists. A setting with Magic A Is Magic A may have the magic users applying the scientific method to magic to improve their understanding of it.

There are several other key elements to the scientific process which fiction and news media have been known to ignore for the sake of convenience or sensationalism:

• Repeatability. Despite popular portrayals, no one experiment can or should single-handedly overturn a theory, or create a new one. After all, experiments involve numerous variables and events; the results from just one won't necessarily tell you what you assume they are (and you may have simply made a mistake). By repeating experiments, you get a better sense of whether the phenomenon you're studying occurs throughout the year, with different entities, etc. If it keeps happening, the phenomenon is "repeatable", and the more repeatable it is the more likely it is to be confirmed. (This is one of the hurdles faced by any Einstein Sue whose hypotheses contradict the current consensus.)

• Consilience. Whatever your findings are, they ought to match those of other scientists in other fields, or else something somewhere is askew. A geneticist comparing bird DNA to reptile DNA should have similar results as a zoologist comparing their bones, because the theory of evolution predicts that these types of data match up. When concilience isn't happening, it raises some flags, as with the apparent contradictions between the current models of relativity and quantum mechanics. This can be frustrating, but it also makes many scientists excited for the opportunity to fill a hole in our understanding and maybe win a Nobel prize or two.

• Control. Your experiment may not be very meaningful if you only observe changes in a single subject. It's good to have a "neutral" subject, kept as similar to the original as possible, for the sake of comparison. For example, in order to be approved by the United States FDA, a drug has to go through a test wherein one group of people take the drug and another (unknowingly) take a fake pill called a placebo. Without the control group, it would be harder to say whether any improvement was actually caused by the drug, as opposed to the placebo effect (when people feel better because they think they took the drug) or just because people with that condition happen to get better over time anyway. Post hoc ergo propter hoc is the fancy Latin term for the fallacy of assuming, "X happened, then Y happened, therefore X caused Y!"; in English we say, "correlation does not prove causation." Fictional science falls prey to the fallacy with depressing frequency, and in real life, control is one means of fighting it. One everyday situation in which non-scientists use "control subjects" is diagnosing problems with a piece of technology. E.g., if a DVD isn't playing properly, you might keep everything the same but switch the DVD player with a different one, or try a different DVD, etc, until you isolate the cause.

• Falsifiability. This one can be counter-intuitive. Falsifiability does not mean "it's false, period", it asks whether your theory also has the possibility of being shown false by an experiment. For example, a claim that an ancient Japanese coin is buried somewhere in the soil of Mars could hypothetically be proven, but it is for all practical purposes impossible to disprove ("the soil of Mars" being essentially an infinite resource that we can never finish digging up; the coin could always be just in the next shovelful). Why is falsifiability important? If your theory or experiment is designed to prevent your idea being wrong every time rather than show it as a natural occurrence, people will accuse you of faking, Moving the Goalposts or advocating God of the Gaps. Notice how this connects to the "make a prediction" step in the scientific method; you need to make sufficiently specific predictions at least partially to give someone room to disprove you.

None of these various principles is so ironclad that an experiment or observation that lacks it is necessarily "not science". Sometimes, a hypothesized event, such as the conditions of the universe in the seconds after the Big Bang, or the evolution of amphibians from fish, can't be repeated, but this doesn't mean there's no warrant to speculate about them based on the evidence those events left behind in the form of cosmic radiation and fossils. Other times, the nature of the subject prohibits control; climatologists can't create a second Earth to see how things would go without human industry, but this doesn't mean there can be no science of climatology.

Of the four things mentioned here, perhaps the most controversial is falsifiability, which is tightly associated with the philosopher Karl Popper. Under Popper's model, true science does not involve confirmation, only dis-confirmation; some scientists have called this too extreme. Meanwhile, various theories, especially "big" ones that cover a wide range of stuff (relativity, evolution, etc) have been accused of being unfalsifiable. It's possible that no single observation would totally disprove such theories (but if you've been reading along, you would remember that from the section above on repeatability). It's a matter of debate whether or not Einstein's theories truly overturned Newton's or subsumed them; like Newton's, some theories may be so well established that future observations could only "modify" them rather than destroy them altogether.

Altogether, the whole process is a lot more tedious than media would have it, but it's precisely the steady persistence over boredom and easy ignorance that give us robust models of nature which help us cure diseases and travel in space.