Replication Crisis

I have been thinking more about science since my last post about it, and the more I think about it, the more I get discouraged.

I don’t know if you’ve heard but there is a crisis afoot in science: the Replication Crisis (see also this video). Basically, scientists are struggling to replicate, or reproduce, results obtained by other scientists.

To see why this is important, let’s do a quick review of the steps of the scientific method:

1. Start with a question about the natural world.
2. Come up with a plausible explanation, called the hypothesis.
3. Test the hypothesis.
4. If the hypothesis passes the test, go to step 3 and test it some more.
5. If the hypothesis does not pass the test, go to step 2 and formulate another hypothesis.

While the order and possible inclusion of these steps should not be considered set in stone, the steps themselves should not be altered in any way.

But if the above steps are the pillars of science, there is a cornerstone: replicability and reproducibility. These together form the foundation of science; they are the reason that science is trusted so much in the modern world.

And for good reason.

But…replication is happening less and less. And sometimes, when it is tried, it fails. That’s the Replication Crisis.

Retraction Crisis

This is disheartening, but that isn’t the only problem.

There is a massive amount of scientific misconduct, so much so that we have what I call a Retraction Crisis, as illustrated by a blog that is only about retractions of “scientific” papers. And the authors of that blog have created a database of retractions that has 21,000 separate instances and about 50,000 authors!

Motivation

Why is this happening? I think the answer is motivation: the motivation of the scientists behind the papers.

There are, of course, good and bad motivators. I will not talk about good motivators because I believe they produce good science.

Let’s talk about two bad motivators.

Money

I don’t think anyone would disagree that money is a bad motivator. I would go further and add that it doesn’t matter where the money comes from, whether giant corporations or governments.

If money is a scientist’s primary motivator, he is not going to do good science.

Ideology

The other bad motivator is ideology. If a scientist already believes something, then they are subject to confirmation bias, which means they will not be impartial. And impartiality is crucial.

What I Personally Do Now

I have a lot of faith in the scientific method, but these crises scare me; they have eroded some of my trust in the scientific community. I trust the method, but I can’t say I completely trust the imperfect humans in the scientific community.

It seems that in order to know the truth for myself, I can no longer blindly trust the experts; I can only trust myself.

Because I can no longer blindly trust the experts, I am not sure how to convince myself of the veracity of scientific results without replication. As such, I have a new personal policy:

I will not trust any new scientific result until and unless either two independent parties replicate the results or I replicate them personally.

On top of that, I will start identifying and investigating fishy results accepted by the scientific consensus.

To do this, I will need to relearn a lot of important skills and knowledge, like statistics, calculus, experimental methods, proof methods (in mathematics), and other essential skills and knowledge for scientists.

I suggest you do the same.

A Question for Scientists

But before I go, I have a question for scientists: why should I trust you? Why should I have enough faith in you to not have to replicate all of the results you give me?

Any serious and honest scientist would give only one answer: that I should not trust them.

If scientists were to be honest with themselves, they would have to acknowledge that these two crises are so large that the loss of confidence in the scientific community is unfortunately deserved. Only dishonest scientists would continue assuring us that all is well.

Why am I so confident in that conclusion? Because I used the scientific method to reach it. You see, when I started questioning my trust in the scientific community, I did the following:

1. Formulated a question: can I trust the scientific community?
2. Came up with a hypothesis: yes, I can because the scientific community hasn’t had any egregious scandals.
3. Tested the hypothesis: I started looking for counter examples by looking for scandals.
4. Failed the hypothesis: I found many egregious scandals.
5. Formed a new hypothesis: most scientists are trustworthy, but I don’t know which ones, so I will need to adjust my trust accordingly.

And the solution I found was that I could only trust that which had been replicated because honest experiments by honest scientists will (almost) always be replicable and reproducible.

Conclusion

Scientists, if you want to restore trust, you must earn it back. You must replicate results many times before claiming them to be correct. And you must remove academic dishonesty. Period.

And until you do, then I, for one, only trust myself. And the Prophet.