Note: The ideas for this article originated from the Self-Correction Science Working Group of the Association for the Improvement of Psychological Science. A follow-up blog will provide more detail on this new organization working to improve psychological science.
Contributors to the ideas presented here include Lisa Oster-Gusman, Andy Fonash, Marcel Van Assen, Richie Lin, Joe Hilgard, Victor Keller, Tracy Mann, Alex MacDiarmid, Alexa Tollett, and Simon Columbus. However, all errors, outrages, misstatements, and other problems in this article are entirely mine.
Why You Should Care About Whether Psychology Is a Self-Correction Science
First, for you, my non-specialist audience. You know all those great advice columns, with titles like (I made this up, but it’s easy to find stuff like this): Six Ways to Tell If Your Partner Is a Narcissist; Four Ways to Have Better Sex; What Matters to Teens?
The information in such articles is either true or false; sometimes it can be misleading oversimplifications. Scientific research, if it exists on the subject, is always the best way to determine whether something is true, false, or even somewhat true. But there’s a problem. Scientific “facts” are not the permanent, unchanging things they seem to be. In 1900, scientists concluded that the sun is made of the same stuff as the Earth. By 1950, it was well understood that the sun is mostly hydrogen and helium. In 1980, the best minds in medical science concluded that stress causes ulcers; in 2005, Barry Marshall won the Nobel Prize in Physiology or Medicine for proving that bacteria cause ulcers (although stress can make tissues more susceptible to bacterial infection). Why is this important? One of the great claims that science makes for its special status and credibility is that it is “self-correcting”—that is, when new data appear that show something that contradicts scientific conclusions, scientists are supposed to change their conclusions. But often they don’t, at least not easily. Accepted scientific wisdom changes, but usually because new scientists embrace the correction, not because the old guard “corrects itself.” Do you think I’m kidding? Then consider what the Nobel laureate and nuclear physicist Max Planck once said: “Scientific truth triumphs not by convincing its opponents and making them see the light of day, but because its opponents eventually die and a new generation grows up knowing it.”
This means that you, as a consumer of psychology, may routinely be “sold” claims that are dubious at best, and downright false at worst. Do you think I’m kidding? You’ve probably come across some of these claims:
- Power posture can change your life
- Stereotype threat explains racial differences in academic achievement
- Implicit biases that we’re not aware of are a powerful source of discrimination and inequality in the real world
- Smiling improves your mood
- Feelings of disgust influence moral judgments
- Conservatives deny science far more than liberals do
- Stereotypes are inaccurate
- People interpret their behavior as a result of their attitudes; they interpret other people’s behavior as a result of their characteristics
- Diversity programs are highly effective at reducing discrimination
- Microaggressions are a pervasive and entrenched form of discrimination
- Type A behavior is bad for your health
All of these claims have either failed or proven dubious, with inconclusive evidence, or are controversial among scientists.
And that doesn’t even include the large percentage of self-help advice promoted in books and blogs.
Why and How Do Dubious Claims Get Promoted?
- Honest Mistakes. Sometimes the evidence seems to support a claim, and subsequent research identifies problems with the original studies or proves that the result cannot be replicated. Science, even at its best, can resemble quasi-systematic trial and error. This means that sometimes results that were honestly produced, even with the best methods, turn out to be incorrect.
- Promote dramatic results too early. The problem of honest mistakes would be less serious if scientists didn’t rush to promote them before intense skeptical scrutiny from the scientific community. This scrutiny can often take a decade or more. That may seem like a long time, but consider this timeline:
Year 0. Some studies have been published. It takes a few years for people to learn about this and grasp its significance.
In years two and three, others begin to design replicas and skeptical tests of the claims.
In years four and five, the first follow-up studies begin to be published.
In years six and seven, follow-up studies begin to spread throughout the field.
In years eight and beyond, a more complete understanding of the original phenomenon and its limitations begins to emerge.
Why, then, do so many psychologists rush to promote their findings in blogs, press releases, major newspapers, and TED talks before they have been fully examined? Do you have to ask? First, there are all the usual selfish reasons. If Dr. X can make a name for himself with some dramatic results, get grants and tenure, receive a large consulting fee, and write a best-selling book by the time the rest of the world discovers that X’s original result is not what he advertises, then X is a department chair at a major university. And if someone complains about this, X can respond by saying, “Well, that’s how science works. We don’t always get it right the first time. One of the great strengths of psychological science is that it is self-correcting.”
But I often doubt that psychologists do this in all seriousness. They mean well, they are excited about their results, they think they have found something important, and they want to tell the world. But ultimately, when it comes to self-correction, it doesn’t matter much whether scientists jump to conclusions and rush to tell the world for good or bad reasons. Either way, a healthy dose of self-correction is needed.
- Convincing narrative. Scientists strive to write well, and some do so brilliantly. Good writing is a critical skill for communicating with other scientists and the general public. Yet most of us have learned to write convincing narratives, rather than acknowledge the messy or inconclusive truth of most data. This has probably created a gap between the strength of the narrative and the strength of the evidence, at least in my home field of social psychology. Many studies have used samples so small that they should all be viewed as tentative, awaiting replication.
- Refuse to accept disconfirmation. The number of times critics find errors in research, and the original authors respond with some change like “Thanks for the correction; of course, our main results still stand” is endless. Another response we hear often is “[Thanks but] the failure to replicate simply indicates that our results, which still stand, cannot be generalized to other contexts.” Or the suggestion that the scientists who performed the replication failed to carry out the research well.
This conversation recently took place on NPR’s Radiolab, which featured a segment on stereotype threat, a controversial area of research whose proponents claim that fear of confirming negative stereotypes about race or gender undermines the academic achievement of African Americans and women. About 32 minutes in, you’ll hear:
Radiolab: “Since repetition was a topic of discussion that you unfortunately had to deal with, did that change your opinion of the work?”
Claude Steele (inventor of the concept of stereotype threat): “No. I don’t think there’s anything that would make me think that this whole thing is wrong.”
Well then. If evidence can’t change a scientific claim, what can?
Conclusion on Scientific Self-Correction and Stereotype Threat
Shortly after I published this, I discovered that Claude Steele published an article that, depending on one’s interpretation, could be seen as either distancing himself from that observation or explaining why it was ultimately justified. You can find the full article here.
Since the purpose of this post was to address the general issues of scientific self-correction, I will leave that issue here for now.
I have a follow-up article on efforts to improve psychological science here.
If you find this article interesting, you can follow me on Twitter, where I focus on issues of stereotyping, bias, discrimination, science reform, free speech, and the academy.
