December 13, 2010 New Yorker article: “The Truth Wears Off: Is there something wrong with the scientific method,” by Johan Lehrer.
Mr. Lehrer reports on a strange effect beginning to be noticed in science journals: replicated experiments failing to confirm results reported in earlier papers. It’s not a matter of mistake or academic fraud. These are cases where protocols were followed as claimed and fully peer reviewed. What is going on?
Part of the effect may be probability. If 20 experiments reject the null hypothesis with 95 percent confidence, one of them is likely to be wrong. With more than 400,000 scientific papers published annually, you can do the math!
Another cause may be our reluctance to report experiments with negative results. “We looked, but didn’t find anything” is not exactly a hot ticket to tenure. So, we selectively write about positive results, and those Type I errors add up. To our credit, JAWRA has been willing to report negative findings, especially where they fail to confirm what one might expect. However, the vast majority of submittals remain positive findings.
I like to think peer-reviewed journals are the gold standard or journalism, especially compared to, say, Fox “News.” But, even we have limits. I guess the lesson here is to always be skeptical of what you read, and always be prepared to let the facts change your theory.
No related posts.
Related posts brought to you by Yet Another Related Posts Plugin.