Irreproducible research spurs reassessment of editorial guidelines

Publish or perish, they say. But what if the pressure to do so is undermining the scientific process?

C. Glenn Begley, senior vice-president at TetraLogic Pharmaceuticals in Malvern, Pa., is acutely aware of this possibility. Whenever he asks professors what’s the most important thing for students and postdoctoral fellows to do while at their institution. The answer, almost invariably, is to get a paper into Nature or Science or Cell. Doing so guarantees the next position for the student or postdoc, and the next grant for the principal investigator.

This of course leads to competing interests, beyond the direct financial interests researchers are typically required to disclose: The pressure to publish can outweigh the desire to report only the most rigorously tested and carefully presented research. As a result, we are beginning to see a degree of carelessness creep into scientific publications — and sometimes something slightly more sinister.

“I’ve seen situations where the PI and the postdoc know they are publishing information that’s at least misleading and possibly even wrong,” Begley said. And they’re OK with that, “because by the time it gets corrected the postdoc will have the job and the PI will have the grant.”

The upshot? There’s this, for starters: In a 2012 Nature Comment, Begley and Lee Ellis described a study in which Amgen scientists were able to reproduce the findings in only 11 percent of 53 published papers in preclinical research. Even more strikingly, they found that some of the original authors of the papers could not reproduce their own work, in their own labs using their own reagents.

The irreproducibility of research has become a significant problem. It has the effect of sending researchers down blind alleyways, eating up resources and generally slowing scientific progress. It is, Begley told me, “a profound distraction.”

Taking steps to address the problem
Other researchers — including a group at Bayer Healthcare, writing in Nature Reviews Drug Discovery — have reported similar numbers of irreproducible studies in top-tier journals, as have scientists from Novartis and Astra Zeneca speaking at the American Association for Cancer annual meeting in 2012. The problem is pervasive, they say, involving a great many labs and journals. And it is likely due to the pressure on researchers to publish.

So what can be done? In a second Nature Comment, published last month, Begley outlined six questions that every author, editor, reviewer and reader should ask themselves when evaluating a paper. For example: Were the experiments performed blinded? Were all of the results presented? And, were there positive and negative controls?

Also, journals themselves are beginning to review their own protocols and how these impact the reproducibility of the studies they publish. Nature is introducing editorial measures to address the problem, ensuring that key methodological details are reported, for example, and encouraging authors to include their raw data with publications.

It shouldn’t stop there, though. Granting agencies should be more serious about looking at work no one has been able to reproduce and “crossing it off the researcher’s CV,” Begley told me. And institutions should be prepared to mentor and counsel their investigators about carelessness in research.