Peer review goes under the lens

How can we improve the filter in scientific publishing?

Imagine receiving a stack of raw research abstracts roughly the height of the Statue of Liberty with a note on top:

“Find the important information in here without quitting your day job.”

This scenario has roots in reality. Journals published more than 756,000 life science articles in 2007 alone, according to PubMed, a database service maintained by the U.S. National Library of Medicine. Gathered into a single pile, with each abstract on a single page, these articles would approach the tip of Lady Liberty’s torch.

Luckily, the editorial peer review process spares scientists from sifting through the entire stack by shuttling each study to a journal with a known mission and reputation. There, an editor typically asks two or three scientists who work in the author’s field to review the manuscript, assuming it’s interesting and relevant. These “peer reviewers” comb through the manuscript to determine if it meets the journal’s standards. If they think the article is appropriate for the publication, they often recommend ways to improve it.

“We have a system where we get several people together to think critically about a paper and give them the anonymity to say what they really think,” says Whitehead Member Iain Cheeseman.

As a result, significant studies usually appear in respected journals while lesser work is relegated to obscure journals with smaller circulations.

But peer review isn’t perfect.

“It’s like democracy—it’s flawed, but it’s better than the alternative,” says Whitehead Member Robert Weinberg.

Blinded by bias?
In a scientific utopia populated by totally selfless and objective researchers, peer reviewers would always make papers better. In reality, however, reviewers bring their own biases to the process.

Stanford University’s Michael Clarke ran into resistance, for example, when he discovered cancer stem cells in solid tumors and submitted a paper to Nature. “To my mind, Michael Clarke’s paper is one of the most important of the decade, and although he produced solid data, Nature rejected it soundly,” says Weinberg. It was eventually published in the Proceedings of the National Academy of Sciences (PNAS).

Whitehead Member Harvey Lodish also has seen reviewers pan papers that challenge conventional thinking, regardless of the supporting evidence. In 2000, his lab showed that adiponectin—a hormone produced by fat cells—causes weight loss by altering muscle metabolism. Nature rejected the study, so Lodish published it in PNAS. “The reviewers at Nature didn’t have anything technically against the study; they just didn’t believe it,” says Lodish.

The following year, two labs independently confirmed the adiponectin finding and Nature Medicine published their papers.

Emilie Marcus, editor-in-chief of Cell, notes that the system relies on reviewer advice from diverse perspectives, so it is not uncommon for reviewers to have differing opinions on a paper. She adds that the responsibility of the editor is to carefully evaluate the science behind both the supportive and the critical reviewer comments, often with the help of further discussions with authors and reviewers before arriving at a final decision.

Given the importance of publishing in a researcher’s career and the inherent disappointment of a negative editorial decision, peer review is a natural target of criticism and resentment, she says. “It’s easier for a scientist to criticize the process than to admit that his paper wasn’t right for the journal,” adds Marcus. “I’m sure that bias exists in peer review, but its role is overestimated.”

Reviewing the reviewers
When scientists disagree, they typically turn to empirical data. But there’s little on the pitfalls of peer review.

“I would love to publish articles on this topic, but I’m not getting any,” says Joan Sieber, editor of the Journal of Empirical Research on Human Research Ethics. “Part of the problem is that while the notion of getting to the bottom of what’s wrong with peer review sounds very exciting, the individual hypotheses that one can examine are fairly mundane.”

Take one of the most common complaints: Peer review favors well-known authors. Typically, biomedical journals disclose the identities of authors while granting peer reviewers anonymity. According to the Publishing Research Consortium, 56 percent of academics believe the author’s identity should be concealed from reviewers. A handful of journals have experimented with this “double-blind” review, but the jury is still out on the benefits. In many cases, reviewers can guess the identity of the author based on references in the manuscript to previous work.

Sieber believes researchers need to gather more empirical data before changing the system. Even the benefits of peer review remain undocumented.

In 2002, a rare systematic review of the literature published in Journal of the American Medical Association revealed that editorial peer review is largely untested and its effects on quality are uncertain. A 2007 review by the Cochrane Collaboration—a U.K.-based international health care analysis group—confirmed this conclusion.

Of course, since few researchers have examined the effects of peer review, future studies also may well produce empirical evidence that it works well overall.

Most scientists remain confident that peer review serves a useful purpose. In fact, 90 percent of academic researchers believe that it improves the quality of the published paper, stated a 2008 survey by the Publishing Research Consortium.

Room for improvement
Despite overwhelming support for peer review, just 32 percent of academic researchers agree that the current peer review system is the best we can achieve.

One might expect the ensuing discussion to focus on detecting fraud, given the well-publicized scandals involving scientists who cheated the peer review system. But Sieber argues that fraud detection is an impossible requirement. “I don’t know how a journal could possibly detect fraud in each case,” she says. “Journals don’t have the resources to police scientists.”

“Implicit in the review process is that one trusts what the author has written,” notes Weinberg. “Peer reviewers should be alert to fraud, but cannot be expected to catch all guilty parties.”

Thus, within the scientific community, many of the proposed improvements target bias rather than fraud.

Whitehead Member Susan Lindquist, for example, favors an open model, under which peer reviewers sign their comments. She predicts the literature would benefit from a dialogue between authors and reviewers, although she acknowledges the exchange would also have some negative side effects.

“Junior faculty members might be reluctant to criticize senior investigators in public,” she says. “But I think anonymity makes some reviewers a bit irresponsible.”

Lindquist says she has watched the number of baseless comments in reviews skyrocket over the last decade and believes the harsh funding climate is partly to blame. Many investigators are struggling to secure grants, and she worries that the review process provides an easy target for their frustration. Given the plethora of negative reviews, she also wishes editors would do a better job of evaluating criticisms and arbitrating disputes. As an editor at PNAS, she thinks critically about conflicting reviews and weeds out inappropriate comments.

“A lot of editors don’t seem to make decisions themselves,” says Lindquist. “At PNAS, I think peer review works pretty well because the editors are academic scientists with labs, so they feel more comfortable interpreting reviews.”

Journals published by professional societies, such as the American Society for Cell Biology and the American Society for Biochemistry and Molecular Biology, typically ask members (working scientists) to serve as editors. Lindquist suspects these individuals have the confidence to dissect reviews and disregard unfounded criticism.

“If you’re dealing with a working scientist, you can have a discussion with them if the reviews are negative,” adds Lodish. “They don’t always accept your paper, but at least they give you a fair hearing.”

Linda Miller, executive editor of Nature and the Nature journals, counters that professional editors (scientists who have left the lab) bring a different set of skills to the table, so they’re better qualified to run the peer review process. “The professional editor has a special kind of expertise that’s distinct from that of someone who’s engrossed in one type of research,” she says. Professional editors also become familiar with the strengths and biases of various referees, says Miller.

“I’d agree,” responds Lindquist. “There’s nothing better than a seasoned professional editor. But it’s a tough profession and there’s too much turnover.”

Perhaps the current peer review system is as good as it gets. But researchers won’t know if that’s true until they start methodically testing hypotheses—and, well, review the results carefully with their peers.

Opening up open access

If paired with peer review, the emerging open access movement could significantly improve scientific publishing, say Whitehead Member Harvey Lodish and Whitehead Fellow Kate Rubins. The movement, embodied by the Public Library of Science, calls for journals to make articles available to scientists and the public for free soon after they are published.

“Compartmentalizing information and relegating it to one place hinders scientific progress,” says Rubins. “It also reinforces misconceptions about scientists. If we share more information with the public, maybe people will stop thinking of us as an elite, insular club.”

Most institutions and communities lack the resources to subscribe to every scientific journal. Yet almost every journal contains the results of taxpayer-funded research. Recognizing the irony, the National Institutes of Health recently required authors to submit NIH-supported studies to a free online database (PubMed Central).

Lodish wonders if this new policy will eventually make journals that employ professional editors irrelevant. He wouldn’t mourn their demise.

“The journals run by professional editors rather than scientists compete with each other for the hot papers that will help sell subscriptions and pay the editors’ salaries,” he says. “Their whole criterion for publication is novelty, but many hot papers turn out to be wrong.”

Linda Miller, executive editor of Nature and the Nature journals, counters that professional editors care about the reputation of their journals above all else.

“If they start selecting papers for the wrong reasons, researchers will stop sending them big, significant papers,” she says. “As soon as we become aware of biases, we move away from them because we want to continue publishing what’s best for the field. Money has nothing to do with our decision-making process.”

Lodish hopes that society journals flourish under the new NIH policy. Many of these journals have been posting free copies of articles on the web for years. For example, Molecular Biology of the Cell, which is published by the American Society for Cell Biology, makes papers available to everyone two months after publication. The journal still has a healthy subscription base, according to publications director Mark Leader. Thus, the society can cover publishing costs.

Peer review works better at society journals, where academic scientists serve as editors, declares Lodish (a past president of the American Society for Cell Biology).

But Emilie Marcus, editor-in-chief of Cell, adds that journals that employ professional editors won’t disappear anytime soon, and can coexist nicely with open-access policies. “Cell Press and Elsevier have really worked to come up with solutions to open access that are sustainable and support the open access initiatives of funding agencies and institutions,” she says.