Breaking

Monday, December 12, 2016

Peer review is broken


Peer review is meant to weed out junk science before it reaches publication. Yet over and over again in our survey, respondents told us this process fails. It was one of the parts of the scientific machinery to elicit the most rage among the researchers we heard from.
Normally, peer review works like this: A researcher submits an article for publication in a journal. If the journal accepts the article for review, it's sent off to peers in the same field for constructive criticism and eventual publication — or rejection. (The level of anonymity varies; some journals have double-blind reviews, while others have moved to triple-blind review, where the authors, editors, and reviewers don’t know who one another are.)
It sounds like a reasonable system. But numerous studies and systematic reviewshave shown that peer review doesn’t reliably prevent poor-quality science from being published.
The process frequently fails to detect fraud or other problems with manuscripts, which isn't all that surprising when you consider researchers aren't paid or otherwise rewarded for the time they spend reviewing manuscripts. They do it out of a sense of duty — to contribute to their area of research and help advance science.
But this means it's not always easy to find the best people to peer-review manuscripts in their field, that harried researchers delay doing the work (leading to publication delays of up to two years), and that when they finally do sit down to peer-review an article they might be rushed and miss errors in studies.
"The issue is that most referees simply don't review papers carefully enough, which results in the publishing of incorrect papers, papers with gaps, and simply unreadable papers," says Joel Fish, an assistant professor of mathematics at the University of Massachusetts Boston. "This ends up being a large problem for younger researchers to enter the field, since that means they have to ask around to figure out which papers are solid and which are not."
That's not to mention the problem of peer review bullying. Since the default in the process is that editors and peer reviewers know who the authors are (but authors don’t know who the reviews are), biases against researchers or institutions can creep in, opening the opportunity for rude, rushed, and otherwise unhelpful comments. (Just check out the popular #SixWordPeerReview hashtag on Twitter).
These issues were not lost on our survey respondents, who said peer review amounts to a broken system, which punishes scientists and diminishes the quality of publications. They want to not only overhaul the peer review process but also change how it's conceptualized.

Fixes for peer review

On the question of editorial bias and transparency, our respondents were surprisingly divided. Several suggested that all journals should move toward double-blinded peer review, whereby reviewers can't see the names or affiliations of the person they're reviewing and publication authors don't know who reviewed them. The main goal here was to reduce bias.
"We know that scientists make biased decisions based on unconscious stereotyping," writes Pacific Northwest National Lab postdoc Timothy Duignan. "So rather than judging a paper by the gender, ethnicity, country, or institutional status of an author — which I believe happens a lot at the moment — it should be judged by its quality independent of those things."
Yet others thought that more transparency, rather than less, was the answer: "While we correctly advocate for the highest level of transparency in publishing, we still have most reviews that are blinded, and I cannot know who is reviewing me," writes Lamberto Manzoli, a professor of epidemiology and public health at the University of Chieti, in Italy. "Too many times we see very low quality reviews, and we cannot understand whether it is a problem of scarce knowledge or conflict of interest."
Perhaps there is a middle ground. For example, eLife, a new open access journal that is rapidly rising in impact factor, runs a collaborative peer review process. Editors and peer reviewers work together on each submission to create a consolidated list of comments about a paper. The author can then reply to what the group saw as the most important issues, rather than facing the biases and whims of individual reviewers. (Oddly, this process is faster — eLife takes less time to accept papers than Nature or Cell.)
Still, those are mostly incremental fixes. Other respondents argued that we might need to radically rethink the entire process of peer review from the ground up.
"The current peer review process embraces a concept that a paper is final," says Nosek. "The review process is [a form of] certification, and that a paper is done." But science doesn't work that way. Science is an evolving process, and truth is provisional. So, Nosek said, science must "move away from the embrace of definitiveness of publication."
Some respondents wanted to think of peer review as more of a continuous process, in which studies are repeatedly and transparently updated and republished as new feedback changes them — much like Wikipedia entries. This would require some sort of expert crowdsourcing.
"The scientific publishing field — particularly in the biological sciences — acts like there is no internet," says Lakshmi Jayashankar, a senior scientific reviewer with the federal government. "The paper peer review takes forever, and this hurts the scientists who are trying to put their results quickly into the public domain."
One possible model already exists in mathematics and physics, where there is a long tradition of "pre-printing" articles. Studies are posted on an open website called arXiv.org, often before being peer-reviewed and published in journals. There, the articles are sorted and commented on by a community of moderators, providing another chance to filter problems before they make it to peer review.
"Posting preprints would allow scientific crowdsourcing to increase the number of errors that are caught, since traditional peer-reviewers cannot be expected to be experts in every sub-discipline," writes Scott Hartman, a paleobiology PhD student at the University of Wisconsin.
And even after an article is published, researchers think the peer review process shouldn't stop. They want to see more "post-publication" peer review on the web, so that academics can critique and comment on articles after they've been published. Sites like PubPeer and F1000Research have already popped up to facilitate that kind of post-publication feedback.
"We do this a couple of times a year at conferences," writes Becky Clarkson, a geriatric medicine researcher at the University of Pittsburgh. "We could do this every day on the internet."
The bottom line is that traditional peer review has never worked as well as we imagine it to — and it’s ripe for serious disruption.

No comments:

Post a Comment

loading...