Replicating results is crucial. But scientists rarely do it.


Replication is another foundational concept in science. Researchers take an older study that they want to test and then try to reproduce it to see if the findings hold up.
Testing, validating, retesting — it's all part of a slow and grinding process to arrive at some semblance of scientific truth. But this doesn't happen as often as it should, our respondents said. Scientists face few incentives to engage in the slog of replication. And even when they attempt to replicate a study, they often find they can’t do so. Increasingly it’s being called a "crisis of irreproducibility."
The stats bear this out: A 2015 study looked at 83 highly cited studies that claimed to feature effective psychiatric treatments. Only 16 had ever been successfully replicated. Another 16 were contradicted by follow-up attempts, and 11 were found to have substantially smaller effects the second time around. Meanwhile, nearly half of the studies (40) had never been subject to replication at all.
More recently, a landmark study published in the journal Science demonstrated that only a fraction of recent findings in top psychology journals could be replicated. This is happening in other fields too, says Ivan Oransky, one of the founders of the blog Retraction Watch, which tracks scientific retractions.
As for the underlying causes, our survey respondents pointed to a couple of problems. First, scientists have very few incentives to even try replication. Jon-Patrick Allem, a social scientist at the Keck School of Medicine of USC, noted that funding agencies prefer to support projects that find new information instead of confirming old results.
Journals are also reluctant to publish replication studies unless "they contradict earlier findings or conclusions," Allem writes. The result is to discourage scientists from checking each other's work. "Novel information trumps stronger evidence, which sets the parameters for working scientists."
The second problem is that many studies can be difficult to replicate. Sometimes their methods are too opaque. Sometimes the original studies had too few participants to produce a replicable answer. And sometimes, as we saw in the previous section, the study is simply poorly designed or outright wrong.
Again, this goes back to incentives: When researchers have to publish frequently and chase positive results, there’s less time to conduct high-quality studies with well-articulated methods.

Fixes for underreplication

Scientists need more carrots to entice them to pursue replication in the first place. As it stands, researchers are encouraged to publish new and positive results and to allow negative results to linger in their laptops or file drawers.
This has plagued science with a problem called "publication bias" — not all studies that are conducted actually get published in journals, and the ones that do tend to have positive and dramatic conclusions.
If institutions started to reward tenure positions or make hires based on the quality of a researcher’s body of work, instead of quantity, this might encourage more replication and discourage positive results chasing.
"The key that needs to change is performance review," writes Christopher Wynder, a former assistant professor at McMaster University. "It affects reproducibility because there is little value in confirming another lab's results and trying to publish the findings."
The next step would be to make replication of studies easier. This could include more robust sharing of methods in published research papers. "It would be great to have stronger norms about being more detailed with the methods," says University of Virginia’s Brian Nosek.
He also suggested more regularly adding supplements at the end of papers that get into the procedural nitty-gritty, to help anyone wanting to repeat an experiment. "If I can rapidly get up to speed, I have a much better chance of approximating the results," he said.
Nosek has detailed other potential fixes that might help with replication — all part of his work at the Center for Open Science.
A greater degree of transparency and data sharing would enable replications, said Stanford’s John Ioannidis. Too often, anyone trying to replicate a study must chase down the original investigators for details about how the experiment was conducted.
"It is better to do this in an organized fashion with buy-in from all leading investigators in a scientific discipline," he explained, "rather than have to try to find the investigator in each case and ask him or her in detective-work fashion about details, data, and methods that are otherwise unavailable."
Researchers could also make use of new tools, such as open source software that tracks every version of a data set, so that they can share their data more easily and have transparency built into their workflow.
Some of our respondents suggested that scientists engage in replication prior topublication. "Before you put an exploratory idea out in the literature and have people take the time to read it, you owe it to the field to try to replicate your own findings," says John Sakaluk, a social psychologist at the University of Victoria.
For example, he has argued, psychologists could conduct small experiments with a handful of participants to form ideas and generate hypotheses. But they would then need to conduct bigger experiments, with more participants, to replicate and confirm those hypotheses before releasing them into the world. "In doing so," Sakaluk says, "the rest of us can have more confidence that this is something we might want to [incorporate] into our own research."
Share on Google Plus

About ANON JEKLOY

    Blogger Comment
    Facebook Comment

0 comments:

Post a Comment

loading...