Breaking

Monday, December 12, 2016

Too many studies are poorly designed. Blame bad incentives.


Scientists are ultimately judged by the research they publish. And the pressure to publish pushes scientists to come up with splashy results, of the sort that get them into prestigious journals. "Exciting, novel results are more publishable than other kinds," says Brian Nosek, who co-founded the Center for Open Science at the University of Virginia.
The problem here is that truly groundbreaking findings simply don’t occur very often, which means scientists face pressure to game their studies so they turn out to be a little more "revolutionary." (Caveat: Many of the respondents who focused on this particular issue hailed from the biomedical and social sciences.)
Some of this bias can creep into decisions that are made early on: choosing whether or not to randomize participants, including a control group for comparison, or controlling for certain confounding factors but not others. (Read more on study design particulars here.)
Many of our survey respondents noted that perverse incentives can also push scientists to cut corners in how they analyze their data.
"I have incredible amounts of stress that maybe once I finish analyzing the data, it will not look significant enough for me to defend," writes Jess Kautz, a PhD student at the University of Arizona. "And if I get back mediocre results, there's going to be incredible pressure to present it as a good result so they can get me out the door. At this moment, with all this in my mind, it is making me wonder whether I could give an intellectually honest assessment of my own work."
Increasingly, meta-researchers (who conduct research on research) are realizing that scientists often do find little ways to hype up their own results — and they’re not always doing it consciously. Among the most famous examples is a technique called "p-hacking," in which researchers test their data against many hypotheses and only report those that have statistically significant results.
In a recent study, which tracked the misuse of p-values in biomedical journals, meta-researchers found "an epidemic" of statistical significance: 96 percent of the papers that included a p-value in their abstracts boasted statistically significant results.
That seems awfully suspicious. It suggests the biomedical community has been chasing statistical significance, potentially giving dubious results the appearance of validity through techniques like p-hacking — or simply suppressing important results that don't look significant enough. Fewer studies share effect sizes (which arguably gives a better indication of how meaningful a result might be) or discuss measures of uncertainty.
"The current system has done too much to reward results," says Joseph Hilgard, a postdoctoral research fellow at the Annenberg Public Policy Center. "This causes a conflict of interest: The scientist is in charge of evaluating the hypothesis, but the scientist also desperately wants the hypothesis to be true."
The consequences are staggering. An estimated $200 billion — or the equivalent of 85 percent of global spending on research — is routinely wasted on poorly designed and redundant studies, according to meta-researchers who have analyzed inefficiencies in research. We know that as much as 30 percent of the most influential original medical research papers later turn out to be wrong or exaggerated.

Fixes for poor study design

Our respondents suggested that the two key ways to encourage stronger study design — and discourage positive results chasing — would involve rethinking the rewards system and building more transparency into the research process.
"I would make rewards based on the rigor of the research methods, rather than the outcome of the research," writes Simine Vazire, a journal editor and a social psychology professor at UC Davis. "Grants, publications, jobs, awards, and even media coverage should be based more on how good the study design and methods were, rather than whether the result was significant or surprising."
Likewise, Cambridge mathematician Tim Gowers argues that researchers should get recognition for advancing science broadly through informal idea sharing — rather than only getting credit for what they publish.
"We’ve gotten used to working away in private and then producing a sort of polished document in the form of a journal article," Gowers said. "This tends to hide a lot of the thought process that went into making the discoveries. I'd like attitudes to change so people focus less on the race to be first to prove a particular theorem, or in science to make a particular discovery, and more on other ways of contributing to the furthering of the subject."
When it comes to published results, meanwhile, many of our respondents wanted to see more journals put a greater emphasis on rigorous methods and processes rather than splashy results.
"I think the one thing that would have the biggest impact is removing publication bias: judging papers by the quality of questions, quality of method, and soundness of analyses, but not on the results themselves," writes Michael Inzlicht, a University of Toronto psychology and neuroscience professor.
Some journals are already embracing this sort of research. PLOS One, for example, makes a point of accepting negative studies (in which a scientist conducts a careful experiment and finds nothing) for publication, as does the aptly named Journal of Negative Results in Biomedicine.
More transparency would also help, writes Daniel Simons, a professor of psychology at the University of Illinois. Here’s one example: ClinicalTrials.gov, a site run by the NIH, allows researchers to register their study design and methods ahead of time and then publicly record their progress. That makes it more difficult for scientists to hide experiments that didn’t produce the results they wanted. (The site now holds information for more than 180,000 studies in 180 countries.)
Similarly, the AllTrials campaign is pushing for every clinical trial (past, present, and future) around the world to be registered, with the full methods and results reported. Some drug companies and universities have created portals that allow researchers to access raw data from their trials.
The key is for this sort of transparency to become the norm rather than a laudable outlier.

No comments:

Post a Comment

loading...