Jun 8, 2015What Research Can Be Trusted?
With the success of for-profit journals, poorly designed research studies can make headlines today. A science journalist reveals how his bogus study on eating chocolate for weight loss went mainstream.
John Bohannon, a science journalist, in collaboration with two German reporters, designed, published, and disseminated a weak scientific study in order to reveal the lack of rigor in both scientific journals and in science and health journalism. The study claimed that eating chocolate every day accelerates weight loss and was published under a pseudonym by a for-profit scientific journal. It was quickly picked up by the media without any probing questions.
Bohannon revealed the plot in an essay at io9.com. The idea came from German television reporter Peter Onneken and his collaborator Diana Löbl.
“They wanted me to help demonstrate just how easy it is to turn bad science into the big headlines behind diet fads,” Bohannon wrote.
The study was based on a clinical trial run by Dr. Gunter Frank, who was in on the stunt. The trial, however, was purposefully poorly designed. There were only 15 participants with more than 18 variables measured, and many more variables were not controlled for. Participants were split into three groups: one was put on a low-carb diet; another on a low-carb diet with chocolate; and another group was told not to change their diet.
Once the 21-day trial was over, Onneken had a financial analyst examine the results. The group with the low-carb diet with chocolate did lose weight 10 percent faster than those on the low-carb diet alone. But Bohannon says these results, like the results in many poorly designed health studies, mislead.
“Here’s a dirty little science secret: If you measure a large number of things about a small number of people, you are almost guaranteed to get a ‘statistically significant’ result,” writes Bohannon. “Our study included 18 different measurements from 15 people. That study design is a recipe for false positives.
“You might as well read tea leaves as try to interpret our results,” Bohannon adds. “Chocolate may be a weight loss accelerator, or it could be the opposite. You can’t even trust the weight loss that our non-chocolate low-carb group experienced versus control. Who knows what the handful of people in the control group were eating? We didn’t even ask them.”
Bohannon and his collaborators took their findings and submitted them to 20 or journals that don’t do peer review and are often for-profit, and received multiple offers for publication within 24 hours. They chose the International Archives of Medicine, which charged them 600 Euros for publication. While the journal claims to have a rigorous review process, the study was published in two weeks, and no editing had been done, even with “chocolate” misspelled more than once, reported The Washington Post.
Bohannon’s next step was spreading the story to the media. He wrote a press release that was accurate but did not mention the sample size. While major news outlets didn’t pick up the story, the findings were published on a number of websites and in health magazines, such as Shape, Bild, Huffington Post’s India and Germany websites, and Cosmopolitan’s Germany website.
Bohannon blames the dissemination of bad science on the journalists who fail to fact check. He says only a reporter from Ohio asked him about the sample size and a fake institute Bohannon created for his pseudonym.
“It’s the reporters and ultimately the editors. People who are on the health science beat need to treat it like science, and that has to come from the editors. And if you’re reporting on a scientific study, you need to actually look at the paper. You need to talk to a source who has real scientific expertise,” Bohannon told The Washington Post. “The whole point is that this was as bad as a lot of science that is considered ‘real’ science. It gets reported without people asking the right questions.”
