How reliable are medical research checks?

24 May 2014 Last updated at 04:15 By Gabriela Torres BBC Mundo health reporter Research into statin drugs is the latest to trigger concern Recent controversy over figures used in the British Medical Journal which were later withdrawn renewed questions over the peer review system - the way medical studies are checked prior to publication.

The BMJ is now investigating two articles that claimed statins, used to reduce cholesterol, could cause harmful side effects in 18-20% of people who took them.

However, it was later discovered that this figure was unreliable - and scientists are still debating what the true level is.

The flaw in the studies was not picked up when they were assessed for publication.

The system used to check papers before they appear in journals is called peer review.

It is a way of validating their work through the scrutiny of the methodology that was used by other experts.

'Worst punishment'

But things do go wrong, and the statins debate is not an isolated case.

It reflects an increasing problem in the scientific community: a dramatic increase in the number of papers retracted - or taken back - by journals.

When a paper is retracted, it means that the research has so many flaws that it has to be withdrawn by the publications in question. Many consider this action as the worst punishment for a scientist.

In recent years the number of retractions has increased dramatically - in 2000 there were 30 but in 2010 this number had risen to 400.

However, the number of studies published increased by only 44% in that period, according to the journal Nature.

This has serious implications for science.

Although these retractions still represent less than 2% of what is published each year - around 30,000 studies - the way the scientific community works amplifies the impact of any mistakes.

Researchers base their new studies on previously published work, which means that one paper with flaws can influence dozens of others, whose work will then be used by many others before someone spots the mistake.

Research is reviewed by others before publication

To some, the large increase in retractions is not particularly surprising.

"It could be, as we and others have argued, a good thing and a sign that science is interested in putting the record straight," says Ivan Oransky, editorial director of the US-based organisation MedPage Today.

"That being said, it is often very difficult for critics to get editors to take their concerns seriously, and retractions are taking longer and longer. So it's a complex story," he adds.

Japanese scientist Haruko Obokata became something of a celebrity last January after publishing a study on stem cells.

It was supposed to be a ground-breaking discovery, but turned out to be a case of fraud.

Ms Obakata's work had passed peer review.

But as soon as another research team tried to replicate Obokata's work, it became evident that there had been an "inappropriate handling of the data".

She has recently been found guilty of misconduct.

Replication

"Science relies on others to replicate the work, it doesn't necessarily have to be exactly the same, but in a way that guarantees that other results can be replicable," Nigel Hooper, professor of biochemistry School of Molecular and Cellular Biology, told BBC Mundo.

"We don't go out there and ask someone to repeat the study; the replication is a natural part of the scientific process," he adds.

Damian Pattison, executive director of the PLOS One journal, says: "The thing is that peer reviews aren't exempt from problems.

"It is very difficult to understand what's going on in a paper, unless you see all the data.

"And the problem is that in the majority of cases, the scientists don't include all the data in their report.

"So there may be many reasons why a study may have a good peer review and later it turns out that it is unreliable."

'Selective' presentation

Dr Elizabeth Iorns, a cancer biologist and CEO of Science Exchange (where scientists can share information), says the literature often does not correct itself.

She believes that the peer review system has several issues, with each reviewer expected to assess a publication that contains data generated by a team of researchers with highly specialised backgrounds.

The individual peer reviewer may not have expertise in all of the techniques used, in particular complex statistical analyses.

And she says: "There is evidence that researchers sometimes selectively present data that supports a specific hypothesis.

"This means that reviewers do not have access to all of the data generated from the research study and cannot determine if the information provided accurately represents the complete dataset."

A couple of years ago, Ivan Oransky founded Retraction Watch with science journalist Adam Marcus. The mission of this blog is to highlight cases of retractions, mistakes and corrections.

These journalists have a lot work ahead of them, judging by the study made by Danielle Fanelli, a natural scientist who specialises in the study of scientific misconduct, bias and related issues and a member of the Research Ethics and Bioethics Advisory Committee of the National Research Council in Italy.

Not all data is shown to reviewers

According to a study of his, published in 2009 in PLOS One and based on the metadata of several surveys, about 2% of scientists have admitted to falsifying, making up or modifying elements at least once.

And one third confessed to other "questionable practices", including "to have 'modified research results' to improve the outcome, then to have reported results they 'knew to be untrue'."

Additionally, more than 70% of scientists say they have witnessed irregular behaviour from their colleagues.

If this is the case, how reliable are the studies published every day? "I can't really answer that question," says Damian Pattinson.

He admits there is an issue with the availability of the data: "We are campaigning to make all the research data available."

Ivan Oransky admits peer review "certainly has its flaws."

But he said: "As Churchill said of democracy, it's the worst possible system except for all those other forms that have been tried from time to time.

"People have suggested a number of ways to improve peer review.

"One, that we champion, is for journals to embrace post-publication peer review more than they currently do, so that the publishing record really reflects how science works."

Another option, he said, was to make peer review open, not anonymous, in the interests of transparency.

But there is general agreement that, in most cases, flaws in the peer review system are due to human error and have little to do with malpractice.

View the original article here

Posted in: checks, medical, reliable, research