New Study Reveals One Third of All Studies Are Inaccurate
I'm not sure I trust the accuracy of this most recent study! The abstract is here.
Seriously, this is exactly the way science is supposed to work. Claims presented must be falsifiable, and our store of scientific knowledge is increased when they are in fact falsified.
Seriously, this is exactly the way science is supposed to work. Claims presented must be falsifiable, and our store of scientific knowledge is increased when they are in fact falsified.
posted by Mister Spark at 12:25 AM
2 Comments:
Research: Third of study results don't hold up
Wednesday, July 13, 2005; Posted: 12:06 p.m. EDT (16:06 GMT)
CHICAGO, Illinois (AP) -- New research highlights a frustrating fact about science: What was good for you yesterday frequently will turn out to be not so great tomorrow.
The sobering conclusion came in a review of major studies published in three influential medical journals between 1990 and 2003, including 45 highly publicized studies that initially claimed a drug or other treatment worked.
Subsequent research contradicted results of seven studies -- 16 percent -- and reported weaker results for seven others, an additional 16 percent.
That means nearly one-third of the original results did not hold up, according to the report in Wednesday's Journal of the American Medical Association.
"Contradicted and potentially exaggerated findings are not uncommon in the most visible and most influential original clinical research," said study author Dr. John Ioannidis, a researcher at the University of Ioannina in Greece.
Ioannidis examined research in the New England Journal of Medicine, JAMA and Lancet -- prominent journals whose weekly studies help feed a growing public appetite for medical news.
Experts say the report is a reminder to doctors and patients that they should not put too much stock in a single study and understand that treatments often become obsolete with medical advances.
"The crazy part about science and yet the exciting part about science is you almost never have something that's black and white," said Dr. Catherine DeAngelis, JAMA's editor-in-chief.
Editors at the New England Journal of Medicine added in a statement: "A single study is not the final word, and that is an important message."
The refuted studies dealt with a wide range of drugs and treatments. Hormone pills were once thought to protect menopausal women from heart disease but later were shown to do the opposite, and Vitamin E pills have not been shown to prevent heart attacks, contrary to initial results.
Contradictions also included a study that found nitric oxide does not improve survival in patients with respiratory failure, despite earlier claims. And a study suggested an antibody treatment did not improve survival in certain sepsis patients; a smaller previous study found the opposite.
Ioannidis acknowledged an important but not very reassuring caveat: "There's no proof that the subsequent studies ... were necessarily correct." But he noted that in all 14 cases in which results were contradicted or softened, the subsequent studies were either larger or better designed. Also, none of the contradicted treatments is currently recommended by medical guidelines.
Not by accident, this week's JAMA also includes a study contradicting previous thinking that stomach-lying helped improve breathing in children hospitalized with acute lung injuries. The new study found they did no better than patients lying on their backs.
DeAngelis said she included the study with Ioannidis' report to highlight the issue. She said the media can complicate matters with misleading or exaggerated headlines about studies.
Ioannidis said scientists and editors should avoid "giving selective attention only to the most promising or exciting results" and should make the public more aware of the limitations of science.
"The general public should not panic" about refuted studies, he said. "We all need to start thinking more critically."
Copyright 2005 The Associated Press. All rights reserved.This material may not be published, broadcast, rewritten, or redistributed.
Contradicted and Initially Stronger Effects in Highly Cited Clinical Research
John P. A. Ioannidis, MD
JAMA. 2005;294:218-228.
Context Controversy and uncertainty ensue when the results of clinical research on the effectiveness of interventions are subsequently contradicted. Controversies are most prominent when high-impact research is involved.
Objectives To understand how frequently highly cited studies are contradicted or find effects that are stronger than in other similar studies and to discern whether specific characteristics are associated with such refutation over time.
Design All original clinical research studies published in 3 major general clinical journals or high-impact-factor specialty journals in 1990-2003 and cited more than 1000 times in the literature were examined.
Main Outcome Measure The results of highly cited articles were compared against subsequent studies of comparable or larger sample size and similar or better controlled designs. The same analysis was also performed comparatively for matched studies that were not so highly cited.
Results Of 49 highly cited original clinical research studies, 45 claimed that the intervention was effective. Of these, 7 (16%) were contradicted by subsequent studies, 7 others (16%) had found effects that were stronger than those of subsequent studies, 20 (44%) were replicated, and 11 (24%) remained largely unchallenged. Five of 6 highly-cited nonrandomized studies had been contradicted or had found stronger effects vs 9 of 39 randomized controlled trials (P = .008). Among randomized trials, studies with contradicted or stronger effects were smaller (P = .009) than replicated or unchallenged studies although there was no statistically significant difference in their early or overall citation impact. Matched control studies did not have a significantly different share of refuted results than highly cited studies, but they included more studies with "negative" results.
Conclusions Contradiction and initially stronger effects are not unusual in highly cited research of clinical interventions and their outcomes. The extent to which high citations may provoke contradictions and vice versa needs more study. Controversies are most common with highly cited nonrandomized studies, but even the most highly cited randomized trials may be challenged and refuted over time, especially small ones.
Author Affiliations: Department of Hygiene and Epidemiology, University of Ioannina School of Medicine, Ioannina, Greece, and the Institute for Clinical Research and Health Policy Studies, Department of Medicine, Tufts-New England Medical Center, Boston, Mass.
Post a Comment
<< Home