January 10, 2012
BehindTheMedspeak: "Why Most Published Research Findings Are False"
The abstract follows.
There is increasing concern that most current published research findings are false. The probability that a research claim is true may depend on study power and bias, the number of other studies on the same question, and, importantly, the ratio of true to no relationships among the relationships probed in each scientific field. In this framework, a research finding is less likely to be true when the studies conducted in a field are smaller; when effect sizes are smaller; when there is a greater number and lesser preselection of tested relationships; where there is greater flexibility in designs, definitions, outcomes, and analytical modes; when there is greater financial and other interest and prejudice; and when more teams are involved in a scientific field in chase of statistical significance. Simulations show that for most study designs and settings, it is more likely for a research claim to be false than true. Moreover, for many current scientific fields, claimed research findings may often be simply accurate measures of the prevailing bias. In this essay, I discuss the implications of these problems for the conduct and interpretation of research.
Read the paper in its entirety, download it as a PDF, and/or print it out here.
Free, the way we like it.
January 10, 2012 at 04:01 PM | Permalink
TrackBack URL for this entry:
Listed below are links to weblogs that reference BehindTheMedspeak: "Why Most Published Research Findings Are False":
Even more interesting than I had expected. A study of multi variant causation of error in research methodology. Not dated a bit for a 2005 study.
Now, if we only applied data validation to our daily decisions / advertising messages.
Posted by: 6.02*10^23 | Jan 11, 2012 12:31:55 PM
I'm looking forward to reading the linked paper.
I expect that statistical rigor - a wonderful tool that increased processor access time affords us - may well be the central issue.
Well, that and the fact that the majority of major research issues have been researched to death (when did anybody see tertiary syphilius (outside of that immoral study) after 1970?
How many bolts fail, power lines fall and municipal water systems / roadbeds / primary services fail? We have studied all of the primary problems and arrived at working solutions. The current problems involve vastly more complex variables / huge data sets or else are so minuscule that obtaining reliable data from the limited data set presents the real problem. Climate change falls into the former group and the effect (if any) of emf on livestock pastured under high tension lines falls into the latter group.
We consider the "Twinkie defense" an outlier (or, outright lie)....
Posted by: 6.02*10^23 | Jan 11, 2012 2:16:13 AM
The comments to this entry are closed.