I have just watched a video of a fascinating presentation by
Prof. John Ioannidis at NIH entitled: "Great Teachers:
Translation, Replication and Credibility of Research Findings".
It is available on the web at:

   http://videocast.nih.gov/PastEventDetail.asp?13691

I think this is available to all, but I'm not certain because I'm
watching it from inside the NIH firewall.

Ioannidis has done a series of studies of medical publishing
extending over quite a few years.  His method is to do an
extensive search of the literature on a specific topic, identify
all studies on that topic, isolate the findings of each study,
and perform statistical analysis to find out how often and with
what degree of correlation initial results are confirmed by
follow on studies.

I'm not a statistician but I hope I'm presenting a reasonable
precis of what he said.

What he said was that the vast majority of "statistically
significant" results reported in the scientific literature (89%
if I remember correctly) turn out not to be statistically
significant at all after later studies are done to attempt to
replicate the results.  He believed that the most common source
of "significance" in results turns out to be bias of one of
several kinds.  Typical sources of bias are:

   Sample size bias:

       Many studies are reporting effects that are small enough
       that the sample sizes are insufficient to justify the
       reported effects.

   Selection bias:

       The authors analyze and report findings selectively,
       leaving out data that they regarded as "insignificant",
       but which in fact often is significant.

   Publication bias:

       Journals look for articles that show highly statistically
       significant results, therefore biasing publishing in the
       direction of those articles that contain lots of outliers
       in their findings.

   Fraud:

       He said that there were reported cases of this and seemed
       to think that more of the bias that authors introduce is
       not as inadvertent as we would hope that it is.

One thing that I found particularly interesting in his
presentation was that he derived numerical values for bias.  For
example, in one specific field, he compared early reports with
a huge, well conducted study of thousands of patients, and
found that there was a built in bias of 1.3 in the results of the
early reports.  If I understand that correctly, he is saying
that, in that field, unless a particular effect is at least 30%
greater than the controls, it is indistinguishable from the
built-in bias in that field.

When a very bright Chinese student came to work in his lab, he
did a parallel study on Chinese publishing in the same field and
found the bias there was 3.0 (compared to 1.3 in European and
American journals.)  In that field, the Americans and Europeans
were doing a bad job, but the Chinese were atrocious.

What all this means to me is that I'm going to add a lot more
grains of salt to all of the latest research findings that we see
reported daily in the scientific press.

   Alan