One of my favorite new blogs this year is Retraction Watch, written by Adam Marcus and Ivan Oransky, both carrying substantial science editing and journalism credentials. If you’re a scientist and you’re not following it, you really should. Anyway, last week brought the retraction of another high-profile and controversial immunology paper (which happened to be the fourth retraction from Nature this year and came in the same issue as an editorial about the increase in retractions). Oransky posted a response from Tom Decoursey, an author of a study that challenged the findings of the now-retracted paper. Decoursey makes a strong case for the importance of removing wrong answers from the literature. There was one thing he said that struck a dissonant chord with me. Decoursey says, in reference to this particular, retraction (emphasis mine):

Despite the fact that a few insiders had doubts about the Ahluwalia et al (2004) paper from the outset, and this was discussed heatedly at numerous international (specialty) meetings, it is incorrect to assume that most people knew what the real story was. Very few people are expert enough, or confident enough, to evaluate opposing claims. Even after two groups (Femling et al, 2006; Essin et al, 2007) had published papers thoroughly disproving every major conclusion reached in the Nature paper, the stock position taken by authors who published papers subsequently was, “There is controversy in the field” or “Group A says this, but Groups B & C say the opposite.” It is not clear how many papers must pile up to refute one incorrect study. Certainly the number is greater when the original study was published in Nature or Science.

I agree that it’s unclear how many “rights” negate a “wrong” so to speak. There are so many variables in experiments–many of which aren’t actually reported in manuscripts–that it can prove quite difficult to distinguish between variation in conditions (concentrations, lots, strains, etc.), honest mistakes, and intentional misconduct. But is the burden of proof higher (e.g. greater number of contradictory studies required) if the original finding in question is published in a GlamourMag? Should it be?

It doesn’t mean that the study or peer-review was more rigorous just because it is published in Cell, Nature, or Science. GlamourMags reject more manuscripts because they receive so many submissions and publish so few articles (compared to society journals). The standards for novelty, impact, and breadth (in theory) are higher because of the broad audiences for these journals. However, as far as I can tell, the peer-review process is much the same as it is for lower impact factor journals. Perhaps there are more rounds of review, higher expectations about the data presented, which sometimes result in experiments being thrown together at the last minute, trying to beat the editor’s deadline (thus those experiments might not be as stringently designed and controlled as one might like).

In my mind, the more important factor in determining the burden of proof is the type of study and the variables involved. Any study involving animals will generally require more evidence to refute than one based on in vitro studies. The more complex the methods, the more variables for which you have to account, perhaps the more evidence we should accumulate before collectively refuting a study–at least in the absence of evidence of fabrication or falsification. It’s still important to publish contradictory results, as this case has demonstrated. But the burden of proof should be determined by the methods not where the study is published.