There are many well documented cases showing that both Nature and Science are particularly susceptible to these types of shenanigans. I don't know if this is unique to them because of their particular competitive spirit, or if distaste in examination extends to most journals and it's just that Nature and Science see far more scrutiny than other journals.
These high profile journals serve the purpose of publicizing discoveries that are thought to have far-reaching interest in science. However, at the same time, they have the strictest limitations on publications, meaning that the methods section, the foundation of science, are rarely described in any detail. So they have some of the worst practices for publication while supposedly chronicling the most important discoveries.
So while Nature/Science papers make the news and make people's careers, it's important to downweight the certainty of the finding until there is independent verification.
From the horse's mouth: only 6 out of 53 "landmark" oncology studies were significantly reproducible [1]. Bayer did a similar study, found that they couldn't reproduce the results of about 2/3s of the papers they tried to use for drug discovery R&D [2].
This is a near universal consequence of academia's incentives and extends into many of the sciences (although biology is most susceptible due to its complexity).
>This is a near universal consequence of academia's incentives and extends into many of the sciences (although biology is most susceptible due to its complexity).
It is not simply a matter of incentives (although they play a part). Another issue is pure publication bias. If you get 'landmark' results, a paper is much more likely to publish then if you got the results everyone would expect. In many ways this is one of the necessary functions of the journals, as it allows people to look at the new stuff without sifting through many articles of expected stuff, but it does create a bias.
Add to that the incentives of needing to get published, and constrained resources, and a willingness to publish what could be sub-par methodologys (of course they are par), and it is clearly not the best system.
From a Bayesian perspective, papers that are demonstrating phenomena that are both (1) novel and (2) significant are the ones that are most likely to be false positives. If you have a journal built around publishing (1) and (2), you're bound to have an excess of studies that can't be reproduced.
That's a very insightful way of looking at it. If Nature wanted to get out of this unfortunate position, it would have to counteract this with stricter requirements on the articles that it publishes. It could require that papers be replicated before publication; it could require the release of any relevant code and data; it could be really diligent about publishing failure-to-reproduce papers. This would be a hassle, but would probably make Nature more prestigious, so I'm sure they could get a lot of researchers to go along with whatever requirements they want.
Nature's and Science's shenanigans are not publishing papers that turn out to be wrong, it's that (1) when contrary data comes up they refuse to publish it, as in this case, and (2) really questionable editorial policies, where articles get peer-reviewed by the wrong audience, and on-the-face obviously-wrong to anybody in the field papers still get published.
Every journal publishes novel and statistically significant findings, and probably encounter similar rates of type 1 error as Nature and Science. It's just that Nature and Science are more susceptible to human politicking and don't seem to face up to it.
> This would be a hassle, but would probably make Nature more prestigious...
Prestige is a relative thing, and there's nothing more prestigious than Nature (or Science). Your suggestions are good, but I think they would be more likely to be adopted first by a journal that isn't Nature or Science, that possibly wanted to compete with them.
This is true. I join AAAS in order to get access to Science and some of the related journals and have become quite disillusioned at their practice of either hiding, or outright omitting, papers or studies that contrast something that appeared 'first' in their pages. I was always taught that a scientist should never fear being wrong, they should only fear being not right. The discourse and disagreements, the questions and reproduction things. make things more real.
These high profile journals serve the purpose of publicizing discoveries that are thought to have far-reaching interest in science. However, at the same time, they have the strictest limitations on publications, meaning that the methods section, the foundation of science, are rarely described in any detail. So they have some of the worst practices for publication while supposedly chronicling the most important discoveries.
So while Nature/Science papers make the news and make people's careers, it's important to downweight the certainty of the finding until there is independent verification.