>"we pay attention to how we construct patient recruitment, sample demographics, ... etc"

I was told the opposite the other day:

"the fact that clinical trials seek people maximally healthy but for the target conditions rather than a representative sample of those with the target condition is a very frequent complaint." https://news.ycombinator.com/item?id=16868660

>"Anyone who wishes to lend a brain cell is welcome."

Its pretty misleading to promise this. People who disagree with the "publish an endless series of statistically significant differences" paradigm are not going to find it at all welcoming.

> "the fact that clinical trials seek people maximally healthy but for the target conditions rather than a representative sample of those with the target condition is a very frequent complaint."

Those aren't quite in contradiction. So, where you start, is with a trial that tries to eliminate meaningful comorbid conditions to allow it to answer the question: "Does this drug work for Disease X?", rather than the question "Does this drug work for Disease X when accompanied by Diseases Y and Z?" There are just far, far too many permutations of X, Y, Z, etc. to allow them all into a study and even pretend to have answered the question about the drug. So, it's not where you start. But, you're absolutely paying attention to patient demographics here, that's part of the point: can we come up with a reasonably homogenous patient pool for step 1? That's not really about being "optimally healthy," it's about homogeneity, except where the latter implies "not a giant variety of comorbid diseases."

But, as I hope I've made clear in the way I reiterated it, that's the initial step. Because, hey, we really do need to answer the question about Drug A for Disease X before we start working on the various permutations of Disease X with other diseases. The latter is usually studied through retrospective analysis rather than RCT. Why?

A given study has only a certain sample size to work with. There's this sort of odd implicit idea that small sample size is merely an issue of study design, but not really: in practice, a research group only has a catchment area of a certain size - they'll only be able to catch so many patients per unit time, and you can't run a study forever (both because you run out of grant funding and because the state of the disease and therapy moves on.) The more specific you get - e.g., people with Disease X and Y and Z, but neither Y and Z to be more than moderately severe - the fewer patients you can potentially recruit, to the point where the final result is statistically meaningless. This isn't really the worst thing ever: the less capable you are of recruiting patients because of their rarity, the less useful such a study would be to the public because of their rarity. As a practical effect, you can't really build a study to control for every meaningful confounding comorbidity. So, we tend to wait for data to accrue and analyse it retrospectively. It's not as good as an RCT, but it's the best we can do under practical constraints.

It's one of those things worth criticizing, in theory, but continues because we don't have the means to do better. Ideally we'd have studies of 20k+ people selecting for significant sub-groups of each permutation of common comorbid conditions. That's just beyond any hope of organizing, running, or paying for.

>>"Anyone who wishes to lend a brain cell is welcome."

>Its pretty misleading to promise this. People who disagree >with the "publish an endless series of statistically >significant differences" paradigm are not going to find it >at all welcoming.

That's simply untrue. I know people working in operations research who are developing novel predictive models for ED patients to predict where they'll need to be sent after workup (and how to prep the appropriate hospital dept. for their arrival, and to estimate when that will be); the VA does shit-tons of operations research. I know people working in the basic sciences who are working on developing ever-more-sophisticated models of the current state of physiology/biochemistry, to allow for increased accuracy of predicting a drug's effectiveness and toxicities before moving on to in vitro testing. And an interesting negative result is about as easily published these days as an interesting positive result (even Easterbrook's big '91 paper on publication bias found that it was for observational and lab studies, but couldn't identify bias in RCT publication.)

The full variety of interesting work to be done in clinical trial design isn't super obvious from the outside. Especially since the folks moaning and whining about it tend to be academics, and a big chunk of good clinical trial work doesn't involve many university academics, and the other cool work (e.g., operations research) is being done in industry and at the VA. But "not super visible to university academics, who are the ones talking about publish or perish" and "non-existent" are very different things.

There's a lot being claimed here that seems to go directly against my experience, I'll focus on what is easily verifiable:

>"(even Easterbrook's big '91 paper on publication bias found that it was for observational and lab studies, but couldn't identify bias in RCT publication.)"

I'm not familiar with the Easterbook 1991 paper but that is a pretty strange thing to claim since it is even one of those criticisms that academia has managed to turn into it's own area of study. It didn't look closely at any of these papers but just for example:

>"we review and summarise the evidence from cohort studies that have assessed study publication bias or outcome reporting bias in randomised controlled trials. [...] Direct empirical evidence for the existence of study publication bias and outcome reporting bias is shown. There is strong evidence of an association between significant results and publication; studies that report positive or significant results are more likely to be published and outcomes that are statistically significant have higher odds of being fully reported. " https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3702538/

>"We conclude that RCTs investigating antipsychotic drugs suffer from substantial outcome reporting bias" https://www.nature.com/articles/tp2017203

In the case of vaccines specifically, there is just a surprising amount of things we still don't understand.

Vaccination is positively ancient technology (Smallpox circa 1796), but it is still difficult to say why e.g., some vaccines are broadly effective at producing immunity and some aren't (and what the factors at play are in the recipients), or why we can produce successful vaccines to some diseases and not others.

A specific example: the efficacy of BCG vaccination (tuberculosis) varies by manufacturer and the reasons for why that is are still unclear.

There are a number of large NIH-funded initiatives that aim at understanding the complex cellular and molecular interactions that result in immunity as a result of vaccination under the Human Immunology Project Consortium (a bunch of high profile papers in Cell, etc. over the past few years).