How Researchers Try to Outsmart Confounding

A treatment that wins overall and loses everywhere

In a well known study of two treatments for kidney stones, one treatment came out ahead when researchers looked at the overall success rate across all patients. But when the same data were split by stone size, the other treatment actually did better, in both the small stone group and the large stone group. The same dataset pointed in opposite directions depending on how you looked at it, and neither answer was a calculation error.

What happened was that doctors did not assign treatments randomly. They tended to use the treatment that looked better overall on the easier cases, the smaller stones, and reserved the other treatment for the harder cases, the larger stones. Stone size affected both which treatment a patient was likely to receive and how likely that patient was to recover well regardless of treatment. It was a confounder, exactly the kind of variable we introduced last time, and it was distributed unevenly enough between the two treatment groups to flip the entire conclusion.

Simpson’s Paradox, defined

This reversal has a name: Simpson’s Paradox, a situation where a trend that appears in pooled data disappears, or even reverses, once the data is separated into meaningful subgroups. It is not a statistical trick or a rare curiosity. It is what confounding looks like when it is severe enough and the confounder is distributed unevenly enough between groups.

The kidney stone example is memorable because the reversal is so complete, one treatment wins the overall tally and loses every subgroup, but the underlying mechanism is the same as any other confounded comparison. If the introduction of bias in the last post explained what confounding is, this post is about what researchers actually do to keep it from quietly rewriting their conclusions.

Randomization: preventing the problem before it starts

The most reliable defense against confounding is also the simplest in concept: assign patients to treatment groups randomly rather than letting clinical judgment, patient preference, or disease severity decide who gets what. In a well conducted randomized controlled trial, every measured and unmeasured characteristic, age, comorbidities, stone size, genetics, anything at all, tends to be balanced across groups purely by chance, especially as the sample grows.

This is the real reason randomized trials sit at the top of most evidence hierarchies. It is not tradition or convention. Randomization is the only tool on this list that protects against confounders nobody thought to measure, simply because it does not rely on measuring them at all.

Stratification: splitting the data apart

Randomization is not always possible. Plenty of important clinical questions, the long term effects of a rare exposure, the outcomes of a treatment nobody would ethically randomize patients away from, can only be studied by observing what already happened. That is where stratification comes in: analyzing the data separately within subgroups defined by a suspected confounder, exactly what revealed the true pattern in the kidney stone data.

Stratification works well when there are only one or two suspected confounders. Split patients into small stones and large stones, or into smokers and nonsmokers, and compare outcomes within each group rather than across the whole pool. The tradeoff is that every additional confounder you stratify by shrinks the number of patients in each subgroup, and with enough subgroups the numbers become too small to draw a reliable conclusion from.

Multivariable adjustment and propensity matching

When there are too many confounders to stratify by hand, researchers turn to multivariable regression, a statistical model that estimates the relationship between an exposure and an outcome while mathematically holding several other variables constant at once. In practice, this is what a study means when it reports an outcome as adjusted for age, sex, and comorbidities: the model is attempting to isolate the exposure’s effect from the effect of those other variables.

Propensity score matching takes a related but distinct approach. Instead of adjusting for confounders directly, researchers calculate each patient’s probability of receiving the treatment based on their baseline characteristics, then match treated and untreated patients who had a similar probability. The goal is to recreate something that looks statistically closer to a randomized comparison, even though the treatment assignment was never actually random.

What none of these tools can fix

Stratification, adjustment, and propensity matching share the same fundamental limitation: they can only account for a confounder that someone thought to measure. If stone size had never been recorded in the kidney stone data, no amount of statistical adjustment could have uncovered the reversal, the paradox would have simply looked like one treatment being better, full stop.

This is called residual confounding, or unmeasured confounding, and it is the honest asterisk that belongs on every adjusted observational study. Adjustment can only ever be as good as the list of variables the researchers thought to collect.

Adjustment can only correct for the confounders you thought to measure. Randomization is the only tool that protects you from the ones you did not.

Key Takeaways

• Simpson’s Paradox occurs when a trend in pooled data reverses once the data is split into meaningful subgroups, usually because a confounder is unevenly distributed across the groups being compared.

• Randomization is the strongest defense against confounding because it balances both measured and unmeasured characteristics across groups by design.

• Stratification compares outcomes within subgroups defined by a suspected confounder, but becomes unreliable as more confounders and smaller subgroups are added.

• Multivariable adjustment and propensity score matching allow researchers to account for several confounders at once in observational data.

• Every adjustment method, no matter how sophisticated, can only correct for confounders that were actually measured, leaving residual confounding as a permanent limitation of observational research.

Up Next:

Next, we will shift gears from research design to diagnostic testing, starting with sensitivity and specificity, the two numbers that determine whether a positive or negative test result actually changes what you believe about a patient.