Interpreting unbiased comparisons

A fair treatment comparison is one that avoids biased comparisons. This entails taking steps to minimise biases due to differences between the patients compared, and biases due to differences in the way treatment outcomes are assessed.

Even if these biases have been avoided, however, interpreting unbiased comparisons is often not straightforward. For example, have any differences between treatments intended and treatments received been taken into account, and has account been taken of the play of chance?

Sometimes, a new study provides very strong evidence of the effects of a treatment. For example, tens of thousands of people participated in a remarkable study that showed that an aspirin tablet could substantially reduce the risk of death among people who are experiencing heart attacks (ISIS-2 1988). It is only very rarely, however, that a single study provides such strong evidence, so it’s important when reading reports of most studies to ask whether the new evidence has been integrated in systematic reviews of all other relevant evidence. If so, have steps been taken during that process of synthesis to minimise the impact of biased reporting of the available evidence and biased selection from the available evidence? Has the potential for reducing the play of chance using meta-analysis been considered?

Reference

ISIS-2 Second International Study of Infarct Survival Collaborative Group (1988). Randomised trial of intravenous streptokinase, oral aspirin, both, or neither among 17187 cases of suspected acute myocardial infarction: ISIS-2. Lancet 2: 349 60.