Agreement of Observational and Randomized Trial Results

Critics of epidemiology often claim that the results of observational studies and randomized trials often conflict. However, besides obvious settings in which randomized trials are fantastically ethical, such as randomizing patients in clinical trials to adverse toxins, radiation, cigarette smoking, or no parachute use to prevent death and major trauma related to gravitational challenge, how true is the claim, on a more evidence-based scientific standard, that observational studies and trials too often conflict?

Fortunately, there were several prominent analyses in the NEJM and BMJ exactly focused on this question. In a BMJ paper in 2001 by Ioannidis et al. , the authors performed an analysis directly comparing the odds ratios of 25 separately clinical investigations in which both observational and randomized studies were conducted and found that the correlation between the odds ratio of randomised trials and the odds ratio of observational designs is actually 0.84, P less than 0.001. See Figure 1 below. This article notably reviewed 2 prominent studies in the NEJM, which both found that results from observational epidemiologic studies have been highly consistent with results from randomized trials-- see Figure 2, and references 2 and 3 below for more information. Indeed this body of evidence quite strongly supports the notion that prospective observational epidemiology is as reliable as randomized trials.

Furthermore, even in the field of digestive surgery, with relatively few large and long term cohorts, it was found in a review by Shikata et al. 2006 that only 4 of 16 observational studies differed significantly from randomized trials, with 75% of observational studies still in agreement with surgical trials, which often cannot be perfectly double-blinded themselves. In addition, there was also significant between-study heterogeneity in the body of these surgical randomized trials, suggesting the minor inconsistencies between study designs may be due to effect modification by underlying patient characteristics rather than bias from residual confounding in observational studies.

On the other hand, critics have frequently argued that epidemiologists got the answer wrong on the effects of estrogen, particularly as the large WHI estrogen trial was published after the 2001 study by Ioannidis et al. However, Grodstein et al.'s NEJM review article comparing the summary evidence about estrogen in prospective observational vs. randomized trials again clearly indicates strong agreement of results.

Notably, observational studies of estrogen therapy found nearly identical or similar associations for: 1. breast cancer, 2. colorectal cancer, 3. hip fracture, 4. stroke, 5. pulmonary embolism-- with the exception of only coronary heart disease, see Figure 2 summary below from Grodstein et al. However, the previous review of the estrogen and CHD issue discusses the likely explanation of the divergent results between observational studies and the WHI trial.

Although there are indeed some other observational studies that conflict with trials, most of such observational studies were either a) ecologic in design- the weakest study design of all and not considered rigorous, b) retrospective case-control design, which has many major limitations regarding recall and selection biases, c) were small in study size and thus more likely to exhibit false-positive results, and d) not carefully controlled for a wide range of confounders. In contrast, large and long term prospective cohort studies with >20 years of followup have generally provided the most reliable and reproducible results.

As we can see the prominent examples reviewed in the several BMJ and NEJM articles, the majority of the evidence indicates that observational epidemiology and clinical trials indeed provide generally consistent results.

[We encourage interested readers to subscribe for updates on new articles]

(click images to enlarge figures)

Figure 1. (from Ioannidis et al.)















Figure 2. (Benson and Hartz)















Figure 3. (from Grodstein et al.)











References:

1. Ioannidis JP, Haidich AB, Lau J. Any casualties in the clash of randomised and observational evidence? BMJ. 2001 Apr 14;322(7291):879-80.

2. Benson K, Hartz AJ. A comparison of observational studies and randomized, controlled trials. N Engl J Med 2000; 342: 1878-1886.

3. Concato J, Shah N, Horwitz RI. Randomized, controlled trials, observational studies, and the hierarchy of research designs. N Engl J Med 2000; 342: 1887-1892.

4. Shikata S, Nakayama T, Noguchi Y, Taji Y, Yamagishi H. Comparison of Effects in Randomized Controlled Trials With Observational Studies in Digestive Surgery. Ann Surg. 2006 Nov;244(5):668-676.

5. Grodstein F, Clarkson TB, Manson JE. Understanding the divergent data on postmenopausal hormone therapy. N Engl J Med 2003;348:645-650.