Issues in Comparisons Between Meta-analyses and Large Trials

ContextThe extent of concordance between meta-analyses and large trials on the same topic has been investigated with different protocols. Inconsistent conclusions created confusion regarding the validity of these major tools of clinical evidence.ObjectiveTo evaluate protocols comparing meta-analyses and large trials in order to understand if and why they disagree on the concordance of these 2 clinical research methods.DesignSystematic comparison of protocol designs, study selection, definitions of agreement, analysis methods, and reported discrepancies between large trials and meta-analyses.ResultsMore discrepancies were claimed when large trials were selected from influential journals (which may prefer trials disagreeing with prior evidence) than from already performed meta-analyses (which may target homogeneous trials) and when both primary and secondary (rather than only primary) end points were considered. Depending on how agreement was defined, kappa coefficients varied from 0.22 (low agreement) to 0.72 (excellent agreement). The correlation of treatment effects between large trials and meta-analyses varied from -0.12 to 0.76, but was more similar (0.50-0.76) when only primary end points were considered. When both the magnitude and uncertainty of treatment effects were considered, large trials disagreed with meta-analyses 10% to 23% of the time. Discrepancies were attributed to different disease risks, variable protocols, quality, and publication bias.ConclusionsComparisons of large trials with meta-analyses may reach different conclusions depending on how trials and meta-analyses are selected and how end points and agreement are defined. Scrutiny of these 2 major research methods can enhance our appreciation of both for guiding medical practice.