Commentary
Real-world effectiveness of new medicines should be evaluated by appropriately designed clinical trials

https://doi.org/10.1016/j.jclinepi.2009.07.013Get rights and content

Abstract

Objectives

Health care providers, policy makers, and importantly patients themselves are increasingly interested in the outcomes of clinical trials yet often expect different questions to be addressed than those commonly asked in conventional phase 3 trials.

Study Design and Setting

Review of methodological articles.

Results

Conventional randomized controlled trials (RCTs) emphasize internal validity through standardization and control but by design reduce external validity, that is, generalizability of results and conclusions. Ongoing uncertainty about effectiveness or safety of medical interventions in the real world is the major driver for developing improved phase 3b and phase 4 study designs. Factors that should improve the relevance of these real-world trials (RWTs) include choice of endpoints; investigator specialty, appropriate patient selection criteria; emphasis on patient–physician interaction; admittance of relevant interventions in all study groups; and more flexible, simple, and possibly event-driven study visits and procedures, while maintaining randomization as a critical element to address confounders.

Conclusion

Although we do not believe that RWTs will supplant conventional RCTs, properly designed RWTs will enrich our understanding of the effectiveness of new health care interventions and better inform patients and health care providers alike.

Introduction

What is new?

  • Health policy makers in many countries require strong evidence of effectiveness and cost effectiveness in order to recommend reimbursement of new health technologies.

What this adds to what was known:

  1. Conventional randomised controlled trials conducted for registration purposes often provide only limited evidence to support reimbursement decisions because of artificial treatment settings.

  2. Real World Trials aim to apply the methodological rigour of conventional randomised trials to clinical practice settings which mimic clinical practice.

  • What is the implication, what should change now?

  • Real World Trials should be more commonly undertaken, and should be available early in the decision making process to inform reimbursement decisions.

  • It is anticipated that Real World Trials will complement rather than replace conventional randomised controlled trials.

Randomized controlled trials (RCTs) are the gold standard for obtaining regulatory approval and to pursue efficacy or safety as the primary objective, usually attempting a medical or biological hypothesis, preferably by comparing an experimental drug vs. placebo rather than facilitating the decision making between different medical treatments. These trials tend to be conducted in tertiary care sites even when the treatment will ultimately be used in other treatment settings. The emphasis on internal validity through standardization and control, however, carries a price on execution of the trial, reduces relevance of the question addressed, and thus external validity, that is, generalizability of results and conclusions. As a result, as early as 1967, the deficiency of drug development programs to address this issue has been recognized, and a “pragmatic” approach within the context of RCTs suggested [1]. Unfortunately, regulatory guidelines emphasize placebo-controlled trials and warn that active-controlled trials may obfuscate differences between treatments as a result of “poor diagnostic criteria, poor methods of measurement, poor compliance, medication error, or poor training of observers” [2]. As a result, industry and academia have not very much focused on developing robust trial designs that ask questions relevant to medical decision making. To make things worse, both have tolerated poor quality trials, often without comparator groups or careful data collection, calling these trials loosely “pragmatic,” “practical,” or “observational” even though the value of nonrandomized observational trials (NROTs) remains controversial. Two meta-analyses [3], [4] claimed that observational studies did not overestimate the size of the treatment effect compared with their randomized counterparts. NROTs even appeared to have highly concordant odds ratios, whereas the results of the randomized trials were often more heterogeneous. The correlation coefficient between the odds ratios of randomized and observational trials was 0.84 (P < 0.001) [5]. However, there were significant caveats of those meta-analyses including missing summary point estimates for some observational studies and discordant pooled point estimates in other cases, that is, pointing into different direction vis-à-vis risk–benefit [6].

There are also a few obvious examples where NROTs clearly failed to deliver reliable information: NROTs with interventions such as beta carotene and alpha tocopherol crashed when the corresponding hypotheses were tested in RCTs [7], [8].

Randomization allocates both known and unknown confounders between experimental groups on the basis of the play of chance. This does not make the groups “the same” because there will be chance differences between the groups. However, randomization reduces the differences between groups to the experimental treatment of interest or the play of chance; thus, any observed differences at the end of the study may only be attributable to chance or the experimental treatment. This orthogonal property of the observed difference proves to be a massive simplification, dealing in one step with the potential risks of confounders (both known and unknown). Nonrandomized studies can only attempt to “adjust” for the effects of known confounders and are largely powerless to deal with the effects of unknown confounders or those that are related completely with another factor of interest (e.g., the so-called “confounding by indication” where higher risk subjects are preferentially given treatments and the outcome of interest is also related to disease severity) [9].

Thus, even though observational studies incorporate useful real-life elements such as choice of treatment and relevant treatment settings, concerns about bias and confounding prevent confident conclusions and recommendations [10]. Is there a place for trials designed to have the protection from bias achieved by randomization, while reflecting more accurately standard treatment settings than the conventional phase 3 trial?

Section snippets

The problem

Ongoing uncertainty about efficacy or safety of medical interventions in the real world is the major driver for phase 3b and phase 4 trials. In addition to traditional groups such as regulatory agencies and physicians, health services and insurers are increasingly interested in the effectiveness of new health care technologies in a realistic clinical setting so as to predict impact on health care utilization [11], [12]. These new groups prefer dedicated trials to address these questions as

A solution: the real-world trial

Real-world trials (RWTs) are heretofore ill defined as a class and, when conducting literature searches, appear to include a large design spectrum ranging from uncontrolled studies or NROTs (stand-alone or follow-up of RCTs) to properly randomized trials that differ only in a few aspects from conventional phase 3 trials. Their stated objective frequently includes the term “effectiveness” as opposed to efficacy, implying that assessment of benefit or risk is taking place in a setting closer to

References (17)

  • D. Schwartz et al.

    Explanatory and pragmatic attitudes in therapeutic trials

    J Clin Epidemiol

    (2009)
  • J. Lau et al.

    Summing up evidence: one answer is not always enough

    Lancet

    (1998)
  • Drug study designs

  • K. Benson et al.

    A comparison of observational studies and randomized, controlled trials

    N Engl J Med

    (2000)
  • J. Concato et al.

    Randomized, controlled trials, observational studies, and the hierarchy of research designs

    N Engl J Med

    (2000)
  • J.P. Ioannidis et al.

    Any casualties in the clash of randomised and observational evidence?

    BMJ

    (2001)
  • The effect of vitamin E and beta carotene on the incidence of lung cancer and other cancers in male smokers

    N Engl J Med

    (1994)
  • S. Yusuf et al.

    Vitamin E supplementation and cardiovascular events in high risk patients. The Heart Outcomes Prevention Evaluation Study Investigators

    N Engl J Med

    (2000)
There are more references available in the full text version of this article.

Cited by (46)

  • Introduction

    2021, Pragmatic Randomized Clinical Trials: Using Primary Data Collection and Electronic Health Records
  • One year of mepolizumab. Efficacy and safety in real-life in Italy

    2019, Pulmonary Pharmacology and Therapeutics
View all citing articles on Scopus
View full text