The Move from Accuracy Studies to Randomized Trials in PET: Current Status and Future Directions

Published Studies

The search in MEDLINE, EMBASE, and the Cochrane Library resulted in 3,311 citations at first and 2,228 after the duplicate check. Screening of titles and abstracts resulted in 63 potentially relevant papers. Screening of these resulted in identifying 30 papers describing 14 trials that fulfilled our inclusion criterion. Table 1 summarizes the results of our investigation for the 14 published studies identified. Ten studies were from oncology, 3 from cardiology, and 1 from neurology. Six studies considered staging, 5 treatment planning, and 3 follow-up evaluation. One publication (14) reported a study protocol, not a finished study. The study of Plewnia et al. (15) used a cross-over design.

In 5 of the 14 studies, the outcome was not a patient-important one. The number of positive scans used by Mullani et al. (16) covered both false-positive and true-positive scans, so that any increase might have been due to just an increasing number of false-positive scans, which is not an advantage for the patients. Correct upstaging used by Maziak et al. (17) implies avoiding unnecessary surgery and, hence, is an advantage, but its frequency has to be balanced against the frequency of incorrect upstaging, which implies that patients miss the advantage of surgery. Indeed, this study observed a significant increase in both correct upstaging by PET and incorrect upstaging by PET. Recurrence during follow-up was the primary outcome in the study of Sobhani et al. (18). In treatment trials, this is a patient-important outcome reflecting morbidity. However, Sobhani et al. did not use PET as a means to prolong the true time until recurrence but rather tested whether PET could shorten the time until detection of recurrences, which is a diagnostic decision and not a therapeutic effect. Actually, the authors reported that only 27 of the 44 recurrences they observed could be confirmed by biopsy or surgery. Therefore, since it remains unclear whether their patients actually benefited from the earlier detection, this is not a patient-important outcome either. Similar arguments apply to one further study (19). Finally, one study (20) considered the number of investigations needed to finalize staging as primary outcome. Reducing this number may be beneficial to the quality of life of the patient. However, this advantage may be counterbalanced by a poorer quality of staging. Consequently, this outcome can be regarded as patient-important only if we are sure that there was no loss in diagnostic accuracy. The authors provided additional information on the agreement between clinical and final staging in both arms addressing this issue. However, with limited sample sizes it is difficult to ensure true equivalence.

Nine studies used a patient-important outcome. Three of these studies (21–23) used long-term outcomes related to (disease-free) survival. One study (15) applied a disease-specific quality-of-life measure. The outcome used in the study of Viney et al. (24) was the performance of a thoracotomy. As that is a management decision, we have to be careful: if a diagnostic method never points to performing a thoracotomy, this method may turn out to be the best one. However, the authors listed in their Table 3 an overview of the reasons for not performing a thoracotomy, and in the text they provided an explicit reason why the thoracotomy would be futile for each patient lacking thoracotomy. Hence, in this case, to regard the outcome as being patient-important seems justified, as it reflects avoidance of an unnecessary invasive procedure. Another 3 studies (9,25,26) used the number of futile thoracotomies or futile laparotomies as outcome. Again, we have to be careful because reducing the number of surgeries also reduces the number of futile surgeries. However, there is only limited doubt about the fact that this is a patient-important outcome because the authors provided detailed information on why the surgical procedure would have been futile for each patient not exposed to it. Moreover, in all 3 studies, the definition of futile surgery also took into account recurrence or death within 6 or 12 mo, and a reduction in recurrence or death actually contributed substantially to the reduction in futile surgeries. Therefore, this outcome also represents improved patient management after surgery beyond the simple avoidance of surgeries. Consequently, this outcome is close to a long-term outcome such as survival, and all in all, we can regard it as being patient-important. The published study protocol (14) defines futile indications for direct laryngoscopy as primary outcome. The final decision on patient importance depends on whether the authors will be able to provide sufficient evidence for the validity of no-surgery decisions.

A sample size calculation was performed for most studies. However, only 2 studies succeeded in presenting explicit justification for the assumed effects in both arms. Moreover, if we look at the effects to be assumed to reach a power of 90% (Table 2), we observe that in most of these studies relative risks or hazard ratios of less than 0.5 have to be assumed to reach this power. In treatment research, such large effects are rare, and Djulbegovic et al. (27) suggested regarding the few studies that were able to demonstrate effects of this magnitude as “breakthroughs.” On the other hand, all 3 studies using futile surgery as primary outcome (9,25,26) actually estimated relative risks in the magnitude of 0.5. A Cohen d of 0.3 is typically regarded as a rather moderate effect.

With respect to the choice of the comparator, we could identify appropriate guidelines for 9 studies from oncology (Table 1). In all cases, the comparator chosen was not in conflict with the existing guidelines. In one study (26), CT was applied in all patients of the control arm, although this was not yet recommended in the guidelines. For 2 cardiologic studies, the onset of the study was not sufficiently specified to locate the corresponding guideline. The comparator in the study of Beanlands et al. (21) was in accordance with existing guidelines.

Registered Studies

The search in the trial registers resulted in 263 studies (clinicaltrials.gov, 63; International Clinical Trials Registry Platform, 166; ISRCTN, 34). Removing duplicates and screening the information available in the trial register resulted in 20 eligible trials. Among these, 5 were already included in the 14 published studies. The remaining 15 studies are summarized in Table 3. Fourteen studies are from oncology, and one from cardiology. In these studies the clinical situation was treatment planning for 5 studies, staging for 4, primary diagnosis for 2, follow-up for 2, and response evaluation for 1. In most studies, the primary outcome was related to accuracy, diagnostic decisions, or management decisions and, therefore, could not be regarded as patient-important. Only 3 studies used an unequivocally patient-important outcome. For 2 further studies, this decision depends on whether the events considered will be validated. One study used the response rate, which is today typically regarded as a surrogate outcome and not a true patient-important outcome (28). However, the description of the primary outcome was often close to a description of the general aim of the study, and hence we cannot exclude that the primary outcome actually chosen in the analysis will be patient-important.

Three clinical situations contributed at least 2 studies both among the published studies and among the registered studies: staging, treatment planning, and follow-up. The median sample sizes in the published studies were 188, 210, and 130, respectively. In the registered studies, the corresponding median sample sizes were 220, 288, and 214, suggesting a slight increase.

Comparison with Results of Scheibler et al.

The review of Scheibler et al. (10) identified 12 published studies. We identified 3 additional studies that were probably excluded by the authors for the following reasons: use of ¹⁸F-FDG coincidence imaging by dual-head γ camera rather than a real PET scanner (19), focus on risk factors instead of modality comparison (16), and the nature of a study protocol (14). One study included in the review of Scheibler et al. was excluded in our review, as PET was administered to all patients (29). Scheibler et al. identified 42 registered studies, in contrast to 20 identified by us, 4 of which were not included in their review. The much larger number of Scheibler et al. probably resulted from their searching for all randomized trials, whereas we included the term diagnos* in the search strategy. Moreover, we included only comparative RCTs, comparing PET with a competitor. Of the 28 additionally identified studies, 18 used an enrichment or interaction design; that is, they were not comparative in our sense. Six studies considered treatment planning, and 10 studies considered response evaluation. If we focus on primary diagnosis, staging, and follow-up evaluation—the most common context of accuracy studies—there is only one additional comparative study (ClinicalTrials.gov identifier NCT00329706) in the review of Scheibler et al. This study compared the impact of making the result of a PET scan available to the clinician immediately or after 2 y. This study actually was identified in our search but was excluded, because both arms were PET-based.