Doing More Harm than Good? Do Systematic Reviews of PET by Health Technology Assessment Agencies Provide an Appraisal of the Evidence That Is Closer to the Truth than the Primary Data Supporting Its Use?

Examples of Erroneous Representation of Factual Data

Errors of fact constitute the most potent source of bias in scientific reports and must be corrected if identified. We have found evidence of material factual errors in several influential HTA systematic reviews scrutinized in this evaluation and include misrepresentation both of primary data and of the authors’ conclusions.

Some of the factual errors in Medical Services Advisory Committee (MSAC) 2001 have been published previously (10). In its evaluation of the role of ¹⁸F-FDG PET in the staging of lung cancer, the Danish Centre for Evaluation and Health Technology Assessment (DACEHTA) stated incorrectly that the study of Pieterman et al. (13) was a randomized controlled trial and that it examined cost efficacy, and DACEHTA incorrectly asserted that Marom's study (14) was unblinded when blinding was clearly described in the study methodology. DACEHTA incorrectly stated that the study of Saunders et al. (15) included 17 malignant patients and 67 benign patients when the study in fact included “84 patients with biopsy proven lung cancer and 13 with a high suspicion of lung cancer.” DACEHTA's errors are admittedly identified in an unofficial translation from Danish. However, the same translation formed a primary source of evidence for the Health Technology Board for Scotland (HTBS) evaluation of PET, which included an extensive verbatim presentation of the DACEHTA findings. DACEHTA's findings have also been referenced by several other HTA systematic reviews.

Further, NIHR erroneously categorized the PET study of Porceddu et al. (16) in restaging patients with SCC of the head and neck as one of diagnostic accuracy. The clearly stated aim of this study was to evaluate the incremental management impact and therapeutic decision-making potential of PET in a specific clinical context. Because of this aim, spectrum bias precluded derivation of the usual diagnostic test metrics such as sensitivity and specificity. Nevertheless, NIHR reinterpreted the available published data erroneously, stating that “PET's sensitivity was five out of ten (50%).” This misrepresentation of the facts would create the impression that PET is not an accurate or useful test, especially as NIHR does not reference Porceddu's conclusion, as follows: “Patients who have achieved a complete response at the primary site but have a residual anatomic abnormality in the neck that is negative on PET scan approximately 12 weeks after treatment do not require neck dissection and can be safely observed.”

MSAC Head and Neck Cancer 2009 used NIHR as a starting point for its own analysis. MSAC replicates the NIHR errors, adding several inaccuracies of its own. The study of Porceddu et al. is classified as retrospective when prospective data collection was clearly documented. Claims that no data were provided on the accuracy of PET for detecting distant metastases in this population conflict with the detailed verification procedure documented in the original publication. These inaccuracies were material to MSAC's classification of this primary study as being of only fair quality.

Examples of Injudicious Appraisal of the Quality or Quantity of Available Studies

Before the publication of STARD guidelines, HTA systematic reviews were potentially constrained by poorly validated quality filters for assessing the risk of bias in primary studies of PET. Yet before publication of these criteria, the quality filters adopted by many PET HTA systematic reviews were not applied in a judicious, explicit, and conscientious manner. Nevertheless, these HTA systematic reviews are widely referenced and were often incorporated into later reviews.

The quality filters developed for the Veterans Affairs Technology Assessment Program (VATAP) review were subsequently influential in several later HTA systematic reviews (e.g., DACEHTA and Institute for Clinical Evaluative Sciences [ICES]). To illustrate failures inherent in the application of VATAP's grading system, we cite the 1995 paper of Valk et al. (17) examining the value of PET in presurgical staging of patients with non–small cell lung cancer (NSCLC). This study was graded as D (on an A–D scale for quality), suggesting multiple serious methodologic flaws with a high risk of bias. However, the summary table records a + against each of 3 quality filter items, signifying compliance. VATAP evaluators claimed the final diagnosis was not determined independently of the PET scan result, yet the authors indicate that histology was the reference standard for all 76 patients in whom the accuracy of PET and CT for detecting mediastinal nodal metastases was assessed. This validation was also supplemented by clinical and imaging follow-up. VATAP claimed that comprehensiveness of the nodal sampling was unclear, when in fact the paper contains very detailed descriptions. VATAP indicated that failure to enroll an equal number of patients with and without cancer indicated methodologic weakness. One can assess the merits of VATAP's decision by reference to a contemporary publication (18) that advises, “Almost any test can distinguish the healthy from the severely affected; this ability tells us nothing about the clinical utility of a test. The true, pragmatic value of a test is therefore established only in a study that closely resembles clinical practice.”

In a compounding factual error, VATAP incorrectly stated that the study included 76 patients rather than the true number, 99. The discussion of Valk et al. emphasized the clinical importance of the finding that PET uniquely detected distant metastases in 11 patients, yet VATAP described the data of Valk et al. as anecdotal and excluded the information from evidence summary tables because of small patient numbers. It seems probable that multifactorial misjudgment of Valk's primary study resulted in this seminal high-quality data, detailing the potential patient benefits of whole-body PET in NSCLC, being unfairly judged as having little or no scientific value by VATAP authors.

ICES adopted a modification of the VATAP 4-point scale for grading the quality of primary studies. An “a priori decision to concentrate on A and B grade articles” was made, presumably to reduce risk of bias. Otherwise relevant primary studies of PET in oncology, not judged to be A or B grade, were not referenced in any way in the report, creating the impression that these studies did not exist. ICES (Susan Garfinkel, written communication, 2005) provided us with the reviewers’ summary score sheets after we questioned the validity of their decisions in relation to published primary studies from our institution. Of the multiple peer-reviewed articles from our group, ICES graded only 12. Of these, 11 were subsequently excluded, including all 3 published in The Journal of Nuclear Medicine (19–21). Despite the ICES protocol requiring several or multiple flaws to attract C or D scores, respectively, only a single flaw was ascribed to 9 of the 11 excluded articles. This fault was a purported lack of independence between PET and the reference standard. However, all the excluded studies used a compound reference standard including pathology, if available, correlative imaging, and a long period of follow-up. Since none of this information was available at the time of PET scan reporting, the PET scans were, by definition, interpreted without knowledge of the reference standard and therefore could not have been subject to bias. Another primary study in which lack of independence was deemed to constitute a high risk of bias (22) was a cohort study that compared survival in 2 groups of NSCLC patients, one whose chemoradiation therapy was planned with and the other without PET. When PET was used to plan radiation treatment, that process could not have taken place with knowledge of the reference standard, which was the duration of the patient's subsequent survival.

In contrast, an abstract pertaining to a randomized trial of PET in NSCLC was awarded an A for quality. This evidence was included in the evidence tables and discussion even though the limited detail in the abstract precluded thorough assessment of methodologic quality. The final published results of this trial (23) will be discussed in more detail below.

The development of new standards for review of the diagnostic imaging literature has not prevented subsequently published HTA systematic reviews from including unreasonable assessments of the strength of the available evidence base. MSAC CRC 2008 failed to evaluate the paper of Kalff et al. examining the clinical impact of PET in restaging colorectal cancer (24) even though the primary study was cited. The methodology that underpinned this study subsequently formed the basis for a scientific study protocol that MSAC endorsed in 2003 as part of its National PET Data Collection process, with the paper arising from that multicenter study being subsequently published in The Journal of Nuclear Medicine (25). This study was discussed in great detail within this HTA systematic review, whereas the earlier paper was ignored.

When questioned about this omission, MSAC claimed that it had missed this reference because the reference was not cited in the “high quality” NIHR 2007 review that was used as a starting point to identify prior publications. However, Kalff's primary study was described in NIHR's discussion section as a high-quality prospective study of a consecutive series of patients that was well designed and well documented. When MSAC was made aware of the erroneous omission, it indicated that the exclusion of the primary study was not pertinent to its conclusions. MSAC CRC 2008 also omitted a cost consequence analysis, commissioned by a grant from the Department of Health and Aging, which demonstrated that PET could reduce health-care costs by approximately $4,000 per patient by virtue of avoidance of inappropriate surgery. Although this analysis was an internal document, this omission is inexplicable given that the study's chief author was also an author of MSAC CRC 2008 and that EBM places great value on finding and reviewing unpublished studies (26).

MSAC CRC 2008 also identified a systematic review of PET in recurrent colorectal cancer, authored by Dietlein et al. (27), that included 15 primary studies and hundreds of patients. The accuracy of PET was one of the parameters appraised, and pooled values for sensitivity, specificity, and likelihood ratios for PET and CT (as the clinical comparator) were detailed. PET was found to have superior sensitivity and specificity, with minimal overlap of the 95% confidence levels and markedly superior positive and negative likelihood ratios. This high-level evidence was omitted from the “Accuracy” discussion in MSAC's executive summary, which details positive predictive values from only 4 primary studies. Similarly, the executive summary does not refer to the systematic review finding of Dietlein et al. that management change occurred in 34% of patients (95% confidence interval, 31%–38%), mentioning only a single primary study and an unpublished randomized controlled trial. MSAC's justification for this omission of apparently relevant, high-quality evidence from its conclusions is that Dietlein et al. evaluated comparative accuracy, whereas MSAC indicated that they were addressing the question of incremental accuracy. Ironically, this is an approach that MSAC had specifically criticized in studies from our institution and was given by ICES reviewers as a reason for excluding our publications from analysis in their review.

Improved reporting is evident in MSAC Lymphoma 2010, with the adoption of the guidelines of the Quality of Reporting of Meta-Analyses conference (28). These allowed some quantitative understanding of the potential evidence base. There were also rudimentary qualitative reasons given for evidence exclusion. Incorporation of results from the MSAC 2001 HTA systematic review led to the inclusion of 38 publications before 2000. An updated literature search was then performed, extending from 2000 to 2009. This identified 2,319 potentially relevant studies. Of these, 2,107 (91%) were excluded for a variety of reasons without publication retrieval. Of 212 remaining studies selected for detailed analysis, only 16 (0.7% of the original sample), plus an unpublished report from the Australian Data Collection Study, were included in the HTA systematic review. Five systematic reviews and an HTA (NIHR) were also mentioned (but not HTBS or ICES). Although the MSAC 2001 systematic review was graded as a high-quality systematic review, the findings pertaining to the 38 studies reviewed were not detailed in the executive summary or results section. Furthermore, neither this evidence nor other systematic reviews or HTA appraisals were added to the evidence summary matrices.

That such a large number of primary studies and other evidence were excluded in this review raises questions about the validity of the analysis to clinical practice. Importantly, this failing was specifically remarked on in the final report under the heading “Expert Opinion” as follows: “Many or most of the excluded studies indicate a positive impact of PET on staging, response assessment or prognostic evaluation, so that the clinical utility of PET may have been underestimated.…Several studies which failed to meet the inclusion criteria have shown a dramatic relationship between early PET response and outcome.”

As an example, the multicenter study of Gallamini et al. detailed 260 patients with Hodgkin lymphoma, demonstrating a highly significant relationship between treatment outcome and ¹⁸F-FDG PET scan results after 2 cycles of chemotherapy (29). Multivariate analysis showed PET status to be the only significant predictor of treatment outcome (P < 0.0001). PET was found to have a 93% positive predictive value and 92% negative predictive value for 2-y progression-free survival. However, MSAC excluded this study on the basis of the comparator used for assessment. Presumably, this was because the primary study used the International Prognostic Score and not simply CT as the comparator for PET. However, the exact reason is not stated explicitly enough for certainty. Similarly, the 2006 study by Zinzani et al. (30) examining the same clinical question, with concordant findings, was also excluded by MSAC for the same reason.

Similar errors of omission are apparent across most of the HTA systematic reviews that we have evaluated. That such a large body of the published literature is routinely excluded from evaluation in HTA systematic reviews is a matter of concern regarding either the experimental methods accepted by major peer-reviewed journals such as The Journal of Nuclear Medicine or, alternatively, the process by which evidence is evaluated within the EBM process by HTA agencies.

Evidence of Injudicious Appraisal of the Consistency of the Evidence Base

The primary studies of PET are remarkable for their consistency in demonstrating superior diagnostic accuracy in relation to contemporary clinical comparators (2). These data have been recapitulated and, in many cases, improved by the introduction of PET/CT (3). In many of the papers cited, clinicians have also documented or imputed marked management changes.

This consistency receives little emphasis in the HTA systematic review appraised. Rather, undue emphasis is placed on the risk of bias, thereby questioning the validity of the primary study. For example, with respect to the value of PET for staging of NSCLC, HTBS notes that 33 papers were identified, with the commentary that “almost all reported that PET is more accurate and more sensitive than CT,” and 2 positive meta-analyses were also cited. These statements were, however, downplayed by the comment, common to many other HTA systematic reviews, that “…these results need to be confirmed in larger randomised trials.”

The HTA systematic reviews also stress apparent inconsistencies between studies. For example, although HTBS noted a published randomized controlled trial (31) reporting a statistically significant (P = 0.003) reduction in patients undergoing futile thoracotomies in the PET group, this was discounted under the heading “Limitations of the Evidence” by reference to an apparently “conflicting” randomized controlled trial available only in abstract form at that time. Assessment of quality of this trial was not possible, as not all patients had been fully evaluated. Indeed, detailed review of the subsequently published primary data from this trial (23) indicates spectrum bias, with most patients having very early stage disease. Also, the primary endpoint was the number of thoracotomies avoided in patients randomized to PET staging, and yet thoracotomy was frequently performed even in patients for whom PET made accurate incremental findings of mediastinal nodal and distant metastatic disease. If the institution's standard of care for NSCLC was thoracotomy irrespective of disease stage, the ethical and scientific context for performing the randomized controlled trial could be questioned.

A major benefit claimed for systematic reviews is to resolve uncertainty when original research, reviews, and editorials disagree (32), yet PET and PET/CT studies are remarkably consistent in supporting the clinical utility of these techniques. Paradoxically, HTA systematic reviews commonly follow the HTBS example of emphasizing apparent heterogeneity as a justification for calling for more definitive evidence. However, when evaluated in a clinical context, variability in results reflects appropriate clinical selection of a patient population to address a specific clinical question.

Evidence of Injudicious Appraisal of the Clinical Importance of Evidence

Injudicious appraisal of the strength and consistency of the evidence base that we have pointed to must introduce a risk that HTA systematic review judgments about the clinical importance of the evidence will also be biased. One threshold question that has not been judiciously addressed when HTA systematic reviews were formulating their decisions about the merits of PET is as follows: accepting that estimates of the diagnostic performance of PET contain some bias, what is the likelihood that the consistently higher accuracy of PET and the 30%–40% documented or imputed change in management related to more accurate disease classification is really a null finding?

Injudicious assessment of the clinical impact of the primary study arises because several HTA systematic reviews, both before STARD (VATAP and ICES) and after STARD (NIHR), fail to ask a clinically focused question specifically addressing patient-important outcomes. For example ICES simply states, “We examined the evidence for clinical applications of PET among 7 commonly occurring categories of cancer.” NIHR 2007 replicates this failure in the post-STARD era, stating its research questions as, “What is the clinical effectiveness of FDG PET for the management of the following cancers…?”. NIHR's conclusion that, as of 2007, there were no published studies demonstrating that ¹⁸F-FDG PET leads to improved patient outcomes is potentially a biased appraisal of the evidence that, in part, relates to the impossibility of providing a valid judgment of the clinical relevance of the evidence across such a wide spectrum of cancers and clinical management questions.

We believe that injudicious appraisal of the evidence pertaining to the clinical importance of PET in the context of individual patients is intimately related to the differences between HTA systematic review methodologists and clinicians regarding the types of evidence required to make valid judgments about the patient benefits of diagnostic tests. In short, HTA analysts have remained highly skeptical of the clinical view that studies validating diagnostic accuracy with a low risk of bias can be used to make sound judgments about potential patient benefit, if reduction in false-negative and false-positive results can be imputed (or shown in trial-based evidence) to have a direct and material impact on patient-important outcomes. Clinicians intuitively understand that false-negative results can delay appropriate introduction of effective treatments or result in patients receiving inappropriate treatment based on inadequate knowledge of the presence or extent of disease, whereas false-positive results often lead to unnecessary anxiety, institution of sometimes morbid and expensive investigations and therapy, or denial of potentially curative treatment.

Convergence to the clinical view that randomized controlled trials may be neither ethical nor essential for judging the patient benefits of diagnostic technologies appears to be gradually occurring among EBM and HTA methodologists (33). The Grading of Recommendations Assessment, Development and Evaluation Working Group (34) still considers randomized studies to be the ideal method for comparing diagnostic approaches but recognizes that observational studies comparing alternative diagnostic strategies with assessment of direct patient-important outcomes can be the basis for strong recommendations about the patient benefits of a more accurate diagnostic test.

Although the continuing evolution of the EBM literature toward a greater focus on patient-important outcomes is encouraging, most HTA systematic reviews continue to negatively judge the evidence supporting PET for want of evidence from randomized controlled trials. As an example, MSAC CRC 2008, referring to the potential for improving patient outcomes related to the demonstrated ability of PET to produce a significant number of additional true-positive findings for extrahepatic metastases in comparison to best conventional evaluations, comments, “In the absence of randomised controlled clinical trials of alternative treatments in this patient group, it is not known whether the quality of life benefits from avoiding surgery and instigating alternative management outweigh any potential benefit of surgery in providing local disease control.”

Yet, in the context of contemporary standards of clinical care of hepatic recurrence of colorectal cancer, extrahepatic metastases (detected clinically or by conventional imaging) usually preclude attempted curative surgery because there is no reliable evidence of improved outcomes in this situation. MSAC acknowledges PET to be safe, and also quantifies the harm associated with hepatic resection as a death rate of 1%–2% and a catastrophic morbidity of approximately 10%. With the information available, it seems an inescapable conclusion that attempted hepatic resection will do the patient more harm than good if performed on patients for whom PET has uniquely defined extrahepatic metastases, yet MSAC creates unnecessary uncertainty by emphasizing the lack of data from randomized controlled trials. As a consequence of this approach, the evidence supporting PET's beneficial effect in this context is said to be supported only by expert opinion.

The NIHR judgment that there are no published studies demonstrating that ¹⁸F-FDG PET improves patient outcomes rests heavily on discrediting the clinical relevance of the many nonrandomized controlled trial–based primary studies and systematic reviews that were evaluated, as well as discounting completely the randomized controlled trial of van Tinteren et al. (31) that clearly shows improved patient outcomes.

Evidence That Factors Unrelated to Patient-Important Outcomes Risked Introducing Bias into HTA Systematic Reviews

EBM values mandate that evidence evaluation be conducted first and foremost from the perspective of individual patients. However, several HTA systematic reviews (VATAP, MSAC 2001, HTBS, and NIHR) adopted appraisal protocols that give societal benefit greater value than individual-patient benefit. For example, both MSAC 2001 and NIHR used the hierarchy of diagnostic efficacy established by Fryback and Thornbury (35) to quantify the level of evidence available. This hierarchy is a 6-point scale that accords the highest value to demonstration of societal benefit as defined by cost efficacy, cost benefit, or cost utility. Studies designed to assess the diagnostic accuracy and predictive value of PET were relegated to levels 2 and 3 under that schema. HTA systematic reviews of PET repeatedly judged such studies as low-level evidence, even though they contain knowledge that is crucial to decision making in the context of individual patient management and therefore of paramount importance within a true EBM context. We believe this decision about classifying the value of evidence has significant potential to prejudice judgments about the strength consistency and clinical relevance of evidence in relation to patient-important outcomes.

Contrary to the tenets of EBM that require the needs of individual patients to be paramount (36), it appears that at least one of the reviews was structured to produce findings and recommendations that were beneficial to those with responsibility for funding PET. Although MSAC was established to be an “independent multidisciplinary scientific committee,” documents obtained under Freedom of Information Legislation show that for MSAC 2001 “the objective was to retain funding at the current level.”