Evidence-Based Assessment of PET in Germany

TO THE EDITOR: In his Invited Perspective, Professor Wolfgang Weber calls for biostatisticians and imagers to “reflect on one’s own deficiencies” in order to make progress in the evaluation of evidence about imaging (1). Weber points out several deficiencies on the side of the biostatisticians but neglects to examine any deficiencies in his own perspective. In our view, one barrier to communication between clinicians and methodologists is misunderstanding what the other side is doing, and we would like to address some of these apparent misunderstandings.

Weber points out that the conclusions of one agency, the German Institute for Quality and Efficiency in Health Care (IQWiG), on the use of PET/CT in various malignant diseases are in conflict with clinical practice in the United States and Europe. He argues that the use of the quality assessment of diagnostic accuracy studies (QUADAS) instrument leads to the wrong conclusion that there are insufficient data to determine the diagnostic accuracy of ¹⁸F-FDG PET. In particular Weber argues that health technology assessment agencies or the reviewers commissioned by them “do not judge the [clinical] content of the reviewed publications but rather assess their quality solely by formal criteria as described by QUADAS.” This remark can actually be considered defamatory, and being reviewers for IQWiG, we hope it was not meant as it comes across. Weber is very well aware that both IQWiG and its external reviewers get input from clinicians who advise them in every single project. Indeed, guidance on the use of both QUADAS and its successor QUADAS-2 requires reviewers to consider the relevance of criteria to the clinical topic and to provide topic-specific criteria where needed.

As authors of both QUADAS and QUADAS-2 (QUADAS-2 replaced the original QUADAS last year), we are also very well aware that the QUADAS instrument has limitations, as indeed do all risk-of-bias tools (2,3). Some limitations of QUADAS have been improved on in QUADAS-2. Other limitations, such as most of those mentioned by Weber, are clearly described in the QUADAS publication and advice is given to reviewers on how to handle them. Examples are given for each item of situations in which the item does not apply. Weber also makes plainly wrong statements about the development of QUADAS. The experts who participated in the Delphi procedure are not “anonymous” but clearly mentioned and acknowledged in the QUADAS paper (2). The development of QUADAS was also not solely dependent on expert opinion. Evaluations of existing tools and of the empiric evidence about the sources of bias and variation in diagnostic accuracy research were performed before, and informed, the Delphi process (4,5).

As Weber correctly points out, there are indeed many different study designs that can be relevant for particular diagnostic questions, and IQWiG allowed a range of study types to be considered in order to evaluate PET. Although some QUADAS items may be applicable in other types of diagnostic studies, it is clearly a tool that is intended for accuracy studies. It is not only in diagnostics that there are clinical situations that make the design of the perfect study (in the sense of having low risk of bias) difficult or impossible. Some therapeutic interventions are impossible to evaluate in a double-blind way, or even in a randomized trial, and health technology assessment agencies are well aware of the real-life limitations to designing the perfect study. Using the “best available evidence” is a pragmatic approach in such situations. The IQWiG assessments aimed to use (if available) both randomized and controlled observational studies assessing the benefits of PET and, in addition, searched for diagnostic accuracy studies and studies addressing prognosis.

Weber argues that the lack of evidence on the clinical benefit of PET and PET/CT derives from the lack of a requirement for formal assessment at the time of introduction. Although this may be true, it does not invalidate the case for formal evaluation. Long-established diagnostic devices have been shown to be ineffective or even harmful for the patient’s overall management when finally tested in randomized clinical trials (6). This also holds true in applications of PET for which a randomized trial found no patient-relevant benefit (7).

Few countries can still afford everything in medicine that is available. With budget constraints, tough decisions have to be made about what is of the best value for the money. Whenever a choice is made to use one technology, it inevitably means that something else will be displaced. Health technology assessment agencies have the unenviable job of helping the decision-making process that leads to such choices. Good-quality evidence is the crucial element in making these informed decisions. Clinicians (or imagers, in the case of PET) have the unenviable job of helping to generate that evidence. We fully agree with Weber’s goal of fruitful collaboration; Wolfgang, the door is open for your help in our next imaging project!

• © 2012 by the Society of Nuclear Medicine, Inc.