Reply: Evidence-Based Assessment of PET in Germany

REPLY: I would like to thank Drs. Scheibler and Kleijnen for their letters. The very intention of my editorial (1) was to stimulate a discussion on the methodology for the assessment of new imaging technologies.

In response to the letter by Scheibler et al., I am convinced that the German Institute for Quality and Efficiency in Health Care (IQWiG) uses a “diligent process of selecting and commissioning external partners” for its reviews, and I had no intention to question this in my editorial. For the preparation of the reports on PET, IQWIG has awarded contracts for “preparation of indication-specific background, screening of the primary studies, data extraction and summary and evaluation of the clinical relevance of included studies and their results in an executive summary.” These services had to be performed according to the “methods of the Institute for Quality and Efficiency in Health Care” (2). Four contracts were awarded to Kleijnen Systematic Reviews, “an independent research company that produces and disseminates systematic reviews, cost effectiveness analyses and health technology assessments of research evidence in health care” (3). Therefore, I do not think that the statement in my editorial on the role of “companies specialized in preparing systematic reviews” is incorrect.

It is correct that the reports on PET published by IQWIG are prepared in collaboration with at least one experienced clinician, but the role of this expert is merely advisory. Specifically, the formal evaluation of the primary studies according to the methodology requested by IQWIG cannot be changed by the clinical expert’s input. This formal evaluation is performed by companies or university institutes specialized in systematic reviews. I have no reason to question that these institutions are highly experienced and have diligently applied the methodology requested by IQWIG after consultation with the clinical experts.

However, I am concerned that the generic “evidence-based methodology” as mandated by IQWIG is not applicable to imaging studies related to cancer staging. Even the best-qualified experts will reach incorrect conclusions if they have to use inappropriate methodology.

Kleijnen et al. take offense at my statement that reviewers who were commissioned by IQWIG did not “ judge the content of the reviewed publications but rather assess their quality solely by formal criteria as described by [quality assessment of diagnostic accuracy studies (QUADAS)]”. In response, I would like to emphasize that a formal evaluation of clinical trials is entirely appropriate if the used methodology is adequate for the studied question. My remarks were certainly not meant to be defamatory, and I do not think that they can be construed to be so. Kleijnen et al. mention the role of the clinicians in the preparation of the reviews. However, as I described in response to Scheibler et al., the role of the clinical experts was only advisory. If the quality of a study was considered poor (based on QUADAS), there was little the clinical expert could do to change this assessment.

Kleijnen et al. also complain that I made “plainly wrong statements” about the development of QUADAS. However, I did not state in my editorial that the experts developing QUADAS were anonymous. My statement about “anonymous experts” was in the context of a general explanation of the Delphi method, which frequently relies on anonymous experts (4). I am not sure which other “plainly wrong statements” Kleijnen et al. are criticizing. I did write that a consensus of experts is generally considered as poor evidence according to the standards of evidence-based medicine and that QUADAS was developed by a consensus of experts. Of course, “expert opinions” should be informed by previous research and empiric evidence. Otherwise, the term expert would hardly seem justified. Therefore, I do not think it was necessary to specifically mention that the experts participating in the QUADAS panel were making their recommendations based on “evaluations of existing tools and of the empiric evidence about the sources of bias.”

Unfortunately, Kleijnen et al. respond with only the following 3 sentences to my critique of QUADAS and its application in the IQWIG reports: “ Some limitations of QUADAS have been improved on in QUADAS-2. Other limitations, such as most of those mentioned by Weber, are clearly described in the QUADAS publication and advice is given to reviewers on how to handle them. Examples are given for each item of situations in which the item does not apply.” QUADAS-2 was not used for the IQWIG reports and is therefore not relevant for this discussion. It remains open which of the limitations that I discussed are “clearly described in the QUADAS publication” and which are not.

In any event, these limitations of QUADAS were apparently not realized by the IQWIG reports. For example, the report on PET in melanoma (5) uses QUADAS items 3–7, 10, 11, 13, and 14 for the evaluation of studies on staging of melanoma with ¹⁸F-FDG PET (see appendix E2 of this report). In my editorial, I argued that items 3–7 and 11 of QUADAS are a priori violated in most studies on cancer staging with imaging techniques. If this limitation of QUADAS was clearly described in the original publication from 2003, why were these items still used in the IQWIG report from 2011?

I fully agree with Kleijnen et al. that it is necessary to critically evaluate new and existing imaging tests with respect to a patient-relevant benefit. I also agree that health-care resources are limited and that informed decisions need to be made regarding the reimbursement of imaging tests. Diagnostic accuracy is a key parameter for making these difficult decisions. Clinicians and biostatisticians should therefore work together to develop a more appropriate framework for the evaluation of the diagnostic accuracy of imaging tests for cancer staging. Otherwise, it will become impossible to introduce novel imaging technologies that not only can improve patient management but also have the potential to reduce overall health-care costs by avoiding expensive but ineffective therapies.

• © 2012 by the Society of Nuclear Medicine, Inc.