Interobserver Agreement of Qualitative Analysis and Tumor Delineation of ¹⁸F-Fluoromisonidazole and 3′-Deoxy-3′-¹⁸F-Fluorothymidine PET Images in Lung Cancer

DISCUSSION

This study was performed in 18 nuclear medicine centers in France connected with a dedicated network. A specific network for the transfer and exchange of PET/CT images and radiotherapy VOIs in multicenter studies was specifically built by the SFMN. Its objective is to favor and foster the implementation of studies requiring the transfer of images (e.g., scintigraphy, radiotherapy images, and delineation of boundaries). This network used Web access and dedicated software and workstations. This study demonstrated the robustness of this network and its ease to use in interobserver agreement studies and multicenter clinical research in the fields of nuclear medicine and radiotherapy. There are 2 major issues to take into consideration with respect to the interpretation of hypoxia data. The first, is the tumor hypoxic (or proliferative)? The second issue is, what is the true hypoxic (or proliferative) subvolume? Both points were assessed subsequently.

This research demonstrated that visual analysis of low-contrast PET images (¹⁸F-FMISO and ¹⁸F-FLT) is feasible and reproducible when the images are analyzed using a binary scale (positive or negative). The interobserver agreement was slightly higher for ¹⁸F-FDG images, but it could also be partly explained by the relatively limited experience of physicians with respect to ¹⁸F-FMISO or ¹⁸F-FLT image analysis. In contrast, the volume delineation was different from one physician to another for these tracers. Among the physicians, the best method found was 1.5 × SUVmax of the aorta for ¹⁸F-FMISO and 1.3 × SUVmax of the muscle for ¹⁸F-FLT. The methods using an SUV of 1.4 and the method using 1.5 × SUVmax of the aorta could be used for both tracers (¹⁸F-FMISO and ¹⁸F-FLT).

The κ-value is a chance-adjusted index of agreement, which can be used in medical research and especially to assess the reproducibility of visual analysis among physicians. It is well known that Cohen’s κ-values do not assume equal classification proportions for different observers, but they depend on the characteristics of marginal distributions. A limitation of our study is that the rate of true-positives in the study population and the bias of one reader relative to another could have significantly affected our results (18,19).

However, the data analysis doesn’t show any evidence of bias, because the different observers systematically differed in their classification. Most studies assessing multirater agreement involve between 3 and 8 readers. We believe that the large number of independent reviewers in this study should provide confidence in our results.

This study, performed in 18 different nuclear medicine centers, demonstrated high reproducibility of ¹⁸F-FMISO or ¹⁸F-FLT PET images when the images where interpreted using a binary scale. The reproducibility of the visual analysis was greater for ¹⁸F-FDG images than for ¹⁸F-FMISO and ¹⁸F-FLT acquisitions, no matter whether the images were interpreted using a 2- or 5-point grading scale. These results could be explained by a much steeper learning curve in the reading of ¹⁸F-FDG PET/CT images by physicians and by the lower contrast of ¹⁸F-FMISO and ¹⁸F-FLT images than of ¹⁸F-FDG. In our population, the SUVmax of ¹⁸F-FDG images was 6.1 (95% confidence interval, 4.0–8.3) before radiochemotherapy and 4.3 (95% confidence interval, 3.2–5.4) during radiochemotherapy. For ¹⁸F-FLT acquisitions, the mean SUVmax was 4.7 (95% confidence interval, 3.6–5.7) before radiochemotherapy and 2.2 (95% confidence interval, 1.5–2.8) during radiochemotherapy. For ¹⁸F-FMISO, the SUVmax was lower, at 1.5 (95% confidence interval, 1.3–1.7) before radiotherapy and 1.4 (95% confidence interval, 1.2–1.7) during radiochemotherapy. The low contrast of ¹⁸F-FMISO and ¹⁸F-FLT PET images could most likely explain the best reproducibility of the analysis of PET/CT images occurring before treatment, compared with images obtained during treatment (Table 3). The interpretation appeared to be easier for primary tumors, compared with nodes, for ¹⁸F-FDG, but it was also difficult for ¹⁸F-FMISO and ¹⁸F-FLT when using a 5-grade scale (Table 2). In all of the analyses, the reproducibility dramatically increased using a binary scale, compared with a 5-point scale. This is an important result for the robustness and future development of these tracers in nuclear medicine. Similar results were previously shown by our group for the reproducibility of ¹⁸F-FDG PET images in a series of patients with squamous cell esophageal carcinoma (20).

This study clearly demonstrated the lack of reproducibility by physicians of the delineation of hypoxic or proliferative tumoral volumes on PET/CT images. Many studies (21,22) demonstrated increases in the reproducibility of GTV delineation using ¹⁸F-FDG PET images, compared with GTV delineation with CT alone. Moreover, the pretherapeutic volume is important to determine precisely because some authors have suggested that the site of highest ¹⁸F-FDG uptake could be a site of recurrence (23). With ¹⁸F-FMISO PET/CT, our group designed a study to boost the hypoxic volume with external maximal-dose radiotherapy (clinicaltrials.gov NCT01576796). Other researchers have proposed boosting cervical radiotherapy in patients with head and neck cancer with the doses then escalated in the hypoxic GTV, as measured by ¹⁸F-FMISO PET/CT images (12). Similar approaches were performed (8,9) with head and neck cancers. Hendrickson et al. (9) increased the external radiotherapy dose to 60 Gy on cervical nodes.

Up until now, there have been no data published regarding the reproducibility of visual analysis of ¹⁸F-FMISO and ¹⁸F-FLT PET/CT images. Because this study demonstrated the poor reproducibility of analyses with these tracers (Figs. 2A and 2B), it appears important that each nuclear medicine department sets up a learning phase and evaluates the reproducibility among nuclear physician readers. A multidisciplinary meeting, with nuclear physicians and radiation oncologists, for volume delineation could be an interesting way to proceed, as suggested by Nestle et al. (5).

In ¹⁸F-FDG PET/CT, a universal method for GTV delineation has not been definitively validated for all situations (e.g., pretherapeutic vs. midtreatment, high vs. low contrast, and large vs. small tumors). The visual (manual) segmentation method is dependent on the experience of the nuclear physician and on the contouring protocol used, which can lead to significant variation in the assessment of GTV with ¹⁸F-FDG PET (24). To reduce these discrepancies, many authors favor the use of semiautomatic segmentation methods, which are considered to be more reproducible, taking into account the lesion volume, the level of SUV distribution heterogeneity, the tumor-to-background ratio, the background activity level, and the region-growing initialization (25,26). The use of these segmentation methods has revealed significant variations between CT-based and PET/CT-based radiotherapy treatment plans for GTV determination. Because of a lack of consistency in tumor contour delineation on PET, interpretation of the available data is difficult, and how PET imaging should be integrated into treatment planning remains uncertain, in the absence of a consensus on the use of a definitive segmentation method (27,28). Many studies have compared ¹⁸F-FDG PET/CT segmentation methods with CT and with MR imaging (29), and only few of these studied used pathologic examination data as references (10,29). Moreover, lung PET/CT imaging applications in thoracic or abdominal pathologies can be affected by motion artifacts related to respiratory movement (signal smearing). This effect can lead to the underestimation of a lesion’s SUVmax or overestimation of GTVs (30), even if different methods are applied to overcome respiratory motion (31). Carlin et al. recalled in a recent review the difficulties in defining a hypoxic volume with ¹⁸F-FMISO (32).

With ¹⁸F-FMISO and ¹⁸F-FLT, few methods have been tested to determine hypoxia and proliferative volumes. Manual methods and methods less sophisticated than ¹⁸F-FDG have been used with these tracers. Been et al. determined a proliferative volume of 40% of the SUVmax of ¹⁸F-FLT images. Nehmeh et al. (33) used a volume obtained with a value of the SUV at 1.2 × background measured in blood plasma. Grosu et al. (34) used, for ¹⁸F-fluoroazomycin arabinoside (FAZA), a volume determined by 1.5 × SUVmax of the cervical muscle. The issue to resolve for muscle is the variability of determination, thus Choi et al. preferred using cerebellar tissue (8). However, cerebellar tissue is not systematically in the field of view. For the esophagus, Han et al. obtained the best volume with an SUV of 1.4 for ¹⁸F-FLT acquisitions, as Tian et al. and Xu et al. (35,36) did for lung cancer with the same tracer. These results are in accordance with our results.

These methods, which have been studied previously, were tested and compared in this study of lung tumors. For methods using background comparison, 3 backgrounds were tested (aorta, contralateral lung, and muscle) and were used successively with SUVmean and SUVmax. The method of Lee et al. (9,12), using comparison to the background and using blood samples, has not been tested. More than 1,400 VOIs were drawn in this preliminary study, and no other complex methods were tested.

Our study clearly showed that all of these methods yielded different results for the volumes of ¹⁸F-FMISO (Fig. 3) and ¹⁸F-FLT (Fig. 4). It was difficult to validate the best method definitively because we did not have a gold standard for in vivo measurement of hypoxic or proliferative volumes. The invasive method, using Eppendorf electrodes, was insufficient because it tested only part of the tumor (37). A comparison with pathology (pimonidazole immunohistochemistry, for example) was not possible because it would have been available only after surgery. Therefore, we used the physicians’ validation as the gold standard. The ability of the methods to find hypoxic or proliferative volumes was compared with the panel of physicians. Excluded were all methods that yielded a hypoxic or proliferative volume when more than 50% of the physicians did not find it and all of the methods that did not determine a hypoxic or proliferative volume when more than 50% of the physicians found it. This approach was similar to calculating sensitivity and specificity (Table 4). We postulated that the hypoxic or proliferative volume could not be superior to the ¹⁸F-FDG GTV. Therefore, all of the methods yielding hypoxia or proliferative volumes superior to the ¹⁸F-FDG GTV, measured at a threshold of 40%, were excluded from the analysis. Thus, for ¹⁸F-FMISO, ¹⁸F-FMISO_40%, and methods using the SUVmean of lung background, comparisons were excluded (Fig. 3). For ¹⁸F-FLT, methods using comparison with background SUVmean (aorta, lung, and muscle) and lung SUVmax_max were excluded (Fig. 4). For the methods using the lung for the background, the SUVs were low (SUVmean and SUVmax) and thus responsible for large volumes. For ¹⁸F-FMISO, the methods using comparison of the background with aorta SUVmax and muscle SUVmax were most likely not efficient because they resulted in the absence of a hypoxic volume (volume equal to zero). Among physicians, the best method found was 1.5 × SUVmax of the aorta for ¹⁸F-FMISO and 1.3 × SUVmax of the muscle for ¹⁸F-FLT. The methods using an SUV of 1.4 and the method using 1.5 × SUVmax of the aorta could be used for the 2 tracers (¹⁸F-FMISO and ¹⁸F-FLT) with relatively good sensitivity and specificity. It is worth noting that, for a slightly larger value (1.6 × SUVmax), the sensitivity declines from 0.71 to 0.14. The threshold change (˜7%) is within the range of the statistical fluctuations of the SUVmax, indicating that even the best method for ¹⁸F-FMISO is unstable.

An optimal fraction for all reference values (max/mean aorta/lung/muscle) can be estimated empirically in a given study sample. It is likely that a different fraction would be optimal if the sample changed (e.g., different PET scanner, reconstruction, scan time). The issue is the degree of stability of these methods if there are small variations. The method stability is especially important in the context of multicenter trials. The above results indicate that the stability is different for ¹⁸F-FMISO and ¹⁸F-FLT. Therefore, an absolute threshold with an SUV of 1.4 is deemed preferable in view of its simplicity.

For these images with low contrast, statistical methods, such as random walk as defined by Onoma et al. during the International Symposium on Biomedical Imaging (ISBI) Congress 2012 or fuzzy locally adaptive Bayesian (FLAB) methods (38), might generate some interest if tested in future work.