Observer Variation in Interpreting ¹⁸F-FDG PET/CT Findings for Lymphoma Staging

MATERIALS AND METHODS

RESULTS

Results are summarized in Table 3. Intra- and interobserver agreement was high for overall Ann Arbor stage (κ_w = 0.79–0.91). For intraobserver agreement, 95% confidence intervals were within the “almost perfect” range of κ-values, whereas for interobserver agreement, there was crossover into the substantial-agreement range. Similarly, agreement was high for the total number of nodal groups involved (intraclass correlation coefficient, 0.83–0.93) and for presence of extranodal involvement (intra- and interobserver κ_w = 0.82–0.84 and 0.74–0.86, respectively).

For specific nodal groups, there was substantial or greater agreement for cervical (κ_w = 0.77–0.86), axillary (0.69–0.80), pelvic (0.65–0.82), inguinal (0.69–0.82), mediastinal (0.73–0.77), periaortic (0.75–0.81), and mesenteric (0.61–0.67) nodal groups. Agreement was lower for hilar (0.56–0.82) and infraclavicular (0.14–0.55) nodal groups. For extranodal involvement, there was substantial or greater agreement for spleen (0.69–0.84) and bone marrow (0.76–0.94). Agreement was lower for lung (0.58–0.90), liver (0.59–0.95), and bowel (0.37–0.71). The higher κ-values in these ranges reflect greater intraobserver agreement.

In the postanalysis review of patients with variability of infraclavicular nodal classification, all patients had distant disease and the variability did not change the overall stage. On consensus review, the variability was clearly due to variable definition of the boundary between infraclavicular nodes and adjacent nodal regions, including medial axillary and supraclavicular (Fig. 1). Review of the 8 patients with disagreement on bowel involvement indicated that 2 were related to gastric involvement and 6 to large-bowel involvement. On consensus review, all these patients had either stage 3e or 4 disease. In the cases of large-bowel involvement, the disagreement was related to differentiating colonic involvement from adjacent mesenteric nodal disease (Fig. 2).

• Download figure
• Open in new tab
• Download powerpoint

FIGURE 1. 

Axial (A) and coronal (B) PET and PET/CT images demonstrating focal increased metabolic activity in subcentimeter node (arrow) that was variably reported as supraclavicular, infraclavicular, or axillary. This did not result in a change in overall stage due to the presence of widespread disease, and this variability would therefore not change management.

• Download figure
• Open in new tab
• Download powerpoint

FIGURE 2. 

Axial PET and PET/CT images demonstrating large-bowel lymphomatous involvement (arrow) in addition to mesenteric nodal involvement. Some reviewers reported only mesenteric nodal involvement, but involvement of bowel wall was clearly evident on review.

DISCUSSION

Few studies have assessed observer variability in lymphoma imaging. Fletcher et al. (9) evaluated interobserver variability for CT detection of cervical–thoracic Hodgkin disease. For individual nodal stations, agreement ranged from poor to moderate, with κ-scores ranging from 0.13 for left paratracheal nodes to 0.72 for right lower cervical nodes. Agreement between the majority of reviewers and the primary report was poor (κ < 0.40) for two thirds of the sites. Zijlstra et al. (10) measured observer variation for PET staging and restaging. For experts using sensitive and conservative models, concordance was 61% and 56%, respectively, for staging and 82% and 94%, respectively, for restaging.

In this study, intra- and interobserver agreement was high for Ann Arbor stage, number of nodal regions involved, and presence of extranodal disease. Most of the study patients (89%) had high-grade lymphoma, in which high glucose use results in high lesion-to-background contrast. This facilitates easy visual perception of active sites of disease and is further assisted by the use of contemporaneous CT, which allows precise anatomic correlation and the identification of physiologic variants or other pathologic processes.

The lower agreement for extranodal bowel involvement appears to be due largely to difficulty in differentiating bowel involvement from adjacent mesenteric nodal disease. This difficulty is not unexpected on imaging alone and is unlikely to alter the management strategy. Interpretation is limited by the low prevalence of patients with bowel involvement (<5.5%); κ-statistics also tend to weigh disagreements more heavily when the prevalence of a positive finding approaches zero (11).

For specific nodal regions, agreement was lowest for infraclavicular nodes, as is consistent with previous findings for CT staging (9). The cause was disagreement about the definitions of infraclavicular, supraclavicular, and axillary nodal regions, but this disagreement did not change the overall stage, and locoregional radiotherapy would not have been considered. There was also lower agreement about hilar nodes than about other nodal stations. Hilar ¹⁸F-FDG uptake is not uncommon, because of inflammatory changes secondary to a reactive or granulomatous process (12). In staging lung carcinoma with PET/CT, we have previously described lower agreement in the hilar region than in mediastinal nodal stations (13). Caution is thus warranted in interpretation of hilar activity, especially if the intensity is discordant with uptake at other sites.

This study had several potential sources of error. The form-based method of data collection may have reduced errors by ensuring systematic review and standardization of terminology. PET/CT findings were not compared with histology or patient follow-up, and therefore no comment can be made on accuracy. Other studies, however, have demonstrated high sensitivity and specificity (1). In this study, reviewers were unaware of clinical history and were not provided with correlative imaging results—a discrepancy with routine clinical practice. Availability of this information, however, would likely result in higher agreement. For example, knowledge of lymphoma subtype will help the reviewer refine expected distribution, likely improving agreement.

This study was limited by the use of experienced PET/CT observers from a single center. As such, they could be expected to show a significant degree of concordance in their approach, especially as the center adopts a dual-reporting system. The findings are still of interest, as multiinstitutional trials increasingly use a central core laboratory for reporting. It would be useful to extend the study to assess variation across different centers and also investigate agreement with less experienced observers or trainees.

Our study did not assess observer agreement for restaging lymphoma after chemotherapy. Observer agreement may be lower in these patients because the intensity of ¹⁸F-FDG uptake may be low. Although the Imaging Subcommittee of the International Harmonization Project in Lymphoma has published positivity criteria for restaging after completion of chemotherapy (14), defining positivity when using PET for restaging during a course of therapy is less well established. Defining appropriate criteria and assessing observer variability in these patients warrant further investigation.

CONCLUSION

Among experienced physicians in a single center, there was a high level of intra- and interobserver agreement using PET/CT for lymphoma staging. This result complements the results of other studies demonstrating a high accuracy and supports the use of PET/CT as a robust noninvasive staging tool. Further research is needed to evaluate observer variability for restaging during and after completion of chemotherapy.