Automated Segmentation of Baseline Metabolic Total Tumor Burden in Diffuse Large B-Cell Lymphoma: Which Method Is Most Successful? A Study on Behalf of the PETRA Consortium

DISCUSSION

The principle aim of this study was to determine the best segmentation method to measure MTV in DLBCL at baseline using automated software. MTV is a robust predictor of patient outcome in DLBCL irrespective of the delineation method, but the absolute values for MTV and the optimal cutoffs to divide good-prognosis groups from poor-prognosis groups differ (11,26). Measurement of MTV in patients with DLBCL takes around 3–6 min per scan, depending on the method, but complex cases can take 10–20 min (11). There is presently no agreement about which method to use; however, there is a consensus that reproducible and rapid automated measurements are needed to explore MTV for prognostic stratification in prospective trials and, ultimately, for clinical application (20). MTV measurement methods can be assessed using simulated and phantom data for which true volumes are known (16), to try to overcome challenges in segmentation of PET images that have limited spatial resolution, causing partial-volume effects, when developing contouring algorithms (31). However, phantoms are not representative of the clinical situation, which includes variations in contrast, heterogeneity, and the shapes and sizes of lesions and patients. Moreover, recent studies suggest that the actual MTVs resulting from using different methods do not affect prognostic performance (10) and, thus, that bias in observed MTV data is clinically less relevant than good reproducibility. Therefore, we chose to rate the success of an automated tool with a fixed color table and SUV scale to delineate visible tumors satisfactorily in patients with DLBCL according to the opinion of experienced observers—a method that represents how these tumors would be assessed in everyday practice (32). We devised a method to rate the success or failure of the Accurate tool a priori. To our knowledge, this is the first report to evaluate the success of an automated method in this way. Furthermore, we considered whether the choice of the best method was influenced by scanning conditions (i.e., uptake time and compliance or noncompliance with standardized recommendations (21)) and whether patients experienced later progression, or did not, using a case-control design.

There were significant differences in performance for automated measurement of MTV with different segmentation methods. The best method for successful segmentation was SUV4.0; however, MV2 also performed well. A majority-vote method was included because no single method can be expected to perform optimally for every patient or every lesion, but a majority method is likely to provide a good approximation of tumor delineation that will be close to the best-performing method in most patients. Consensus approaches using a majority vote and the Simultaneous Truth and Performance Level Estimation (STAPLE) algorithm are being explored in radiotherapy planning (30) and performed better than segmentations based on a single algorithm in an imaging analysis challenge to contour a large dataset comprising simulated, phantom, and clinical images of solitary tumors (31).

Automatic delineation using SUV4.0 was successful in over three quarters of patients using single clicks to remove any uptake present in the brain, urinary tract, and heart. In the remainder, an automated process was not completely reliable, generally because lesions were adjacent to areas with high physiologic uptake requiring user interaction. For some cases, the volume was altered substantially during editing, but for most, only 1 or 2 additional steps were needed to obtain a reasonable estimate of MTV. The other segmentation methods did not perform as well. Failure of SUV2.5 was mostly due to the complex editing needed to remove physiologic uptake, for a reasonable approximation of tumor burden. For SUV4.0, this situation was encountered in only 1 patient, because spillover of counts into other tissues was less common and, when it did occur, the overlap between tumor and physiologic uptake was less extensive and required 1 or 2 editing steps. Failure of the 41% and A50P methods was usually due to underestimation of tumor because of heterogeneity, with more than 50% of the tumor having uptake less than the chosen threshold. The 41% method performed slightly worse in patients scanned at 90 min, probably because uptake in tumor rises over time and, thus, for heterogeneous tumors with areas of high uptake, fewer voxels would be included in 41% of the SUV_max, compounding the problem of underestimation. The influence of uptake time could possibly explain the preference of different groups for particular methods in reported studies (20). Other scanning conditions did not influence how well methods performed.

The absolute values for MTV varied among methods, as previously reported (20,26,33). Across the whole range of volumes, a positive bias was seen for SUV2.5 and a negative bias for the other methods, in comparison with SUV4.0 and MV2, which performed similarly to each other. MV2 selects voxels included in at least 2 segmentation methods (SUV2.5, SUV4.0, A50, or 41%) and commonly included voxels delineated using SUV4.0 and the next method that came closest to delineating a similar volume, usually SUV2.5. MV3 selects voxels included in at least 3 of the segmentation methods, and it segmented volumes that were similar to those of the 41% method, which was most likely to delineate more of the same voxels as 2 or more of the other methods. MV2 performed similarly to SUV4.0 but required delineation using more than 1 segmentation method. This process is fully automated within Accurate but could be less easy to implement across software platforms than a single SUV4.0 threshold method. Clinically available software currently can measure MTV using SUV4.0, although Accurate has additional features to enable the user to quickly review the maximum-intensity-projection image and add missed lesions or remove physiologic uptake with a single click, speeding the segmentation process. Correlation among all the other thresholds was moderate or good but not sufficient to allow the MTV from different segmentation methods to be used interchangeably by a simple linear transformation.

All methods had similar discriminative power, as previously reported (11). Because we used a case-control design, with an oversampling of patients with progression, we cannot express results as positive and negative predictive values to decide which method is best for clinical use. Yet, our results seem to confirm previous findings (9,11), although the receiver-operating-characteristic curves demonstrated lower discriminative power than was found by Ilyas et al. (11), possibly because the latter used cases from a single institution and performed manual editing in most. Nonetheless, the fact that all methods predicted prognosis equally well suggests that selection of the best method can be based on success rate, ease of use, and time or user interaction to obtain total tumor burden.

Limitations are that the research software developed by our group is not yet widely available but the software has been designed as a tool that could be implemented across software platforms after discussions with manufacturers. Only classic segmentation methods published in lymphoma datasets were assessed, whereas more sophisticated methods may give more reliable estimates of tumor volume (31). However, as we realized that a single method may not be able to reliably delineate all lesions for all patients, we included majority-vote–based approaches that have been shown to outperform single-method segmentations. Yet, we observed that at baseline, MTV measurements in DLBCL patients were equally feasible using MV2 and SUV4. MTV was assessed by 2 experienced observers with high concordance, mirroring the high reproducibility reported by others (11,25). However, the observers were aware of which method was being applied; masking was not possible because the delineation method was often obvious. We have assumed that the cases evaluated are representative, but the case-control design meant that 50% of patients progressed; the rate of progression would be lower in the clinic. This overrepresentation is likely to accentuate the challenges of measuring MTV, as patients who progress later would be expected to have a higher disease burden and more extranodal disease than the average clinical population.