Distinction of Lymphoma from Sarcoidosis on ¹⁸F-FDG PET/CT: Evaluation of Radiomics-Feature–Guided Machine Learning Versus Human Reader Performance

Patients

The study was approved by the Ethics Committee of the University Hospital of Liège. The need for written informed consent was waived because of the retrospective and noninterventional design of the study. We retrospectively collected consecutive ¹⁸F-FDG PET/CT scans obtained at the University Hospital of Liège between April 2010 and February 2020 of patients with HL, DLBCL, or sarcoidosis at initial diagnosis, before any treatment. The diagnosis was based on pathology in all lymphoma cases and in most cases of sarcoidosis. The diagnosis of the remaining sarcoidosis cases was based on clinical evidence and follow-up. Exclusion criteria were radiotracer extravasation, artifacts in pathologic areas, absence of a delineated volume of interest (VOI) after semiautomatic segmentation, and absence of relevant information in the DICOM files. Basic clinical data (age, sex, and weight) were collected from the information obtained routinely on the day of the PET/CT scan. Figure 1 shows the flowchart of the study.

• Download figure
• Open in new tab
• Download powerpoint

FIGURE 1.

Study flowchart.

Imaging

¹⁸F-FDG PET/CT scans were acquired using 2 cross-calibrated PET/CT systems, a GEMINI TF Big Bore and a GEMINI TF 16 (Philips), 66 min on average (range, 58–92 min) after intravenous injection of ¹⁸F-FDG (mean injected activity, 245 MBq, depending linearly on the patient’s weight). The patients fasted for at least 6 h before the injection, and the median glycemia was 92 mg/dL (range, 59–195 mg/dL). A low-dose CT scan (slice thickness, 5 mm; tube voltage, 120 kV; and tube current–time product, 50–80 mAs, depending on the patient’s weight) was performed without injection of intravenous contrast agent, followed by a PET emission scan of 90 s per bed position (50% overlap), extending from the upper thighs to the skull base. All images were acquired and reconstructed according to the EARL (European Association of Nuclear Medicine Research Ltd.) guidelines for both PET/CT systems. Images were reconstructed with standard 4 × 4 × 4 mm voxels (slice thickness, 4 mm) using an iterative list-mode algorithm (blob ordered-subset time-of-flight), and corrections for attenuation, dead time, random events, and scatter events were applied without postreconstruction smoothing.

Lesion Segmentation and Clinical Diagnosis

The entire cohort of anonymized patients was randomly distributed into 4 groups (groups A–D). The scans of each group were assigned to 4 different nuclear medicine physicians (observers A–D), who were unaware of any clinical information or diagnosis and had 6 y (observer A), 3 y (observer B), 15 y (observer C), and 10 y (observer D) of experience. In a first step, from visual interpretation of the PET/CT scans, the physicians attributed a diagnosis to each patient of their cohort. For that purpose, they first chose either sarcoidosis or cancer, and if the latter was selected, they chose HL or DLBCL. This evaluation was based solely on the experience of each physician. No reading guidelines or visual or semiquantitative interpretation criteria were provided to the readers within the framework of the study. For each answer, the physicians indicated their level of confidence (0, possible; 1, probable; or 2, certain).

In the next step, the physicians segmented PET VOIs for their assigned patient population using MIM software, version 7.0.5 (MIM Software Inc.), with the following 4 steps. The first was automatic selection of all regions using an absolute threshold SUV_max of at least 3 within a rectangular VOI manually drawn on the whole body. The second step was automatic exclusion of VOIs smaller than 2 cm³. The third step was manual exclusion of all physiologic VOIs (e.g., brain, heart, and kidneys). The fourth step was manual modification of some pathologic VOIs, that is, removing physiologic activity in continuity with the pathologic VOI but never enlarging the VOI. In the absence of literature references for this combination of diseases, especially considering sarcoidosis, the thresholds of SUV_max and volume were decided after tests were performed on a sample of images with the aim of including as many lesions as possible while limiting the need for manual modifications. A VOI of 20 cm³ was also drawn on the healthy liver.

Radiomics Extraction and Model Elaboration

Two hundred fifteen features were extracted from the segmented PET volumes using the RadiomiX Toolbox (Radiomics SA), coded with Matlab (MathWorks), and aligned with the Imaging Biomarkers Standardization Initiative, with, however, some additional features (a list of all features can be found in the supplemental materials, available at http://jnm.snmjournals.org). We also studied the ratio of the feature values calculated in the tumor and in the liver (tumor-to-liver ratio [TLR]), except for the shape features. For calculation of the texture matrix-based features, the intensities were discretized using 2 different methods according to the recommendations of the Imaging Biomarkers Standardization Initiative: fixed bin numbers of 32 and 64 and fixed bin widths of 0.05, 0.1, 0.2, and 0.5 SUV.

Since each patient could have more than 1 lesion, 2 radiomics approaches were tested. In a first approach (lesion-based), each lesion was considered as 1 observation and the goal was to classify each lesion as, first, belonging to the sarcoidosis or lymphoma class and, second, as belonging to the HL or DLBCL class. In the second approach (patient-based), the radiomics features of each lesion and for each patient were merged using their minimum, maximum, mean, and median values, and clinical data (age, sex, and weight) were added to the radiomics features. Here, the aim was to classify each patient into the sarcoidosis, HL, or DLBCL group.

We also evaluated whether combining different feature selection (FS) approaches and ML classifiers would allow for a radiomics signature to differentiate sarcoidosis from lymphoma patients and to differentiate HL from DLBCL. For that purpose, we tested a different set of models, which differed in feature type, that is, original radiomics or TLR radiomics; FS and ML classifier method; and the effect of adding clinical data before FS. Seven different FS methods were tested: accuracy decrease obtained from the embedded FS of the random forest (RF) classifier; Gini impurity decrease obtained from the embedded FS of the RF classifier; forward FS using the minimum redundancy maximum relevance (MRMR) method with Pearson correlation; backward FS using MRMR with Pearson correlation; forward FS using MRMR with Spearman correlation; backward FS using MRMR with Spearman correlation; and forward MRMR based on the mutual information. We also used 4 ML classifiers: RF, support vector machine with radial kernel, naive Bayes, and a logistic regression (16). The dataset was stratified with the same percentage of classes, avoiding unbalanced data, and was randomly divided into training and test sets (80% and 20%, respectively). We tested different models that differed in the FS, ML, and intensity discretization method and in the number of features, which was between 2 and 20 with intervals of 2. We used 5-fold cross validation in our training data, and we chose the best radiomics signature according to the best mean 5-fold cross validation area under the precision-recall area under the receiver-operating-characteristic curve (AUCpr). For each classifier, the default hyperparameter values were used in their respective R packages. Finally, for each of the different models with distinct selected features, all training data were bootstrapped to derive the corresponding 95% CIs for each performance metric and tested on the independent test set. The number of bootstrap repetitions was set to at least 1,000. Since images came from only 2 scanners (same manufacturer and model, same acquisition protocol), which were also cross-calibrated, we did not consider it necessary to perform data harmonization. As recently suggested by Buvat and Orlhac (17), we performed a T.R.U.E. checklist (Is it true? Is it reproducible? Is it useful? Is it explainable?) to assess the potential impact of our findings.

Statistical Analysis

The homogeneity in age and weight across the A, B, C, and D populations was assessed by the Kruskal–Wallis nonparametric test, whereas χ² test association was performed for sex and the final diagnosis (gold standard). Additionally, the homogeneity in age, sex, and weight across cancer and sarcoidosis patients and between HL and DLBCL patients was also evaluated using the χ² and Kruskal–Wallis tests. Statistical significance was assigned for P values of 0.05 or less.

The diagnostic performance of all observers pooled together and each individual observer against the gold standard was calculated using sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV). In addition, diagnostic performance was evaluated by calculating the percentage of agreement (or accuracy) and the AUC. To calculate the predicted probabilities, we fitted a logistic regression model with the observer’s classification as the predictor. We additionally bootstrapped the data to measure the AUC 95% CIs. The 95% CIs for the sensitivity, specificity, PPV, and NPV were calculated using exact binomial confidence limits.

To test variability among observers, we applied the CI approach in sample size estimation for interobserver agreement with binary outcomes (18). Because of a lack of literature on the established agreement, to calculate the sample size in the case of 4 observers we assigned an expected κ of 0.70, indicating moderate agreement (19), a lower bound of 0.50 with an unknown upper bound, and a significance level of 0.05. With a prevalence of 0.3 for sarcoidosis versus cancer and 0.6 for HL versus DLBCL, a sample size of 27 and 23 was required, respectively. On the basis of this estimation, we randomly selected a subgroup of 34 patients who were subsequently analyzed by all 4 observers to evaluate the interobserver variability. Because of the misclassification for the sarcoidosis versus cancer, only 21 patients remained in the evaluation of interobserver variability in the analysis of HL versus DLBCL. Fleiss κ was used to investigate overall agreement among 4 observers in the classification (for sarcoidosis/cancer and for HL/DLBCL) and intraclass correlation coefficients for degree of certainty. Finally, the Hotelling T² test was used to test the difference in agreement between pairs of observers.

For radiomics, we evaluated the performance of the models using AUC, AUCpr, sensitivity, specificity, PPV, and NPV, with a 0.5 probability threshold on the test set for lesion-based and patient-based approaches. Statistical and ML analyses were performed for the 2 clinical tasks (sarcoidosis/cancer and HL/DLBCL) using R software, version 4.0.3.