Visual Abstract
Abstract
Our purpose was to determine and compare the interobserver variability of 3 clinically frequently used radiotracers targeting the prostate-specific membrane antigen (PSMA), namely 18F-DCFPyL, 18F-PSMA-1007, and 68Ga-PSMA-11, in primary prostate cancer (PCa) staging. Methods: Patients with newly diagnosed PCa in whom PSMA PET/CT was performed for primary staging purposes were retrospectively included. All PSMA PET/CT images were centrally overread within a high-volume PCa center, and original reports (from referring hospitals) were compared with overread reports (from the overreading hospital). To assess the interobserver variability, a Cohen κ analysis was used. To study possible differences in interobserver variability between the 3 applied PSMA radiotracers, multivariate logistic regression analyses were used. Results: In total, 584 patients with newly diagnosed PCa were included in the analysis. 18F-DCFPyL, 18F-PSMA-1007, and 68Ga-PSMA-11 were used in 205 (35.1%), 168 (28.8%), and 211 (36.1%) patients, respectively. The overall agreement (Cohen κ analysis) for locoregional lymph node metastases, distant lymph node metastases, bone metastases, and visceral metastases was 0.86, 0.86, 0.80, and 0.46, respectively. 18F-PSMA-1007 showed a significantly increased interobserver variability regarding bone metastases, compared with 18F-DCFPyL and 68Ga-PSMA-11 (P = 0.001 and 0.03, respectively). Additionally, 18F-PSMA-1007 showed a significantly increased interobserver variability regarding overall agreement and locoregional lymph node metastases, compared with 18F-DCFPyL (P < 0.001 and P = 0.01, respectively). Conclusion: Interobserver variability differs among the 3 clinically frequently used PSMA radiotracers (18F-DCFPyL, 18F-PSMA-1007, and 68Ga-PSMA-11) in patients with newly diagnosed PCa. The agreement in bone metastases is significantly worse for 18F-PSMA-1007, mainly due to nonspecific tracer uptake in osseous structures. On the basis of our findings, PSMA PET/CT scans undertaken with 18F-PSMA-1007 in primary staging should be interpreted carefully, and training on interpreting this specific PSMA radiotracer is strongly advised.
Prostate cancer (PCa) is the second most common malignancy in men worldwide. Imaging has a pivotal role in staging and selection of the appropriate management strategy in men with primary diagnosed PCa. After its clinical introduction in 2011, PET/CT imaging with agents targeting the prostate-specific membrane antigen (PSMA), a transmembrane folate hydrolase on the surface of PCa cells, has shown increasing adoption for use in staging and restaging of PCa (1,2). Compared with conventional imaging using CT and bone scans, PSMA PET/CT has shown superior accuracy; higher sensitivity and specificity, more frequent management changes, fewer equivocal findings, and lower radiation exposure (3).
The interobserver variability, defined as the absence of consensus among nuclear medicine physicians regarding oncologic staging, gives an indication of the reliability and reproducibility of the assessment and is an essential indicator of the clinical value of PSMA PET/CT scans. Interobserver variability is affected by the ability of nuclear medicine physicians to recognize potential false-positive sources of uptake, such as false-positive bone findings, already frequently described with PSMA radiotracers (4–12). Forestalling these false-positive findings and consequently limiting the interobserver variability are dependent on tracer and training and crucial to ensuring high-quality diagnostics.
Currently, several PSMA radiotracers, including 18F-DCFPyL, 18F-PSMA-1007, and 68Ga-PSMA-11, are being used for PET/CT imaging. Even though these PSMA radiotracers are well established, agreement on which tracer is optimal in primary PCa staging is lacking. Few studies have been published regarding the interobserver variability of PET imaging with PSMA radiotracers, and these studies have focused primarily on 68Ga-PSMA-11 (1,13–15). No large studies have compared interobserver variabilities of different PSMA radiotracers for staging purposes. The aim of this study was to determine the interobserver variability in primary PCa staging among the 3 clinically most frequently used PSMA radiotracers (i.e., 18F-DCFPyL, 18F-PSMA-1007, and 68Ga-PSMA-11) and identify the PSMA radiotracer with the least interobserver variability for use in PCa staging.
MATERIALS AND METHODS
Study Design and Patient Population
A retrospective cohort study was performed at The Netherlands Cancer Institute (NCI) on the interobserver variability of PSMA PET/CT scans in patients with newly diagnosed PCa.
Patients who had been referred to the NCI between January 2018 and December 2020, and whose PSMA PET/CT scans were overread (defined as a secondary interpretation) by 1 of 3 nuclear medicine physicians of the NCI, were retrospectively included. Patients were excluded from analysis when staging was performed using tracers other than 18F-DCFPyL, 18F-PSMA-1007, or 68Ga-PSMA-11 and when PSMA PET/CT scans were not performed for primary staging purposes (Fig. 1). The institutional review board approved this retrospective study, and the requirement to obtain informed consent was waived.
Flowchart of study design.
Being a high-volume PCa center, the NCI received patient referrals on a case-by-case basis, either for treatment or treatment advice, from 44 hospitals in The Netherlands (academic or nonacademic and high- or low-volume). Together with the referral letter, DICOM images were securely sent to the NCI using CD-ROMs. After uploading the DICOM images to the electronic system, we destroyed the CD-ROMs. All PSMA PET/CT scans, initially performed, interpreted, and reported in referring hospitals, were overread in the NCI for clinical purposes and in line with the PROMISE criteria (16). Given the clinical purpose of the overreads, data were not anonymized. The assessments of nuclear medicine physicians from both referring hospitals and overreading hospitals were performed in a nonmasked manner. All PSMA PET/CT scans were discussed within a multidisciplinary consultation in the NCI with radiologists, nuclear medicine physicians, urologists, radiation oncologists, and medical oncologists. Overreads were considered the reference to be followed.
Patient characteristics were collected from patient charts, and PSMA PET/CT results were collected from nuclear medicine reports. According to the PROMISE criteria, a scan was reported as positive when the lesion was consistent with or suggestive of being PCa (16). Original reports (from referring hospitals) were compared with overread reports (from the overreading hospital—i.e., NCI) regarding molecular imaging TNM (miTNM) classification. Agreement scoring was based on the miTNM classification irrespective of the number or location of lesions per patient. Overall agreement was defined as complete agreement in miTNM classification irrespective of the number or location of lesions per patient.
PET Imaging and Analysis
All PET images were acquired from mid thigh to skull base or vertex. Most patients selected for PSMA PET/CT had a biopsy Gleason score of at least 4 + 3 = 7 (International Society of Urological Pathology [ISUP] ≥ 3), an initial prostate-specific antigen (PSA) value of at least 20 ng/mL, or clinical or radiologic disease that was at least T3.
Different tracer incubation times and doses were used for different tracers; a median of 60 min (interquartile range [IQR], 60–120 min) after a median dose of 217 MBq (IQR, 205–305 MBq) for 18F-DCFPyL, a median of 90 min (IQR, 89–120 min) after a median dose of 271 MBq (IQR, 231–301 MBq) for 18F-PSMA-1007, and a median of 60 min (IQR, 59–60 min) after a median dose of 137 MBq (IQR, 117–156 MBq) for 68Ga-PSMA-11. PET images were combined with either a low-dose CT scan (120–140 kV, 40–80 mAs) or a diagnostic CT scan (130 kV, 110 mAs) for anatomic correlation and attenuation correction.
Statistical Analysis
Categoric variables were reported as frequency distributions and percentages, and continuous variables were reported as medians with IQR.
First, the characteristics of the 3 different patient populations staged with 1 of the 3 PSMA radiotracers were compared to check for case-mix variation, using a χ2 test for dichotomous and categoric variables or a Mann–Whitney U test for continuous variables. To assess the interobserver variability of the 3 clinically most frequently used PSMA radiotracers, a Cohen κ analysis was performed for each miTNM category. As conventionally classified, κ values of 0–0.20 defined poor agreement; 0.21–0.40, fair agreement; 0.41–0.60, moderate agreement; 0.61–0.80, substantial agreement; and 0.81–1.0, nearly perfect agreement (17). To study possible differences in interobserver variability between the 3 applied PSMA radiotracers, multivariate logistic regression analyses were used, taking into account potential differences in initial PSA level, cT stage, cN stage, and biopsy ISUP grade. For this analysis, the degree of agreement was dichotomized.
A P value of less than 0.05 was considered to indicate statistical significance. All statistical analyses were performed with the statistical package SPSS (version 27; IBM) for MacOS (Apple).
RESULTS
In total, 584 patients staged with PSMA PET/CT for primary, untreated PCa were included in the analysis. Their characteristics are presented in Table 1. 18F-DCFPyL, 18F-PSMA-1007, and 68Ga-PSMA-11 were applied in 205 (35.1%), 168 (28.8%), and 211 (36.1%) of the 584 patients and in 15 (34.1%), 27 (61.4%), and 17 (38.6%) of 44 hospitals, respectively. A comparison among radiotracers showed a case-mix variation in clinical tumor stage (cT stage) and clinical nodal stage (cN stage); 18F-DCFPyL was applied in patients with significantly lower cT and cN stages than was 18F-PSMA-1007 (P = 0.01 and P < 0.001, respectively) or 68Ga-PSMA-11 (P = 0.01 and 0.01, respectively).
Patient Characteristics
Interobserver Variability
According to the local PSMA PET/CT interpretation, 99.0% (578/584) of patients were considered positive, of whom 31.8% (186/584) had locoregional lymph nodes metastases (miN1), 9.1% (53/584) had distant lymph node metastases (miM1a), 12.0% (70/584) had bone metastases (miM1b), and 0.5% (3/584) had visceral metastases (miM1c; lung, liver, and pleura). After overreading all scans, 98.6% (576/584) were considered positive, of which 32.7% (191/584) had miN1, 9.8% (57/584) had miM1a, 10.6% (62/584) had miM1b, and 1.0% (6/584) had miM1c (lung and liver, Table 2).
PSMA PET/CT Results
The overall agreement for all tracers combined for the assessment of a positive scan, miN1 status, miM1a status, miM1b status, and miM1c status was 0.71, 0.86, 0.86, 0.80, and 0.46, respectively (Cohen κ analysis, Table 3). For 18F-DCFPyL, the Cohen κ for a positive scan, miN1 status, miM1a status, and miM1b status was 0.66, 0.89, 0.81, and 0.88, respectively. No κ for miM1c status could be calculated within the 18F-DCFPyL cohort, since no visceral metastases were found by nuclear medicine physicians from referring hospitals. For 18F-PSMA-1007, the Cohen κ for a positive scan, miN1 status, miM1a status, miM1b status, and miM1c status was 0.66, 0.82, 0.79, 0.73, and 0.50, respectively. For 68Ga-PSMA-11, the Cohen κ for a positive scan, miN1 status, miM1a status, miM1b status, and miM1c status was 1.00, 0.86, 0.91, 0.82, and 0.49, respectively.
PSMA PET/CT Interobserver Variability
When comparing the interobserver variability of the 3 clinically frequently used radiotracers, significant differences were found regarding overall agreement, miN status, and miM1b status (Table 4). 18F-PSMA-1007 showed a significantly higher interobserver variability regarding miM1b status than did 18F-DCFPyL or 68Ga-PSMA-11 (P = 0.001 and 0.03, respectively). Additionally, 18F-PSMA-1007 showed a significantly higher interobserver variability regarding overall agreement and miN status than did 18F-DCFPyL (P < 0.001 and P = 0.01, respectively).
PSMA PET/CT Interobserver Variability
Clinical Confirmation
In 29 (5.0%) patients, referred from 18 different academic or nonacademic and high- or low-volume centers, no agreement regarding miM1b status was observed between the original report and the overread report. 18F-PSMA-1007 was used in 17 of these 29 (58.6%), of whom 7 underwent a robot-assisted radical prostatectomy with pelvic lymph node dissection. After surgery, 6 of the 7 patients had undetectable PSA levels, making the presence of bone metastases unlikely. In 1 patient, PSA levels remained detectable (0.12 μg/L). Pathology reports showed a pT3aN1 tumor with 3 of 19 positive locoregional lymph nodes and a clear margin. In this patient, PSMA PET/CT reported 3 locoregional lymph nodes but also a dubious paraaortic lymph node. Consequently, in this patient, the presence of bone metastases remains equivocal.
DISCUSSION
Since the introduction of PSMA PET/CT in clinical practice, various types of PSMA radiotracers have been developed, implemented, and used in parallel. However, these interchangeable applications are prone to bias and may impact the therapeutic management of patients with PCa at initial staging. To the best of our knowledge, this was the first study comparing the interobserver variability of the 3 clinically most frequently used radiotracers in PSMA PET/CT imaging. We observed superior interobserver agreement for the radiotracers 18F-DCFPyL and 68Ga-PSMA-11 compared with 18F-PSMA-1007 in regard to miN status, miM1b status, and overall agreement.
No studies comprising large patient cohorts have been conducted yet on the interobserver variability of 18F-PSMA-1007 and 18F-DCFPyL for use in patients with newly diagnosed PCa. Studies on the interobserver variability of 68Ga-PSMA-11, on the other hand, have been conducted. Basha et al. (13), Derwael et al. (14), Demirci et al. (18), and Gültekin et al. (19) described substantial to nearly perfect agreement for miN, miM1a, miM1b, and miM1c status (κ values of 0.63–0.94). Compared with the aforementioned studies, we found better interobserver agreement for 68Ga-PSMA-11; only nearly perfect agreement was found (κ values of 0.82–1.00). Notably, we found a relatively low Cohen κ for the assessment of a positive scan and miM1c status using PSMA PET/CT imaging, whereas a high interobserver agreement was observed. This is due to a previously described κ paradox: the higher the agreement, the higher the possibility of finding agreement by chance. Since the Cohen κ corrects for chance, a higher agreement can create a smaller Cohen κ (20).
The overall agreement for the clinically used PSMA radiotracers regarding miN1, miM1a, and miM1b status was moderate to high. Accordingly, a nearly perfect interobserver agreement was found for both miN and miM1a status (both κ values of 0.86). This is in contrast to the interobserver agreement of other diagnostic modalities frequently used in PCa staging, such as MRI. Regarding locoregional and distant lymph node staging, Johnston et al. found decreased κ values for N and M1a status using whole-body MRI (κ = 0.79 and 0.68, respectively) compared with our κ values for miN and miM1a status (21). This higher interobserver variability relative to PSMA PET/CT may be explained by the multiple sequences of MRI (dynamic contrast-enhanced/diffusion-weighted/T2-weighted) and the lower specificity of MRI than of PSMA PET/CT for the detection of distant PCa metastases.
This higher interobserver variability of 18F-PSMA-1007 is at least partly caused by nonspecific uptake of this tracer, most frequently seen in osseous structures, as illustrated in Figure 2; increased uptake of 18F-PSMA-1007 is observed, whereas a clear correlation on CT images is missing. Initially, 18F-PSMA-1007 was a promising PSMA radiotracer because of its hepatobiliary clearance. This is different from 18F-DCFPyL and 68Ga-PSMA-11, since both tracers are excreted in the urine, resulting in physiologic biodistribution in the ureters and bladder. Because of the absence of urinary excretion in 18F-PSMA-1007, detection of local disease or recurrent disease might be improved (22,23). However, increasing evidence points to a major disadvantage of using 18F-PSMA-1007 in primary staging. Significantly higher numbers of lesions with increased tracer uptake, attributed to a benign (i.e., fibrous dysplasia, posttraumatic, or degenerative change) or physiologic origin, are found for 18F-PSMA-1007 (24–28). Although nonspecific skeletal uptake is described more often in the literature with regard to 18F-PSMA-1007 (27), it is also seen in 18F-DCFPyL and 68Ga-PSMA-11 (4–9). The exact mechanism of this nonspecific skeletal 18F-PSMA-1007 uptake is yet unknown. However, it is hypothesized that the higher affinity of 18F-PSMA-1007 for the PSMA receptor, which was shown in preclinical studies, may result in a higher signal from benign skeletal lesions (25).
Comparison of 18F-DCFPyL (A and C) and 18F-PSMA-1007 (B and D) in single patient. 18F-DCFPyL images were acquired 48 d after 18F-PSMA-1007 PET/CT scan. No suspected skeletal lesions were found using 18F-DCFPyL, whereas suspected skeletal lesions were found in os ischium and acetabulum dome using 18F-PSMA-1007 (B and D, respectively). After robot-assisted radical prostatectomy with extended pelvic lymph node dissection, pT2N0 ISUP 3 was found. Postoperative PSA levels were undetectable, making presence of bone metastases unlikely and indicating presence of nonspecific uptake of 18F-PSMA-1007 in bone.
Our study was not devoid of limitations. First, it was a retrospective study in which the presence of selection bias cannot be ruled out. Although we performed multivariate logistic regression analyses, patients assessed and staged by 18F-PSMA-1007 could have had a significantly higher interobserver variability in miM1b status, due to overall more aggressive disease features. Second, we conducted a retrospective cohort study in which overreads were conducted as part of a second opinion or treatment advice. The results may be affected by this selection, since hospitals may refer patients with doubtful PSMA PET/CT scans more frequently. In addition, assessments were performed in a nonmasked manner by only 2 nuclear medicine physicians. In contrast to a multiple-reader method, the use of only 2 nuclear medicine physicians could have affected the interobserver variability. Another important limitation of our study was the lack of uniform systematic interpretation criteria. During the original report in referring hospitals, different interpretation criteria were used. The presence and clinical use of different interpretation criteria may impact the interobserver variability since each criterion focuses on slightly different aspects of PSMA PET/CT scans. However, recent introduction of the E-PSMA, the European Association of Nuclear Medicine standardized reporting guidelines for PSMA PET, will enable harmonization of diagnostic interpretation criteria by combining the PSMA visual with the quantitative and semiquantitative expression (23). This combination will facilitate data reproducibility and thus support therapeutic management decisions. Lastly, not only the PSMA radiotracer but also the nuclear medicine physician may impact interobserver variability. Especially in questionable situations, in which the presence of metastases is doubted, experience is important. One could argue that less experienced nuclear medicine physicians are more likely to misinterpret PSMA PET/CT scans (i.e., misinterpret the 18F-PSMA-1007 uptake in benign [skeletal] lesions as being compatible with bone metastases). Considering the fact that 18F-PSMA-1007 has been introduced more recently, experience in reporting PSMA PET/CT scans made with this tracer is likely less than for 18F-DCFPyL or 68Ga-PSMA-11. Therefore, lack of experience with this tracer cannot be excluded as a cause of the higher interobserver variability.
Unlike other studies on the interobserver variability of PSMA PET/CT scans, our study was based on a considerably larger cohort. In addition, patients were referred from different hospitals throughout The Netherlands, and overreads were conducted by several experienced nuclear medicine physicians. The interobserver variability therefore cannot be attributed to individual interpretations.
CONCLUSION
Interobserver agreement on PSMA PET/CT for use in PCa staging was moderate to high. However, agreement on assessment of PSMA PET/CT imaging by 2 nuclear medicine physicians regarding staging of newly diagnosed PCa differs among 3 frequently used PSMA radiotracers (18F-DCFPyL, 18F-PSMA-1007, and 68Ga-PSMA-11). Agreement on miM1b status is significantly worse for 18F-PSMA-1007 than for 18F-DCFPyL or 68Ga-PSMA-11, mainly because of nonspecific tracer uptake in osseous structures. On the basis of our findings, bone lesions seen on PSMA PET/CT scans with 18F-PSMA-1007 in primary staging should be interpreted carefully, and training—to gain experience in interpreting this specific PSMA radiotracer—is strongly advised.
DISCLOSURE
No potential conflict of interest relevant to this article was reported.
KEY POINTS
QUESTION: Does the interobserver variability differ among the 3 clinically frequently used PSMA radiotracers (18F-DCFPyL, 18F-PSMA-1007, and 68Ga-PSMA-11) in primary-PCa staging?
PERTINENT FINDINGS: In a retrospective analysis with 584 patients staged with PSMA PET/CT for primary untreated PCa, a significantly higher interobserver variability regarding assessment of bone metastasis was observed in patients assessed and staged by 18F-PSMA-1007.
IMPLICATIONS FOR PATIENT CARE: Bone lesions seen on PSMA PET/CT scans with 18F-PSMA-1007 in primary staging should be interpreted carefully, and training—to gain experience in interpreting this specific PSMA radiotracer—is strongly advised.
ACKNOWLEDGMENT
We thank Drs. Zing J. Cheung and Marcel P.M. Stokkel for overreading the PSMA PET/CT scans used for this retrospective study.
Footnotes
Published online Aug. 25, 2022.
- © 2022 by the Society of Nuclear Medicine and Molecular Imaging.
REFERENCES
- Received for publication August 30, 2021.
- Accepted for publication February 10, 2022.
