Abstract
The effect of the procedural variability in image acquisition on the quantitative assessment of bone scan is unknown. Here, we have developed and performed preanalytical studies to assess the impact of the variability in scanning speed and in vendor-specific γ-camera on reproducibility and accuracy of the automated bone scan index (BSI). Methods: Two separate preanalytical studies were performed: a patient study and a simulation study. In the patient study, to evaluate the effect on BSI reproducibility, repeated bone scans were prospectively obtained from metastatic prostate cancer patients enrolled in 3 groups (Grp). In Grp1, the repeated scan speed and the γ-camera vendor were the same as that of the original scan. In Grp2, the repeated scan was twice the speed of the original scan. In Grp3, the repeated scan used a different γ-camera vendor than that used in the original scan. In the simulation study, to evaluate the effect on BSI accuracy, bone scans of a virtual phantom with predefined skeletal tumor burden (phantom-BSI) were simulated against the range of image counts (0.2, 0.5, 1.0, and 1.5 million) and separately against the resolution settings of the γ-cameras. The automated BSI was measured with a computer-automated platform. Reproducibility was measured as the absolute difference between the repeated BSI values, and accuracy was measured as the absolute difference between the observed BSI and the phantom-BSI values. Descriptive statistics were used to compare the generated data. Results: In the patient study, 75 patients, 25 in each group, were enrolled. The reproducibility of Grp2 (mean ± SD, 0.35 ± 0.59) was observed to be significantly lower than that of Grp1 (mean ± SD, 0.10 ± 0.13; P < 0.0001) and that of Grp3 (mean ± SD, 0.09 ± 0.10; P < 0.0001). However, no significant difference was observed between the reproducibility of Grp3 and Grp1 (P = 0.388). In the simulation study, the accuracy at 0.5 million counts (mean ± SD, 0.57 ± 0.38) and at 0.2 million counts (mean ± SD, 4.67 ± 0.85) was significantly lower than that observed at 1.5 million counts (mean ± SD, 0.20 ± 0.26; P < 0.0001). No significant difference was observed in the accuracy data of the simulation study with vendor-specific γ-cameras (P = 0.266). Conclusion: In this study, we observed that the automated BSI accuracy and reproducibility were dependent on scanning speed but not on the vendor-specific γ-cameras. Prospective BSI studies should standardize scanning speed of bone scans to obtain image counts at or above 1.5 million.
More than 80% of metastatic prostate cancer patients have skeletal metastasis (1). In clinical routine, bone scan is the most common imaging modality to monitor bone metastasis. In therapeutic clinical trials, bone scan is the standard imaging modality to assess eligibility and progression for patients with metastatic prostate cancer. However, the clinical utility of bone scan is limited due to its subjective assessment, which is susceptible to the interobserver variability, and in its inability to assess response. The development and qualification of a quantitative assessment of bone scan can standardize the assessment of on-treatment change that is clinically relevant.
The bone scan index (BSI) is a fully quantitative assessment of bone scan, and it represents the total skeletal tumor burden as percentage skeletal mass (2). Despite showing clinical utility as a prognostic and response biomarker (3,4), the manual BSI has not been adopted in routine clinical practice because of the laborious process of manual calculations, which on average takes 20–40 min per scan. To overcome the limitations of manual assessment, the BSI methodology was automated with a computerized image analysis system that used an artificial neural network to detect and classify metastatic hotspots in bone scans (5). With the computer automation, the time of detecting metastatic lesion and calculating the BSI was reduced to 3–5 s per scan. The recent analytic validation study of automated BSI demonstrated that the platform has consistent accuracy in low- to high-burden disease and that with minimal manual supervision the automated BSI can minimize the interoperator variability (6,7). The analytic validation study was a vital step forward in the continual effort to validate automated BSI as an imaging biomarker in patients with metastatic prostate cancer.
However, the consistent analytic performance of the automated BSI platform is still predisposed to the procedural variability of bone scan. In bone scan image acquisition, the procedural variabilities are largely reflected in the use of vendor-specific γ-camera, in the scanning speed, and in the time interval between the tracer injection and scanning. The impact of variability in these preanalytical factors on the quantitative assessment of bone scan via automated BSI is largely unknown. A recent Japanese study found that the BSI assessment of bone scan was dependent on the time between the tracer administration and scanning and recommended that the time interval should be near constant in a clinical follow-up study (8). Qualification of BSI as an imaging biomarker hinges on the identification and subsequent standardization of such procedural factors that significantly affect the performance of the automated BSI assessment.
Here, in a multiinstitutional collaboration, we have developed and performed preanalytical studies to assess the impact of the variability in the vendor-specific γ-camera and in the scanning speed on the reproducibility and accuracy of the automated BSI. The study is part of the continual effort to analytically and clinically validate the automated BSI as an imaging biomarker to assess clinically relevant change in bone scan.
MATERIALS AND METHODS
Preanalytic Study Design
Two distinct observational studies were developed and performed to evaluate the impact on the reproducibility and accuracy of the automated BSI due to the variability in vendor-specific γ-camera and in scanning speed. The predefined objectives and design of each study are summarized in Table 1. The local ethical boards of Sahlgrenska University Hospital, Gothenburg, Sweden, and of Yamaguchi University Hospital, Yamaguchi, Japan, approved the study, and individual patient consent was obtained from all subjects to perform the repeated bone scans.
Objective, Study Design, and Analysis of Preanalytic Study
Patient Study
The objective of the patient bone scan study was to compare the impact on automated BSI reproducibility due to variability in vendor-specific γ-camera and in scanning speed. Prostate cancer patients with skeletal metastasis, who were referred to routine clinical follow-up bone scan, were enrolled for repeated bone scans at Sahlgrenska University Hospital and at Yamaguchi University Hospital. Both bone scans, the original (first scan) and the repeated (second scan), were obtained sequentially, 3 h after a single injection of 600 MBq of 99mTc-methylene diphosphonate. A low-energy high-resolution collimator was kept constant for all scans, and the energy discrimination was also kept constant at a 10% window centered on the 140-keV photo peak.
The first group (Grp1) of patients served as the control of the patient study design. In Grp1, the repeated scan (second scan) had the same γ-camera and the same scanning speed as that of the original scan (first scan), which maintained the image counts at or above 1.5 million. In the second group (Grp2) of patients, the repeated scan was twice the speed of the original scan, with the same scanning camera as that used in the original. In third group (Grp3) of patients, the repeated scan replaced the Siemens Symbia (used in the original scan) with Philips BrightView, and the speed was the same as that of the original scan, which maintained the image counts at or above 1.5 million.
Simulation Study
The objective of the simulation study was to use an analytic standard to compare the impact on the accuracy of automated BSI quantification due to the variability in scanning speed and its corresponding image counts and due to the variability of vendor-specific γ-camera. The simulations were performed using the SIMIND Monte Carlo program (9) and the Extended Cardiac-Torso (XCAT) phantom (10), representing a standard male. Randomly distributed focal lesions were inserted in the XCAT phantom skeleton by a MATLAB (The MathWorks) script. The tumors were confined to the skeleton volume. A restriction was set so that no tumor was placed below the femur or below the humerus. Fifty different phantoms with predefined tumor burdens in the range from 0.10 to 13.0 BSI were created. The predefined tumor burden (phantom-BSI) was calculated as described in the original BSI study (2).
Variable Image Count Simulations
The SIMIND Monte Carlo program was set to mimic a Siemens Symbia system. A 9.2% energy resolution at 140 keV, a 256 × 1,024 image matrix with a 2.4-mm pixel size, and a low-energy, high-resolution collimator setting were used for the simulation. For each of the 50 phantoms, Poisson noise was added after the simulations, corresponding to measurements with counts at 0.2, 0.5, 1.0, and 1.5 million, in the anterior image. Anterior and posterior whole-body images were simulated for every phantom.
Vendor-Specific γ-Camera Simulation
Each of the 50 phantoms was simulated with 2 different γ-camera settings, corresponding to a Philips BrightView camera and a Siemens Symbia camera. A 256 × 1,024 image matrix with a 2.4-mm pixel size and a vendor-specific low-energy high-resolution collimator were used for the simulations. The specifics of the 2 simulated γ-cameras are summarized in Table 2.
Specific Simulation Details of 2 Respective γ-Cameras in Simulation Study
For each simulation, the activity concentration in the bone, kidneys, bone marrow, and tumors was set to 18, 9, 2.5, and 72, respectively, in relation to the remainder of the body. To mimic real measurements, all the simulations were performed with sufficient histories to avoid Monte Carlo noise.
Automated BSI Analysis
Commercially available EXINIboneBSI version 2, developed by EXINI Diagnostics AB, was used to analyze the bone scans and to generate automated BSI. The methodology of the automated platform has been described in detail in a previous study (11). In summary, the different anatomic regions of the skeleton are segmented, followed by detection and classification of the abnormal hotspots as metastatic lesions. The mass fraction of the skeleton for each metastatic hotspot is calculated, and the BSI is calculated as the sum of all such fractions. Manual supervision and the manufacturer’s BSI assessment instructions were used to generate the automated BSI from bone scan images of patient and phantom studies.
Statistical Analysis
In the patient study, BSI reproducibility data were calculated as the absolute value of the difference between the observed automated BSI values of the 2 repeated scans in all the 3 predefined groups. The interquartile range (IQR), mean, median, SEM, and SD of the absolute BSI differences were calculated in each group. Given the moderate scale of the patient study with an independent measure design, the nonparametric Mann–Whitney U test analysis was used to compare the reproducibility data that were observed in each predefined group. In the simulation study, accuracy was assessed as the absolute value of the difference between the observed automated BSI and the predefined phantom-BSI. The IQR, mean, median, SEM, and SD of the absolute BSI difference were calculated for each of the simulation studies. Because the same 50 phantoms were subjected to variable conditions of the simulation study, the nonparametric Wilcoxon signed-rank test was used to compare the accuracy data.
As a preanalytical observational study, no prior assumptions were made for performance of the automated BSI to render power calculations. A statistical significance level of P less than 0.05 (2-tailed) was applied in each test. All statistical analyses were performed using R software (version 3.1.2).
RESULTS
Patient Study
Seventy-five patients with skeletal metastasis (BSI: mean, 0.61; IQR, 0.20–3.15) were enrolled in the study. Twenty-five patients with skeletal metastasis were enrolled in each of the predefined groups. The average scanning time for the repeated scans were observed to be 20 min (IQR, 19–21 min), 12 min (IQR, 10–13 min), and 27 min (IQR, 24–29 min) for Grp1, Grp2, and Grp3, respectively. An illustrative example of Grp1 and Grp2 with the corresponding image counts and automated BSI scores is shown in Figure 1. The repeated bone scan in Grp2, which was obtained with twice the speed of the original scan, corresponded to an approximate 50% reduction of image counts (mean ± SD, 0.81 million; IQR, 0.64–1.02 million), compared with that of the original scan (mean ± SD, 1.70 million; IQR, 1.52–2.04 million). BSI reproducibility data for each predefined groups are detailed in Figure 2 and Table 3. The BSI reproducibility data of Grp2 (mean ± SD, 0.35 ± 0.59) was found to be significantly different from that of Grp1 (mean ± SD, 0.10 ± 0.13) and that of Grp3 (mean ± SD, 0.09 ± 0.10; P < 0.0001). No significant difference was observed in the reproducibility data in Grp3, with repeated bone scan using a different γ-camera, when compared with the reproducibility data of patients enrolled in Grp1 (P = 0.388).
Anterior and posterior (A/P) image count and associated automated BSI values in repeated bone scans of patients with same scanning speeds (Grp1) (A) and in repeated bone scans of patients with twice scanning speed of original scan (Grp2) (B). Doubling of scanning speed results in image count that is approximately half that of original scan. Lesions detected and classified as metastatic by automated platform, for BSI calculation, are highlighted in red.
Box plots representing reproducibility data in 3 predefined patient study groups.
Reproducibility Data in 3 Predefined Patient Study Groups
Simulation Study
An illustration of the bone scan simulation at the respective image counts representing varying scanning speed and the observed automated BSI values is shown in Figure 3. The automated BSI accuracy data for each predefined simulation study are detailed in Tables 4 and 5. The accuracy data of the bone scan simulation at 0.2 million image counts (mean ± SD, 4.67 ± 0.85) and at 0.5 million image counts (mean ± SD, 0.57 ± 0.38) were observed to be significantly different from that observed in the bone scan simulation at 1.5 million image counts (mean ± SD, 0.20 ± 0.26; P = <0.0001; Fig. 4). There was no significant difference observed in accuracy data of the bone scan simulation with vendor-specific γ-camera (P = 0.266).
Respective automated BSI reads of simulated bone scans at 0.2, 0.5, 1.0, and 1.5 million image counts with predefined tumor burden corresponding to phantom-BSI of 3.0.
Accuracy Data in Simulation Study at Respective Image Counts
Accuracy Data in Simulation-Study with Vendor-Specific γ-Camera Settings
Box plot representing accuracy data under respective image count simulation.
DISCUSSION
Despite the prolific growth in the availability and use of novel imaging agents within nuclear medicine, there is no standard imaging biomarker in prostate cancer to monitor clinically relevant change in metastatic disease. An imaging biomarker, or the measurement index, that reliably quantifies the specific tracer activity would also help realize the true clinical potential of the imaging modality. However, the qualification of the imaging biomarker hinges on the preanalytical procedural factors of the imaging modality that can significantly affect the performance characteristics of the intended biomarker. Therefore, the need to clinically validate an imaging biomarker, in turn, spurs the need to standardize the procedures of the imaging modality.
In bone scan, the subjective or semiquantitative assessment of on-treatment change is susceptible to interpreter-dependent variability (12). Multiinstitutional clinical trials have attempted to circumvented the issue of interpreter-dependent variability by centralizing the assessment of change in bone scan. However, the assessment of the central interpreter might be limited in comparison to that of the local interpreter who has the access to the complete medical history of the patient and is aware of the nonmalignant or degenerative tracer activity. Additionally, the centralized visual assessment of a bone scan, in a multisite study, does not account for the deviations from the bone scan procedure guidelines. Recently, in the analytic validation study of automated BSI, we have demonstrated that with minimal manual supervision the automated BSI can minimize the interoperator variability in assessing change in bone scans (7). In this preanalytical study, we have identified scanning speed and the resulting image counts as a procedural factor in bone scan image acquisition that can affect the reproducibility and accuracy of automated BSI assessment and therefore needs subsequent standardization for prospective clinical studies.
The concept of preanalytical variability is often related to the handling and processing of blood-based biomarkers and its standardization plays a significant role in qualification of such biomarkers (13). In comparison, the investigation of preanalytical variabilities of an imaging modality, in the context of generating a consistent quantitative imaging biomarker, is limited. The Society of Nuclear Medicine and Molecular Imaging and European Association of Nuclear Medicine identify 3 essential guidelines for image acquisition of whole-body bone scans: 3–5 h between tracer injection and scanning, the scanning speed corresponding to image counts of 1.5 million, and a vendor-specific high-resolution collimator setting of the γ-camera (14,15). In multiinstitutional and multi–time point clinical follow-up studies, the procedural variability that results in deviation from these technical guidelines is inevitable (16). Effect of variation or deviation from the procedural guideline on the automated BSI assessment could be consequential to its clinical relevance.
A recent study found that the automated BSI was dependent on the time between the tracer administration and scanning. BSI values obtained at 2, 4, and 6 h after injection with 99mTc-methylene diphosphonate were significantly different from each other. Therefore, the study recommended that time interval should be near constant in a clinical follow-up study (8). The present study is part of the continual effort to identify the procedural factors in image acquisition that significantly influence the BSI quantification of bone and therefore need standardization in prospective multiinstitutional clinical studies.
Our study incorporated bone scans from both patients and simulation studies. As a novel approach, we have used the existing simulation technology to validate automated BSI against the preanalytical variability of a bone scan procedure. The XCAT phantom, with predefined tumor burden (phantom-BSI), and the SIMIND Monte Carlo simulation represent a true analytic standard, which is the known ground truth. In this study, we have taken advantage of the ability of the SIMIND Monte Carlo program to emulate a γ-camera setting for the desired preanalytical variable conditions: simulation under variable image counts and under vendor-specific γ-camera resolution.
Despite its advantages as an analytic standard, the simulated bone scan does not represent the real clinical scenario and patient-dependent variabilities. Therefore, our study design also incorporated patients with skeletal metastasis to assess the impact of the preanalytical procedural variabilities of the bone scan image acquisition on the reproducibility of automated BSI. Additionally, the use of the patient and simulation bone scans allows for independent confirmation of the findings related to the performance of automated BSI.
Both patient and simulation studies demonstrated that the accuracy and reproducibility of the automated BSI assessment were dependent on the image counts resulting from the scanning speed. The bone scans from simulation and patient studies demonstrated increased automated BSI values at lower image counts that resulted from faster scanning speed (Figs. 1 and 3). With lower count density, the false-positive detection and classification of artifacts as metastatic lesion increased and led to higher automated BSI values. In the patient study, the reproducibility data in Grp2, in which the repeated scan speed was increased to twice the speed of the original scan and resulted in a 50% reduction of image count density, were observed to be significantly different from that of the control Grp1, in which the scanning speed of the repeated bone scan was maintained to obtain image counts at or above 1.5 million. The finding is concordant to the earlier studies that have shown that the proficiency of subjective assessment of bone scans varies with the total number of counts acquired (17). Higher-speed low-count images might be sufficient for a follow-up staging of a known metastatic patient who is unable to remain still for a longer time. However, such images cannot be relied on to generate consistent and reproducible automated BSI values. The results of the simulation study independently verified the findings of the patient study, as the automated BSI values at lower image count simulations were significantly higher than the true BSI (phantom-BSI) values (Fig. 3). As a result, the accuracy data of the simulation bone scan at 0.2 million counts and 0.5 million counts were significantly different from those observed at 1.5 million counts. Additionally, the findings of the simulation study demonstrated that the accuracy data at 1.0 million image counts were not significantly different from those observed at 1.5 million image counts. The finding pointed toward a possible threshold for BSI accuracy at image resolution corresponding to 1.0 million image counts, below which the automated BSI accuracy is significantly variable and therefore cannot be reliable. The observation needs further validation in patient studies.
In a SPECT γ-camera, the collimator and the detector, consisting of the crystal and the photomultiplier tubes, are the 2 major components. The bone scan procedure is routinely performed in the low-energy high-resolution collimator setting (15). However, the vendor-specific collimator design, the crystal, and the photomultiplier tubes also determine the resolution and sensitivity in bone scan. The availability of various γ-cameras, across sites and sometimes within the same institution, creates scenarios in which the follow-up treatment bone scans are performed using different γ-cameras. No data are available on the impact of variability due to the use of various vendor-specific γ-cameras on the quantitative assessment of bone scan images. The findings in both the patient and the simulation studies indicated that the BSI quantification was not affected due to the change in vendor-specific γ-camera. The use of Philips BrightView or Siemens Symbia γ-cameras, in their routine clinical setting of vendor-specific low-energy high-resolution collimator, did not appear to significantly affect the reproducibility or accuracy data of automated BSI.
Our study was limited in its scope of assessing the effect of the procedural variabilities in the technical aspect of image acquisition on quantification of bone scan via automated BSI. The potential effect of procedural variabilities in the radiopharmaceutical preparation and the patient-related factors have not been addressed here. Additionally, our study was restricted with the availability of only 2 vendor-specific γ-cameras. However, in this study, we have developed a preanalytical study design that can be expanded to identify and subsequently standardize the procedural factors that can significantly influence the performance of the intended imaging biomarker. The study design with patient and simulation bone scan has the advantage of assessing the effect on both the reproducibility and the accuracy, which are 2 separate performance parameters of automated BSI that can be influenced independently. The data of BSI performance against the preanalytical variability of bone scan are vital for routine clinical setting and for prospective multiinstitutional clinical studies that are aimed to qualify automated BSI as an imaging biomarker.
CONCLUSION
In this study, we observed that the BSI accuracy and reproducibility were dependent on scanning speed and not on vendor-specific γ-camera. Prospective clinical studies with automated BSI should adhere to guidelines of Society of Nuclear Medicine and Molecular Imaging and European Association of Nuclear Medicine and ensure a scanning speed that corresponds to 1.5 million image counts or more. The study adds to the empiric evidence toward the standardization of bone scan image acquisition for robust quantitative BSI assessment. The data presented here serve as the analytic foundation for prospective clinical investigations aimed to validate automated BSI as a quantitative imaging biomarker indicative of assessing change in bone scan that is clinically relevant.
DISCLOSURE
The costs of publication of this article were defrayed in part by the payment of page charges. Therefore, and solely to indicate this fact, this article is hereby marked “advertisement” in accordance with 18 USC section 1734. No potential conflict of interest relevant to this article was reported.
Footnotes
Published online Jul. 21, 2016.
- © 2016 by the Society of Nuclear Medicine and Molecular Imaging, Inc.
REFERENCES
- Received for publication April 19, 2016.
- Accepted for publication June 18, 2016.
Jump to section
Related Articles
Cited By...
- Assessing Radiographic Response to 223Ra with an Automated Bone Scan Index in Metastatic Castration-Resistant Prostate Cancer Patients
- Denoising of Scintillation Camera Images Using a Deep Convolutional Neural Network: A Monte Carlo Simulation Approach
- Reply: A Common Mistake in Assessing the Diagnostic Value of a Test: Failure to Account for Statistical and Methodologic Issues
- A Common Mistake in Assessing the Diagnostic Value of a Test: Failure to Account for Statistical and Methodologic Issues