Reply: A Common Mistake in Assessing the Diagnostic Value of a Test: Failure to Account for Statistical and Methodologic Issues

REPLY: Thank you for your interest and comments regarding our work (1). We agree that reproducibility/reliability and accuracy/validity are two different methodologic issues and should be evaluated using appropriate tests for each. Further, we also concur that for quantitative variables, the intraclass correlation coefficient should be used to test reliability and the interclass correlation coefficient (Pearson r) should be used to test validity. Since the automated bone scan index (BSI) is a quantitative parameter, the comments on qualitative assessments are inapplicable.

Our analytic validation study demonstrated the validity of the automated BSI with a Pearson r of 0.99 (P < 0.0001) and with associated parameters of linear regression (slope, 0.80 [95% confidence interval, 0.78–0.83]; intercept, 0.38 [95% confidence interval, 0.25–0.51]) (2). In this study, Cohen κ-agreement and other well-known standard tests for imaging biomarkers (3) were also used to evaluate the reliability of the automated BSI platform. These tests demonstrated a coefficient of variation (<20%) with a consistent linearity—satisfying the homoscedasticity and Shapiro–Wilken test—from low-burden to high-burden disease. The Bland–Altman plot of absolute difference in BSI readings of repeated scans revealed the analytic noise with an SD of 0.15. The study concluded that in the clinical setting, the assessment of change in BSI above the noise threshold (2 SDs, or 0.30) had high interobserver agreement (Cohen κ = 0.96).

Despite the analytic validation of BSI, variation in the imaging procedure can significantly affect the analytic performance of the imaging biomarker. Therefore, in a subsequent publication, we analyzed the effect of variability in scanning speed and γ-camera settings on the BSI assessment (1). The objective of our preanalytic study was not to evaluate the performance characteristics of BSI but to use appropriate statistical tests to evaluate the effect of procedural variability on the analytically validated BSI values. The Wilcoxon/Mann–Whitney test (4) demonstrated that the noise in the BSI value was significantly higher because of variability in scanning speed in comparison with the known analytic gold standard (accuracy performance) and with repeated measurement of test–retest patient bone scans (reproducibility performance). The P value in our study, from the Wilcoxon/Mann–Whitney test, did not imply clinical significance but referred to the statistical significance of the degree of systematic difference between the two BSI readings of the repeated bone scans that were performed in a specific study design to assess the effect of procedural variability on the known performance characteristics of BSI.

In conclusion, the statistical methodology must be reviewed in the context of the study design and its objective. We have performed a comprehensive assessment to demonstrate the reliability and validity of BSI, and with the preanalytic study, we demonstrated that the BSI performance characteristics were dependent on scanning speed. The study added empiric evidence toward the standardization of bone scan acquisitions for robust quantitative BSI assessment in multiinstitutional studies. Together, the analytic and preanalytic studies served as the foundation for prospective clinical investigations aimed at validating automated BSI as a quantitative imaging biomarker.

• © 2017 by the Society of Nuclear Medicine and Molecular Imaging.