A Common Mistake in Assessing the Diagnostic Value of a Test: Failure to Account for Statistical and Methodologic Issues

TO THE EDITOR: I was interested to read the paper by Anand et al. in the December 2016 edition of The Journal of Nuclear Medicine (1). The purpose of the authors was to assess the impact of variability in scanning speed and in vendor-specific γ-camera settings on the reproducibility and accuracy of the automated bone scan index (BSI) (1). They measured reproducibility as the absolute difference between repeated BSI values, and they measured accuracy as the absolute difference between observed BSI values and phantom BSI values. Descriptive statistics were used to compare the generated data.

Reproducibility (reliability) and accuracy (validity), as two completely different methodologic issues, should be assessed using appropriate tests. It is crucial to be aware that, regarding reliability, one should use the intraclass correlation coefficient for quantitative variables and the weighted κ-test for qualitative variables. However, regarding validity, one should use the interclass correlation coefficient (Pearson r) for quantitative variables whereas the most appropriate tests for qualitative variables may include sensitivity, specificity, positive and negative predictive value, positive and negative likelihood ratio, diagnostic accuracy, and odds ratio. Moreover, in analyzing reliability, one should apply an individual-based approach using single-measure intraclass correlation coefficient agreement because applying a global-average approach (absolute difference) can be misleading. A test may indicate high validity, yet there may be no reliability at all (2–8).

Anand et al. enrolled 25 patients in each of 3 groups and observed a significantly lower reproducibility for group 2 (mean ± SD, 0.35 ± 0.59) than for group 1 (0.10 ± 0.13; P < 0.0001) or group 3 (0.09 ± 0.10; P < 0.0001). However, no significant difference in reproducibility was observed between group 3 and group 1 (P = 0.388) (1). Statistical significance and clinical importance are two completely different issues, and in clinical research—especially reliability analysis—we should not put the emphasis on significance level (P value) (2–8).

They concluded that the accuracy and reproducibility of automated BSI were dependent on scanning speed but not on vendor-specific γ-camera settings. Such a conclusion should be supported by the above-mentioned statistical and methodologic issues. Otherwise, in clinical practice, misdiagnosis and patient mismanagement may occur.

• © 2017 by the Society of Nuclear Medicine and Molecular Imaging.