Marker Selection Based on Only Reproducibility Can Be Questioned

TO THE EDITOR: An excellent recently published paper (1) suggests a role for image heterogeneity markers, with an eye toward therapy monitoring. The reproducibilities of 24 individual measuring processes were determined from esophageal tumor scans obtained a few days apart. The “Conclusion” section singles out 5 markers as being preferred, as justified solely from their being the most reproducible (i.e., the 5 with the lowest tabulated SD for percentage reproducibility).

Although readers can examine the text beyond the “Conclusion,” they might preferably consult a related interesting and thorough investigation published earlier by the authors (2) to see a fuller picture for choosing the best markers. This earlier work on selecting markers correctly used receiver-operating-characteristic (ROC) curves requiring outcome information that indicates therapy responders versus others. For the particular circumstances encountered here, this ROC-based selection methodology gave results similar to a solely reproducibility-based selection methodology. The “Conclusion” does not emphasize that only ¹⁸F-FDG PET esophageal studies were conducted.

The need to also have outcome information seems intuitive, as is mathematically supported by the exclusive analytic dependency (3) of the ROC area on a ratio (referred to as the Mahalanobis distance in classification theory and popularly also relatedly encountered in t testing between groups):difference in average marker values of 2 states[(measuring process SDs)2+(states' SDs)2]1/2.

Noteworthy besides the measurement reproducibility term are 2 biologically based terms: how much the markers of 2 states (e.g., for responders vs. others) differ on average, and what variability in markers is additionally encountered because of biologic randomness. It cannot be assumed that the relative importance among these 3 terms for all tumors, marker types, and protocols will resemble that encountered for the heterogeneity markers of esophageal tumors on ¹⁸F-FDG PET scans.

Finally, the traditional full-range ROC area, though quite good, may not be the entire story for marker selection in all instances. Practical clinical considerations may sometimes lead an investigator to conclude that only some judiciously chosen sensitivity or specificity range, defining a partial area, is of importance. Also, though complex to study, cost-to-benefit considerations can play a role when markers come from substantially differing protocols.

• © 2012 by the Society of Nuclear Medicine and Molecular Imaging, Inc.