Quality Factors in Radiomics Studies
| Factor | Description |
| Imaging | |
| Standardized imaging protocols | Imaging acquisition protocols are well described and ideally similar across patients. Alternatively, methodologic steps are taken toward standardizing them. |
| Imaging quality assurance | Methodologic steps are taken to incorporate only acquired images of sufficient quality. |
| Calibration | Computation of radiomics features and image-processing steps matches benchmarks of the IBSI. |
| Experimental setup | |
| Multiinstitutional/external datasets | Model construction or performance is evaluated using cohorts from different institutions, ideally from different parts of world. |
| Registration of prospective study | Prospective studies provide the highest level of evidence supporting clinical validity and usefulness of radiomics models. |
| Feature selection | |
| Feature robustness | Robustness of features is evaluated against segmentation variations and varying imaging settings (e.g., noise fluctuations and interscanner differences). Unreliable features are discarded. |
| Feature complementarity | Intercorrelation of features is evaluated; redundant features are discarded. |
| Model assessment | |
| False-discovery corrections | Corrections for multiple testing comparisons (e.g., Bonferroni or Benjamini–Hochberg) is applied in univariate analysis. |
| Estimation of model performance | Teaching dataset is separated into training and validation sets to estimate optimal model parameters (e.g., using bootstrapping, cross-validation, and random subsampling) |
| Independent testing | A testing set distinct from the teaching set is used to evaluate performance of complete models (i.e., without retraining and without adaptation of cutoffs). Evaluation of performance is unbiased and not used to optimize model parameters. |
| Performance results consistency | Model performance in training, validation and testing sets is reported. Consistency checks of performance measures across the different sets are performed. |
| Comparison to conventional metrics | Performance of radiomics-based models is compared against conventional metrics such as tumor volume and clinical variables (e.g., staging) to evaluate the added value of radiomics (e.g., by assessing the significance of AUC increase with the DeLong test). |
| Multivariable analysis with nonradiomics variables | Multivariable analysis integrates variables other than radiomics features (e.g., clinical information, demographic data, and panomics). |
| Clinical implications | |
| Biologic correlate | Relationship between macroscopic tumor phenotypes described with radiomics and underlying microscopic tumor biology is assessed. |
| Potential clinical application | Current and potential applications of proposed radiomics-based models in clinical setting are discussed. |
| Material availability | |
| Open data | Imaging data, tumor region of interest, and clinical information are made available. |
| Open code | Software code for computation of features, statistical analysis, machine learning, and exact reproduction of results, is open-source. Code package is ideally shared as easy-to-run organized scripts pointing to other relevant pieces of code, along with useful sets of instructions. |
| Open models | Complete models are available, including model parameters and cutoffs. |
AUC = area under the receiver-operating-characteristic curve.