Unveiling the Black Box: A Synergistic Exploration of Deep Learning Features in Relation to Radiomics and Clinical Information for Enhanced Interpretability in Medical Imaging

Introduction: Linking deep learning features (DF) with radiomics (RF) or clinical features (CF) can enhance interpretability and transparency, fostering trust and collaboration between advanced AI and traditional clinical practices. This fusion ensures that deep learning insights align with established medical knowledge, facilitating a more intuitive interpretation of AI-generated findings in clinical decision-making.

Methods: We analyzed 221 lung cancer PET/CT scans from TCIA and our local clinical database. PET images were first registered to CT, and SUV correction, clipping and normalization were then applied to images (Fig. 1). We employed 2 categories of well-known features including RFs and CFs to interpret DFs. DFs adapt to diverse masks due to their autonomous learning. By contrast, RFs, manually crafted for specific patterns, are conventionally extracted from segmented regions. This distinction arises from the inherent adaptability of DFs and the predefined nature of RFs in medical imaging analysis. A 3D Autoencoder neural network architecture was used to extract 1024 DFs from the bottleneck layer through 3 masks, including whole (W), cropped (C) (32×32×32 mm³) and manually segmented (S) tumor area on PET/CT images, while 215 standardized RFs were extracted from each segmented tumor area. 20 CFs were collected from categories such as surgical, biopsy, clinical history, tumor staging, chemo&radiotherapy, disease duration, and demographics information (datasets summarized in Table 1). Pearson correlation was used to identify relation between CFs, RFs, and DFs. Features with correlation coefficient > or < 70% were considered as similar or independent features, respectively. Moreover, various hybrid machine learning systems (HMLS) including 3 feature selection algorithms (regulated on selection of 20 features) followed by 10 regression algorithms were employed to predict overall survival (death).

Results: Table 1 indicates relatively similar features between DF and RF datasets; 4 DFs in PET-C-DF (DFs extracted from cropped PET) were relatively similar with 4 RFs such as Mean (Intensity histogram), kurtosis (Intensity histogram), 90th percentile (Statistics) and mean (Statistics) in PET-RF (RF extracted from segmented PET). Furthermore, there were no similar features between PET-S-DF (DFs extracted from Segmented PET) and PET-RF. The 4 abovementioned features from PET-RF were similar to 4 DFs of PET-W-DF (DF extracted from Whole PET). 5 DFs from CT-C-DF (DF extracted from Cropped CT) were similar to 5 RFs such as skewness, minimum, energy, inverse difference moment normalized and skewness in CT-RF dataset. Moreover, 12 DFs in CT-S-DF (DF extracted from segmented CT) were similar to 12 RFs from CT-RF including 5 morphology features and 7 Co-occurrence matrix features. Finally, 2 DFs from CT-W-DF (DF extracted from whole CT) were similar to 2 RFs including Asphericity (Morphology) and Difference entropy (Co-occurrence matrix) in CT-RF dataset. In total, 26 of our DFs were interpretable through RFs. Furthermore, we showed no correlation coefficient over 0.7 between CFs vs. either RFs or DFs. The highest overall survival (OS) prediction performance (Table 1) of 0.19 ± 0.04 years [outcome range: 0.11-6.6 years] was achieved by CT-S-DF involving Mutual Information feature selection algorithm+Extra Trees Regressor. 3rd column of Table 1 indicates the contribution of 13 DFs similar to RFs contributing to highest performance HMLSs.

Conclusions: Our study showed that some important DFs could be interpreted by RFs, while neither DFs nor RFs were interpretable via CFs. In short, 26 of the DFs indicated high correlation with RFs, and 13 out of these 26 DFs were selected by HMLSs.

• Download figure
• Open in new tab
• Download powerpoint