Ethical Considerations for Artificial Intelligence in Medical Imaging: Data Collection, Development, and Evaluation

Consent to Secondary Analysis

Obtaining explicit informed consent from every data subject for each specific reuse of deidentified data would be costly. Some have therefore argued that patients should be asked to provide broad consent to the use of their deidentified data for future secondary research projects (12). More radically, some argue that after deidentification there is no owner of the data and that consent thus should be replaced by an enhanced duty of care to ensure data use is appropriate (13). Although there is currently no settled view on the appropriate ethical standard to apply to secondary research, these proposals suggest 3 issues researchers should consider.

First, individual subjects should be notified that their data will be included in AIMD datasets. This notification may be accomplished by notification of the possibility of reuse during initial data collection (i.e., during medical treatment) or at the time of each specific reuse. When possible, notification should include a statement of the risks raised by the intended reuse and the opportunity to refuse or withdraw (14). When this is not possible, educational programs to raise public awareness regarding the use of anonymized medical data should be adopted (2). It is likely that a single mechanism is insufficient given that different data sources may be more or less ethically sensitive (e.g., cranial imagery or whole-body CT). Instead, multiple opportunities to notify data subjects will better support the trustworthiness of AIMD development.

Second, data subjects should be provided the opportunity to withdraw their data from research datasets. Currently, many medical imaging datasets are fully anonymized on collection, allowing sharing of the dataset without human subject research oversight. Although this arguably promotes subject confidentiality, it also deprives subjects of control of their data, a question of particular concern when public datasets are used to create closed-source AIMDs. Though there may not be the opportunity to remove data from a trained model, some subjects may wish to remove their imagery from datasets to prevent further use. Providing opportunities for blanket withdrawal of data from secondary datasets is a key mechanism for demonstrating trustworthiness and working to repair damaged relationships with oppressed communities. The medical imaging research community should consider exploring models—such as those adopted in genomics and biobanking (15)—that allow for this kind of withdrawal of consent. These kinds of data repositories require infrastructure that securely maintains coded links between subject contact information and subjects’ deidentified data alongside staff to facilitate withdrawal of data from the repository. This is more costly than open access, permanently anonymized datasets that allow unrestricted access.

Third, in the absence of individual consent or withdrawal, researchers should develop other mechanisms to remain accountable to subject populations. A data subject advisory board may be formed to express concerns and exercise oversight related to the use of deidentified data. This may be particularly important in the context of data from vulnerable or historically marginalized populations. Marginalized communities may have a warranted mistrust of medical researchers because of serious historical abuses. Black and indigenous scholars have thus suggested enhanced ethical obligations with respect to the use of health data from marginalized populations, including seeking explicit consent for each subject or engaging in community consultation (16,17).

Privacy and Reidentification

Clinical data used for the development of AIMDs may contain sensitive personal health information. Nuclear medicine and molecular imaging generate datasets that include whole-body scans, diagnoses, and neurologic imaging—all of which may expose highly sensitive information about a patient’s sex, behavior, life expectancy, or insurability. Moreover, there are particular sensitivities surrounding data connected to behavioral or neurologic health. Creating a framework to guide approaches to security, deidentification, and privacy for nuclear medicine and molecular imaging data is thus necessary if we are to increase public trust in the use of these data for AIMD development (2,18). We highlight 3 strategies for increasing the privacy of data subject information.

First, whereas many AI-based approaches include patient demographic features as inputs into decisions, the inclusion of features that could directly or indirectly identify a subject in training data should be limited to cases for which there is a specific rationale for the use of that feature. Given the increasing integration of data mining technologies into our daily lives and the high dimensionality of AIMD datasets, caution should also be exercised with respect to indirect identifiers. For instance, the National Institutes of Health’s All of Us precision medicine database identifies a tier of “data elements that may not, in their own right, readily identify individual participants but that may increase the risk of unapproved re-identification” (19). Limiting the dimensionality of a dataset may thus be a key privacy safeguard. However, deidentification carries with it risks of degrading performance. For instance, quasiidentifiers such as patient weight raise reidentification risk (20) but enable calculations (e.g., SUVs) that are essential for making sense of molecular images. Although it is not always easy to determine a priori whether a feature will influence AIMD performance, if early proof-of-concept projects show that particularly sensitive features have a low degree of influence, then there is a strong case for excluding them from training datasets used in later work.

Second, when indirectly identifiable information (i.e., weight or CT of the head) is required, differential privacy techniques can be used to obscure patient-level identifiers while preserving population-level regularities (21). Differential privacy aims to remove the identifiability of individual subjects while preserving the underlying structure of the data. However, differential privacy may also involve performance trade-offs given that identifiable data may be necessary to ensure model utility in the clinical environment.

Finally, even when researchers use differential privacy techniques, the storage of large amounts of patient data—so-called data lakes—carries the risk of malicious intrusion. Although such data lakes promise to improve the speed and efficacy of AIMD development, they also offer tempting attack surfaces for malicious actors. One tool for minimizing this risk may be federated learning, whereby AIMDs can be trained on a federation of smaller datasets, without formally aggregating the datasets (22). A combination of federated learning and differential privacy techniques may help minimize the risks of reidentification and help build community trust in the secondary reuse of data.

Procedural Fairness

Standard principles of medical ethics require that patients be treated with equal consideration, regardless of their race, sex, religion, or other protected characteristics (38). In the context of AIMDs, this is normally understood as a prohibition on the use of these features to make predictions or decisions about patients during deployment. This has led many developers to attempt to exclude race or sex as features within a training dataset. Two considerations suggest that ensuring procedural fairness requires more nuance.

First, the high dimensionality of AIMD input data, alongside the pervasive impact of race and sex on people’s lives, means almost all features will be partially correlated with race or sex. For instance, in the United States, race is correlated with residential address, insurance status, and prior access to PET/CT. Although procedural fairness may require identifying and removing close correlates of race and sex, identifying and eliminating all the features correlated with race or sex is likely to severely reduce accuracy. In this respect, some have suggested that rather than a focus on whether sensitive features cause predictions, procedural fairness requires that models be well calibrated for all racial and sex groups (39,40). In this view, ensuring that predictions have the same meaning (i.e., predictive value) for different subpopulations ensures that subsequent decision-making treats these individuals with equal consideration.

Second, explicit use of sensitive features during model training may be justifiable as a way of measuring or improving the performance of algorithms for marginalized groups. Historically, some medical decision-making tools have used features such as patient race or ethnicity as direct inputs. Although the use of race was initially rooted in racist biologic essentialism, the medical community’s understanding of the influence of race and ethnicity on diseases has progressed. Some now argue that race can act as a proxy for the influence of racist oppression on patient’s health, opening the possibility that race (or other sensitive attributes) can be used to improve predictive accuracy for these groups (41). Notably, however, the categorically driven capture of this information (e.g., patients identified as Black, White, Asian, or other) can often be imprecise and can in fact contribute to the reification of social stereotypes.

Distributive Fairness

Another potential concern is the differential impact of AIMD systems on marginalized populations. AIMDs have tremendous potential to mitigate health disparities in the United States; however, discrepancies in access and use of AIMDs in patients who differ by race or sex have raised concerns regarding AI contributions to health-care disparities (42). There are well-documented inequities in medical imaging studies on the topics of coronavirus disease 2019, mammography, lung cancer screening, and missed care opportunities (43). For instance, privately insured Hispanic/Latinx and Black women are more likely to obtain mammography at facilities with less favorable characteristics, which subsequently correlates with higher breast cancer mortality rates in this population (44). Likewise, common training and text datasets for AIMDs (e.g., Medical Information Mart for Intensive Care, version IV) reflect societal biases in the diagnosis and treatment or patients—biases that can distort algorithms optimized to perform well in those test datasets (45). To underscore this issue, a recent study demonstrated a 4-fold discrepancy between Black and non-Hispanic White patients in their access to prostate cancer molecular imaging with ⁶⁸Ga-PSMA-11 PET (46). Since PET is not immune to these challenges, neither are PET-based AI systems. We identify 3 points within the development pipeline at which technical or policy interventions may promote distributive fairness.

First, preprocessing strategies attempt to remediate biases introduced during the training phase. For instance, the development and characterization of standardized testing datasets that are more broadly representative than those available to single research teams is a high priority (e.g., the All of Us research program (15)). Such broadly representative datasets may limit sampling bias and societal biases that are localized to a particular study site. It is, however, fundamentally misleading to think that unbiased or debiased training sets will eliminate the potential for disparate impact. Even though preprocessing strategies are likely important considerations in the development of successful AIMD, they are not sufficient.

Second, inprocessing and postprocessing seek to remediate outcome disparities by fine-tuning model performance in different subgroups. This is often achieved by constraining the model to satisfy a statistical metric of fairness (i.e., demographic parity, error rate parity, or calibration parity). Although these metrics may sometimes help diagnose problems (39), they suffer 2 important limitations as methods for achieving fairness. First, they cannot be mutually satisfied under ordinary conditions (47) and thus require a value judgment about the appropriate metric to adopt in a given domain. Second, satisfying these metrics does not reliably reduce health disparities. For instance, in the famed study by Obermeyer et al. (39), the underlying inequality was generated by an inappropriate proxy for health risk (i.e., health expenditure), which would not show up as unfair on these tests. Thus, although failing to satisfy these fairness metrics may provide evidence of inappropriate bias, satisfying them is not sufficient to ensure that AIMDs reduce health disparities.

Finally, ensuring distributive fairness requires careful consideration of the context of deployment. For instance, an AIMD trained using imagery from a high-resolution PET/CT scanner may perform well in all demographic subgroups in a test dataset. Nonetheless, if it is with imagery derived from lower-resolution PET/CT scanners, it may exhibit higher error rates. This may increase disparities between high- and low-resource contexts despite a putatively fair algorithm. As we discuss in a companion piece (3), training, imaging infrastructure, and structural biases all contribute to the effect of an AIMD on health disparities.