Ethical Considerations for Artificial Intelligence in Medical Imaging: Deployment and Governance

Intended Use and Performance

Although AIMDs are emerging as incredibly powerful new tools in health care, increasingly able to make diagnostic or treatment recommendations, the nature of the physician–patient relationship requires that those at the bedside retain responsibility and accountability for potential errors in AI-based medical diagnosis or risk stratification. Although developers and regulators carry an ethical burden to ensure that AIMD performance claims are warranted (15), the clinician who is credentialed by the appropriate professional body is responsible for the clinical action. This suggests 3 considerations.

First, clinicians should be knowledgeable about the intended use of an AIMD system. The Food and Drug Administration has done substantial work to define a typology of AIMD (what they call Software as a Medical Device), based on the level of computer-aided detection (CADe) or computer-aided diagnosis (CADx) that the AIMD is intended to provide (Table 2) (16). Non-CADe systems provide measurement or annotation of imagery without interpretation. CADe refers to an AI device that intends to identify abnormalities but does not attempt diagnosis or treatment recommendations. CADx systems attempt to directly diagnose the presence (and severity) of a disease (4).

Grading AIMDs on this scale allows clinicians to identify the level of risk associated with using a specific AIMD in a clinical workflow. Consider AIMD systems involved in PET workflows. Most systems at the non-CADe level, such as a PET quantification tool, may pose lower risks since they simply provide additional information that the physician incorporates into decision-making. Although even at this level, AIMDs may inadvertently eliminate or de-emphasize malignant features in imagery (17). At the CADe level, the risk increases since physicians may deviate from their judgment on the basis of overreliance on AI-based detection and segmentation of tumors on, for example, ¹⁸F-FDG PET. The highest risk comes with CADx systems, since they provide binary (or categorical) diagnostic information and may obscure the underlying evidence or reasoning for the diagnosis from the clinician. In all cases, clinicians should not use an AIMD outside its intended use.

Second, responsible use of AIMDs requires that clinicians be familiar with the task-specific performance of an AIMD within the population that the clinician serves. In a previous paper (15), we noted that AIMDs should ultimately be evaluated by their performance on clinical tasks in representative clinical contexts and populations. To avoid inappropriate use, clinicians should familiarize themselves with these performance data, including differences in accuracy for race or sex subpopulations.

Finally, clinicians must consider the risks of automation bias (18). Automation bias occurs when users come to unquestioningly accept the output of AIMD, without appropriate regard for predictive errors or uncertainties. In general, clinicians should act with an appropriate level of skepticism with respect to the outputs of AIMDs, until such time as they are well integrated into routine clinical practice. Indeed, most Food and Drug Administration–approved AIMDs specifically include a statement that the software is not intended to diagnose or treat a disease and may only be applied as a measurement tool. Nonetheless, as CADx systems start to appear, and as AIMDs begin to demonstrate better accuracy than physicians at a specific task, automation bias may become difficult to resist (19). It is critical to remain attentive to the fact that ethical clinical decision-making requires sustaining the shared decision-making paradigm, where AI is but one source of information in a set of considerations that, together, contribute to a decision (20).

Patient Best Interest

The outputs of AIMDs will inform clinician decision-making about a host of tradeoffs in medical imaging: for example, between false-positive and false-negative diagnoses, or acceptable dosage of radioisotopes relative to investigational value. Two considerations are relevant.

First, minimizing harm and maximizing benefit require that we recognize imbalances in the harm of false positives and false negatives for a specific task. For example, in a cancer diagnosis task, false negatives will often have higher costs for patients than false positives (21). This suggests that common performance measures for AIMDs may not provide sufficient information to clinicians and patients involved in shared decision-making. For instance, the area under the receiver-operating-characteristic curve is a threshold agnostic performance metric that treats false positives and negatives as equivalently weighty (which is rarely true in the clinical context) (22). Nor does task-specific selection of simple metrics (e.g., minimizing false negatives) solve the problem, since for almost all tasks both false negatives and false positives harm patients (i.e., through under- and overtreatment). Instead, clear communication of confusion matrices may be a necessary component of clinical evaluation, to ensure that doctors and patients can navigate the complex assessment of costs and benefits themselves.

Second, minimizing harm and maximizing benefit require careful consideration of the different ways patients make tradeoffs between the risks and benefits of interventions. Many interventions in nuclear medicine carry grave tradeoffs between longevity and quality of life. In this respect, AIMDs—and especially CADx or CADe systems (23)—should avoid unnecessarily hard coding judgments about the appropriate risk and benefit tradeoffs (24). For instance, during radiation therapy planning, an AIMD that segments tumors in PET/CT could provide an estimate of how much diseased tissue is present in each voxel of the image (25), allowing caregivers and physicians to discuss risk tolerances with patients. Of course, not all value judgments can be avoided in the development of an AIMD. Tasks such as image denoising or instrument calibration (26)—although they affect the error rates of downstream diagnosis or intervention—are too abstracted from patient outcomes for meaningful dialogue with each individual patient to occur. In these cases, reasonable effort should be made to ensure that embedded value judgments—that is, aggressiveness of denoising, or sensitivity to patient motion—reflect broadly held standards. If well-established standards do not exist to guide the selection of critical thresholds, developers should seek to involve stakeholders, including patients and providers, in the selection process (27).

Notification and Risk Declaration

Clinicians have an obligation to notify patients regarding the use of AIMDs in a clinical workflow when the clinicians have reason to believe that this information would be material to the patients’ decision-making. First, performance information should be clinically relevant (e.g., false-positive and -negative rates in clinical contexts) and not simply abstract performance metrics (e.g., area under the receiver-operating-characteristic curve). This will enable informed discussions with patients about the relative risks and benefits of AIMD use and the relevance of an AIMD’s findings to the overall prognosis or treatment plan. Second, performance limitations for racial or sex subpopulations should be declared to patients who are members of the disadvantaged class. This requires that the performance of the AIMD be evaluated in subpopulations that are likely to be encountered. Moreover, alternatives to the use of the AIMD, and the relative performance of these alternatives, should also be provided to patients.

Explainability

The black box nature of deep learning means that detailed information about the decision procedure of AIMDs is not always accessible or interpretable, arguably undermining clinician and patient understanding (28,29) (the standard practice for preparing informed consent forms is to use an eighth-grade reading level). Explainability refers to a cluster of techniques that aim to help physicians understand and explain the AI’s internal decision-making process. Although explainability techniques may sometimes be useful, we argue that understanding the performance and limits of the AI system is likely more important. We do so for 2 reasons.

First, currently existing explainability techniques are not able to reliably explain predictions at the individual level (30). Calls for greater explainability often assume that these techniques can describe the precise computational pathway between individual inputs and outputs (31). One purported strategy is to create a parallel regression model that identifies the statistical association of particular input features to an AIMD’s output. Another is to develop heat maps that purport to identify the areas of an image that were relevant to the prediction. It is unclear, however, whether these techniques deliver information about individual predictions or about the general parameters of the model (6,19). Moreover, emerging evidence indicates that explainability techniques may actually compromise clinicians’ ability to identify incorrect outputs and thus worsen automation bias (19).

Second, calls for explainability often conflate the different forms of explanation that are desirable in different contexts (32). Ferretti et al. (28) distinguish among 3 forms of opacity in medical AI: lack of disclosure, epistemic opacity, and explanatory opacity. Lack of disclosure refers to instances in which patients are unaware that diagnostic or interventional decisions are being made with the aid of an AIMD. These can be dealt with through simple notifications on the use and performance of AIMD. Second, epistemic opacity refers to the inability to inspect the precise computational pathway (i.e., feature weights and parameters) between inputs and predictions. Although this information may be useful to developers, this level of transparency about decision pathways is rarely demanded of other medical interventions (33). Finally, explanatory opacity refers to the inability to explain why the input data are causally connected to the prediction—a problem of particular importance in machine learning, which relies on identifying statistical regularities that may not have well-characterized causal explanations. It is unclear, however, whether principles of informed consent require clinicians to explain to patients the precise causal pathway between diseases and diagnostic tests (33,34). Moreover, a detailed explanation of the causal mechanisms underlying imaging results and disease diagnosis would require a greater understanding of statistics and nuclear medicine than most patients possess. These considerations suggest that explainability techniques—although of technical interest to developers—may not be necessary to satisfy existing informed consent practices.

Procedural Fairness

Procedural fairness requires that patients be treated with equal consideration, regardless of their race, sex, religion, or other protected characteristics (12). Unfortunately, there is some evidence that clinicians (both in nuclear medicine and in other specialties) have often failed to live up to this requirement with respect to the provision of medical imaging (38). Although structural barriers mean that equal access does not ensure equal opportunity to benefit from AIMD, procedural fairness requires that the medically indicated use of AIMDs be offered to patients regardless of their protected characteristics.

Procedural fairness also discourages the use of a patient’s protected characteristics as an input feature. Historically, some medical decision-making tools have used features such as patient race or ethnicity as direct inputs (39,40), informed by ill-conceived genetic and biologic understandings of race (41). For instance, a common breast-cancer screening tool uses race alongside family history, age, and the Breast Imaging Reporting and Data System breast density score (42), with the result that it may underestimate risk in nonwhite patients (40). Although some now argue that race can act as a proxy for the influence of racist oppression on patient’s health, naïve attempts to try to correct systemic bias may unintentionally introduce biases of their own (43). For instance, it can impose tradeoffs on marginalized groups: by using features such as race to improve the accuracy of risk predictions, we may reduce access to desired treatments or interventions for that group (39). Moreover, the capture of demographic information (e.g., through self-identification or physician assessment) is imprecise and can contribute to the reification of stereotypes among clinicians about patient risk. As our understanding of the link between health, race, and other socially salient attributes improves, we believe that the use of these categories to make clinical decisions requires careful justification. These justifications should include robust knowledge of the data sources for demographic information, the causal structure of the association between the attribute and health, and the effect on marginalized people of using socially salient attributes to make clinical decisions.

Distributive Fairness

Another potential concern is the accuracy of AIMDs by ethnicity, socioeconomic status, or sex. Encouragingly, there has been much recent work on technical methods to remove bias from AIMDs (37). In a companion paper (11), we will note that a variety of techniques during the data collection and development phase can help ameliorate biases introduced during the training phase. Nonetheless, these techniques are not sufficient to ensure that AIMDs reduce health disparities, for at least 2 reasons.

First, the impact of an AIMD on disparities will be dependent on its precise pattern of accessibility. There are well-documented inequities in structural access to medical imaging for coronavirus disease 2019 diagnostics, mammography, and lung cancer screening (8). The cost of AI implementation is not only limited to the cost of developing or purchasing the AIMD but also includes supportive technology infrastructure, training staff, and patient education. Careful assessment of whether AIMDs should be adopted is especially critical for low-income or rural areas, where implementing expensive AI technology may divert resources from lower-tech interventions, with a greater impact on patient outcomes (44).

Second, even if an AIMD is equally accurate and accessible for all demographic subgroups, it may exacerbate existing structural inequalities (45). For instance, an AIMD designed to schedule future appointments based on a no-show predictor may schedule patients with a history of missed appointments to overbooked days. Of course, missed appointments are often related to a patient’s structural determinants of health—an inability to cover transportation costs or childcare or to take time off from work—and hence the very patients who may need additional care must now experience longer wait times or overbooked clinics. Thus, even if the AIMD predicts absenteeism correctly regardless of race or socioeconomic status, its deployment may widen health disparities due to background structural injustice.