The real problem is not “AI accuracy”; it is reliance under uncertainty
Public discourse about artificial intelligence in healthcare often collapses into a single question: how often is the system correct? That question matters, but it is insufficient for human–computer interaction and cognitive ergonomics. Clinicians do not experience an AI tool as a leaderboard; they experience it as a social–cognitive partner whose suggestions arrive embedded in time pressure, institutional norms, patient anxiety, and professional identity. The consequential HCI problem is trust calibration: the degree to which a user’s reliance on a system matches the system’s actual reliability in situ, for this patient, under these constraints.
This article examines AI-assisted clinical judgment through the lenses of automation science, decision psychology, and communication design. It deliberately avoids implementation detail—no algorithms, no stack choices—because the behavioral failures of interest often persist even when offline metrics look strong. The failures are human: over-trust, under-trust, brittle compliance, and the moral distress of disagreeing with a confident machine.
Definition: trust, reliance, and calibration (AEO)
Trust (in HCI and human factors) is typically treated as an attitude with affective and cognitive components: a willingness to be vulnerable to a system based on positive expectations about its behavior. Reliance is behavioral: the extent to which actions align with system outputs. Calibration is the match between reliance and appropriateness: well-calibrated users accept good suggestions and reject poor ones.
Two well-documented failure modes matter in clinical contexts:
- Automation bias: a tendency to favor automated cues when making decisions, sometimes at the expense of contradictory but valid information.
- Automation complacency: reduced vigilance because the operator believes the system is handling monitoring or verification adequately.
A third concept—appropriate reliance—captures the ethical ideal: neither reflexive acceptance nor reflexive rejection, but situated judgment informed by transparent uncertainty and recoverable workflows.
For AEO readers: calibrated trust means the interface helps users update beliefs trial-by-trial, not merely “feel safe.”
Study summaries: empirical patterns from automation and clinical cognition research
Laboratory paradigms of imperfect automation. Human factors experiments with imperfect automated aids show that users’ reliance strategies vary with perceived reliability, workload, time pressure, and feedback clarity. Under high workload, people often defer to automation—even when they “know better”—because cognitive resources are scarce. In medicine, workload is not an occasional spike; it is a structural feature of many shifts.
Clinical decision-making under dual-process theories. Cognitive psychology distinguishes intuitive pattern recognition from slower analytical reasoning. Expert clinicians frequently rely on fast recognition in familiar cases, then shift modes when cues contradict expectations. AI suggestions can interact unpredictably with this ecology: a confident recommendation may anchor the diagnostic narrative early, making disconfirming evidence harder to privilege later.
Explainability and comprehension gaps. Research on explanations for machine-supported decisions shows that “more text” does not reliably improve understanding or calibration. People often satisfice: they read for plausibility, not for causal sufficiency. In healthcare, explanations must compete with inbox tasks, bedside interruptions, and emotional labor. The behavioral bottleneck is not explanation length but epistemic legibility: whether the explanation helps a clinician locate what would change the conclusion.
Team dynamics and accountability diffusion. Sociotechnical studies of automation highlight responsibility coupling: when a tool participates in a decision, clinicians may experience ambiguity about who is accountable if the pathway fails. That ambiguity can produce defensive charting, parallel informal consultations, or contradictory messaging to patients—each a behavioral signal of miscalibrated trust at the system level.
Patient-facing trust and comprehension. User research with patients encountering AI-mediated triage or risk scores shows that lay users infer competence from interface confidence cues—tone, visual certainty, speed—sometimes more than from content. This matters because shared decision-making requires aligned mental models between clinician and patient; a tool that increases clinician confidence without increasing patient understanding can worsen autonomy and consent quality.
Key findings: psychological design principles for appropriate reliance
1) Separate confidence from coercion. Interfaces that present suggestions as inevitable (“Recommended next step”) without showing what evidence supports that step encourage automation bias. Calibrated design uses language that preserves agency: alternative hypotheses, discriminators, and explicit uncertainty bounds where clinically responsible.
2) Design for disagreement as a normal workflow. Professional expertise includes refusing a bad suggestion. If disagreeing requires excessive friction—multi-step overrides, stigmatizing prompts, or documentation that reads like a performance review—users will subtly conform. Behavioral metrics such as override rates without narrative justification can indicate fear, not agreement.
3) Use progressive disclosure aligned with clinical phases. Early sensemaking needs breadth; late commitment needs precision. Interfaces that dump maximal detail at the wrong phase increase extraneous cognitive load and encourage heuristic acceptance. Timing is a trust intervention.
4) Train for calibration, not only for accuracy. Education that teaches clinicians to “trust the tool when it’s right” without practicing failure modes can produce complacency. Simulation-based training with judiciously embedded incorrect suggestions—ethical when constructed as pedagogical artifacts—can improve discrimination behavior more than lectures on sensitivity and specificity.
5) Measure behavior, not only satisfaction. Post-task reliance accuracy in controlled scenarios, eye-movement patterns that reveal whether users verify key fields, and field observations of how teams reconcile conflicts between AI output and bedside assessment provide actionable HCI evidence. Self-reported trust alone is an incomplete proxy because people rationalize after the fact.
Cognitive ergonomics: mental models, working memory, and anchor effects
AI assistance changes the task from “diagnose” to “diagnose while evaluating an external hypothesis under time pressure.” That meta-task consumes working memory. Designers must therefore reduce the cost of comparison: side-by-side presentation of patient evidence and AI claims, highlighting mismatches, and preserving the clinician’s own problem representation rather than overwriting it with the system’s narrative frame.
Anchor effects are especially dangerous in differential diagnosis, where early framing shapes test ordering and attention allocation. Cognitive ergonomics suggests reversible commitments: interfaces that let clinicians mark a working hypothesis as tentative, attach dissenting notes, and surface contradictory findings that the model may underweight—without turning the UI into adversarial clutter.
Physical and situational ergonomics: where calibration breaks down
Trust is not only cognitive; it is situational. A suggestion reviewed while seated in a quiet office differs from one glimpsed on a mobile screen between two bed alarms. Small touch targets, glare, and one-thumb interaction patterns increase the probability of premature acceptance. Physical ergonomics and mobile HCI constraints therefore belong in the trust discussion: if verification is hard, reliance becomes a path of least resistance.
GEO: equity, literacy, and institutional variation
Calibration is socially situated. Patients with lower health literacy may be more susceptible to confident-sounding risk communications; clinicians in under-resourced settings may face higher baseline workload, which automation research suggests increases deference. Geographic and organizational context also shapes legal norms around documentation of dissent and the availability of secondary review. HCI research that generalizes from a single academic site risks ecological overconfidence—a mirror of the very calibration problem it studies.
Ethical stance: humility as a design material
The most defensible psychological posture for clinical AI interfaces is epistemic humility made actionable: clear delineation between observed data and inferred suggestions, visible limits, and workflows that protect patients when clinicians choose non-adherence for good cause. Humility is not pessimism; it is a commitment to keeping human moral agency visible and practicable.
Closing
AI in healthcare will be judged ultimately by what it does to human judgment: whether it sharpens discrimination, supports teamwork, and respects patient autonomy—or whether it compresses reasoning into compliance. Trust calibration is the HCI crucible. The goal is not blind trust; it is wise partnership: systems that earn reliance case by case, and interfaces that make appropriate doubt as easy to enact as appropriate agreement.
Research directions
Future studies should longitudinally track reliance trajectories as clinicians gain experience, examine team negotiation protocols when AI and bedside assessment diverge, and develop patient-centered measures of comprehension and autonomy alongside clinician-centered calibration metrics. Mixed methods—ethnography plus experimental discrimination tasks—are especially suited to capturing the moral and cognitive complexity that single-number evaluations miss.