From P(doom) to P(harm) in healthcare: An operational surveillance framework for accumulative clinical risk
Debates about artificial intelligence (AI) safety often reach for P(doom), a shorthand in AI-safety discussions for the probability of catastrophic AI risk. Healthcare needs a nearer and more testable companion: P(harm), a proposed surveillance framework for cumulative clinical risk. We define accumulative risk as harm that arises from repeated small shifts in diagnosis, triage, clinician behavior, training, or workflow rather than a single catastrophic failure. We operationalize P(harm) as a family of workflow-specific, severity-weighted conditional estimates: in a declared workflow and model version, it is the estimated probability that an independently reviewed AI-exposed encounter has an adverse outcome of at least a prespecified severity within a prespecified time horizon, after stratifying or adjusting for acuity, case mix, and local workflow covariates. In pilot use, the estimator should be prespecified, for example, a risk-adjusted empirical incidence with confidence or credible intervals or a hierarchical logistic/Bayesian model; P(harm) is not a causal-attributable fraction unless paired with a valid comparison design. The framework is organized around four local pathways—AI–clinician discordance in complex patients, automation bias, skill erosion and never-skilling, and consumer AI triage failure—plus a fifth, less mature pathway for correlated vendor or foundation-model failures across institutions. The motivating evidence is early and heterogeneous, often retrospective or vignette-based, so each pathway is framed as a surveillance hypothesis rather than proof of generalized harm. The purpose is not to slow beneficial AI, which may reduce errors and administrative burden, but to make clinical drift visible early enough for health systems to adjudicate, recalibrate, roll back, or retrain before repeated small failures accumulate.

- Ord T. The Precipice: Existential Risk and the Future of Humanity. Grand Central Publishing; 2020.
- Kasirzadeh A. Two types of AI existential risk: decisive and accumulative. Philos Stud. 2025;182(7):1975-2003. doi: 10.1007/s11098-025-02301-3
- Topol EJ. High-performance medicine: the convergence of human and artificial intelligence. Nat Med. 2019;25(1):44- 56. doi: 10.1038/s41591-018-0300-7
- Chatziisaak D, Burri P, Sparn M, Hahnloser D, Steffen T, Bischofberger S. Concordance of ChatGPT artificial intelligence decision-making in colorectal cancer multidisciplinary meetings: retrospective study. BJS Open. 2025;9(3):zraf040. doi: 10.1093/bjsopen/zraf040
- Zeltzer D, Kugler Z, Hayat L, et al. Comparison of initial artificial intelligence and final physician recommendations in AI-assisted virtual urgent care visits. Ann Intern Med. 2025;178(4):498-506. doi: 10.7326/ANNALS-24-03283
- Goh E, Gallo R, Hom J, et al. Large language model influence on diagnostic reasoning: a randomized clinical trial. JAMA Netw Open. 2024;7(10):e2440969. doi: 10.1001/jamanetworkopen.2024.40969
- Dratsch T, Chen X, Mehrizi MR, et al. Automation bias in mammography: the impact of artificial intelligence BI-RADS suggestions on reader performance. Radiology. 2023;307(4):e222176. doi: 10.1148/radiol.222176
- Budzyń K, Romańczyk M, Kitala D, et al. Endoscopist deskilling risk after exposure to artificial intelligence in colonoscopy: a multicentre, observational study. Lancet Gastroenterol Hepatol. 2025;10(10):896-903. doi: 10.1016/S2468-1253(25)00133-5
- Hough J, Culley N, Erganian C, Alahdab F. Potential risks of GenAI on medical education. BMJ Evid Based Med. 2025;30(6):406-408. doi: 10.1136/bmjebm-2025-114339
- Abdulnour REE, Gin B, Boscardin CK. Educational strategies for clinical supervision of artificial intelligence use. N Engl J Med. 2025;393(8):786-797. doi: 10.1056/NEJMra2503232
- Ke Y, Jin L, Ong JCL, et al. AI-induced never-skilling in medical education. Nat Med. 2026;32(6):1997-2006. doi: 10.1038/s41591-026-04438-y
- Ramaswamy A, Tyagi A, Hugo H, et al. ChatGPT Health performance in a structured test of triage recommendations. Nat Med. 2026;32(5):1671-1675. doi: 10.1038/s41591-026-04297-7
- HTI-1 Final Rule. Office of the National Coordinator for Health Information Technology. Accessed June 24, 2026. https://healthit.gov/regulations/hti-rules/hti-1-final-rule/
- Clinical Decision Support Software: Guidance for Industry and Food and Drug Administration Staff. US Food and Drug Administration. Published January 2026. Updated January 29, 2026. Accessed June 24, 2026. https://www.fda.gov/regulatory-information/search-fda-guidance-documents/ clinical-decision-support-software
- Mitigating automation bias in physician-LLM diagnostic reasoning using behavioral nudges. ClinicalTrials.gov. Accessed June 24, 2026. https://clinicaltrials.gov/study/NCT07328815
- Loaiza-Bonilla A, Penberthy S. Harnessing Moravec’s paradox in health care: a new era of collaborative intelligence. NEJM AI. 2025;2(5):AIp2500005. doi: 10.1056/AIp2500005
