AccScience Publishing / JCI / Online First / DOI: 10.36922/JCI026260008
Cite this article
52
Views
Related Info Links
More by Authors Links
Journal Browser
Volume | Year
Issue
Search
News and Announcements
View All
PERSPECTIVE ARTICLE

Calibrated refresh systems for clinical artificial intelligence: A sociotechnical framework for expert escalation

Arturo Loaiza-Bonilla1,2,3*
Show Less
1 Massive Bio, Inc., Boca Raton, Florida , United States of America
2 Department of Hematology and Medical Oncology, St. Luke’s University Health Network, Bethlehem, Pennsylvania , United States of America
3 Lewis Katz School of Medicine at Temple University, Philadelphia, Pennsylvania , United States of America
Received: 28 June 2026 | Revised: 29 July 2026 | Accepted: 12 October 2026 | Published online: 26 October 2026
© 2026 by the Author(s). This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution -Noncommercial 4.0 International License (CC-by the license) ( https://creativecommons.org/licenses/by-nc/4.0/ )
Abstract

Expert attention is scarce, yet clinical artificial intelligence often escalates by fixed thresholds. Hereby present Calibrated Refresh Systems (CRS), a systems-level design pattern for calibrated expert review rather than a new estimator or validated intervention. CRS accepts a low-resolution action only when calibrated residual bounds cannot cross an action threshold; otherwise, the case receives high-resolution review. It couples selective prediction, learning to defer, conformal calibration, reference-label governance, accepted-case audit, drift arbitration, subgroup safeguards, and never-fade rules. A synthetic capacity-matched mechanism check illustrates how harm-aware budgets can reduce a prespecified weighted disagreement proxy without implying improved patient outcomes. CRS yields testable operational artifacts and a staged validation roadmap, generating artifacts suitable for safety governance and regulatory documentation while preserving attention for rare, high-harm cases. Its claims depend on exchangeability, adequate calibration samples, credible reference processes, workflow capacity, and prospective clinical evaluation.

Graphical abstract
Keywords
Clinical decision support
Alert fatigue
Learning to defer
Conformal prediction
Selective deferral
Artificial intelligence governance
Funding
None.
Conflict of interest
The author is Co-Founder and Chief Medical AI Officer of Massive Bio, Inc., which develops AI-enabled clinical-trial matching and oncology decision-support products; holds AI-related patents filed in 2025; and is Co-Founder of Aequis AI Inc. None of these entities funded or directed this work, and Calibrated Refresh Systems as described is a general framework not tied to any product. The author declares no other competing interests related to this manuscript.
References
  1. Wong A, Otles E, Donnelly JP, et al. External validation of a widely implemented proprietary sepsis prediction model in hospitalized patients. JAMA Intern Med. 2021;181(8):1065-1070. doi: 10.1001/jamainternmed.2021.2626
  2. Ostermayer DG, Braunheim B, Mehta AM, et al. External validation of the Epic sepsis predictive model in 2 county emergency departments. JAMIA Open. 2024;7(4):ooae133. doi: 10.1093/jamiaopen/ooae133
  3. Ancker JS, Edwards A, Nosal S, et al. Effects of workload, work complexity, and repeated alerts on alert fatigue in a clinical decision support system. BMC Med Inform Decis Mak. 2017;17(1):36. doi: 10.1186/s12911-017-0430-8
  4. Geifman Y, El-Yaniv R. SelectiveNet: A deep neural network with an integrated reject option. Proceedings of the International Conference on Machine Learning. PMLR. 2019;97:2151-2159. Accessed June 27, 2026. https://proceedings.mlr.press/v97/geifman19a.html
  5. Madras D, Pitassi T, Zemel R. Predict responsibly: Improving fairness and accuracy by learning to defer. Adv Neural Inf Process Syst. 2018;31:6147-6157. https://proceedings.neurips.cc/paper/2018/hash/09d37c08f7b129e96277388757530c72-Abstract.html
  6. Mozannar H, Sontag D. Consistent estimators for learning to defer to an expert. Proceedings of the International Conference on Machine Learning. PMLR. 2020;119:7076-7087. Accessed June 27, 2026. https://proceedings.mlr.press/v119/mozannar20b.html
  7. Verma R, Nalisnick E. Calibrated learning to defer with one-vs-all classifiers. Proceedings of the International Conference on Machine Learning. PMLR. 2022;162:22184-22202. Accessed June 27, 2026. https://proceedings.mlr.press/v162/verma22c.html
  8. Vovk V, Gammerman A, Shafer G. Algorithmic Learning in a Random World. Springer; 2005. doi: 10.1007/b106715
  9. Angelopoulos AN, Bates S. Conformal prediction: A gentle introduction. Found Trends Mach Learn. 2023;16(4):494-591. doi: 10.1561/2200000101
  10. Angelopoulos AN, Bates S, Fisch A, Lei L, Schuster T. Conformal risk control. International Conference on Learning Representations. In: The Twelfth International Conference on Learning Representations (ICLR 2024). Vienna, Austria; 2024. Accessed June 27, 2026. https://proceedings.iclr.cc/paper_files/paper/2024/hash/f3549ef9b5ff520a7e41ff3cc306ab2b-Abstract-Conference.html
  11. Straitouri E, Wang L, Okati N, Gomez Rodriguez M. Improving expert predictions with conformal prediction. Proceedings of the International Conference on Machine Learning. PMLR. 2023;202:32633-32653. Accessed June 27, 2026. https://proceedings.mlr.press/v202/straitouri23a.html
  12. Verma R, Barrejón D, Nalisnick E. Learning to defer to multiple experts: Consistent surrogate losses, confidence calibration, and conformal ensembles. Proceedings of the International Conference on Artificial Intelligence and Statistics. PMLR. 2023;206:11415-11434. Accessed June 27, 2026. https://proceedings.mlr.press/v206/verma23a.html
  13. Sittig DF, Singh H. A new sociotechnical model for studying health information technology in complex adaptive healthcare systems. Qual Saf Health Care. 2010;19(Suppl 3):i68-i74. doi: 10.1136/qshc.2010.042085
  14. Sendak MP, Gao M, Brajer N, Balu S. Presenting machine learning model information to clinical end users with model facts labels. npj Digit Med. 2020;3:41. doi: 10.1038/s41746-020-0253-3
  15. Caroprese L, Vocaturo E, Zumpano E. Argumentation approaches for explanaible AI in medical informatics. Intell Syst Appl. 2022;16:200109. doi: 10.1016/j.iswa.2022.200109
  16. Guzzo A, Rullo A, Vocaturo E. Process mining applications in the healthcare domain: A comprehensive review. WIREs Data Min Knowl Discov. 2022;12(2):e1442. doi: 10.1002/widm.1442
  17. Tschandl P, Rinner C, Apalla Z, et al. Human-computer collaboration for skin cancer recognition. Nat Med. 2020;26(8):1229-1234. doi: 10.1038/s41591-020-0942-0
  18. Reverberi C, Rigon T, Solari A, et al. Experimental evidence of effective human-AI collaboration in medical decision-making. Sci Rep. 2022;12:14952. doi: 10.1038/s41598-022-18751-2
  19. Obermeyer Z, Powers B, Vogeli C, Mullainathan S. Dissecting racial bias in an algorithm used to manage the health of populations. Science. 2019;366(6464):447-453. doi: 10.1126/science.aax2342
  20. Teeple S, Smith A, Toerper M, et al. Exploring the impact of missingness on racial disparities in predictive performance of a machine learning model for emergency department triage. JAMIA Open. 2023;6(4):ooad107. doi: 10.1093/jamiaopen/ooad107
  21. Anderson JW, Visweswaran S. Algorithmic individual fairness and healthcare: A scoping review. JAMIA Open. 2024;8(1):ooae149. doi: 10.1093/jamiaopen/ooae149
  22. Tabassi E. Artificial Intelligence Risk Management Framework (AI RMF 1.0). National Institute of Standards and Technology; 2023. NIST AI 100-1. doi: 10.6028/NIST.AI.100-1
  23. US Food and Drug Administration. Marketing Submission Recommendations for a Predetermined Change Control Plan for Artificial Intelligence-Enabled Device Software Functions: Guidance for Industry and Food and Drug Administration Staff. 2025. Accessed June 27, 2026. https://www.fda.gov/regulatory-information/search-fda-guidance-documents/marketing-submission-recommendations-predetermined-change-control-plan-artificial-intelligence
  24. European Parliament and Council of the European Union. Regulation (EU) 2024/1689 of 13 June 2024 laying down harmonised rules on artificial intelligence (Artificial Intelligence Act). Off J Eur Union 2024;L:2024/1689. 2024. Accessed June 27, 2026. https://eur-lex.europa.eu/eli/reg/2024/1689/oj/eng
  25. Lekadir K, Frangi AF, Porras AR, et al. FUTURE-AI Consortium. FUTURE-AI: International consensus guideline for trustworthy and deployable artificial intelligence in healthcare. BMJ. 2025;388:e081554. doi: 10.1136/bmj-2024-081554
  26. Adams R, Henry KE, Sridharan A, et al. Prospective, multi-site study of patient outcomes after implementation of the TREWS machine learning-based early warning system for sepsis. Nat Med. 2022;28(7):1455-1460. doi: 10.1038/s41591-022-01894-0
  27. Henry KE, Adams R, Parent C, et al. Factors driving provider adoption of the TREWS machine learning-based early warning system and its effects on sepsis treatment timing. Nat Med. 2022;28(7):1447-1454. doi: 10.1038/s41591-022-01895-z
  28. Levin S, Toerper M, Hamrock E, et al. Machine-learning-based electronic triage more accurately differentiates patients with respect to clinical outcomes compared with the Emergency Severity Index. Ann Emerg Med. 2018;71(5):565-574.e2. doi: 10.1016/j.annemergmed.2017.08.005
  29. Vasey B, Nagendran M, Campbell B, et al. DECIDE-AI expert group. Reporting guideline for the early-stage clinical evaluation of decision support systems driven by artificial intelligence: DECIDE-AI. Nat Med. 2022;28(5):924-933. doi: 10.1038/s41591-022-01772-9
  30. van de Sande D, Chung EFF, Oosterhoff J, van Bommel J, Gommers D, van Genderen ME. To warrant clinical adoption AI models require a multi-faceted implementation evaluation. npj Digit Med. 2024;7:58. doi: 10.1038/s41746-024-01064-1
Share
Back to top
Journal of Clinical Informatics, Published by AccScience Publishing