Futurism logo

Ontario Audit Finds AI Medical Scribes Hallucinate Referrals and Miss Key Diagnoses

All 20 government approved vendors failed accuracy tests raising patient safety concerns as accuracy counted for only 4 percent of approval score

By Behind the TechPublished 6 months ago • 4 min read

Read Time 6 minutes Tags AI Scribes Healthcare AI Hallucination Medical Records Patient Safety Ontario Audit In recent years many overworked doctors have turned to so called AI medical scribes to help automatically summarize patient conversations diagnoses and care decisions into structured notes for health record logging But a recent audit by the auditor general of Ontario found that AI scribes recommended by the provincial government regularly generated incorrect incomplete and hallucinated information that could potentially result in inadequate or harmful treatment plans that may potentially impact patient health outcomes In a recent report on Use of Artificial Intelligence in the Ontario Government the auditor general reviewed transcription tests of two simulated patient doctor conversations performed across 20 AI scribe vendors that were approved and pre qualified by the provincial government for purchase by healthcare providers All 20 of those vendors showed some issue with accuracy or completeness in at least one of these simple tests including nine that hallucinated patient information 12 that recorded information incorrectly and 17 that missed key details about discussed mental health issues What the audit found One Hallucinated and incorrect clinical data In the report the auditor general points out multiple concerning examples of mistakes in those summaries that could have a direct and negative impact on a patient subsequent care That includes situations where an AI scribe hallucinated nonexistent referrals for blood tests or therapy incorrectly transcribed the names of prescription medication and missed key details of mental health issues discussed in the simulated conversations One particularly alarming example was a hallucinated referral for a total heart removal which illustrates how plausible sounding but false content can enter a medical record if left unchecked Two Scoring system weighted away from accuracy Across all approved vendors the average tested AI scribe scored only a 12 out of 20 on the accuracy of medical notes generated section of Supply Ontario evaluation rubric But that seemingly key accuracy metric was only responsible for about 4 percent of a vendor overall score making it easy to meet the minimum threshold for approval even if an AI scribe scored a zero on the accuracy metric A separate metric measuring domestic presence in Ontario was worth 30 percent of the overall scoring This weighting meant vendors could pass approval without demonstrating reliable clinical accuracy Three Inadequate evaluation process All these factors contributed to the auditor general overall finding that these AI scribes were not evaluated adequately In a display of restraint and understatement the report notes that it is important that AI scribe systems are tested to provide assurances as to the quality of their generated notes and to minimize inaccuracies It also recommends that IT departments using these scribes force doctors to confirm their review of the notes produced before committing them to patient logs Why this matters for clinical practice One Risk to patient safety Incorrect prescriptions missed mental health flags and phantom referrals can lead to delayed treatment unnecessary procedures or medication errors In high volume clinics where doctors rely on scribes to keep up with documentation load the temptation to accept notes without deep review is high Two Automation bias and trust drift When a tool is branded as AI and approved by government procurement processes clinicians may over trust the output A single hallucinated diagnosis can propagate through referrals and specialist consults creating downstream harm that is difficult to trace back to the original error Three Legal and liability exposure If a patient is harmed due to a hallucinated note the question of liability falls on the clinician who signed off the record and the institution that procured the tool Current malpractice frameworks assume human authorship and review but the speed of AI assisted documentation can outpace meaningful human oversight What the audit reveals about procurement One Misaligned incentives The audit shows how procurement rubrics can prioritize policy goals like local economic presence over clinical safety If domestic presence is worth 30 percent and accuracy 4 percent vendors have little incentive to invest in model robustness or clinical validation Two Lack of real world testing The evaluation used two simulated conversations which are a weak proxy for the variability of real patient encounters Accents medical jargon overlapping speech and emotional content all increase error rates in deployed settings Three Absence of continuous monitoring Approval was treated as a one time event rather than an ongoing requirement for monitoring error rates drift and failure modes in production What needs to change One Mandatory human review The auditor general recommendation that doctors confirm their review before notes enter patient logs is a minimum bar but compliance will be uneven without workflow changes that surface discrepancies and require acknowledgment Two Stricter accuracy thresholds Approval should require passing accuracy benchmarks on diverse real world transcripts with penalties for hallucination of medications referrals and diagnoses Vendors should publish error rates by category and allow independent audits Three Clinical context integration Scribes must be tested for handling mental health content medication names and multi party conversations Errors in these domains were the most common and have the highest clinical impact Four Governance and liability clarity Institutions need policies that define who is responsible when an AI scribe introduces an error and how incidents are reported and remediated Without clear lines clinicians will either avoid the tools or use them unsafely Broader implications One Scaling caution Ontario is not alone in piloting AI scribes Health systems in the US UK and Australia are rolling out similar tools under pressure to reduce clinician burnout The audit is a warning that speed of deployment cannot substitute for validation Two Model limitations remain Even state of the art models hallucinate under pressure and struggle with low resource languages rare conditions and sensitive topics Clinical documentation requires higher reliability than general chat applications Three Trust erosion If patients learn that notes may contain fabricated referrals confidence in both clinicians and the health system will erode Trust is already fragile in mental health and chronic care contexts where documentation accuracy matters most For clinicians the takeaway is to treat AI scribe output as a draft not a record For procurement teams the lesson is to weight accuracy and safety far above administrative criteria For regulators the challenge is to create standards that keep pace with deployment Do you think mandatory post encounter clinician review is enough to contain risk or should AI scribes be banned from clinical use until accuracy improves Share your view in the comments

artificial intelligencetech

About the Creator

Behind the Tech

Enjoyed the story? Support the Creator.

Subscribe for free to receive all their stories in your feed.

Subscribe For Free

Reader insights

Comments

There are no comments for this story

Be the first to respond and start the conversation.

Sign in to comment
    Written by Behind the Tech