Futurism logo

AI Medical Scribes Show Hallucinations in Testing Raising Safety Concerns for 5000 Ontario Doctors

All 20 government approved platforms produced inaccuracies in procurement tests though officials say errors have not appeared in live visits

By Behind the TechPublished 6 months ago • 4 min read

Read Time 6 minutes Tags AI in Healthcare Medical Scribes AI Hallucination Patient Safety Ontario Health AI Ethics Clinical Documentation If you ve been to a medical appointment in the past two or three years chances are high that your doctor was using an AI scribe software that listens into the conversation transcribing it and structuring it into the format of medical notes In theory it s a cool idea but pain points abound Earlier this week Ontario s auditor general an accountability officer acting under the Legislative Assembly of Ontario released a special report warning that AI medical scribes were not evaluated adequately and may present fabricated information to medical professionals First reported by Global News the audit took a look at 20 AI scribe platforms and found that all AI scribe systems from the 20 government approved vendors showed one or more inaccuracies at the procurement testing phase such as hallucinations fabrication incorrect information or missing or incomplete information Inaccuracies in medical notes generated by AI Scribe systems could potentially result in inadequate or harmful treatment plans that may potentially impact patient health outcomes the report declared The gap between testing and deployment Muddying the waters Ontario s Minister of Public and Business Service Delivery and Procurement Stephen Crawford noted that the hallucinations were observed during testing by state regulators and had not been recording during actual medical visits Let s be very clear about that s not actually in operational use with doctors that s in the optional stage where we re reviewing the various scribes Crawford told Global News Still the auditor general Shelley Spence noted that the various scribes are nonetheless in use by around 5000 doctors across Ontario Talking to reporters Spence said she went so far as to ask her physician to please look at the transcript when you re done with my own visit That news comes as another AI scribe system OpenEvidence faces growing scrutiny in the US over hallucinations and incomplete answers As several doctors told NBC News for example OpenEvidence can occasionally draw overly strong conclusions from medical studies with relatively small sample sizes While many physicians express appreciation for the new tool it remains to be seen how they fare under real world conditions and how the medical world will judge them once the AI hype wears off Why hallucinations matter in clinical notes One Clinical risk Clinical documentation drives diagnosis treatment plans medication orders and referrals If an AI scribe inserts a symptom that was never mentioned or misses a critical allergy the downstream decision can be harmful The auditor general warned that inaccuracies could potentially result in inadequate or harmful treatment plans Two Liability and trust Physicians remain legally responsible for the note even if it was generated by AI This creates a burden to review and edit every transcript which can erode the time savings that made scribes attractive in the first place If doctors stop reviewing due to fatigue the risk of error compounds Three Evaluation blind spots The audit found problems during procurement testing but not in live use That could mean testing protocols are more adversarial than real visits or it could mean errors are occurring and going undetected The lack of routine post deployment auditing is a gap Four Vendor diversity did not reduce risk All 20 approved vendors showed issues This suggests the problem is not limited to one model or one company but is systemic to how current large language models handle free form medical conversation Patterns seen in other AI deployments One Overconfidence on weak evidence OpenEvidence has been criticized for drawing strong conclusions from small sample studies This mirrors a broader issue where models present probabilistic output with deterministic certainty A physician unfamiliar with the underlying literature may not catch the overreach Two Data leakage and context errors Models can conflate patients mix up temporal sequences or insert plausible but false details These errors are hard to detect because they read naturally and fit the clinical tone Three Feedback loops If inaccurate notes are fed back into training or used for downstream decision support the errors can propagate across systems What would make deployment safer One Mandatory human review Every AI generated note should require a clinician sign off with a clear audit trail showing what was edited and why Review should be tracked as a quality metric not just an optional step Two Real world evaluation Ontario s testing occurred at procurement but ongoing monitoring in live settings is needed Logs of corrections discrepancies and adverse events linked to scribes should be collected and analyzed quarterly Three Clear scope limits Scribes should be restricted to transcription and structure leaving diagnosis and plan generation to the clinician Models should be tuned to refuse when audio is unclear or when critical information is missing rather than fill gaps with plausible guesses Four Vendor transparency Regulators should require vendors to publish error rates on standardized test sets and to disclose training data sources and safety mitigations Black box procurement makes it impossible to assess risk Five Patient notification Patients should be informed when an AI scribe is used and given the right to request a human reviewed transcript This aligns with informed consent norms for other technologies used in care Broader implications for AI in medicine One Automation bias Physicians may over trust AI output especially under time pressure The design of the tool should make errors visible and easy to correct not buried in long transcripts Two Workflow impact If review takes longer than manual note taking the technology fails on its primary value proposition Adoption will stall unless accuracy improves or review workflows are streamlined Three Regulatory precedent Ontario s audit is one of the first government level evaluations of AI scribes at scale Other jurisdictions will likely follow How Canada handles vendor approval monitoring and liability will shape global standards Four Public trust Trust in clinical encounters depends on patients believing the record reflects what was said If hallucinations become public the backlash could slow adoption of beneficial AI tools across healthcare For clinicians the immediate takeaway is to treat AI scribe output as a draft not a final record For health systems the takeaway is to invest in monitoring and governance before scaling For vendors the takeaway is that accuracy on clinical transcription is a prerequisite not a nice to have Do you think AI scribes can become reliable enough for routine use with proper safeguards or is the risk of hallucination too high for clinical documentation Share your view in the comments

artificial intelligencetech

About the Creator

Behind the Tech

Enjoyed the story? Support the Creator.

Subscribe for free to receive all their stories in your feed.

Subscribe For Free

Reader insights

Comments

There are no comments for this story

Be the first to respond and start the conversation.

Sign in to comment
    Written by Behind the Tech