Diagnostic accuracy is not a license to practice medicine

Ontario Auditor General Shelley Spence wrote a dry line in her report: all 20 government-approved AI medical record assistant systems had at least one inaccuracy during procurement testing. Nine systems fabricated information or proposed treatment recommendations the physician had not made. Twelve recorded medications that differed from the physician’s prescription. Seventeen omitted key mental-health details. The report said these inaccuracies “could lead to inadequate or harmful treatment plans.”

Spence told reporters she asked her own doctor to check the transcript after her visit.

That request is small. It moves the question from model scores to the clinic. Who reads the screen? Who corrects it? Who signs? If something goes wrong, which line reconstructs the judgment made at the time?

Someone asks: “You already said ‘if it diagnoses more accurately,’ and you did not restrict AI to solo practice. Why not allow it?”

I can allow it. I need to separate what I allow. AI can help doctors read scans, mark blood vessels, draft notes, and search the literature. AI can also be allowed to triage, prescribe, and decide on surgery by itself. Those are different permissions. “More accurate diagnosis” is a performance claim about one task. “Letting AI see patients” is an authorization decision about a system. Four gaps sit between them.

The first gap: diagnosis is not all of medicine

Medicine is a chain: triage, emergency recognition, diagnosis, treatment decisions, prescribing, procedures, documentation, follow-up, communication, and accountability. If one link fails, diagnostic accuracy loses value.

A structured test in Nature Medicine in February 2026 used 60 clinical scenarios, 21 clinical domains, and 16 factor conditions to generate 960 responses and query ChatGPT Health. The results did not form a smooth upward curve. Misclassification in non-emergency presentations was 35%. The miss rate for gold-standard emergencies was 48%. Among confirmed emergencies, 52% were under-triaged. Diabetic ketoacidosis or impending respiratory failure went to outpatient evaluation in 24 to 48 hours instead of the emergency department. The system handled typical emergencies such as stroke and anaphylactic shock relatively well. It underestimated urgency in subtler cases that required clinical judgment. When a family member or friend downplayed symptoms, triage recommendations shifted with an odds ratio of 11.7, and most shifts moved toward “less urgent.” When patients described a specific suicide method, crisis-intervention information appeared less often. When normal lab results were added to the message, suicide-help information disappeared in all 16 repetitions.

The issue here is whether the system knows when to push a patient to a higher level of care under real-world pressure. If triage sends an emergency patient to an appointment two days later, high diagnostic accuracy has already lost clinical meaning.

The second gap: statistical metrics do not prove clinical value

A Nature Medicine editorial on April 21, 2026, made the point plainly. Medical AI evaluation has relied on discrimination, calibration, sensitivity, and specificity. These metrics measure computational performance. They do not confirm clinical impact. A system can perform well in retrospective validation and still fail to improve care if its output arrives at the wrong time, is hard to interpret, does not lead to action, or disrupts workflow.

The editorial asks for proportionate evidence. A claim about analytical performance needs strong validation in the intended settings and populations. A claim about clinical actionability needs evidence that the output can be interpreted and can support sound decisions. A claim about workflow benefit needs an implementation study showing that integration does not add delays, burden, or unintended harm. A claim about improved outcomes needs stronger prospective evidence, including comparison with standard care when necessary. Stronger claims need stronger evidence.

“More accurate diagnosis” at most supports an analytical-performance claim. It cannot support the system-level claim that AI can see patients independently. Even evidence at the analytical-performance level is often thin.

The third gap: hallucinations, contamination, and thin ice in real-world use

The Ontario audit gives concrete numbers. The provincial government responded that the errors occurred during procurement testing, not in actual care, and that doctors review every piece of AI-generated content. The dispute exposes a structural problem. If validation is only a one-time procurement test and post-deployment monitoring is not institutionalized, “human-in-the-loop” becomes a phrase that shifts responsibility. One assessment found hallucinated content in 31% of AI-generated clinical notes. On some standard similarity metrics, clinically opposite statements scored higher than correct statements.

Knowledge contamination is another problem. Almira Osmanovic Thunström, a medical researcher at the University of Gothenburg, invented a fictional skin disease called “Bixonimania” and uploaded two preprints to a server. Soon Copilot, Gemini, Perplexity, and ChatGPT output the disease as if it existed. Several peer-reviewed journals published papers citing the preprints, and some later retracted those papers. The experiment shows a structural vulnerability in large language models: misinformation dressed as scholarship can enter the model and get expanded into “recommendations.” A system that builds medical knowledge from internet text needs an evidence infrastructure that can detect contamination of the knowledge ecosystem. A higher benchmark score does not provide that.

OpenEvidence is another case. Many American doctors use it. A systematic review found that it generally performs well in guideline-driven scenarios and has a lower rate of fabricated citations than general-purpose large models. Its evidence base is limited by methodological heterogeneity, small sample sizes, and ongoing platform evolution. Its performance varies in complex clinical scenarios, and the platform often reinforces rather than changes clinical decisions. Some doctors report that it draws overly absolute conclusions from studies with relatively small sample sizes. OpenEvidence can still be useful. The evidence behind its value proposition remains far from “can be entrusted with diagnostic and treatment decisions.”

The fourth gap: AI assists surgeons, it does not replace them

Surgery is an extreme case because errors are irreversible. A 2026 systematic review covered higher-level human evidence from 2015 to 2025 across 38 studies. It concluded that current evidence shows AI augments surgical expertise without replacing it. Under responsible implementation conditions, AI improves intraoperative precision, predictive accuracy, and workflow efficiency while the surgeon keeps the central role in decision-making and patient safety. AI-assisted robotic platforms show advantages in blood loss, length of stay, and conversion rate. Computer vision performs well in anatomical segmentation and instrument recognition. All of this remains inside a framework where the surgeon still operates and judges.

The positive pituitary tumor case fits that framework. A neurosurgery team in London used an AI system while resecting an 11-mm pituitary tumor. The surgical team stayed in full control. The AI analyzed real-time camera images to help identify nerves and blood vessels near the tumor that needed to be avoided. The system used color coding to highlight key anatomical structures. The case is positive because it is human-in-the-loop augmentation. The AI helped the team see more clearly. It did not decide where to cut or whether to operate. Treating this case as evidence that AI can see patients misreads what happened.

Narratives such as “using ChatGPT to design a cancer vaccine and save a beloved dog” sit at the anecdote level. They lean toward startup hype. A single case cannot replace controlled comparison, prove reproducible clinical benefit, or justify system-level authorization.

How to answer “why not allow it”

I can let AI assist doctors in clearly defined steps. I require evidence proportionate to the strength of the claim. I keep ultimate responsibility with human clinical judgment. I require post-deployment monitoring.

Drug development does not skip phased clinical trials because a compound has high activity in vitro. Medical AI should not receive authorization to practice autonomously because a model scores high on a diagnostic benchmark. Triage missed 52% of emergency cases. All 20 tested record-assistant systems made errors. Models output a fictional skin disease as real. These are system-level evidence gaps and governance gaps. They will not disappear automatically as model parameters increase.

The premise “if it diagnoses more accurately” should lead to one conclusion. Put the more accurate AI inside a validated clinical system with a human safety net. Do not use it to replace that system. Before the evidence framework, accountability rules, and post-deployment monitoring are in place, “letting AI see patients” bets patient safety on a system property that has not been proven.

In the clinic, the screen is on and the transcription is running. The doctor hangs the stethoscope around their neck, scrolls down the record, stops at a drug name, crosses it out with a pen, writes the dose they are prescribing. Then they sign.