ER Doctor: Here’s what he found when he asked ChatGPT to diagnose his patients

Artificial intelligence has been touted as the next big thing in medicine, with potential applications ranging from diagnosing diseases to automating tedious tasks. However, the recent experiment conducted by Dr. Josh Tamayo-Sarver, in which he fed the History of Present Illness notes for 35 to 40 patients into OpenAI’s chatbot, ChatGPT, reveals that AI still has a long way to go before it can be used as a reliable diagnostic tool.

Dr. Tamayo-Sarver’s experiment showed that while ChatGPT was able to correctly diagnose some patients, it failed to diagnose others with life-threatening conditions. In one instance, a 21-year-old female patient came into the ER with right lower quadrant abdominal pain, and ChatGPT’s differential diagnosis suggested appendicitis or an ovarian cyst, among other possibilities. However, the patient had an ectopic pregnancy, which ChatGPT failed to diagnose. This could have been fatal, as ectopic pregnancies can result in death caused by internal bleeding if not diagnosed and treated in time.

The experiment showed that ChatGPT worked well as a diagnostic tool when fed perfect information and the patient had a classic presentation. However, most actual patient cases are not classic, and the vast majority of any medical encounter is figuring out the correct patient narrative. This is where AI falls short, as it is unable to extract all the necessary information required to create the right narrative.

While AI may have the potential to be a useful tool in medicine, it is important to note that it is not a replacement for human physicians. The art of medicine lies in extracting all the necessary information from the patient, and using that information to create a narrative that leads to a correct diagnosis. AI may be able to assist physicians in this process by automating tedious tasks and suggesting differentials, but it cannot replace the human element of medicine.

There are also concerns that AI may introduce even worse outcomes if used as a doctor’s assistant. If AI is used to automatically read patient notes during treatment and suggest differentials, it may encourage physicians to miss important questions that are not included in the patient notes. This could result in missed diagnoses and worse outcomes for patients.

It is also important to note that AI is only as good as the data it is trained on. If the data is biased or incomplete, then the AI will be as well. This is particularly concerning in the field of medicine, where biases and disparities already exist. For example, a recent study found that an AI system used to predict which patients would benefit from extra care in a hospital was biased against black patients.

Despite these limitations, AI still has the potential to be a valuable tool in medicine. For example, AI could be used to help physicians analyze medical images and detect abnormalities that may be difficult for the human eye to see. AI could also be used to automate tedious tasks such as transcription and data entry, freeing up physicians to focus on patient care.

However, before AI can be fully integrated into medical practice, there are several challenges that need to be addressed. First and foremost, the technology needs to be more accurate and reliable. This will require better data and algorithms, as well as more rigorous testing and validation. Additionally, there needs to be a greater emphasis on transparency and accountability, so that patients and physicians can understand how AI is being used and make informed decisions about their care.

In conclusion, while AI has the potential to revolutionize medicine, it is not a replacement for human physicians. The art of medicine lies in extracting all the necessary information from the patient, and using that information to create a narrative that leads to a correct diagnosis. AI may be able to assist physicians in this process, but it cannot replace the human element of medicine. It is important to continue to explore the potential applications of AI in medicine