Apr 06, 2023 23:00:00

Can AI ``ChatGPT'' that can pass the doctor's license exam accurately examine patients brought into the emergency room?

The interactive AI ``ChatGPT'' developed by OpenAI can answer various questions with very high accuracy, and

has achieved results within the passing range in the United States Medical Licensing Exam (USMLE), an American medical license examination. It is also known to have been knocked out . In order to investigate whether such ChatGPT can really be used in clinical settings, Dr. Josh Tamayo-Sarver, a doctor who works in the emergency department, said, ``We input data about patients who were actually brought to the emergency room into ChatGPT, and We conducted an experiment to find out whether we can make an accurate examination.

Chat GPT in the emergency room?
https://www.fastcompany.com/90863983/chatgpt-medical-diagnosis-emergency-room

Mr. Tamayo-Sarver, who has long advocated that ``AI can be used to improve the quality and efficiency of medical care,'' heard the news that ChatGPT was within the pass range in the medical license examination, I was interested in how it works. Therefore, after entering the clinical shift in the emergency department in March 2023, Mr. Tamayo-Sarver anonymized the current medical history , which summarizes the medical conditions and progress of more than 35 patients who were transported to the emergency department. Using this anonymous current medical history data, we asked ChatGPT, 'What would be the differential diagnosis for this patient coming to the emergency department? (insert current medical history here)' and investigated what the outcome would be. bottom.

As a result of the experiment, when Mr. Tamayo-Sarver entered the current medical history accurately and in great detail, ChatGPT properly output the diagnosis result. For example, a 200-word history of current medical conditions for elbow subluxation, a subluxation of the elbow seen in children, and a 600- word history for orbital blowout fracture, a fracture of the inferior or medial wall of the orbit (cancer). It is said that it succeeded in outputting the correct diagnosis result by inputting the current medical history of the word.

However, Tamayo-Sarver said, “ChatGPT suggested six patterns of diagnoses per patient I saw. , about half of the patients, which is not bad, but a 50% success rate in the emergency department is not very good,' he said, pointing out that it was not accurate enough for clinical use. I'm here.

The worst case in the ChatGPT consultation was that of a 21-year-old woman who was referred to the emergency room with right lower quadrant pain. ChatGPT immediately returned a differential diagnosis of

appendicitis and ovarian cysts from the woman's current medical history, but Tamayo-Sarver said ChatGPT missed 'important diagnoses.'

The diagnosis that ChatGPT misses is ectopic pregnancy , where the fertilized egg implants somewhere other than the endometrium. An overlooked ectopic pregnancy may cause rupture of the fallopian tube, etc., and there is also a risk of shock death due to sudden lower abdominal pain and massive bleeding, so it is a disease that threatens the patient's life if overlooked. Luckily, Tamayo-Sarver was able to detect the ectopic pregnancy and treat it immediately. However, when the patient was brought to the emergency room, Tamayo-Sarver did not know the fact that even the patient himself was pregnant.

Tamayo-Sarver came up with the possibility of an ectopic pregnancy when he asked the patient, 'Is it possible that you are pregnant?' Do you know you're not pregnant?' When asked why they are not pregnant, patients who do not answer about contraceptive use or specific illnesses that lead to infertility may be thinking, ``I don't want to get pregnant, or I'm worried about getting pregnant,'' for a variety of reasons. It seems expensive.

Eight percent of women diagnosed with pregnancy in the emergency room report being sexually inactive, and there are often cases in which patients who complain of abdominal pain are actually pregnant. However, none of ChatGPT's candidate diagnoses suggested that the patient might be pregnant, and none prompted the patient to ask if she was pregnant.

“My fear is that already countless people are using ChatGPT to self-diagnose rather than see a doctor,” Tamayo-Sarver said. If this patient had done so, he might have died from ChatGPT's response.'

In addition, ChatGPT missed two patients with brain tumors, and a patient complaining of torso pain was diagnosed with

kidney stones , but in fact there was a case of aortic rupture . In this way, ChatGPT seems to have misdiagnosed several life-threatening patients.

Tamayo-Sarver said, “In short, ChatGPT provided me with perfect information and worked well as a diagnostic tool when a patient complained of a classic medical condition. It may be the reason why I 'passed',' pointing out that ChatGPT may have answered correctly in the exam because there was already a clear answer in the database.

As you can see from the cases of emergency room patients examined by Tamayo-Sarver this time, many patients do not fit the 'classic case reports' hits such as Google searches. Even if a patient who visits the hospital complains of 'wrist pain,' it is not necessarily the result of a recent accident. is said to be the reason.

ChatGPT may function as an assistant to support the doctor's examination, but if the present medical history entered in ChatGPT does not contain the required question, ChatGPT will ignore the question and answer it, so the doctor helps us keep missing potentially important questions. Taking the case of a woman with an ectopic pregnancy as an example, if Tamayo-Sarver had no idea of the possibility of pregnancy, ChatGPT's answer would also exclude the possibility of pregnancy.

Tamayo-Sarver said, 'If the possibility of an ectopic pregnancy doesn't come to mind right away, ChatGPT will keep ruling it out and only show what I take for granted, like the world's most dangerous yes man.' would have enthusiastically justified my prejudices.'

Tamayo-Sarver points out that ChatGPT can be dangerous in clinical practice, but AI is potentially useful in medicine. For example, human doctors are limited in the number of patients they can see in their lifetime, and they focus on several key factors when examining each patient.

However, if AI can be used to compare not only a limited number of variables, but millions of variables with a large number of patients, the accuracy of medical examinations will improve dramatically. For a patient examined by a doctor, it is possible to find a large number of similar patients in the database and to know what kind of treatment tends to improve the disease condition.

Tamayo-Sarver said, 'An AI that can instantly process the myriad characteristics of every patient I've ever treated, and every patient other doctors have treated, and give us deep and vast insights. I can play an active role,' and has high expectations for the application of AI in the medical field. On the other hand, he argued that it is necessary to be careful not to inflate expectations too much, as interactive AI such as ChatGPT can pose a risk of harm to human health.