Evaluating the Feasibility and Accuracy of Large Language Models for Medical History-Taking in Obstetrics and Gynecology

In pre-diagnostic settings, effective doctor-patient communication is crucial for gathering essential information, while time-consuming and privacy-sensitive for some health issues. Large Language Models (LLMs) have shown potential in supporting these conversations by streamlining repetitive tasks and enhancing patient comfort.

By utilizing ChatGPT-4o's advanced audio conversational capabilities, this study aims to evaluate its feasibility, effectiveness and accuracy in medical history-taking interactions in obstetrics and gynecology. ChatGPT-4o were set to play both the patients and doctors using 100 cases, simulating the daily doctor-patient communication and generating medical records. The feasibility of using AI-agent for pre-diagnostic medical history-taking was assessed.

We expect the AI-driven conversation to yield a complete and coherent medical record, capturing essential information with minimal human intervention. Full experiments are under progress. However, preliminary results are promising based on the available data collected. The doctor AI agent performed nearly the same as a realistic physician, asking in a professional and logical sequence and engaging patients with a comfortable and respectful tone. The final assessment will include an evaluation of logic, readability, and empathy in the LLMs-generated records and practitioners’ evaluation of accuracy.

Preliminary tests indicate that ChatGPT-4o can produce coherent and structured medical records, though adjustments may be needed to improve accuracy in capturing nuanced medical details. Implementing LLM for history-taking may reduce physicians' workload by automating routine tasks, while enhancing patient comfort in sharing private information. ChatGPT-4o 's effectiveness in automating medical history-taking is evaluated, with potential for improving efficiency and data accuracy in clinical settings.