Comparative analysis of multimodal large language models GPT-4o and o1 versus clinicians in clinical case challenge questions: Retrospective cross-sectional study

Abstract

Generative pretrained transformer 4 (GPT-4) has demonstrated strong performance in standardized medical examinations but has limitations in real-world clinical settings. To address these limitations, the multimodal GPT-4o model integrates text and image inputs, and the multimodal o1 model incorporates advanced reasoning. This study compared the performance of GPT-4o and o1 against that of Medscape respondents (majority vote) in real-world clinical case challenges. This retrospective, cross-sectional study used 1426 Medscape case challenge questions from May 2011 to June 2024. Each case included text and images of patient history, physical examinations, diagnostic tests, and imaging studies. Medscape respondents were required to choose 1 answer from among multiple options, with the most frequent response defined as the Medscape respondent&apos;s decision. GPT models (3.5 Turbo, 4 Turbo, 4 Omni, and o1) were used to interpret the text and images and generate formatted responses. We compared the performances of the Medscape respondents and GPT models using mixed-effects logistic regression analysis. Medscape respondents (majority vote) achieved an overall accuracy of 85.0%, whereas GPT-4o and o1 demonstrated higher accuracies of 88.4% (P = .005) and 94.3% (P < .001), respectively. In multimodal analysis involving images (n = 917), GPT-4o achieved an accuracy of 88.3% (P = .005), while o1 achieved 93.9% (P < .001), both significantly outperforming Medscape respondents. o1 demonstrated the highest accuracy across all question categories, achieving 92.6% (P < .001) in diagnosis, 97.0% (P < .001) in disease characteristics, 92.6% (P = .002) in examination, and 94.8% (P = .005) in treatment. In terms of medical specialty, o1 achieved 93.6% (P < .001) accuracy in internal medicine, 96.6% (P = .030) in major surgery, 97.3% (P = .030) in psychiatry, and 95.4% (P < .001) in minor specialties, significantly surpassing Medscape respondents. Across 5 trials, GPT-4o and o1 provided the correct answer 5/5 times in 86.2% and 90.7% of the cases, respectively. The GPT-4o and o1 models achieved higher accuracy than Medscape respondents (majority vote) in clinical case evaluations, particularly in disease diagnosis. GPT-4o and o1 could serve as valuable tools to assist healthcare professionals in structured scenarios.