Performance of Multimodal Generative AI Models in Addressing Complex Dental Inquiries With Text, Images, and Analytical Data

Abstract

Objective Multimodal large language models (LLMs) have the potential to transform dental learning and decision-making by addressing multimodal dental inquiries that integrate text, images, and analytical data. The purpose of this study was to evaluate the performance of various multimodal LLMs in responding to multimodal dental queries and to identify factors influencing their performance.Materials and Methods Four multimodal LLMs (ChatGPT-4V, Claude 3 Sonnet, Microsoft 365 Copilot 2024, and Google Gemini 1.5 Pro) were evaluated based on their correct answers and passing margin for the Integrated National Board Dental Examination (INBDE) and the Advanced Dental Admission Test (ADAT). Descriptive statistics, chi 2 tests, Cohen's kappa, Kruskal-Wallis tests, and Mann-Whitney U tests were used to analyze the performance across different question types, independent inputs, and picture types (alpha = 0.05).Results Claude 3 Sonnet outperformed the other models in both INBDE and ADAT exams, achieving the highest accuracy, followed by ChatGPT-4V, Microsoft 365 Copilot 2024, and Google Gemini 1.5 Pro. chi 2 tests revealed significant differences between chatbots in the ADAT exam, but not in the INBDE. Cohen's kappa showed weak to moderate model agreement for INBDE and stronger agreement for ADAT, with the highest agreement between Claude 3 Sonnet and ChatGPT-4V (kappa = 0.757) and the lowest between Google Gemini 1.5 Pro and Microsoft 365 Copilot 2024 (kappa = 0.059). Model performance was influenced by question type (theoretical and clinical), with common errors including misinterpreting clinical scenarios, visual data difficulties, and dental terminology ambiguities.Conclusion Multimodal LLMs show potential in answering multimodal dental inquiries, though performance varies across models, with challenges in interpreting clinical scenarios, visual data, and terminology ambiguity.Clinical Significance Large language models canbe applied not only to memorization-type but also interpretation andproblem-solving cognitive questions in dentistry. Tomaximize the utility of these artificial intelligence models, users need bothan understanding of their differences and the ability to manage complexclinical data.