Artificial intelligence (AI), particularly large language models (LLMs), is transforming numerous fields, including medicine.^1,2 These advanced platforms demonstrate significant potential in supporting diagnosis,^3-5 enhancing medical education,⁶ and advancing research in disease management⁷ and decision-making.⁸ Recent studies have explored the performance of LLMs on medical proficiency tests to evaluate their practical applications.^9-13 These tests simulate real-world clinical scenarios, allowing for the assessment of the accuracy, relevance, and practical utility of LLM-generated responses in healthcare settings. However, most of these studies have focused on English^9,10,12,13 or Chinese,^14,15 with limited research addressing other languages, such as Arabic or Spanish.^16,17

Over half of the global population, including approximately 293 million Portuguese speakers, remains underrepresented in English-centric datasets, posing a major challenge. This lack of representation risks amplifying health inequities in the application of LLMs in medicine. Health disparities are particularly prevalent in regions where English is not the primary language.

This approach underscores a critical gap in capturing the diversity of regional diseases,^18,19 particularly within the Brazilian context. In cardiology — a field requiring specialized knowledge due to the complexity and importance of managing cardiovascular diseases — there is a noticeable lack of detailed analyses on the performance of these tools.

Despite global efforts to reduce these gaps, progress has been uneven, often hindered by slow advancements toward universal health coverage.²⁰ In this context, technology has the potential to play a pivotal role.²¹ The integration of LLMs into healthcare systems could serve as a powerful tool for addressing and mitigating these inequalities.

This study aims to explore the application of LLMs in cardiology by evaluating and comparing the performance of 23 LLM platforms. The assessment was conducted using the Cardiology Certification Exam — an official certification exam for the cardiology specialist title —, administered by the Brazilian Society of Cardiology (SBC). Known for its rigor and scope, this exam evaluates the comprehensive knowledge required for clinical practice in cardiology.

The primary objective is to assess how effectively LLMs can address cardiology-related questions, focusing on their abilities in interpretation, reasoning, and decision-making compared to expert-level standards. Additionally, this study emphasizes the challenges and opportunities of deploying LLMs in Portuguese-language contexts. It aims to offer valuable insights for developing more effective AI tools to improve cardiovascular care and support healthcare professionals in non-English-speaking countries.

The 2023 Cardiology Certification Exam, conducted across five rounds by 23 of the world's leading LLMs in November 2024, resulted in a total of 9,200 outputs. Of these, 44.8% were classified as Type 1, 58.3% as Type 2, 11.9% as Type 3, and 10% as Type 4. All Type 4 responses were classified as incorrect based on the evaluation method. This classification directly reflects how effectively each LLM comprehended the task presented in Portuguese. Table 3 presents the number of each question types for each model.

The performance of each proprietary LLM based on our dataset is presented in Table 4. Notably, none of the models reached the exam's passing threshold of 67%. Among the proprietary models, GPT-4o achieved the highest success rate at 62.25%, followed by Claude Opus with 60,25%, as shown in the Center Illustration. In the medium-sized model category, Claude Haiku achieved a success rate of 47.25%.

The results of the open-source models are presented in Table 5. In the large-sized model category, Llama3 70B Instruct achieved a success rate of 53.50%. In contrast, Llama3 8B, categorized as a small-sized model, achieved a success rate of 36.25%. Despite achieving 35% accuracy, Gemma 7B only provided "A" as the answer, suggesting that the model may have been unable to interpret the questions effectively, instead defaulting to the first option it encountered.

This study evaluated the performance of the leading 23 language models in responding to cardiology-related question in Portuguese. The results, presented in the Central Illustration and Tables 4 and 5, indicate that none of the models exceeded the exam's cutoff score of 67%. This outcome is concerning, especially considering that most of these models achieve scores above 80% when tested in English and within an American medical context.¹⁷

It is important to note that this is a highly challenging specialist exam, with Brazilian practitioners undergoing a minimum of 10 years of rigorous medical training before taking the test. However, the results remain strikingly low, raising concerns about the current capabilities of state-of-the-art LLMs when applied to the Brazilian medical field and its specific context. Figure 2 presents a comparative graph that illustrates the accuracy of the LLMs in relation to their parameter sizes.

Considering the goal of leveraging technology to benefit a broader population, applying LLM models in public hospitals and healthcare centers in Brazil could be a promising approach. These institutions often operate with limited computational resources and face funding constraints that hinder investments in complex systems. As such, the most effective strategy would be to choose an LLM that delivers the best quality while maintaining a minimal parameter count. This would optimize performance within available resources and ensure accessibility to advanced healthcare tools in underserved areas.

The grey area in Figure 2 highlights the LLMs that offer the best cost-benefit ratio for addressing Brazilian medical challenges. Among these, the Llama 3 70B Instruct model demonstrates a solid performance of 53.5% accuracy given its 70 billion parameters. However, the standout model is Claude Haiku, which achieves 47.25% accuracy with only 20 billion parameters. While neither model meets the exam's cutoff score, their results suggest considerable promise, particularly when considering their efficient parameter sizes.

It is important to understand the reasons behind such poor performance. Considering that output types 3 and 4 are generated when the model fails to properly interpret the prompt or command, it can be inferred from Table 3 that models such as Gemma 7B, Llama2 7B, Llama2 13B, Llama2 70B, Llama3 70B, and Mistral 7B struggled with comprehending the command in Portuguese. This indicates that these models faced significant challenges in accurately processing the task in the given language, which led to their lower performance.

Models like Gemma 7B exhibited a particular issue, where they consistently provided only the letter "A" in their answer, indicating that when they partially understood the question, they defaulted to selecting the first available option. Conversely, models like Mistral 7B generated more than 50% of their responses in English, failing to deliver appropriate outputs in Portuguese. This poses a significant limitation for applying these models in the Brazilian public health system, where English responses would be unsuitable and undermine the model's effectiveness in real-world medical scenarios.

Even when the models lack sufficient knowledge to provide a correct answer or fail to understand the question, they often attempt to generate a response, which is frequently inaccurate. In the medical field, this poses a serious risk, potentially leading to misdiagnosis or inappropriate treatment, which could potentially result in misconduct by a healthcare provider. However, some LLMs stood out by acknowledging their limitations, either stating they lacked the necessary knowledge or tools to provide the best answer. These models included all GPT models, Claude Opus, Gemini 1.5 Pro, Llama3 70B, and Mistral 8×7B.

Both proprietary and open-source LLMs performed poorly on the exam, failing to meet the exam's cutoff score, as can be seen in the Center Illustration. In general, although larger models tended to outperform smaller ones, some medium-sized models, such as Llama3 70B Instruct and Claude Haiku, showed relatively strong results given their size. The LLMs demonstrated a lack of specialized cardiology knowledge and struggled with interpreting and providing answers in Portuguese. This reveals a significant gap in global AI capabilities, highlighting the need for further advancements in both domain-specific knowledge and multilingual understanding.

For further studies, a cardiology exam in English could be compared to evaluate whether the primary issue lies in the lack of cardiology knowledge or in the models’ ability to comprehend Portuguese. Parallel investigations are underway in other specialized medical fields, and the development of a Portuguese-specific medical LLM is being considered. However, key challenges in its development include selecting an appropriate training dataset, managing computational costs, and ensuring high-quality responses. This approach aims to pinpoint the underlying challenges and explore potential solutions for improving LLM performance in both medical domains and languages.