Benchmarking Open-Source Large Language Models in Medical French

Large Language Models (LLMs) are increasingly applied in healthcare, yet their evaluation in medical French remains limited. Building on the MedFrenchmark study by Quercia et al. (2024), this work assesses 15 open-source models through a subset of 77 medical questions spanning multiple specialties and reasoning types. All 1,155 generated responses were manually rated on a 0-100 scale for factual accuracy, contextual relevance, clarity, and potential clinical usability. Model performance ranged from 36% to 81%, showing notable differences. While the largest model, GPT-OSS:120B, achieved the highest score, smaller models such as Qwen3:8B and Gemma3:27B reached comparable levels of precision and coherence. Overall, the strongest models demonstrated substantial progress in reasoning and linguistic fluency, although some inconsistencies in terminology and semantic control were still observed. This study offers an updated and transparent overview of open-source LLMs in medical French, underlining both their promising development and their growing potential for future clinical applications.