Two published comparisons worth examining

There is published evidence that OpenEvidence can underperform leading general-purpose models. That is a stronger, more useful claim than assuming that a medical-only product must be the best clinical reasoner. It also needs a task, a date and a defined outcome.

Nature Medicine: knowledge, answer quality and real clinician queries

Vishwanath and colleagues evaluated 500 MedQA items, 500 HealthBench items and 100 real clinician queries. Twelve clinicians reviewed the latter blind. [1]

Tested system MedQA accuracy Mean clinician rating, 1–4
Gemini 3.1 Pro Preview 97.4% 3.62
GPT-5.2 94.2% 3.54
Claude Opus 4.6 90.2% 3.52
OpenEvidence 89.6% 3.24

Gemini and GPT significantly exceeded OpenEvidence on MedQA; Claude's small numerical edge was not such a demonstration. Frontier models led the clinician ratings; clarity was OpenEvidence's weakest dimension. Harmful-content and hallucination flags did not differ significantly.

Public-benchmark exposure, model-based HealthBench judging, interface differences and questions from a general-model deployment limit generalisation. These are answer-level results, not patient outcomes. September correspondence challenged the breadth of the conclusions. [2]

Refractive surgery: a substantial gap on one examination set

Wallerstein and colleagues' 26 October 2026 online publication tested 225 AAO BCSC refractive-surgery multiple-choice questions. OpenEvidence scored 76%; Claude 92%; Grok 89.8%; ChatGPT-5, Copilot and Gemini 88% each; and ChatGPT-4o 86.2%. All six general models significantly exceeded OpenEvidence on that set. The 1,983 programme users averaged 77%. [3]

The authors explicitly limit the result to this task. Exam accuracy does not directly measure open-ended clinical performance. We inspected the indexed abstract; the full publisher text was unavailable, so model versions not specified there should not be guessed.

The counter-evidence belongs in the same discussion

Feng and colleagues' 27 June 2026 preprint used 620 physician queries from OpenEvidence and 187 HealthBench questions. In specialty-matched comparisons by 149 physicians, OpenEvidence led GPT-5.5, Claude Opus 4.8 and Gemini 3.1 Pro across five rated dimensions. The primary win-minus-loss margins were 25–39 percentage points, not accuracy percentages. [4]

OpenEvidence co-developed collection, implemented it and paid respondents. The authors reported no OpenEvidence affiliation, prespecified analysis and blinded analysis of model identities. Those disclosures should travel with the result.

This linked preprint version uses different questions, models and judging from the Nature study. It cannot be treated as a replication that either erases or confirms the earlier results.

What a buyer should conclude

The studies undermine an assumption of automatic specialist-AI superiority. They do not establish that OpenEvidence is universally weak, that frontier systems never make clinical mistakes, or that one architecture is permanently ahead.

Before choosing a tool, ask whether its evaluation resembles the work you need:

  • Does it interpret a difficult clinical narrative, retrieve a fact, or select a multiple-choice answer?
  • Do judges inspect the source support, the next action and the experiment's meaning?
  • Are the questions new to the system, and were answers judged without product labels?
  • Are the actual versions, prompts, failures, latency and costs retained?

Why this matters to MontMedical

MontMedical's case for value is the way its workflow turns findings and research into an inspectable clinical explanation. A published result for a base model is not validation of a product built around models. Neither paper, nor the preprint, tested MontMedical.

Our eight retained comparisons therefore make a narrower argument. In the prothrombin and red-cell channel cases, MontMedical more precisely distinguishes what the experiment establishes from the causal mechanism still being tested. Other cases show agreement, and the CANOMAD pair shows OpenEvidence presenting urgency more effectively.

That is the evidence readers can examine today: particular answers, particular inferences, and the sources behind the critique. A credible future performance claim needs a frozen case set, complete inputs, independent blinded review and results that include failures. See how we compare medical AI.