Define the job before judging the answer
A tool may be asked to summarise records, explain an assay, retrieve literature, compare diagnoses or discuss treatment evidence. Decide which job you are evaluating. A short answer can be excellent for one purpose and inadequate for another.
Write down the intended user and setting as well. An explanation for a family preparing questions has different communication needs from a specialist's molecular-pathology discussion. These differences should shape the assessment without changing the underlying case facts.
Keep the inputs comparable
Give each tool the same version of the case and the same task. Record the date, available settings and whether search or other retrieval was used. If a tool receives additional information, retain it in the comparison record.
Save the complete outputs. An excerpt can be useful for teaching, but it should not replace the material needed to understand how a score was assigned. If a revision is requested, distinguish the first answer from the revised answer.
Assess several separate qualities
| Dimension | A practical question |
|---|---|
| Case fidelity | Did it preserve the timeline, units, treatment status and measured findings? |
| Diagnostic synthesis | Did it identify and compare the important explanations? |
| Experimental interpretation | Did it explain what each experiment actually identifies? |
| Evidence | Do the references exist and support the claims attached to them? |
| Consequences | Do the suggested questions or next steps follow from the findings? |
| Communication | Can the intended reader understand the main point and its basis? |
An overall score can summarise these judgments, but it should not hide a serious error in one dimension. Publish the scoring criteria and examples of what each level means.
Verify the references separately
Check consequential claims against their cited sources. A citation audit evaluates the source relationship; it is a separate task from deciding whether the overall explanation is clinically persuasive. The reference-checking guide provides a practical method.
Historical research has documented fabricated and erroneous AI references. That supplies a reason to check references directly, not a current error estimate for a different tool or configuration. [1]
Keep improvement cases and evaluation cases apart
A case used to revise a prompt has become part of development. Its improved score shows that the revision helped on that example. It is not an independent estimate of performance on future cases.
Reserve additional cases that were not used to choose or refine the instructions. Keep the evaluation criteria fixed while comparing versions, record repeat runs when variability matters and include failures rather than selecting only the best answer. If independent raters are used, describe their expertise and whether they knew which tool produced each answer.
These are proposed assessment practices for this publication. They are not a claim that MontMedical has already completed an independently validated clinical study.
Distinguish written performance from clinical benefit
A synthetic-case comparison measures performance on a selected written task. It does not establish what happens when a tool is used during real care. DECIDE-AI addresses reporting of early live evaluations of AI decision support, where clinical context and human use matter. [2]
TRIPOD+AI addresses transparent reporting for clinical prediction models. Its emphasis on clear reporting is relevant to assessing published claims, but a language-model answer comparison is not automatically a prediction-model study meeting that guideline. [3]
The distinction helps readers understand what an evaluation actually contributes. A carefully annotated example can be valuable educational material without supplying a population-wide accuracy figure.
Include practical performance
Record time to the first useful response, time to the completed review, the information available to the tool and how failures are handled. Check whether completed work remains available and whether a failed section can be distinguished from an unfinished section.
These observations matter to usability. They should be reported alongside clinical-content assessment rather than used to compensate for an unsupported medical conclusion.
Apply the comparison to MontMedical
MontMedical provides an initial discussion, a referenced evidence review and an optional treatment literature search. Its public worked examples illustrate the reasoning questions this library teaches. They are curated educational cases, not a clinical-validation dataset or proof of superiority over another product.
Use the criteria above to examine a case in MontMedical, then check the evidence and reasoning yourself. Explore a clinical case or read how this publication reviews an explanation.