Hello,
Do you remember we talked about the Nature Medicine paper comparing OpenEvidence, UpToDate Expert AI, and frontier LLMs?
That study found that frontier LLMs performed better across medical benchmarks and real clinical questions.
This week, a new preprint adds another angle. When real point-of-care questions were reviewed by specialty-matched physicians, OpenEvidence was rated higher in this evaluation (Read here).
So, it will be interesting to see where this conversation goes next.
Let’s dive into today’s issue.
🤖AIBytes
Two clinical studies that deserve a closer look.
AI Decision Support for Hematologic Malignancies
Agent Supports Hematology Boards
This study tested HemaGuide, a locally deployable AI agent for complex hematology cases.
The goal was to see whether it could generate auditable treatment recommendations using clinical records, guidelines, prior tumor board cases, and molecular evidence.
Methods
HemaGuide extracted information from clinical documents and routed cases into guideline, advanced, or molecular modes.
It was tested on 45 high-complexity cases, 70 missense variants, 555 external cases, and a 1-month silent trial with 64 consecutive cases.
Results
HemaGuide had higher concordance than plain LLMs.
In one ablation study, the plain LLM baseline had 0% concordance. The full HemaGuide agent reached 86.7%.
Concordance was 81.8% in the external cohort and 82.8% in the silent trial. In the silent trial, physicians were blinded to HemaGuide’s recommendations, so this measured agreement, not impact on care.
For molecular interpretation, HemaGuide agreed with expert classification in 56 of 70 variants. No oncogenic variant was downgraded to benign.
Hallucinations occurred in 2 of 664 evaluated cases.
Key Takeaways
HemaGuide was tested in several settings, including an external cohort and a silent trial.
The study shows how a case-grounded AI agent was tested for tumor board preparation, but it did not test patient outcomes.
Hallucinations were rare, but still present.
🔗 Zoller J, Kalz M, Wu X, et al. Clinical decision support in hematological malignancies using a case-grounded AI agent. Nature Medicine. 2026. doi:10.1038/s41591-026-04494-4
Physicians and AI Evaluate LLMs Differently
This study compared how physicians and AI agents evaluated LLM-generated responses to real clinical cases.
The goal was to see whether automated evaluators could help scale medical AI evaluation without replacing physician judgment.
Methods
The study included:
421 physicians from 8 medical institutions
421 AI evaluator agents
7 real de-identified clinical cases
5 LLMs: GPT-4o, Claude 3.5, Qwen-Max, DeepSeek-R1, and OpenAI o1
Both groups ranked the answers across 6 areas: comprehensiveness, coherence, accuracy, helpfulness, harmlessness, and structure.
Results
Physician rankings changed by clinical seniority and practice environment.
AI agents showed broad agreement with physicians overall, but they did not fully match physician judgment.
Overall rank concordance: Kendall’s W = 0.798
AI agents had identical rankings in 76.7% of ranking pairs
Physicians had identical rankings in only 3.3% of ranking pairs
AI agents consistently favored OpenAI o1. Physician rankings varied more, with GPT-4o and DeepSeek-R1 often rated higher in areas like accuracy, coherence, and helpfulness.
Key Takeaways
Physicians did not evaluate LLM responses in one uniform way. Experience and clinical setting changed the rankings.
AI agents may help with early screening or benchmarking, but the study found they could not replace physician-centered evaluation.
The study used only 7 Chinese clinical cases, so the findings need validation in other languages and settings.
🔗 Shi P, Li J, Yang Z, et al. Physicians and artificial intelligence diverge in evaluating large language models on real clinical cases. npj Digital Medicine. 2026. doi:10.1038/s41746-026-02942-6
🧬AIMedily Snaps
Fast updates clinicians should not miss.
Meta Brain2Qwerty uses AI to decode brain activity into text (Link).
AMA: Augmented intelligence in medicine and AI’s assistive role (Link).
OpenAI releases GeneBench-Pro for testing scientific AI agents (Link).
UC San Diego launches Institute for Applied Health Intelligence (Link).
Anthropic launches AI drug discovery program (Link).
Mayo Clinic explores the next phase of healthcare AI (Link).
🧪Research Signals
New papers worth your time.
BMJ: AI Tools for polyp detection and characterization (Paper).
Nature: Large language models are powerful electronic health record encoders (Paper).
The Lancet: AI-supported mammography screening: measuring benefit (Paper).
Nature: Bridging the gap from clinical to home ECG (Paper).
The Lancet: Using LLMs to model mental health processes (Paper).
Nature: Better genome standards for precision medicine (Paper).
🦾TechTools
Medical AI tools
Nucs AI — AI platform for PSMA-PET/CT analysis, tumor burden assessment, and prostate cancer treatment monitoring.
Heidi Dictate — Voice dictation for clinicians that works across apps, including EHRs, letters, emails, and documents.
📈 Productivity AI tool of the week:
Lex — AI writing workspace for drafting, editing, and improving text faster.
🧩TriviaRX
A quick question to test your knowledge.
A 19th‑century obstetrician reduced puerperal fever deaths from ~18% to ~1% by forcing handwashing with chlorinated lime, but was ignored and later died in an asylum. Who was he?
A. Ignaz Semmelweis
B. Joseph Lister
C. Louis Pasteur
D. Robert Koch
Now, the answer from last week: ✅ John Snow traced many cholera cases to the Broad Street water pump in London and helped convince authorities to remove the pump handle.
That’s it for today.
Thank you for taking the time to read.
You’re already ahead of the curve in medical AI — don’t keep it to yourself. Forward AIMedily to one colleague.
Itzel Fer, MD PM&R
Forwarded this email? Subscribe free.