AIMedily 68: AI Took the History—and Google’s Medical AI Got a Major Test

Researchers evaluated three versions of MedGemma: two that work with text and images, and one focused on text.

They compared MedGemma with the original Gemma 3 models to see whether medical training improved performance.

Some tests used completely new datasets that the models had not seen during training.

They also fine-tuned MedGemma for specific tasks to see how well it could adapt, including when only a small amount of training data was available.

On new datasets, MedGemma improved performance by 2.6–10% for medical image questions, 15.5–18.1% for chest X-ray classification, and 10.8% on simulated agent tasks compared with the original Gemma models.

The 27B text model scored 86.2% on MedQA benchmark, one of the highest results reported for an open model.

In physician-reviewed medical responses, inaccurate answers decreased from 18.5% with Gemma 3 to 15.4% with MedGemma 4B.

For chest X-rays, 81% of AI reports were judged likely to lead to the same or better clinical decisions.

Training the model specifically for medicine helped.

The same model could handle very different tasks, from clinical questions to several types of medical imaging.

Because it is open, developers can adapt it for specific uses and potentially run it locally, which may matter for privacy, cost, and control over data.

The results give a strong foundation for the next step: testing how MedGemma performs in real clinical settings.

172 patients with non-emergency eye conditions were randomized to either an AI agent or an ophthalmology resident for pre-consultation.

The AI asked patients questions using voice or text and automatically created a clinical note.

They evaluated history-taking, communication, documentation, patient satisfaction, and test recommendations.

A senior ophthalmologist then reviewed the chart, examined the patient, and completed the assessment.

The AI scored higher for overall pre-consultation quality, with an adjusted difference of 17.7 points on a 100-point scale.

Patients rated the AI higher for patience and empathy, while residents were better at reducing anxiety. Overall satisfaction was similar.

The AI took much longer: 11.2 minutes compared with 3.1 minutes for residents.

Using history alone, the AI had higher diagnostic accuracy. But once physical examination findings were added the difference became much smaller.

AI collected a more complete history, while residents were much faster and more focused.

This could make AI useful before the visit, helping gather information so physicians can spend more time on the exam and clinical decisions.

But the physical exam still made a difference. Residents gained more diagnostic value from the ocular findings than the AI did.

The U.S. Department of Health and Human Services awarded funding to a project focused on bringing agentic AI into clinical care. (Link)

Google introduced new Earth AI work that combines environmental and health data to help public-health teams anticipate outbreaks and identify vulnerable communities. (Link)

CHAI released a new playbook to help health systems evaluate AI vendors more consistently before adoption. (Link)

Stanford received up to $14.9 million to develop a system that monitors other AI agents being used in cardiovascular care. (Link)

Teladoc added AI tools for ambient documentation and contactless vital signs to its hospital platform. (Link)

Six major U.S. physician organizations said physician judgment should remain central as AI becomes more involved in patient care. (Link)

Nature: A randomized study found that a respiratory AI chatbot helped people answer simulated diagnosis and triage questions more accurately than regular web search. (Paper)

npj: An AI system for breast ultrasound improved radiologist accuracy and reduced missed cancers in a multinational study. (Paper)

npj: A multicenter randomized trial, AI-assisted telerehabilitation was tested in people with early Parkinson’s disease to see whether remotely delivered, AI-supported rehabilitation could improve care. (Paper)

npj: Clinicians preferred AI-generated discharge summaries in longer hospital stays and rated them higher. (Paper)

npj: Researchers found publicly available performance evidence for only 39% of authorized AI diagnostic tools in pathology and hematology. (Paper)

Stanford: An Evidence API that can search more than 100 million scientific papers and quickly surface evidence supporting or challenging a scientific claim. (Link)