Lancet study tests Google's AMIE chatbot with real primary care patients

Researchers at Beth Israel Deaconess Medical Center (BIDMC) and Google have published what they describe as the first prospective, real-world study of a patient-facing conversational AI in primary care. The work was published Oct. 8 in The Lancet, according to Harvard Medical School.

The study matters because chatbots have mostly been tested in simulations. The authors say it is a step toward judging the technology with the same kind of clinical testing used for other health care interventions. It was designed to test feasibility and safety, not health outcomes.

What the study tested

The team evaluated the Articulate Medical Intelligence Explorer (AMIE), a medical AI chatbot that patients used from home after booking an urgent primary care appointment with a BIDMC physician. The research was done in collaboration with researchers at Google.

According to the BIDMC announcement, the system used a secure text chat. It asked about symptoms, took medical histories and offered possible diagnoses for patients to discuss with their doctor. It also wrote a summary for the clinician before the visit.

The study ran from April to November 2025 and enrolled 114 patients, 98 of whom completed both the AI interaction and their primary care appointment. The paper is titled "Conversational diagnostic artificial intelligence in ambulatory primary care: a prospective feasibility study".

Safety findings

A physician watched every conversation live. A board-certified internal medicine physician, designated the AI supervisor, monitored the interactions in real time. BIDMC says that physician could step in if a patient seemed at risk, became distressed, asked to stop, or if a safety concern arose.

According to BIDMC, none of the 98 completed conversations needed a safety-stop intervention. Supervising physicians identified one hallucination. They added clinical clarification in five cases.

Patient and doctor reactions

BIDMC says patients and physician evaluators rated conversation quality favorably on most measures. Patient attitudes toward AI improved after using the chatbot and stayed higher after the clinician appointment.

Two concerns remained. Patients were less sure about the confidentiality of what they shared. They also questioned the chatbot's honesty and trustworthiness, BIDMC said.

Primary care providers reviewed an AI-generated transcript or summary in 44 cases. Physicians said it helped them prepare in 75% of cases and may have influenced their approach in 57%, BIDMC reported.

In one case, a provider called the interaction somewhat harmful. According to BIDMC, the provider worried that a patient may have become anxious after AMIE listed lymphoma among possible diagnoses.

What the study does not show

The authors stressed the study was built to assess safety and feasibility, not whether AI improves health outcomes. It was a single-centre, single-arm study with no comparison group, according to ASCO's AI coverage.

Patients also used the chatbot only after booking an urgent appointment, with a physician supervising. The results do not cover unsupervised use.

Funding and ties matter here too. Google researchers collaborated on the evaluation. BIDMC states the study was funded by Alphabet. It adds that lead author Adam Rodman served as a visiting researcher at Google during part of the work.

Another recent preprint on a related question reached a more mixed result. A standardized-patient field experiment in China found that AI chatbots had higher diagnostic accuracy than primary care physicians. They were also substantially more likely to recommend unnecessary tests and inappropriate medications. That study tested different chatbots in a different setting, so the two are not directly comparable.

What the authors say

Co-first author Peter Brodeur said the study helps establish baseline characteristics of real-world AI conversations. He said they offer a foundation for future trials on when human-on-the-loop workflows may be appropriate.

Co-senior author Marc Cohen said the study can serve as an example of how to assess AI safety before clinical rollout. He said BIDMC expects clinicians to stay involved in care.

What happens next

Google said larger clinical trials are needed to assess patient-facing AI at scale. The BIDMC team said future work should examine which design features build patient trust. It should also examine whether such tools can strengthen the patient-physician relationship.

Questions about workload, outcomes and unsupervised use remain untested in this study. The authors say those will need larger trials.