When Rebecca Green arrived for her first appointment with a urologist and was asked whether she would agree to having an artificial intelligence system document the conversation, she said yes without hesitation. She did not want to appear difficult or take up more of her doctor’s time.
Only later did she discover that a serious and completely false claim had entered her medical record, noting that she used psychedelic mushrooms.
Gallery
AI transcription of doctor-patient conversations brings major benefits, but also significant challenges and risks. Illustration
(Photo: Shutterstock)
Green found the error after undergoing surgery for kidney stones in March and reading a letter the specialist had sent to her family doctor. The letter said she was microdosing psychedelic mushrooms and suggested that this could explain previous bleeding around the kidney. "I was gobsmacked and in tears. I have never done mushrooms in my life," Green told Australia’s ABC.
For Green, the mistake was more than embarrassing. At the time, she was receiving compensation following a workplace injury and feared that a reference to illegal drug use in an official medical document could affect her case.
She also described the experience as impersonal. At a time when she was injured and needed treatment, she said she felt that the doctor did not really know her or fully understand her condition.
After Green complained, the urologist apologized and corrected the medical correspondence. The doctor wrote that she takes the accuracy of medical documentation seriously but could not determine exactly how the reference to psychedelic mushrooms entered the letter. She said it appeared to stem from an error in the dictation or transcription process.
She also said she would review the way AI was being used in her clinic, though she did not answer ABC’s questions about what changes had been made following the incident.
Green said she felt fortunate to have noticed the error herself and worried that other patients might miss mistakes that could affect their health, employment or insurance. A proper review before the letter was sent, she said, could have caught the problem.
"I love AI, but the essential ingredient is the human," Green told ABC, adding that in her view the main failure was the lack of human involvement.
When AI records the wrong breast
AI medical scribes listen to conversations between doctors and patients, transcribe them and convert the discussion into structured clinical documentation. Depending on the system, the output can be a visit summary, referral letter, discharge letter or note entered directly into the medical record.
Their use is expanding rapidly. In an online survey conducted by the Royal Australian College of General Practitioners in November 2025, 40% of respondents said they regularly used AI scribes, up from 22% in August 2024.
The attraction is clear. Instead of typing throughout an appointment or finishing notes after working hours, doctors can devote more attention to the patient. A study published in JAMA Network Open involving 263 physicians and other health professionals across six U.S. health systems found that after 30 days of using an AI scribe, reported burnout fell from 51.9% to 38.8%.
Participants also reported lower cognitive workload and a greater ability to focus on patients.
The study, however, had important limitations. It compared participants before and after adoption, did not include a control group and relied heavily on self-reported outcomes.
Doctors can devote more attention to the patient
(Photo: Shutterstock)
The technology also introduces a different set of risks. AI scribes can omit information, attribute comments to the wrong person, confuse medical terminology or add details that were never mentioned.
A report published in July by Digital Rights Watch cited, among other examples, a system that identified the wrong breast in the record of a patient with breast cancer and another that incorrectly recorded an epilepsy diagnosis for a patient who did not have the condition.
Dr. Elizabeth Deveny, CEO of the Consumers Health Forum of Australia, said patients sometimes discover such errors by chance and can then struggle to determine where the information came from and how to have it corrected or removed.
She also warned of the risk of automation bias. A doctor may closely review every AI-generated note when first using the technology, but over time that scrutiny can weaken as confidence in the system grows.
Not all AI scribes perform equally. An Australian study published in April compared four commercial AI scribe systems with notes written by doctors during four simulated consultations.
The AI systems received similar and, in some cases, higher scores than the doctors, though the overall difference was not statistically significant. One system was also more prone to including information that had never appeared in the conversation.
The researchers emphasized that the sample was small and that the technology still needs to be tested more extensively in real clinical settings.
Can patients really refuse?
Another concern is whether patients are genuinely giving informed consent. Digital Rights Watch found that in some clinics, the only notice that AI was being used appeared on a sign in the waiting room. In others, consent was incorporated into the online appointment-booking process.
The report also found cases in which patients were told that if they did not agree to the use of the system, they would need to see another provider.
The organization examined the privacy policies of 60 Australian health care providers identified as using AI scribes. Only four explicitly stated that such systems were in use.
The privacy issue extends beyond data storage. Knowing that an outside system is listening may change what patients are willing to disclose, particularly during consultations involving mental health, domestic violence, sexual health or drug use.
How did a psychedelic mushroom end up in the medical record? Ask the AI
(Photo: Shutterstock)
Guidelines issued by the Australian Health Practitioner Regulation Agency, AHPRA, make clear that responsibility remains with the clinician. Doctors are required to obtain informed consent, exercise human judgment and check the accuracy and relevance of AI-generated documentation before it is saved or sent.
The regulatory status of AI scribes is still evolving. In Australia, a system intended only to transcribe and summarize a conversation is generally not considered a medical device. But if it analyzes the discussion or adds a diagnosis, test or treatment recommendation that the doctor did not make, it may be classified as a medical device and require approval from the Therapeutic Goods Administration (TGA).
The regulator is now examining systems marketed in Australia and whether some may effectively be offering capabilities beyond simple transcription without the necessary approvals.
Similar tools are already being tested in Israel
Green’s case illustrates what can happen when AI-generated information enters a medical record without adequate review. But Prof. Ran Balicer, deputy CEO and chief innovation officer at Clalit Health Services, says the risks need to be weighed against the reasons these systems are being adopted so quickly.
“First, it should be said that the use of automated scribes in the doctor’s room is increasingly becoming routine in the United States and other countries,” he said. “Leading companies worldwide have reached a level of quality and accuracy that allows for high-quality transcription in the vast majority of cases, and the emphasis here is on the English language.”
Prof. Balicer said the main promise of the technology is that it can free doctors from administrative work and improve the interaction with patients. “The promise of the automated scribe is also to free the clinician from the keyboard, allow a free-flowing conversation with the patient without interruptions, and capture everything that was said in the medical record without having to remember it afterward,” he said.
Prof. Ran BalicerPhoto: Tal ShaharIn Israel, adoption has taken longer because free-flowing Hebrew and medical jargon are more difficult to transcribe accurately. Prof. Balicer said tools capable of handling those challenges now exist and several are being tested across Israeli health systems, including in Clalit healthcare clinics and hospitals.
But no system is perfect. Prof. Balicer divides the possible failures into three main categories: omitting information that was said, adding information that was never said, sometimes described as an AI "hallucination", and replacing a correct detail with an incorrect one, such as a number, dosage or side of the body.
“Every tool based on artificial intelligence will make mistakes at some rate,” he said. That makes human review essential. But reviewing the output is not necessarily easy. When nearly all of a note is accurate, a single incorrect detail can be difficult to spot.
“Think about how much detective work and concentration it takes to look for the needle in the haystack when, most of the time, there is no needle,” Prof. Balicer said.
Who is responsible for checking the AI
(Photo: Shutterstock)
The rapid spread of these systems raises questions that have yet to be fully answered. How often does each system make errors? How serious are those mistakes? Who should evaluate the technology before it is introduced? And who is responsible for monitoring it once it is in routine use?
Prof. Balicer said regulators across Western health systems are grappling with the same problem. AI systems are fundamentally different from drugs, which can be tested and approved through a relatively fixed process. AI products can change over time, and their performance may vary depending on where and how they are deployed.
“These questions apply not only to scribe-type tools, but to all AI systems entering health care organizations,” he said. “The question is who is responsible for carrying out the testing and evaluation process to make sure they are safe, responsible and that their error rates are acceptable for the specific use case.”
Centralized regulation is particularly difficult, he added. “It is not like approving a drug, where you approve it once and it works the same way everywhere,” Prof. Balicer said. “Regulation of these tools involves a great deal of dynamism and significant differences from one implementation site to another, which makes centralized approval very difficult.”
To address that challenge, Clalit developed an internal framework called Optica for evaluating and monitoring AI systems. “We created an organizational process in which every AI product entering Clalit Health Services undergoes a checklist of up to 77 items, covering all aspects including safety, privacy protection, accuracy and continuous monitoring of the product over the years, because it may change,” Prof. Balicer said.
He added that the World Health Organization recently cited Optica as one of the leading frameworks health systems can use to evaluate and responsibly implement AI solutions.