The Verge – no paywall: “Earlier this month, The Guardian published an investigation that showed Google was serving up misleading and outright false information via its AI overviews in response to certain medical inquiries. Now those results appear to have been removed. According to the original report:
In one case that experts described as “really dangerous”, Google wrongly advised people with pancreatic cancer to avoid high-fat foods. Experts said this was the exact opposite of what should be recommended, and may increase the risk of patients dying from the disease.
In another “alarming” example, the company provided bogus information about crucial liver function tests, which could leave people with serious liver disease wrongly thinking they are healthy. As of this morning, the AI overviews for questions like “what is the normal range for liver blood tests?” have been disabled entirely. Google declined to comment on the specific removal. But this is just one more controversy for a feature that has told people to put glue on pizza, eat rocks, and been the subject of multiplelawsuits…”
See also Ars Technica an 8, 2025: ChatGPT Health lets you connect medical records to an AI that makes things up. “New feature will allow users to link medical and wellness records to AI chatbot. On Wednesday, OpenAI announced ChatGPT Health, a dedicated section of the AI chatbot designed for “health and wellness conversations” intended to connect a user’s health and medical records to the chatbot in a secure way. But mixing generative AI technology like ChatGPT with health advice or analysis of any kind has been a controversial idea since the launch of the service in late 2022. Just days ago, SFGate published an investigation detailing how a 19-year-old California man died of a drug overdose in May 2025 after 18 months of seeking recreational drug advice from ChatGPT. It’s a telling example of what can go wrong when chatbot guardrails fail during long conversations and people follow erroneous AI guidance. Despite the known accuracy issues with AI chatbots, OpenAI’s new Health feature will allow users to connect medical records and wellness apps like Apple Health and MyFitnessPal so that ChatGPT can provide personalized health responses like summarizing care instructions, preparing for doctor appointments, and understanding test results. OpenAI says more than 230 million people ask health questions on ChatGPT each week, making it one of the chatbot’s most common use cases. The company worked with more than 260 physicians over two years to develop ChatGPT Health and says conversations in the new section will not be used to train its AI models…”
