Alongside impressive successes, medical AI has also seen well-documented failures. They show which risks need to be taken seriously – not in order to reject AI, but in order to use it safely.
1. Biased data, biased results
An AI learns from the data it is trained on – including its imbalances. A well-known example was described in a study in Science (Obermeyer and colleagues, 2019): an algorithm widely used in the USA was intended to identify patients with high care needs. Because it used previous treatment costs as its yardstick, and on average less money had been spent on Black patients with the same burden of illness, it systematically underestimated their needs.
Similar problems arise when skin images come predominantly from people with light skin types or when studies mainly include men.
2. Good test results, weak real-world performance
Many AI systems perform better on the data they were developed with than in real-world use. An external validation of a sepsis prediction model widely used in US hospitals (Wong and colleagues, JAMA Internal Medicine, 2021) found that it was considerably less reliable than stated by the manufacturer and failed to detect a large proportion of sepsis cases in time, while at the same time triggering many false alarms. The 2026 Oxford study on chatbots also shows that high scores on exam questions do not guarantee better decisions by real people.
3. Hallucinations
Large language models generate text that sounds plausible – even when it is wrong. They can invent studies, doses or findings. In documentation, the main problems are omissions and incorrect attributions. This is why AI-generated texts in medicine must always be checked by professionals.
4. Automation bias and loss of skill
People tend to trust a machine that is usually right – and then overlook its mistakes. A study from Poland (The Lancet Gastroenterology & Hepatology, 2025) also found evidence that, after regular use of AI, doctors detected fewer precancerous lesions during colonoscopies performed without AI. More on this in AI in colonoscopy.
5. Changes over time
A model that works well today may perform worse tomorrow – for example when new devices, different patient groups or changed treatment standards come into play. Experts call this data drift. AI systems must therefore be continuously monitored even after they have been introduced.
6. Data protection and security
Health data is an attractive target for attacks. Anyone using AI services that process data outside the EU must check particularly carefully how it is protected. In addition, AI systems themselves can be manipulated, for example through deliberately altered inputs.
7. Who is liable?
Responsibility for diagnosis and treatment continues to lie with doctors. Manufacturers are liable for defective products. The new EU Product Liability Directive (EU) 2024/2853 explicitly clarifies that software is also a product; it must be transposed into national law by December 2026. However, many detailed questions – for example regarding self-learning systems – will only be resolved by the courts.
How risks can be limited
- independent testing on data from different hospitals and population groups,
- randomised trials in routine care rather than laboratory tests alone,
- clear rules for human oversight,
- staff training, including in recognising AI errors,
- monitoring of performance after introduction,
- transparency towards patients.
Conclusion
The risks of medical AI are known and can largely be managed – if they are taken seriously. The EU AI Act addresses many of these points. Good medical AI is not the one with the most spectacular test results, but the one that demonstrably helps safely in everyday practice.
More on this topic: AI in Medicine 2026 – an overview and The AI Act and medicine.
Note: This article is for general information only. As of: October 2026.
Sources
- Obermeyer Z et al.: Dissecting racial bias in an algorithm used to manage the health of populations. Science 2019
- Wong A et al.: External Validation of a Widely Implemented Proprietary Sepsis Prediction Model in Hospitalized Patients. JAMA Internal Medicine 2021
- University of Oxford: New study warns of risks in AI chatbots giving medical advice (2026)
- HMP Global: AI Exposure May Reduce Polyp Detection Skills of Endoscopists (2025)
- Richtlinie (EU) 2024/2853 über die Haftung für fehlerhafte Produkte
- Verordnung (EU) 2024/1689 (KI-Verordnung)
Leave a Reply