AI chatbots are worse than search engines for medical advice

There is a clear gap between the theoretical medical knowledge of large language models (LLMs) and their practical usefulness for patients, according not a new study from the Oxford Internet Institute and the Nuffield Department of Primary Care Health Sciences at the University of Oxford. The research, conducted in collaboration with MLCommons and other institutions, involved 1,298 people in the UK.

In the study, one group was asked to use LLMs such as GPT-4o, Llama 3, and Command R to assess health symptoms and suggest courses of action, while a control group relied on their usual methods, such as search engines or their own knowledge.

The results showed that the group using generative AI (genAI) tools performed no better than the control group in assessing the urgency of a condition. They were also worse at identifying the correct medical condition, according to The Register.

About Author

AndyC

Andy Curtis is an award-winning security consultant, researcher and public speaker. He has been working in the computer security industry since the early 1990s, having been employed by state and federal government, leading healthcare and banking providers across three continents. He has given talks about computer security for some of the world’s largest companies, worked with law enforcement agencies on investigations into hacking groups, and is a regular voice on TV and radio explaining IT security threats.

See author's posts