Why Does AI Keep Failing to Live up to the Hype in Medicine?

Diagnosis is a complex task.

Before treatment can even begin, doctors spend hours on testing, reviewing reports, and consulting. Diagnosis, in fact, accounts for up to 90% of the cost and time spent on a given ailment.

While the advent of advanced testing and imaging technologies has improved diagnostic accuracy and reduced time to diagnosis, making a definitive connection between a person’s symptom profile and a specific disease condition remains a complex task.

Machine learning and artificial intelligence are the latest technology to capture imaginations wondering how the diagnostic landscape can be further simplified. With these sophisticated algorithms, what if providers could analyze past data to make inferences about a patient’s symptoms, and perhaps even suggest possible diagnoses?

Some went even further, envisioning specialized AI that could read x-ray and MRI scans, among other complex tasks. In 2015, a venture capitalist famously issued a prediction that AI would replace radiologists in five years. Nearly ten years later, however, the radiology profession as we know it still appears to be safe.

Starting with IBM Watson, companies have invested - often flamboyantly, to the tune of billions of dollars - in these businesses, trying to create AI that can live up to the diagnostic hype.

Today’s large language models (LLMs), which includes ChatGPT, were quickly lauded as ushering in the next diagnostic era. Just as quickly, however, they revealed themselves to be prone to the same failures as their predecessors. The technology simply isn’t advanced enough to replace the need for humans to make the final decisions. LLM-based chatbots, for example, regularly misdiagnose symptoms and make inappropriate suggestions.

The problem is that diagnosis is complex, because diseases are complex. Any physician would say that a disease is not caused by any one single thing. In the same way, diseases cannot be diagnosed based on the appearance of any single symptom or trait.

The MRI scans of a patient with shoulder pain, for example, may show something that could also be present in hundreds of thousands of people who do not have shoulder pain.

If a genetic mutation shows up in the genetic analysis of a patient, it doesn't mean that exact mutation will cause a disease. That mutation is present in hundreds of thousands of people who will never develop the associated disease. Many of these people likely also have different diseases altogether.

The real problem lies in how today’s AI finds patterns in data.

While AI comprises lots of different algorithms, the AI as we commonly see it now is called next word prediction or sequence-to-sequence prediction.

Put in a very simple way, the AI predicts what the next plausible word will be, given a word.

Current AI algorithms need lots of data - multiple millions, or even billions, of data points - to find even simple patterns. To make matters more challenging, medical data is not readily and freely available for training, as it comes with privacy, legal, and other ethical concerns.

The limited availability of medical data for training makes it more difficult to apply traditional AI algorithms. This has led people to design AI with the mindset that there are only a limited number of diseases, and that all symptoms will show up in scans, reports, or genes.

Real-world evidence shows that this is not the case.

While diseases themselves are limited to a certain number, a given disease can present in different patients in innumerable different ways. For example, not everyone who gets COVID is affected in the same way. Some experience severe disease, some have moderate, and some have very mild symptoms or none at all. Even those who have the same disease severity may have different symptoms. Four years after the world’s first diagnosed COVID infection, we are still only beginning to understand why the disease seems to affect people so differently.

To recap the problem briefly:

Medical data available for training traditional AI is limited

The number of ways diseases can present cannot currently be predicted from the data by typical AI methods

The number of ways diseases can present cannot be handled by current AI algorithms

To be useful in medicine, AI needs to be able to tolerate the ambiguity and complexity that accompany diseases in the real world, from multiple underlying root causes, to varying symptom profiles, to the presence (or lack thereof) of different diagnostic biomarkers.

This means being able to recognize all combinations of symptoms, biomarkers, and other diagnostic test results (such as imaging).

However, even if access to medical data were unlimited, the number of possible combinations that would have to be extracted from the data would likely number in the millions - a Herculean task for today’s AI models. Traditional AI which is based on statistical learning is not built to handle searching a dataset of this size. In academic jargon, this is known as Curse of Dimensionality

As an alternative, non-statistical Graph theory based AI is studied. These novel AI models built on graph theory, however, have the capability to search and extract insights from much larger datasets that traditional AI models can. NaturalText AI is one of these novel models.

It uses graph algorithms to extract combinations of disease markers from vast datasets. The graph algorithms also allow NaturalText AI to find patterns in data using only analysis of the data it​self - without needing to be trained like traditional AI models.

We need new innovative algorithms, to solve the problem of analytics in diagnostics. NaturalText AI is starting in that direction.