Not too long ago, in 2015, a leading investor in Silicon Valley made headlines with the claim that AI would replace radiologists by 2020. As of 2022, this and numerous other apocalyptic visions of AI putting humans out of work in various fields have not panned out.
Why?
AI actually has done some astonishing things in the past decade. From filtering spam emails and tagging photos to translating articles to and from various languages, AI’s achievements are undeniable.
However, these achievements are limited to a few domains. Is it because people are stubbornly clinging to the “old way” of doing things? Is AI simply too expensive for the majority of businesses to take advantage of its capabilities?
Actually, the issue is much more fundamental.
Currently, AI’s essential function is to predict the next sequence in a set of data. In technical terms, it is called sequence to sequence prediction. It is a very simple mathematical operation that is performed trillions and trillions of times, requiring massive datasets, to get results.
This and other algorithms behind today’s AI advances are more than half of a century old—yes, you read that correctly. Although they were developed in the 1950s and 1960s, they required today’s level of computing power to be put into use and push AI forward.
(Consider, for example, that what was a multimillion-dollar supercomputer at the time is now what powers our smartphones.)
AI is only now starting to catch up to the algorithmic potential promised decades ago. Meanwhile, the business world faces challenges that are firmly rooted in the 21st century. Most industries have a bigger information problem than an automation problem.
This is not to discount the importance of automation. It remains important to numerous fields—it has plenty of valuable business applications.
It is just that businesses face challenges that go beyond automation alone, specifically when it comes to data. In our strongly knowledge-based economy, businesses need to find more information and even create new information to remain competitive.
Is today’s AI equipped to predict the next sequence—or essentially infer a new sequence—when prior data become irrelevant or do not provide the complete picture being sought?
That question was answered during the pandemic. AI and ML models broke practically overnight because people’s behavior changed so suddenly and dramatically. The models could no longer rely on years’ worth of data collected to define consumer habits and behavior. Automated insights based on outdated data are no longer reasonably predictive.
Some industries, like eCommerce, pharma, fintech, and others, are unique because they have carried on as normally as possible throughout the pandemic—and yet, they experience this same problem with data-driven insights from today’s AI.
What makes these industries exceptional are what they need from AI:
Stronger performance requires uncovering insights in data…but the businesses do not know what to look for in the data to find these insights. (This is also known as the “unknown unknowns” problem.)
These industries generate mixed data that contain both text and numbers. This means that the full application of statistics is not possible in the same way it is with numerical-only datasets.
Things change frequently and each problem is different, so solutions cannot be automated.
Existing AI/ML methods cannot scale automatically for datasets of different sizes. Models must first be tuned and trained to find the proper parameters to analyze a given dataset. They work for small data samples, however.
Each of these issues will be discussed in upcoming posts. The closer we examine these issues, the better we will be able to evaluate, understand, and apply the AI/ML methods needed.