A new wave of startups are anticipating the existential crisis facing the AI industry: What happens when we run out of data?
n 2011, Marc Andreessen, whose venture capital firm Andreessen Horowitz has since invested in some of the biggest startups in AI, wrote that “software is eating the world.” More than a decade later, it is literally doing just that.
Artificial intelligence, specifically the large language models that power it, is a voracious consumer of data. But that data is finite and it is running out. Companies have mined everything in their efforts to train ever more powerful AIs: YouTube video transcripts and subtitles, public Facebook and Instagram posts, copyrighted books and news articles — sometimes without permission, sometimes with licensing deals. OpenAI’s ChatGPT, the chatbot that helped mainstream AI, has already been trained on the entire public internet, roughly 300 billion words including all of Wikipedia and Reddit. At some point, there will be nothing left.
Researchers call this “hitting the data wall.” And they say it’s likely to happen as soon as 2026.
That makes creating more AI training data a billion-dollar question — one that an emerging cohort of startups are looking for new ways to answer.
One possibility: creating artificial data.
That’s five-year-old startup Gretel’s approach to AI’s data problem. It makes what’s known as “synthetic data” — AI-generated data that closely mimics factual information, but isn’t actually real. For years, the startup, now valued at $350 million, has provided synthetic data to companies working with personally identifiable information that needs to be protected for privacy reasons — patient data, for example. But now its CEO Ali Golshan sees an opportunity to supply data-starved AI companies with fake data made from scratch, which they can use to train their AI models.
“Synthetic data was a great fit,” Golshan, a former intelligence analyst, said of the data wall issue. “It solved two sides of the same coin. You could make data high quality and you could make it safe.”
This “AI feeds AI” approach has already been embraced by Anthropic, Meta, Microsoft and Google, which have all used synthetic data in some capacity to train their models. Last month, Gretel announced that it would make its synthetic data available to customers using Databricks, a data analytics platform, to build AI models.
“Junk safe data is still junk data.”
But synthetic data has its limits. It can exaggerate biases in an original dataset and fail to include outliers, rare exceptions that you’d only see in real data. That could make AI’s tendency to hallucinate even worse. Or models trained on fake data could simply fail to produce anything new. Golshan calls this a “death spiral,” but it’s more widely known as a “model collapse.” He requires new customers to provide Gretel with a chunk of real, high-quality data to avoid it. “Junk safe data is still junk data,” Golshan told Forbes.
Another way around the data wall: people. Some startups are hiring armies of them to clean up and label existing data to make it more useful for AI or create more new data.
The heavyweight in the so-called “data labeling” space is $14 billion behemoth Scale AI, which provides human-annotated data to top AI startups like OpenAI, Cohere and Character AI. The company has a giant operation, employing some 200,000 human workers across the globe through a subsidiary called Remotasks. These workers do things like draw boxes around objects in an image, or compare different answers to a question and rate which one is more accurate.
On an even larger scale, Amsterdam-based Toloka has crowdsourced 9 million human labellers or “AI tutors” for similar purposes. Dubbed “Tolokers,” these freelancers from around the world annotate data too — like labeling personally identifiable information in a dataset for use in a community AI project led by Hugging Face and ServiceNow. But they also create data from scratch: translating information into new languages, summarizing it into blurbs and transcribing it from audio to text.
“Nobody likes to deal with human operations.”
Toloka also works with experts like physics PhDs, scientists, lawyers and software engineers to create original domain-specific data for models that target niche tasks. It hires German-speaking lawyers, for example, to create content that can be fed into legal AI models. But it’s a lot of work to wrangle people across 200 countries, to check their work is accurate, authentic and unbiased and translate any academic jargon into language that’s accessible and digestible for AI models.
“Nobody likes to deal with human operations,” Toloka CEO Olga Megorskaya told Forbes. “Everybody likes to build AI models and businesses. But dealing with real humans is not a very common skill in the AI industry.”
There are industry-wide labor issues around this sort of work. Scale workers last year told Forbes about their low pay. Toloka clickworkers contacted for this story had similar complaints. Toloka CEO Megorskaya told Forbes she feels compensation is fair, and Scale AI has similarly said it’s committed to paying workers “a living wage.”
The most obvious solution to the data scarcity problem is perhaps the most obvious: using less data to begin with.
While there is an urgent need for AI training data to feed massive models, some researchers posit that one day, advanced AI might not need as much. Nestor Maslej, a researcher at Stanford University’s Human-Centered Artificial Intelligence, thinks that one of the real issues here isn’t quantity — it’s efficiency.
“You don’t need to take a rocket ship to the grocery store.”
“If you think about it, these large language models, as impressive as they are, they’re seeing millions times more data than any single human would see in their entire life. Yet humans can somehow do some of the things these models can’t,” Maslej, said. “It’s clear from a certain perspective that the human brain works at a level of efficiency that isn’t necessarily being captured in these models.”
That technical breakthrough hasn’t happened yet, but the AI industry is already starting to shift away from massive models. Rather than trying to build large language models that can compete with OpenAI or Anthropic, many AI startups are instead building smaller, more specific ones that require less data. Popular open-source AI model maker Mistral AI, for example, recently launched Mathstral, an AI designed to excel at math problems; it’s a fraction of the size of OpenAI’s GPT-4. Even OpenAI is getting into the mini model game with the launch of GPT-4o mini.
“We’re seeing this race for volume and the big generalist model providers hoovering up more data and trying schemes to generate new data,” said Alex Ratner, CEO of data labeling company Snorkel AI. “The key to getting a model to work really well on a certain task is quality and specificity of data, not volume.”
As a result, Snorkel’s approach is to help companies take advantage of the data they already have and convert it into AI training gold. The startup, which was birthed by Stanford’s AI lab and is now valued at $1 billion, provides software that makes it easier for a company’s staff to label data quickly.
That way, a company’s models are purpose-built for what they actually need. “You don’t need to take a rocket ship to the grocery store,” he said.