Happy Sunday and welcome to Investing in AI. Be sure to check out our AI in NYC podcast if you want to know what’s happening in New York in the AI scene. Since I get a fair number of questions about my work at Neurometric and what we do with Small Language Models, I thought I would take a weekend to write about why SLMs are so awesome.
For the past two years, the narrative of the “AI Revolution” has been dominated by a single word: Large. We have been told that more parameters equal more intelligence, and that to achieve truly useful artificial intelligence, we must build models with trillions of weights housed in massive, power-hungry clusters. In this “bigger is better” paradigm, Small Language Models (SLMs) have been relegated to the periphery—viewed merely as “lite” versions of their larger siblings, intended only for edge devices like smartphones or offline laptops where hardware constraints make LLMs impossible.
But as we move from the era of “chatbots” to the era of “agents,” this narrative is being turned on its head. In the high-performance environment of the modern data center, SLMs are no longer the compromise; they are becoming the competitive advantage. The secret benefit of SLMs isn’t just that they fit on a phone; it’s that they are fundamentally better suited for the repetitive, operational, and narrow tasks that define agentic workflows.
From Conversational Search to Agentic Workflows
To understand why SLMs are gaining ground, we must first look at why we needed LLMs in the first place. The “first killer use case” for generative AI was conversational search and general-purpose reasoning. If you want a model that can write a Shakespearean sonnet about quantum physics one second and debug a Python script the next, you need a generalist. Large models like GPT-5 or Claude 4.6 are incredible at this because their vast parameter counts allow them to “know” a little bit about everything and maintain the nuance required for human-like conversation.
However, the industry is shifting. We are moving away from simple back-and-forth chat and toward Agentic Workflows. In an agentic system, the AI isn’t just talking to you; it is performing a series of tasks—browsing the web, calling APIs, checking database schemas, and validating structured data.
In these workflows, the requirements change. You don’t need a model with a Ph.D. in 18th-century literature to verify if a JSON object contains the correct keys. You need a model that is fast, reliable, and precise. This realization is at the heart of recent research from NVIDIA, which suggests that the future of AI isn’t about one giant model doing everything, but rather a “heterogeneous system” where SLMs do the heavy lifting.
The NVIDIA Perspective: SLMs as the Future of Agents
In the seminal paper “Small Language Models are the Future of Agentic AI” (NVIDIA Research, 2025), researchers argue that the “singleton” approach—using one massive LLM for every step of an agent’s process—is fundamentally inefficient. Their position is grounded in the observation that the majority of subtasks in a deployed agentic system are repetitive, scoped, and non-conversational.
The paper highlights that while LLMs provide the “foundational intelligence” needed for high-level strategy and planning, they are operationally unsuitable for the hundreds of “micro-tasks” an agent must perform. NVIDIA’s research shows that models like the Nemotron-H family or the Hymba-1.5B can match or even exceed the performance of models ten times their size when focused on narrow instructions. By moving these “errands” to SLMs, developers can build systems that are not just cheaper, but more robust.
1. The Speed Advantage: Latency is the New Intelligence
The most immediate benefit of SLMs in a data center is speed. In a standard chatbot scenario, a 2-second delay is annoying. In an agentic workflow, a 2-second delay is a dealbreaker.
Why? Because agents rarely make just one call. An agent tasked with “booking a flight that fits my calendar” might need to:
-
Extract dates from a prompt.
-
Search a database for meetings.
-
Call a flight API.
-
Filter results based on airline preferences.
-
Validate the final selection against a set of business rules.
If each of these steps uses a massive LLM with a high “Time to First Token” (TTFT), the total latency compounds. A five-step loop that takes 4 seconds per step becomes a 20-second wait for the user. By replacing those five calls with specialized SLMs—which can have throughputs 3.5x higher than transformers of the same size—you can reduce that total time to 3 or 4 seconds. In the data center, where milliseconds translate to user retention and system throughput, the low latency of SLMs is a “secret” superpower.
2. The Economic Reality: Cost-Effective Scaling
There is a common misconception that data centers have “infinite” resources, so model size doesn’t matter. In reality, the economics of the data center are what make SLMs so attractive.
As NVIDIA’s research points out, the “10-fold discrepancy” between the capital investment in infrastructure and the actual market for AI services means that efficiency is the only path to profitability. Running a 400-billion-parameter model to perform a simple intent classification is like using a Boeing 747 to deliver a pizza. It works, but the margins are non-existent.
SLMs allow for “Horizontal Scaling” on much cheaper hardware. Because they require significantly less VRAM, you can pack multiple instances of an SLM onto a single GPU or run them on older hardware that LLMs can’t touch. For an enterprise processing millions of agentic steps a day, the shift from a “frontier” LLM to a fine-tuned SLM can reduce operational costs by 90% or more without sacrificing the quality of the end result.
3. The Accuracy Paradox: Better Because They Are Smaller
Perhaps the most surprising finding in the move toward SLMs is that they are often more accurate than LLMs on narrow tasks. This seems counter-intuitive—how can a model with 2 billion parameters beat one with 200 billion?
The answer lies in Fine-Tuning and Focus. A large model is a generalist; it has been trained to be “helpful, harmless, and honest” across every conceivable topic. This generality often leads to “hallucinations” or over-complication when performing structured tasks.
SLMs, however, can be aggressively fine-tuned on specific datasets. If you take a 1.5B parameter model and train it solely on “SQL Query Generation” or “JSON Schema Validation,” it becomes a world-class expert in that one thing. Because it doesn’t have the “distraction” of knowing how to write poetry or explain the French Revolution, it is less likely to deviate from the required format. The NVIDIA paper notes that even a small amount of high-quality, task-specific data allows an SLM to reach parity with—and eventually surpass—generalist models on those specific operational steps.
The Move to “Smarter Systems”
The “Secret Benefits of Small Language Models” aren’t a secret anymore for the teams building the next generation of AI. We are witnessing a shift from “Smarter Models” to “Smarter Systems.”
In this new architecture, the Large Language Model acts as the “CEO” or “Orchestrator”—handling the complex reasoning and the final conversational output. Meanwhile, a fleet of specialized Small Language Models acts as the “Engine Room,” handling the millions of operational tasks with lightning speed, surgical precision, and minimal cost.
I’ve seen this firsthand in AI forward companies. They start with one LLM for their use cases and as inference costs rise they slowly setup task based endpoints – servers with a model specifically for a certain task. It saves money and time, and usually increases performance on that task.
The data center is no longer just a place for giants. It is becoming the home of the “Model Zoo,” where the smallest, most efficient models are the ones doing the real work. If you are building for the agentic future, don’t just look for the biggest model you can find. Look for the smallest model that can get the job done. That is where the true power of AI lies. If you are interested in SLMs, check out our small model marketplace.
Thanks for reading.