The middle layer, not the model, now decides whether enterprise AI works

Enterprises have spent two years discovering that access to a capable foundation model settles almost nothing about whether an AI system will work in production. A model that generates a plausible answer is now the easy part, available to every competitor at a price approaching zero,  but what separates a deployment that survives a compliance audit from one that fails is the software sitting between the model and the business outcome.

This layer decides which tools to call, keeps track of context throughout a task, judges whether an answer is correct, and blocks outputs that cross regulatory lines.

This middle layer is invisible in most product demonstrations, yet it is becoming its own enterprise category, and the executives building it argue it will outlast any model licence.

The global AI orchestration market was valued at roughly $11 billion in 2025 by several research houses. For instance, MarketsandMarkets projects growth to more than $30 billion by 2030 at a compound annual rate above 22%, and Fortune Business Insights forecasts a climb toward $60 billion by 2034. The monitoring category is expanding faster still.

Research and Markets valued the large language model observability platform market at close to $2 billion in 2025 and expects it to pass $9 billion by 2030, while Mordor Intelligence puts the narrower agent observability and governance segment at $1.23 billion in 2025 and forecasts a climb toward $8.6 billion by 2031 at a compound rate near 39%.

The middle layer is where a model's output becomes usable work

So what does the middle layer actually do? Ramprakash Ramamoorthy, Director of AI Research at Zoho Corp, described the middle layer as everything standing between the raw model and the business application.

“The middle layer is everything between the foundation model and the business application: the orchestration that decides which tools to call, the memory that holds context, and the guardrails that keep outputs safe and reliable,” he said. The distinction he drew was between generation and action. “The model generates text, and the middle layer turns it into something that can take real action in an enterprise,” he added. “It is where reasoning becomes useful work.”

This maps onto a live enterprise problem - where a model can produce fluent text but with no guarantee that the text is correct, safe, or connected to the systems where work happens, and the middle layer is the machinery that closes those gaps.

Its prominence has grown as the industry moved from single-model calls toward agentic workflows, where one instruction triggers a cascade of model calls, tool invocations and data retrievals that all need to be coordinated and monitored.

McKinsey’s 2025 global survey found that 23% of respondents had scaled an agentic system somewhere in their organisation, with a further 39% experimenting. That distribution explains why orchestration and observability have become procurement priorities rather than engineering afterthoughts.

For Avinav Nigam, Founder and CEO of TERN Group, the middle layer is not an abstraction but the most defensible part of what his company has built. The UAE-based clinical workforce platform raised $24 million in Series A funding in September 2025, taking its total to $33 million.

Its platform runs AI interviews that assess nursing candidates across 28 attributes for 80 specialised roles, and the component that judges those responses had to be built in-house. “In TERN’s case, the evaluation engine that scores a nursing candidate’s interview response, assessing clinical accuracy, communication quality, behavioural signals, leadership indicators, is something we built ourselves,” Nigam said.

Nothing off the shelf understood what a correct answer looked like for assessing the Gulf healthcare workforce. Getting that judgment right is what lets the business hit a 96% retention rate for placed workers and cut hiring timelines from as much as 12 months to under 10 weeks.

Ownership is the strategic question, and it is still unresolved

The market has not settled who should own this layer. Nigam called the current state fragmented, and he treated that fragmentation as a liability rather than a passing inconvenience. “Right now it is fragmented, and that fragmentation is itself a problem,” he said. Some of the tooling is built internally by larger enterprises, and some is emerging as point solutions from evaluation startups, observability vendors and guardrail providers, but none of it has consolidated into a coherent enterprise stack.

The competitive research supports that reading. Mordor Intelligence describes the agent observability segment as moderately competitive and fragmented, with no single player holding more than a 15% share in 2025, as incumbent monitoring vendors and hyperscalers extend existing platforms while specialist startups compete on developer tooling.

Fragmentation is why enterprises should treat ownership as a strategic imperative, according to Ramamoorthy. “This is the layer enterprises should want to own, because it encodes how their business actually runs,” he said. Zoho has kept its own middle layer in-house, and he detailed what it comprises. “At Zoho, we have built it in-house, including agent memory, tool orchestration across a large tool library, and context management, and we are standardising it into shared components used across our products,” he explained. The logic was about controlling the most sensitive assets. “Owning it keeps the business logic and customer context with our customers and us,” he said.

The two accounts converge despite the businesses being very different. A horizontal software company serving millions of customers and a vertical healthcare platform assessing nurses reached the same conclusion, which is that the layer encoding domain knowledge and business logic is the layer worth building rather than buying.

For TERN, that decision carries commercial weight. The company now competes for government workforce assessment contracts against firms including Palantir Technologies, in a UAE healthcare market projected to spend more than $50 billion by 2029.

Models are commoditising

The case for the middle layer rests on a claim about where the models themselves are heading. Ramamoorthy stated the premise directly. “Foundation models are becoming a commodity, with several capable options that are increasingly interchangeable,” he said.

Analysis drawing on the Open LLM Leaderboard shows the top ten models clustering within a three-point range on the MMLU knowledge benchmark in early 2026, and on the SWE-bench Verified coding benchmark, the leading five models sat within a single percentage point of one another in March 2026. Independent assessments put the performance gap between open and closed models at under 5% on many core tasks, even as frontier inference still costs several times more than optimised alternatives.

Nigam reached the same reading of the direction of travel. “Because the models are commoditising faster than most people expected,” he said. The capability gap between frontier systems is narrowing, and the cost of switching between them approaches zero. What is not commoditised is the encoded knowledge of what good output looks like in a specific vertical under specific regulatory constraints. “That takes years to build. A better foundation model does not erase it overnight,” he said. From that, he drew a direct challenge to the idea that model selection is where advantage lives. “The companies that will have durable AI advantages are the ones that own the middle layer in their domain, not the ones that picked the right model in 2024,” he added.

Ramamoorthy located the same durable advantage in the quality of orchestration. “The lasting advantage sits in the middle layer, in how well the system orchestrates tools, remembers context, and stays reliable, since that is what decides whether AI delivers real outcomes,” he said. As the underlying models converge toward parity, the differentiating work moves upward into the layer that enterprises build around them. “As models converge, this is where one enterprise’s AI will stand apart from another’s,” he said. For a regional ecosystem where sovereign AI ambition and enterprise adoption are accelerating together, the implication is clear.

Competitive advantage will be manufactured in the systems built around commoditised models, not purchased with a licence to the models themselves.