Inference changes everything underneath

Training created the current race for compute, but most enterprises will experience the infrastructure challenge somewhere else. As models move into everyday use, inference turns exceptional demand into a continuous workload that has to be supported wherever the business needs it.

The spectacular part happens before most organisations ever touch the model. Vast clusters run intensive training workloads, drawing enough power and requiring enough specialist hardware to make the infrastructure impossible to ignore. Those projects have shaped much of the debate around data centres, energy consumption and the appetite for GPUs, even though relatively few companies will ever build a frontier model themselves. For most enterprises, the more consequential infrastructure story begins after training has finished.

Every time an employee asks a model a question or an application calls one in the background, inference takes place. An agent may make several such calls while working through a single request, checking information before deciding what to do next. None of those individual interactions resembles a giant training run, but successful deployments multiply them across users and business processes until demand becomes continuous. The infrastructure requirement is no longer concentrated around the exceptional act of creating a model; it sits underneath the ordinary act of using one.

The workload that comes after training

Jad Jebara sees that distinction being lost in much of the discussion around infrastructure. As co-founder and CEO of data centre infrastructure management company HyperView, he works with the physical environments supporting increasingly compute-intensive applications. Training attracts attention because of its scale, but he argues that the commercial value emerges when organisations begin applying what has been created.

“People associate AI with training models, but training models is expensive,” Jebara says. “The value comes from what those models enable in vertical solutions, high-value knowledge and workflow automation. That is where we see the money and the value.”

The change sounds subtle until usage begins to grow. A training cluster may run an enormous workload for a defined period, whereas an application in production is expected to respond whenever it is needed. Demand follows employees, customers and operational systems rather than a research schedule. An assistant used occasionally by a small team presents one profile; the same capability embedded into thousands of transactions every day creates something very different underneath.

That distinction becomes more important as agents take on multi-step processes. A person may make one request, but the system can respond by consulting a model, searching additional information and asking another specialist component to check the answer before an action is taken. What appears to the user as a single interaction can therefore generate a chain of computational activity. Production volumes are determined not only by how many people use the application, but by how much work the software performs on their behalf.

Brian Sathianathan, CTO and co-founder of enterprise AI company Iterate.ai, believes this is why model choice must be treated as an economic decision rather than a contest for maximum capability. Larger models have an important role, but routing every task through the most powerful option can impose a cost that becomes obvious only when the application scales. The right question is what level of capability the particular workload requires.

“If I have a simple classification problem, I do not need to send it to the biggest model,” he explains. “You need to look at the use case and choose the model that gives you the right performance at the right cost.”

That software decision ultimately becomes an infrastructure decision. Calling a smaller specialised model can reduce the compute required to deliver the same business outcome, while unnecessary use of a much larger system increases demand before another server has even been installed. Capacity planning therefore begins partly in the application architecture, not solely in the data centre.

Compute follows the work

The geography of demand also changes once models become part of operational systems. Training can be concentrated where enormous quantities of compute have been assembled, but inference often needs to respond to conditions somewhere else entirely. A hospital cannot always treat patient data like a generic cloud workload, while a system monitoring equipment at a remote site may need to continue functioning when connectivity is uncertain.

Jebara expects those requirements to produce a far more distributed estate than the headline focus on hyperscale facilities suggests. “You will always have the massive cloud data centres,” he says, “but you will also have smaller systems at the edge and various forms of integration between those data centres.”

The important shift is not that edge computing replaces the cloud. Large facilities retain enormous advantages where workloads can be centralised efficiently, while private infrastructure remains appropriate for organisations with different regulatory or operational requirements. What changes is the assumption that all intelligence will naturally gravitate towards the biggest available concentration of compute. As inference becomes embedded into real activity, infrastructure is pulled towards the places where responses are needed.

Latency is one reason, but Jebara also points to sovereignty. Moving information across borders or into an external environment may be unacceptable for particular applications, regardless of how attractive the economics of centralisation appear. “You start thinking about data sovereignty and latency,” he says. “If you are running a system for agriculture, for example, it needs to be in that area. The data centre is becoming part of the fabric of society rather than an isolated warehouse.”

That description changes the way infrastructure should be viewed. A regional facility supporting healthcare or industrial operations is not merely a smaller version of a giant training centre. Its value lies in being located within the service it supports, able to process information with the response time and controls that application demands. The infrastructure estate begins to resemble a connected operating environment rather than a hierarchy with everything flowing inevitably towards one central location.

This is also where workload placement becomes more dynamic than a simple decision between cloud and on-premise systems. Some information may be processed locally before a smaller result is sent elsewhere, while another application can tolerate the journey to a remote model without any meaningful loss of performance. A successful enterprise deployment may therefore use different infrastructure for different stages of the same process. The architecture is shaped by the work rather than by loyalty to a particular computing model.

Efficiency becomes physical

More distributed infrastructure does not remove the pressure created by rising compute demand. If inference becomes embedded into customer service, engineering tools and operational systems across entire industries, apparently modest workloads accumulate quickly. The temptation is to assume that infrastructure can simply expand alongside them, adding more processors and power whenever software requires it.

Professor Jesper N. Tegnér questions whether brute-force expansion can remain the dominant response. A professor of bioscience and computer science at King Abdullah University of Science and Technology, he studies alternative approaches to computational architecture, including lessons drawn from biological systems. His broader concern is that continuing to obtain capability primarily through scale carries an increasing energy and infrastructure burden.

“Scaling works to some extent, but it is also limited and, in particular, very expensive,” Tegnér explains. “Biological systems, including the human brain, do not work exactly in this way. Most likely, we need other kinds of architectures and other modalities.”

For infrastructure planners, the implication extends beyond the design of future models. A system that can achieve the required result using less computation changes the physical resources needed to support it. At sufficient volume, improvements in software efficiency affect electrical capacity, cooling requirements and the amount of hardware that must be installed. The cheapest megawatt may ultimately be the one an application no longer needs.

That makes efficiency a shared problem across layers that are often considered separately. Data centre operators can improve cooling and power utilisation, but application designers also determine whether an expensive model is called unnecessarily. Researchers can develop more efficient architectures while enterprises decide how much reasoning is required for an everyday task. The eventual infrastructure footprint reflects all of those choices long before a capacity shortage appears.

Sathianathan sees specialised models as part of that response. A model tuned to a narrower domain can perform a defined job without carrying the computational burden of a system intended to answer almost anything. For businesses deploying at scale, the appeal is not academic elegance but economics: capability that costs less to invoke can be used more widely without creating the same infrastructure demand.

Jebara expects the resulting environment to contain large and small systems rather than converging on a single form of compute. Frontier training will continue to justify enormous centralised facilities, while inference creates demand in regional and local environments that serve very different purposes. What links them is the need to understand the workload well enough to place and provision it intelligently.

The infrastructure question becomes most important when nobody is thinking about infrastructure at all. A customer asks a question and expects an immediate response. An agent works through another stage of a process without anybody counting the model calls behind it. Multiply those unremarkable interactions across an organisation and inference stops looking like a feature added to software; it becomes a permanent workload that the underlying estate has to absorb.

Training created the visible race for compute because the requirements were extraordinary. Widespread inference may prove more consequential for enterprises precisely because it makes demand ordinary. The challenge is no longer only how to support the biggest training cluster, but how to build an infrastructure estate that can sustain intelligence wherever the work happens.