The AI market is experiencing a fundamental shift where inference tasks now drive the majority of infrastructure demand, exceeding training workloads for the first time. This transition is influencing chip design, power management, and deployment models worldwide.
- Inference workloads now constitute two-thirds of AI infrastructure demand
- Modular, low-power data centers emerge as optimal environments for inference
- Commodity memory chips gain traction over scarce high-bandwidth memory
Market signal
The AI infrastructure market has pivoted sharply toward inference activities, which currently make up approximately 67% of workloads, a significant increase from about 15% in 2023. This trend emphasizes the growing importance of deploying AI where inference performance and operational efficiency matter most. Demand is increasingly shifting away from large, centralized training clusters to smaller, modular setups that can be built quickly and offer better economics for inference.
A key market indicator is the rise of modular data centers supporting inference workloads at power levels between half a megawatt and a few megawatts. These sites provide operators with scalable, rapid-deployment options that align well with inference’s lower power and cooling requirements compared to traditional training infrastructures. Notably, major cloud providers and emerging neocloud vendors are adopting this approach, leveraging inference-optimized hardware to monetize AI token generation swiftly.
Operator impact
Operators face new hardware and deployment considerations as inference workloads dominate. Positron AI leads with inference machines built around commodity memory instead of the limited and expensive high-bandwidth memory standards traditionally used in AI processors. This design supports air cooling and lower power consumption—about 4,500 watts per server—and enables racks of inference servers to operate within roughly 40 kilowatts, a fraction of the power draw typical of training nodes.
These hardware innovations make inference workloads ideal candidates for modular data centers, which can be deployed rapidly with fewer regulatory and infrastructure challenges than large-scale training facilities. Operators can now focus on securing locations with affordable power rather than maximizing raw chip performance alone. The ability to transition from site acquisition to monetizable inference tokens within three to four months is becoming a critical competitive advantage in AI infrastructure deployment.
What to watch next
Attention now turns to how chipmakers will respond to inference’s dominance, especially those developing silicon specifically tailored to these workloads. Positron plans to release its next-generation Titan server with custom silicon in the near term, which could further optimize inference performance and energy efficiency. The adoption of such specialized silicon across more cloud and neocloud providers will be a key indicator of broader market shifts.
Additionally, the spread of modular data center deployments worldwide will be important to monitor, including the geographic and regulatory environments that facilitate rapid build-outs. Developments in memory technology supply chains, particularly any changes affecting commodity versus high-bandwidth memory availability and pricing, will also significantly influence the cost and speed of inference infrastructure expansion.