Nebius achieved a dramatic cut in AI agent rollout collection time by addressing inference bottlenecks and improving parallelism between training and generation, significantly enhancing cloud resource utilization and developer workflow.
- Inference bottlenecks reduced by workload parallelization
- Policy staleness managed through divergence tracking
- Infrastructure key to RL iteration speed and efficiency
Infrastructure signal
Nebius identified inference for rollout generation as the critical bottleneck limiting overall AI agent training throughput despite powerful GPU resources. The rollout phase must complete all agent trajectories before training can proceed, and variable trajectory lengths cause idle GPU time waiting for stragglers. Improving batching, cache management, and task scheduling was crucial to reducing total rollout collection wall-clock time by 66.9%.
A major infrastructure innovation involved overlapping training and inference by allowing rollout generation to use slightly outdated model weights, sidestepping strict synchronization. This approach enabled concurrent GPU utilization across both phases, dramatically improving resource efficiency in cloud environments. Additionally, precise measurement of policy divergence—not just the age of generated data—helped manage tradeoffs around data staleness and sample utility to maintain training quality.
Developer impact
For engineering teams, the results emphasize the importance of viewing inference engines as integral components of reinforcement learning optimizers, rather than isolated systems. Measuring end-to-end time to a trainable batch, instead of average tokens per second, better captures real training throughput including rollout variability and synchronization delays.
Managing policy staleness through divergence metrics requires new API endpoints and monitoring tools to track the offset between generating and current models. Developers must adapt workflows to support asynchronous rollout generation and training pipelines, emphasizing reproducible environments and realistic task states to ensure robust model learning transfer.
What teams should watch
Cloud teams should prioritize infrastructure changes that enable parallel execution of data generation and training workloads to reduce idle GPU time and lower overall compute costs. This includes enhancements in batching logic, cache strategies, and smarter scheduling algorithms that consider trajectory length distributions.
Platform and API teams must invest in tooling for staleness measurement, such as importance sampling corrections and divergence tracking, to safely reuse generated rollout data and designate thresholds for data drop or down-weighting. Continuous refreshment of training environments and tasks is critical to maintain meaningful learning signals as models improve.
Observability improvements focusing on the interplay between rollout generation and training workloads will help teams detect bottlenecks earlier and refine pipeline efficiency. Tracking wall-clock timing metrics alongside traditional throughput measurements is essential for understanding real-world RL system behavior and guiding informed infrastructure investments.