Coinbase has dramatically reduced its AI cloud expenses by implementing a sophisticated multi-model infrastructure that automatically routes tasks between frontier and cost-effective open-weight models. Their approach pairs aggressive caching with task-specific workload distribution, allowing engineering teams to maintain high productivity while minimizing reliance on expensive AI APIs.
- Internal AI spend cut nearly 50% despite growing token usage
- Multi-model gateway automates selection between expensive frontier and cheaper open models
- Caching improvements boost cache hit rate from 5% to 60%, driving major cost reductions
Infrastructure signal
Coinbase’s architecture represents a shift from single-provider AI dependency toward a multi-model ecosystem, where a centralized gateway evaluates requests in real time to select the optimal model. This gateway takes into account task complexity, cost profiles, and cache availability to route each request, striking a balance between performance and expense. Open-weight models like Z.ai’s GLM 5.2 and Moonshot AI’s Kimi 2.7 serve as cost-effective defaults, with self-hosting ensuring full data control and privacy.
Caching is a foundational component of their infrastructure strategy, having increased cache efficiency from 5% to 60%. By keeping context locked to a single model and reusing previously computed results when valid, Coinbase drastically reduces redundant model queries. This reduces overall cloud AI inference compute and associated billing, which is critical at their scale, roughly equivalent to operating 1,200 full-time AI agents normalized against typical work weeks.
Developer impact
Engineering teams at Coinbase enjoy unrestricted access to AI tooling without the friction of manual model selection or usage caps. The intelligent gateway abstracts model choice, automatically assigning tasks to cheaper models for routine execution and reserving more expensive frontier models for complex reasoning workloads. This automation frees developers from costly, manual tuning decisions and improves productivity by simplifying integration with the AI infrastructure.
Additionally, the sharp separation of model inference responsibilities enhances developer experience by delivering consistent performance aligned with the task at hand. With lowered cost pressures and improved throughput, teams can experiment more freely and deploy at scale, accelerating innovation velocity and lowering the barrier for AI-powered feature development.
What teams should watch
Organizations should monitor the expanding viability of self-hosted open-weight models that deliver competitive benchmarks at a fraction of the cost of premium APIs. Improvements in multi-model orchestration and gateway capabilities are making sophisticated cost and performance optimization achievable at scale and should be evaluated alongside vendor lock-in risks.
Teams must also prioritize observability around cache utilization rates, latency, and model performance to tune routing logic effectively. This requires investing in metrics and alerting that reflect workload-specific demands and pricing dynamics, enabling continuous refinement of the AI stack. Finally, seamless integration of these systems into developer workflows is essential to maximize adoption and cost benefits.