Databricks' Zerobus Ingest service redefines telemetry ingestion at petabyte scale by automatically scaling stream connections and directly loading data into Delta Lake tables — eliminating the need for managing Kafka infrastructure and complex partitioning.
- Serverless streaming ingest at 12+ GB/s per table without manual partition tuning
- Ordering guaranteed per stream connection allows seamless autoscaling and load balancing
- Direct data landing into Delta Lake tables simplifies ETL and accelerates query readiness
Infrastructure signal
Zerobus Ingest introduces a paradigm shift in cloud streaming infrastructure by decoupling ordering guarantees from static partition assignments. This design enables autoscaling at the granularity of stream connections rather than partitions, allowing dynamic pod allocation based on demand without impacting data ordering. As a result, the system achieves elastic scalability capable of ingesting over 1 petabyte of telemetry data in under 24 hours, sustaining a continuous throughput of roughly 12 GB per second to a single Delta table.
This model eliminates traditional requirements to pre-configure brokers and partitions and reduces overhead associated with managing Kafka pipelines. Moreover, routing incoming streams heuristically based on pod load ensures balanced resource utilization while maintaining stable latency under heavy workloads. The underlying implementation minimizes data copying and memory allocation inefficiencies, contributing to cost-effective compute utilization and simplified cloud expense forecasting.
Developer impact
For developers, Zerobus Ingest removes much of the operational complexity historically involved with managing high-throughput streaming pipelines. Instead of manually tuning Kafka infrastructure or handling partition topology, engineers push data directly into Delta Lake tables through a simple, push-based API. This workflow drastically reduces time spent on configuring and maintaining streaming infrastructure and provides data ready for querying within seconds.
Key to developer experience is the ordering guarantee scoped per stream connection rather than per partition, enabling streams to be reliably consumed without concern for partition rebalancing or consumer lag related to static sharding. This seamless scalability allows development teams to focus on building data products and analytics without ongoing infrastructure tuning or disruption fears.
What teams should watch
Teams building telemetry, IoT, or high-velocity event ingestion pipelines should evaluate Zerobus Ingest as a compelling alternative to Kafka-based architectures, especially when integrating tightly with Delta Lakehouse for analytics and machine learning. Product and platform teams responsible for cloud cost management will benefit from the autoscaling model’s ability to adjust compute capacity dynamically, minimizing overprovisioning.
Observability and reliability teams should note the stable latency and ordering guarantees even under fluctuating workloads, which support more predictable SLAs for downstream consumers. Database and API architects must consider how coupling ingestion directly with Delta tables impacts data governance, schema enforcement, and cataloging, given Zerobus’s integration with Unity Catalog for unified security and metadata management.