Databricks now enables direct ingestion of OpenTelemetry traces into Unity Catalog, allowing AI teams to analyze detailed telemetry data like prompts, tool calls, and latencies using familiar SQL tools while benefiting from enterprise-grade governance and scalability.
- Direct OpenTelemetry ingestion to Delta Lake simplifies telemetry architecture
- Unified trace data enables long-term analytics, monitoring, and governance
- Serverless Zerobus Ingest removes infrastructure overhead for telemetry pipelines
Infrastructure signal
Databricks introduces a serverless, managed ingestion service named Zerobus Ingest that supports OpenTelemetry protocols natively through gRPC and REST APIs, enabling direct streaming of spans, logs, and metrics to Unity Catalog tables. This single-sink architecture eliminates the need for intermediate messaging layers like Kafka and reduces pipeline complexity, resulting in lower operational overhead and potential cloud cost savings related to telemetry infrastructure.
The ingested telemetry is stored in Delta format within Unity Catalog, leveraging Delta Lake’s scalability, governance features, and integration with existing SQL warehouses. This approach treats trace data as first-class lakehouse assets, streamlining governance and compliance by consolidating telemetry alongside business and model data without requiring separate observability systems or complex data duplication pipelines.
Developer impact
Developers benefit from a unified telemetry ingestion point that decouples instrumentation from storage. They can export OpenTelemetry data from agents running anywhere, including external deployments, directly into Databricks-managed tables. This streamlining improves developer workflow by supporting trace analysis, debugging, and performance evaluation using familiar SQL querying and MLflow experiment interfaces within the same platform.
Because tracing data is tightly integrated with analytics and machine learning tools, teams can rapidly iterate on AI agents by combining real production traces with downstream evaluation. This continuous feedback loop accelerates agent tuning and troubleshooting while minimizing dependencies on specialized observability dashboards or tools, making telemetry more accessible and actionable throughout development cycles.
What teams should watch
Teams operating AI-driven agent workloads in production should evaluate integrating Databricks’ OpenTelemetry ingestion to unify traces with their existing datasets for improved observability and governance. This approach is especially advantageous for scenarios requiring sensitive prompt data to remain under strict control or for long-term trace retention and analytics beyond traditional observability constraints.
Platform and infrastructure teams need to adjust deployment and monitoring strategies to leverage Zerobus Ingest’s serverless architecture, which may reduce reliance on existing telemetry streaming components such as Kafka. Monitoring the performance and reliability of this ingestion layer will be critical to ensure sustained trace throughput and durability while capitalizing on the inherent scalability and SQL-based accessibility offered by Unity Catalog and Delta Lake.