Databricks' implementation of OpenSharing enables live, incremental data consumption through shared Delta tables, eliminating the need for file copies or batch exports. This advancement improves cloud resource usage, supports structured streaming, and simplifies developer deployment and observability in global data and AI infrastructure environments.
- Eliminates redundant storage by reading live shared tables without copying files
- Supports incremental streaming with built-in checkpointing and data lineage
- Simplifies deployment with metastore-based share management and durable checkpoints
Infrastructure signal
OpenSharing exposes Delta tables over an open protocol, allowing recipients to stream changes directly from the provider's cloud storage without creating copies. This approach results in reduced cloud storage costs since data duplication via file exports or API dumps is avoided. The shared tables retain full transactional history, enabling incremental reads aligned with cloud-native structured streaming paradigms.
The provider’s setup requires enabling Change Data Feed optionally for upsert and delete tracking, with Databricks Runtime 19+ automating row-level change capture. Durable checkpoint storage, such as Unity Catalog Volumes, ensures fault tolerance during incremental consumption. Metadata-level privileges manage share creation securely, fostering governance and reducing operational overhead linked with managing multiple export jobs or SFTP endpoints.
Developer impact
Developers benefit from simplified ingestion pipelines by streaming directly from shared Delta tables using Structured Streaming with triggers like AvailableNow. This triggered micro-batch approach ensures jobs only process new increments, cutting unnecessary compute time and making workflows more predictable and efficient. Developers can also configure streaming to trigger on table updates instead of fixed schedules, further optimizing resource use.
Checkpointing mechanisms automatically track processed versions, enabling exactly-once or incremental processing semantics depending on the streaming query options chosen (ignoreChanges or skipChangeCommits). This reduces manual state management complexity and accelerates development velocity. Developers retain access to full change feed information for downstream systems requiring inserts, updates, and deletes, enhancing data accuracy and auditability.
What teams should watch
Data teams and platform engineers should monitor shared tables’ commit histories and checkpoint states to understand data freshness and consumption lag. Proper metastore role assignments are critical as CREATE SHARE and RECIPIENT commands require privileges beyond standard workspace admin roles, impacting cloud governance frameworks.
Observability teams should leverage the structured streaming metrics and logs to track incremental job progress and diagnose pipeline health. Teams planning to replace multiple export endpoints or legacy file-based feeds with OpenSharing should evaluate the impact on cloud costs, workload reliability, and how incremental changelog materialization aligns with downstream analytics needs.