Lakebase Postgres, a disaggregated storage database platform by Databricks, has improved its compute caching architecture to accelerate query performance by expanding Postgres shared buffers to use up to 75% of available DRAM on fixed-size compute units. This change replaces a dual cache tier system, boosting speed and efficiency for large workloads.
- Shared buffers auto-scale to consume up to 75% of DRAM on fixed compute nodes.
- Local file cache layer is disabled, removing cache duplication and latency.
- Optimized caching improves throughput and reduces storage access on S3-based backend.
Infrastructure signal
Lakebase Postgres uses a disaggregated storage architecture where data pages are stored in object storage (like S3) and fetched into compute nodes as needed. Ensuring an efficient cache on the compute side is critical to delivering the low latency and high throughput expected from Postgres workloads.
Previously, the cache architecture combined a static Postgres shared buffers area with an autoscaling local file cache (LFC). The LFC was designed to compensate for the 1 GB max limit on shared buffers by providing a second caching layer in DRAM. Databricks has now eliminated the LFC for fixed-size compute units (80+ compute units) and expanded shared buffers to consume up to 75% of DRAM. This reduces complexity, avoids double buffering, and increases the likelihood that frequently accessed data remains in the fastest memory.
Developer impact
Developers connecting to Lakebase Postgres can expect noticeably faster query execution, especially for workloads involving large, frequently accessed datasets. With the larger shared buffers, data stays cached longer in the native Postgres memory area, substantially reducing round trips to slower storage layers.
The change is transparent at the API and SQL levels; developers do not need to alter queries to benefit. Performance tuning can also be simplified since the shared buffers parameter now scales aggressively with available memory on fixed compute sizes. Observability into cache hit rates and memory utilization will be key to fully leveraging these improvements.
What teams should watch
Cloud platform and infrastructure teams should monitor memory utilization metrics closely when running workloads on Lakebase Postgres, especially on fixed compute tiers 80 CU and above. The removal of the LFC and expansion of shared buffers changes memory pressure and cache eviction behavior.
Data engineers and DBAs managing large-scale analytic workloads should validate that their existing workloads experience improved latency and throughput after this cache model adjustment. Observability tools that highlight cache hits, evictions, and reads from object storage should be updated to track new cache efficiency.
In future phases, Databricks plans to introduce fully dynamic autoscaling shared buffers for flexible compute clusters. Teams should prepare for incremental rollouts that will add autoscaling cache controls and possibly further reduce reliance on local storage layers, enhancing cost efficiency and reliability.