Databricks' internal Slide Hub app uses Lakebase Postgres to combine vector and keyword search on millions of slides and related metadata, providing developers practical guidance for compute sizing and performance under load.
- Hybrid vector and BM25 search runs jointly in Lakebase Postgres with no data duplication
- Compute allocated in units with autoscales from 0.5 to 64 CU impacts RAM and caching
- Ranking and filtering logic consolidated in one SQL statement for atomic, permission-aware queries
Infrastructure signal
Lakebase Postgres integrates extensions for vector and full-text search directly alongside application data, avoiding the complexity of maintaining separate search services or data duplication. This approach leverages IVF partitioning and quantization for efficient approximate nearest neighbor (ANN) search on high-dimensional vector embeddings. The BM25 index enables traditional keyword relevance scoring, allowing hybrid search capabilities without separate databases.
Compute units (CUs) control resource allocation, with a single CU providing roughly 2 GB RAM, CPU, and local SSD. The serverless database autoscales between configured minimum and maximum CUs, with the local file cache dynamically using up to 75% of memory to keep frequently accessed 8 KB pages resident, reducing storage fetches. This sizing model supports workloads from thousands to millions of rows, with Slide Hub demonstrating practical performance at up to 23k rows and benchmarks at 4 million passages.
Developer impact
Developers building search capabilities on Lakebase Postgres benefit from simplified deployment and observability by using a single integrated system. The Slide Hub app’s hybrid search is implemented in one SQL statement that fuses candidate results from ANN vector search and BM25 keyword search, applies relevance weights, filters by user permissions, and joins metadata like votes in one transaction. This reduces complexity in deployment and error handling, streamlining development workflows.
From a cost perspective, understanding how many compute units to allocate upfront is essential, as compute size influences local caching effectiveness and query latency. Developers can scale resources dynamically to balance performance and cost. The extensions are enabled via straightforward CREATE EXTENSION commands, minimizing operational burden while delivering advanced search functionalities within the Postgres environment familiar to many teams.
What teams should watch
Teams responsible for building data-driven apps with embedded search should evaluate Lakebase’s hybrid search to optimize infrastructure footprint, avoiding costly, separate search systems while still supporting scalable vector and text search. Planning compute sizing early based on expected data scale and query patterns is critical, especially given the impact of local file cache hit ratios on performance.
Observability into query performance, ranking accuracy, and cache usage is an area to enhance operational maturity. Platform teams should also monitor cost tradeoffs between fixed and serverless scaling options, especially in workloads with variable demand. Database administrators will want to understand how indexing strategies like IVF with RaBitQ quantization and BM25 interact under hybrid workloads to tune performance and resource consumption.