Financial institutions and asset managers face growing challenges in analyzing sensitive data scattered across on-premises and cloud environments. Databricks Lakehouse Federation addresses this by enabling real-time SQL queries across diverse sources without copying or syncing data, simplifying hybrid cloud analytics and reducing operational costs.
- Query sensitive data across on-premises and cloud in real time without data movement
- Simplify developer workflows by eliminating ETL pipelines and data materialization
- Combine internal holdings and third-party financial data within a unified catalog
Infrastructure signal
Lakehouse Federation introduces a real-time data federation layer that connects on-premises databases like SQL Server, PostgreSQL, and MySQL with cloud data providers such as Snowflake and marketplace datasets. This avoids the need to migrate or copy data into a central cloud repository, which traditionally requires costly ETL pipelines and complex synchronization mechanisms.
By maintaining data in place and executing live queries across federated catalogs, organizations reduce cloud storage and processing costs associated with data duplication. The system supports federated external catalogs within the Unity Catalog framework, enabling consistent security and governance controls across hybrid environments while enhancing infrastructure reliability by minimizing stale or incomplete copies.
Developer impact
Developers experience a streamlined workflow by removing the complexity of building and maintaining ingestion pipelines, materialized views, or specialty gold tables for combined datasets. SQL authors can query federated external tables as if they were native Unity Catalog objects with no additional development overhead for data transformation or synchronization.
The flexibility to join internal portfolio data with subscribed third-party financial data on the fly accelerates dashboard development and analytical experimentation. Integration with natural-language query systems further democratizes data consumption, allowing non-technical users to run complex cross-source queries without bespoke code or manual data preparation.
What teams should watch
Data engineering and cloud platform teams should evaluate Lakehouse Federation as a tool to cut cloud data duplication and operational risks related to compliance, especially where sensitive financial or client data must remain on-premises or behind firewalls. Real-time federation reduces incident impact from ETL failures or delayed pipeline runs.
Analytical and BI teams should monitor how combined federated and marketplace data catalogs simplify SQL querying and dashboarding across distributed data stores. Observability and performance monitoring must adapt to federated query patterns where query execution spans external source systems, requiring new attention to latency and source availability.
Security and governance teams will find the unified Unity Catalog model helpful for enforcing consistent access policies across both federated and cloud-shared data, facilitating compliance audits and reducing manual data handling procedures. Teams integrating AI-driven natural language agents should watch for improvements in end-user query experience powered by live federated source joins.