United Airlines has developed a serverless, near-real-time change data capture pipeline that replicates transactional reference data from Amazon Aurora MySQL to Apache Iceberg tables on Amazon S3. This approach eliminates legacy batch processing delays and improves query performance for hundreds of analysts.
- 5x average query latency improvement and 2x faster ETL jobs
- Serverless streaming pipeline reduces operational overhead
- Scalable CDC using AWS DMS and on-demand Kinesis Data Streams
Infrastructure signal
United Airlines' infrastructure modernization pivots on streaming CDC events from Amazon Aurora MySQL through AWS DMS into Apache Iceberg tables stored on Amazon S3. This new architecture replaces the bottlenecked legacy batch data replication with a scalable and serverless streaming pipeline composed of Amazon Kinesis Data Streams, AWS Lambda for transformation, and Amazon Data Firehose for delivering change events as Parquet files into Iceberg tables.
By leveraging managed services that automatically scale, such as Kinesis on-demand mode and Firehose’s serverless delivery, the airline reduces infrastructure management complexity and cost. The solution also introduces open table format benefits through Iceberg, enabling efficient handling of large datasets exceeding 10 million rows without hitting prior database query limits.
Developer impact
Developers benefit from a streamlined workflow where AWS CloudFormation automates deployment of the entire CDC pipeline, including monitoring with Amazon CloudWatch. The Lambda transformation function decodes the CDC event format and correctly routes changes to destination Iceberg tables, which may require engine-specific customization, but overall simplifies maintenance and evolution of the data pipeline.
Improved query response times by 5x for analyst workloads and a 2x reduction in total ETL job processing duration reduce bottlenecks in data access, enabling faster iteration and business decision-making. The shift to open Iceberg tables also allows easier integration with multiple query engines such as Amazon Athena and Redshift Serverless.
What teams should watch
Data engineering and analytics teams should prioritize adopting serverless, event-driven patterns for CDC pipelines to unlock near-real-time data availability. Teams must focus on managing CDC engine compatibility since Lambda functions parsing incoming events require adjustments when switching CDC technologies or source databases.
Observability improvements via CloudWatch metrics for Kinesis, Firehose, and AWS DMS streams will allow proactive monitoring of data flow health and latency. Additionally, teams should evaluate query workloads against Iceberg table snapshots to balance consistency and performance, ensuring that downstream analytics meet SLAs without overspending on resources.