Data Migration Goes Cloud Native in Automotive Manufacturing

A Terraform-based solution was developed to manage configurations via a YAML file, allowing metadata-driven ingestion without direct IT involvement. This YAML file defines essential parameters such as database connection details, streaming or batch processing methods, and potential future metadata extensions.

AWS Data Migration Service (DMS) subsequently leverages this configuration for efficient Change Data Capture (CDC) operations, depositing data into a landing zone.

Data from the landing zone undergoes further processing through Spark jobs. These jobs generate two critical data layers:

• Historical Layer: Capturing each data mutation.
• Cloud-Native Table Layer: Mirroring the on-premises table structure, enabling consistency and seamless integration within the cloud environment.

This layered approach simplifies data management, providing both historical context and current data states, enhancing analytical capabilities.

Transitioning computational workloads from Glue and EMR to Kubernetes, managed through AWS Elastic Kubernetes Service (EKS), significantly optimized resource allocation. Kubernetes offers a powerful, flexible environment to run Spark workloads, thereby reducing dependency on AWS-specific managed services.

Adopting Kubernetes also addressed concerns about vendor lock-in associated specific cloud-vendor solutions.

By relying on Kubernetes, which is an open standard, the infrastructure can easily be migrated across different cloud providers or even to an on-premises environment, offering the flexibility to adapt to future business requirements or technological advancements without significant re-architecture.

To overcome the IP address limitation of the existing VPC, Cilium was deployed to establish an Overlay Network within the Kubernetes cluster. This approach decouples real VPC IP addresses from virtual addresses, effectively bypassing network constraints and providing near-unlimited scalability for IP address allocation.

Karpenter was introduced for efficient node scaling, dynamically adjusting cluster resources based on workload demands.

By implementing vertical scaling, the number of nodes required was reduced by optimizing resource allocation—fewer nodes with enhanced computational capacity, effectively reducing overhead and costs.

An extensive observability layer was established using AWS CloudWatch, Prometheus, and Grafana. This integration provides detailed visibility into pipeline health, execution success rates, and resource consumption metrics, significantly improving operational transparency.

The observability infrastructure also supports proactive incident management, triggering alerts for pipeline failures or anomalies. This capability ensures timely interventions, minimizing downtime and operational disruptions.

Further, metrics tracking data freshness and quality adherence were implemented, adhering to Service Level Agreements (SLAs). This approach ensures continuous compliance with data contracts, maintaining high standards of data integrity and reliability.

Adopting data mesh philosophies, the migration strategy emphasized self-service data infrastructure and decentralized data governance. This model allows non-technical stakeholders to initiate new database migrations autonomously through metadata-driven YAML configurations.

The infrastructure supports potential enhancements, including comprehensive metadata tagging and advanced marketplace concepts where data consumers can transparently view and manage resource consumption and associated costs.