Data Mesh at ENEL Spa

The Challenge

ENEL Spa, a leading global energy manufacturer and distributor, advanced their cloud native initiative by building the ENEL Digital Platform (EDP). The objective of the EDP is to enable easy integration, standardization, and eliminate the organizational silos that may limit communication in large enterprises. 

The biggest challenge with this project is integrating the immense variety, volume, and velocity of data sources, while maintaining organizational scalability.

To be successful, the EDP must reduce the friction due to point-to-point data integration of their thousand applications. This will eliminate the need to extract data assets from operational sources before copying them towards consumers with the added advantage that decentralized data management will provide enhanced scalability.   

However, the natural tendency for organizations to drift into fragmented departments causes the adoption of multiple, competing data silo solutions, with data lakes and warehouses to centralize data transformation across multiple domains. For ENEL this was a major concern, they wanted to ensure they harvested the most value from their data across the entire organization.

To be successful, ENEL needed to ensure their new platform used not only technology like microservices, but that the platform itself was based in fundamental principles to avoid the pitfalls of fragmentation, silos, and variable standards.

To solve these problems, ENEL Spa selected Agile Lab, the European Data Mesh experts, to design and enable data analytics at scale on their new, microservice-oriented platform based on AWS-services. 

Microservices enable data analytics 

ENEL and Agile Lab worked closely with AWS Services to rapidly innovate the solutions to these challenges.

The ENEL Digital Platform promotes a database per microservice pattern to maximize decoupling and reusability. In this model, an operational database is typically owned by a microservice. Each microservice is required to route data consumers to the best underlying data store based on context and data access type. For example, a microservice selects from any available option, from large access for batch or interactive workloads, to stream processing, based on the requirement. Offloading operational data to analytical data stores leverages CQRS patterns to prepare the microservice to serve read-optimized datasets or events from an event store. 

While microservices live within Amazon EKS, batch and streaming workloads rely on AWS EMR properly configured for autoscaling to adaptively meet consumer needs and optimize utilization cost. AWS MSK serves as the event store and a single microservice can be owner of topics. AWS Fargate allows to bind microservices and stream processing very closely to reach great control over stability and gaining much flexibility in terms of networking options. 

Interactive workloads leverage Dremio from AWS Marketplace for data virtualization while AWS networking capabilities seamlessly embed its features within the whole picture.  

Evolving toward Data Mesh

A data mesh initiative starts from a domain-driven design. ENEL adopted domains at the inception of the EDP, so the project is well-suited for Data Mesh, the emerging data management paradigm. Data Mesh extended EDP’s solid domain-based foundations to close the gap between operational and analytical data planes. After working with AWS to solve the technical challenges and enable analytics, Agile Lab addressed the composition of the data product architectural quantum (the smallest item needed to run an application) including all the necessary part related to data analytics.  

At this stage, microservices are designed by a domain driven approach, so they provide a natural starting point to be extended as source aligned data products. 

Data products can be combined and expanded to create new value for consumers through domain-oriented processes running on EMR as streaming processes or batch workloads. Output ports offer polyglotism through APIs, Kafka topics and Dremio data views. 

Agile Lab designed architecture is flexible for future innovation, and will embrace emerging standards and adapt to promising technology trends. AWS cloud services provide additional adaptability to ensure the success of the ENEL Digital Platform. 

ENEL is a success case on Data Mesh Architecture. Find out how they did it in this video.