Data Mesh at ENEL Spa

The challenge

ENEL Spa, a leading global energy manufacturer and distributor, advanced their cloud-native initiative by building the ENEL Digital Platform (EDP). The objective of the EDP is to enable easy integration, standardization and eliminate the organizational silos that may limit communication in large enterprises.

The biggest challenge with this project is integrating the immense variety, volume, and velocity of data sources, while maintaining organizational scalability.

To be successful, the EDP must reduce the friction due to point-to-point data integration of their thousand applications. This will eliminate the need to extract data assets from operational sources before copying them towards consumers, with the added advantage that decentralized data management will provide enhanced scalability.

However, the natural tendency for organizations to drift into fragmented departments causes the adoption of multiple, competing data silo solutions, with data lakes and warehouses to centralize data transformation across multiple domains. For ENEL this was a major concern, they wanted to ensure they harvested the most value from their data across the entire organization.

To be successful, ENEL needed to ensure its new platform used not only technology like microservices, but that the platform itself was based in fundamental principles to avoid the pitfalls of fragmentation, silos, and variable standards.

To solve these problems, ENEL Spa selected Agile Lab, the European Data Mesh experts, to design and enable data analytics at scale on their new, microservice-oriented platform based on AWS-services.

Microservices enable data analytics

ENEL and Agile Lab worked closely with AWS Services to rapidly innovate the solutions to these challenges.

The ENEL Digital Platform promotes a database per microservice pattern to maximize decoupling and reusability. In this model, an operational database is typically owned by a microservice. Each microservice is required to route data consumers to the best underlying data store based on context and data access type. For example, a microservice selects from any available option, from large access for batch or interactive workloads, to stream processing, based on the requirement. Offloading operational data to analytical data stores leverages CQRS patterns to prepare the microservice to serve read-optimized datasets or events from an event store.

While microservices live within Amazon EKS, batch and streaming workloads rely on AWS EMR properly configured for autoscaling to adaptively meet consumer needs and optimize utilization cost. AWS MSK serves as the event store and a single microservice can be owner of topics. AWS Fargate allows to bind microservices and stream processing very closely to reach great control over stability and gaining much flexibility in terms of networking options.

Interactive workloads leverage Dremio from AWS Marketplace for data virtualization while AWS networking capabilities seamlessly embed its features within the whole picture.

Evolving toward Data Mesh

A data mesh initiative starts from a domain-driven design. ENEL adopted domains at the inception of the EDP, so the project is well-suited for Data Mesh, the emerging data management paradigm. Data Mesh extended EDP’s solid domain-based foundations to close the gap between operational and analytical data planes. After working with AWS to solve the technical challenges and enable analytics, Agile Lab addressed the composition of the data product architectural quantum (the smallest item needed to run an application) including all the necessary parts related to data analytics.

At this stage, microservices are designed by a domain-driven approach, so they provide a natural starting point to be extended as source-aligned data products.

Data products can be combined and expanded to create new value for consumers through domain-oriented processes running on EMR as streaming processes or batch workloads. Output ports offer polyglotism through APIs, Kafka topics and Dremio data views.

Agile Lab designed architecture is flexible for future innovation, and will embrace emerging standards and adapt to promising technology trends. AWS cloud services provide additional adaptability to ensure the success of the ENEL Digital Platform.

Il Data Mesh è il futuro della Data Engineering?

Is Data Mesh the Future of Data Engineering?

To extract real value from data, companies need to change their approach to Data Engineering. That’s why Data Mesh is becoming increasingly popular.

Agile Lab explains what it is.

By Roberto Coluccio – Agile Lab Data Architect

 

Is Data Mesh the key that’s been missing?

Is Data Mesh the key that’s been missing to create a 360-degree view?
There is no doubt that the vast amounts of data generated from our digital lives is one of the greatest assets of our time. One of the many ways to take advantage of the data being collected is through a so-called 360 view. In a nutshell, it allows companies to create a holistic picture of their customers by collecting data from different types of touch points…

By Roberto Coluccio – Agile Lab Data Architect

 

Meetup – Data Mesh 101 – Data Mesh Italia – 29th September 2022 | 19 CET

We’re pleased to announce the Data Mesh Italia group’s first official event: we start with DATA MESH 101, an overview of the paradigm, in Italian, held by Roberto Coluccio, Data Architect at Agile Lab.

It will be an online event, where everyone will have the opportunity to talk, chat, and interact.

See you on 29th September 2022, at 19 CEST.  

Learn more and reserve your seat now!