Data mesh is putting data at the center. As I said Data is the value, data is the product…so the greatest effort is spent in letting people CONSUME data instead of processing and managing them.
The core idea is that we will have only two stakeholders, who are producing data, let’s say selling data, and who are consuming data, buying them.
And we all know that in a real-world the customer is ruling the market, so let’s think about which are the needs of data consumers:
1. they want to be able to consume all the data of the company in a self-service way. Amazon-like, you browse and you buy, so basically you are consuming products from a marketplace.
2. they don’t want to deal with technical integration every time they need a new data product.
3. they don’t want to spend time trying to understand which are the characteristics of the data, e.g. if you are willing to buy something on Amazon, are you going to call the seller to understand better how the product looks like? NO! The product is self-describing, you must be able to get all the information you need to buy the product without any kind of interaction with the seller. That’s the key, because any kind of interaction is tightly coupled, sometimes you need to find a slot in the agenda, wait for documentation, and so on, this requires time thus destroying the data consumer time to market !!!
4. they want to be able to trust the data they are consuming. Checking the quality of the product is a problem for who is selling, not buying it. If you buy a car, you don’t need to check all the components, there is a quality assurance process on the seller side. Instead, I saw many and many times performing quality assurance in the Data Lake after ingesting data from the operational system, doing the same in the DWH after ingesting data from the lake, and again in BI tool maybe….after ingestion from DWH.
Data Mesh is simply enabling this kind of vision around data in a company. Simple concept, but really hard to build in practice: that’s why it’s still not mainstream ( but it is coming ). It has been coined in 2019 and we are still in an experimental phase with people exchanging opinions and trying it. What is truly game-changing for me is that we are not talking about technology, we are talking about practice!