Shaping organizational culture to promote a Data Mesh initiative
This article wants to introduce the theory of organizational culture and leadership by Edgar Henry Shein as a framework to analyze and address cultural actions.
Table of Contents
- Understanding the culture
- Bridging cultures
- How hard it is to evolve culture
Data Mesh is about organization and culture. Yet, many companies are concerned about the perfect data platform out there that can get it done.
While Data Mesh is a great opportunity for technologies in the data engineering sphere, the social aspects suffer from getting the right attention.
I guess there are several reasons for this to happen:
- If talking about technology is difficult, fixing cultural problems is even harder. This is like the personal fear of going to the psychologist to uncover yourself.
- The cultural and social aspects of a company are very hard to analyze and they look unordered, informal, entropic, vanishing, and implicit.
- Rarely do big enterprises with a strong cultural identity recognize all the facets of their own culture. This keeps a relevant chunk of the company hidden.
The science around organizational culture is relatively young and it is being formally explored since the 80s.
Understanding the culture
Following this theory, a company culture manifests itself at different levels: behaviors, exposed beliefs, and implicit values.
The three layers of organizational culture
Roughly speaking, behaviors are about what a community explicitly shows (so-called artifacts). Examples of artifacts are the Agile Manifesto and the company’s values page. They explicitly state how a community, company, or group of people claim to be. Artifacts are everything that explicitly talks about a company and branding is one of the key elements in this sense.
The exposed beliefs are how a community behaves. An open-source community may collect hundreds of developers eager to help each other (even if they have never met in person). This behavior is spread in the open-source culture. Sometimes it turns into artifacts. The attitude to clarify and write things down is by itself an exposed belief. Explaining how to behave within an open-source community is another example of exposed belief.
Finally, implicit values are intrinsically linked to what a community thinks and believes regardless of how it shows beliefs and behaviors.
Being an agile developer makes you close to the concept of uncertainty. You may even feel personally uncomfortable with long-term planning since you do not consider it realistic.
As an experienced software engineer, you will not feel unsafe after a heavy code review because you understand and appreciate that practice. Those behaviors are not exposed. You as a developer believe in such principles. In your community, you do not need to explain yourself about them. It is implicitly clear what to do and how.
The bigger the company, the more complex it is to build values upon solid principles and lead a cultural change. Underestimating cultural changes is a common mistake and ICT faced it many times in the past.
Further, the proliferation of subcultures brings in more complexity.
A big company can have lots of subcultures split by departments, competencies, educational backgrounds, etc. This brings fragmentation.
Often, it happens that C-level, business lines, functions, and organizational silos grow with specific subcultures.
They are interconnected by formal channels, informal channels, or personal acquaintances.
Organizational silos are a peculiar case. They can grow a subculture around attitudes, technologies, and technical backgrounds.
This paragraph is not an attempt to stereotype professional figures but an example of what can happen in some circumstances.
Thus, give the following description the right importance and focus on your own reality trying to project your feeling about a subculture into the three layers.
Such a department is often subjected to the attitude to buy centralized assets and services and manage those items through ITSM-derived practices. They are rooted in bare metals, old-fashion networking, and storage, CAPEX-driven reasoning with strict long-term roundtrips with procurement, etc. Convergence towards Cloud Infrastructure is usually a big deal because of the macro differences approaching problems.
Mainly, buyer and as-a-service attitudes combined with centralization of the company’s assets do not make scale out the organization. Sometimes, big companies prefer to split those departments between old and new to account for the cultural friction.
Renewing the staff is not simple because — regardless of the subculture specificity — such departments preserve valuable company knowledge. It is easy to experience spare artifacts (formal communications), many exposed behaviors (undocumented processes), and very difficult to interpret implicit values (attitude for centralization).
Traditional functions are often served through operational silos that address CRM, ERP, etc. They are usually linked to specific technologies like SAP, Salesforce, Siebel, mainframe-based applications, etc. Employees working within those departments build their culture around those technologies ignoring the rest of the ecosystems.
Those departments are usually populated by consulting services that spread their own company’s culture. This often makes operational silos share the same exposed beliefs and reinforce their implicit values around someone’s core business.
In these cases, if a company does not invest in its own company culture, artifacts are really poor, exposed beliefs are far from representing a shared behavior, and implicit values root in the company’s domain (business people) and underlying technologies (internal developers + consulting services). The domain is the real lever in this context.
On the one hand, they are similar to Infrastructure in how they manage through centralization. In this sense, they must manage requests as a service and inherit some of the ITSM mindset flavors. On the other hand, they are similar to Operational silos since they are often technology or solution-centric (Cloudera, Databricks, legacy, or custom).
In this case, the influence of consulting services is less heavy but the centralization is shifted from managing assets to managing use cases. For instance, a data analytics department could be in charge of a data lake based on Cloudera where data engineering must respond to all business use cases of the business lines and/or functions of the company.
Depending on the size of the company, there could be multiple operational silos split by business line/function or technologies. Those teams cannot rely on a strong domain-driven culture since they work more as technological crossroads for all business cases. They try to maximize the whole “throughput” of data rather than the value chain.
Their exposed beliefs are often influenced by technology (i.e., low code — no code in cases like Talend, Informatica, or other pervasive tools dictating a working modality).
Their implicit values can really change depending on the specific case. Groups of professionals used to working with open-source-based technologies are usually also open to innovation, curious, and eager to learn. This is good but it could turn into a driver for data engineering shadowing.
People more used to working with pervasive tools and frameworks live more in compliance with the company’s standard but this could result in less openness to innovation. Also, macro technical cases can spin off several subcultures. Big data scenarios vs traditional data pipelines — for instance — can bring in the picture frictions between mindsets.
Wherever you go, you will hear about “The Business” as an entity external to IT. Whether they are business analysts, SMEs, process owners, or so; they will surely look outside the scope of IT developments. As I stated in the introduction, Eric Evans wanted to highlight how to make development and IT closer to each other to make agile developments work better. This is far from being a reality in most companies.
A rule of thumb to check if your company managed to work domain-driven is asking a sample of consulting developers for the domain model or a short explanation of the business objective of their piece of software. Business employees are oriented by the capabilities they are responsible for.
They have business problems and do not really care about technology. Their artifacts are usually very clear since they are more exposed to the external world, thus they must excel in communication. Their exposed beliefs are usually transversal to that of technical people while their implicit values are often rooted in their common experiential business background.
It is important for a company to understand its own subcultures and size communication, expectation, explicit and implicit values, and business objectives in a fair trade-off among each other.
The Data Mesh wave is getting more and more important worldwide, but I see that poor importance is given to the cultural plane.
There is a dedicated chapter of Dehghani’s book “Data Mesh: Delivering Data-Driven Value at Scale” that puts emphasis on the cultural impacts by talking about a sociotechnical approach to share, accessing, and managing analytical data in complex and large-scale environments. Let’s analyze some drivers to create interconnections among several subcultures.
DDD is a pretty involved argument. An organization that wants to adapt to this approach is going to establish some boundaries for its domains. In Data Mesh terms, consumer-aligned data products can be built upon other data products from multiple domains.
A new connection between domains is an opportunity to create business value. Still, establishing the ownership of such a data product is very difficult since domain boundaries are blurred. Domains fight to find who should be responsible for this new burden and it is not easy for an organization to always sort this out.
Domain-driven design enforces a company to establish effective rules to implement data ownership. Organizational issues often hide cultural ones. In those cases, the company misses a strong domain-driven culture that builds data products without conflicts.
Adapt your organization to domains and not the other way around.
Domains exist beyond the expected organization. Adapting domains to the current organization does not break existing silos and enforces wrong models across the company.
Working around this is very difficult. Subcultures communicate through artifacts and exposed beliefs. While artifacts can be artificially manipulated, merging communities of different professionals can increase the friction at the level of exposed beliefs.
Find subcultures representatives most supportive of change and get them to cooperate.
Business-driven by domains
Domains must cover the whole company’s business but this does not happen so often.
DDD is patched here and there in big organizations but its real value shows up when the adoption is broad.
Understanding domains starts far from the mere development of microservices or data products. A good domain-driven culture should focus on business capabilities, sharing the main idea of domain boundaries where processes and entities live.
Usually, it happens that DDD prevails as a development practice with an understanding of entities, connections, and business knowledge sharing localized to specific teams or departments.
Build the big picture, do not lose business pieces.
Thus, only part of the organization’s domains is backed by DDD while the rest of the organization may suffer such a clarity.
This easily brings situations where part of the business is not really represented and supported by the current organization. This perhaps happens even at the level of artifacts when several parts of the organization are not aligned with the strategic view and they run for different goals.
Genuinely spreading the foundation of Data Mesh from the top of the organization can make it more effective in the way employees from the several subcultures perceive the change for innovation.
In my opinion, this is the greatest common denominator that can be beneficial even at the level of implicit values. Good managers do a difference in the way initiatives can be embraced.
Founders, C-levels, and top managers must engage with the Data Mesh principles and best represent the four pillars. They genuinely spread the new culture.
A good Data Mesh initiative should address the sense of community across the whole company.
Cultural impact on domains and data product teams
Data product teams are another tricky thing in the organization that is impacted by a cultural aspect. Data silos are populated by data engineers. When it comes to building data product teams, it is necessary to refactor the organization to account for data engineer skills within each data product team and establish a data platform team.
This is very similar to what happens with agile teams that move to DevOps where there is the need to balance the local autonomy of the DevOps team and a software development self-served platform caring about cross-cutting concerns and automated governance.
Having a team able to autonomously build software from development to deployment with the right degree of freedom to implement infrastructure and manage operations is a challenge.
A similar challenge is the one that brings data engineers into teams that can autonomously build a data product with the right focus and context. Organizations that were organized to keep data competencies in a special department struggle to understand the need for a data product to include data engineers.
Data lakes have been built for years around specialized teams that served the whole company. Also, working with a DDD approach should move software and data engineers closer to the domain data model.
This is not as trivial and wanted as we may wish. Technical professionals often refuse to consider functional knowledge as part of their daily job.
Empower data product teams, share knowledge, build intrinsic motivation.
Anyway, having autonomous data product teams improve productivity and knowledge sharing, indeed software and data quality. Success stories can boost belief in the Data Mesh approach. Promoting agile development culture, and enabling self-served platform capabilities should make data product teams feel they can easily tackle their real data domain challenge.
Depending on the cases, this may affect more the sphere of exposed beliefs than implicit values where a good common culture of software engineering practices is spread across the company.
This is not always the case. Anyhow, it should be easier to open discussions explaining which practical benefits a Data Mesh initiative should bring into the daily job to move intrinsic motivations ahead.
People's engagement depends on their own culture. Building a Data Mesh oriented one may require actions to reinforce extrinsic and intrinsic motivations.
Given the several subcultures in your organization, consider diverse actions depending on the specific context. The more you touch implicit values, the more your incentives will be effective. Thus, the balance is not exclusively in favor of more money on the plate. Instead, money can be supportive of bad competition.
At a certain level, Maslow’s hierarchy of needs applies to an organizational unit and its subculture. Positively influencing an organizational unit in favor of its esteem and cognitive needs can ease the change.
For instance, investing in people's growth, creating career paths in the Data Mesh journey, and promoting fair and good competition are actions that move towards the acknowledgment of merit.
Promote fair competition, do not compete for money.
Addressing a Data Mesh initiative is very complex. Distributing the budget strategically can be very challenging. This is the reason why a Data Mesh project should start small. A good suggestion of how to approach complexity comes from system thinking at the organizational level. Of the thousands of parameters you have in your organization, there are only a few of them that creates a positive impact. Once you have distributed the budget across some projects, this gets consumed whether the project's outcome was impactful or not.
Start with small data mesh projects that work as leverage points for company’s business and culture.
Choosing the right starting point makes the difference. Starting from non-relevant business areas does not show the value of the initiative. Starting from areas fighting cultural change will not bring value. This double condition should make you ask yourself whether Data Mesh is a good fit for you.
Platform and technologies
Looking for technologies that solve the data mesh issue is another pain point.
JIRA and Jenkins are great tools but they cannot bring Agile Methodologies and DevOps into a company by themselves. Similar considerations are valid in this case.
Data Mesh is good for data-driven organizations that must keep pace with the market and wants to organize for it. Scaling out means being able to build value faster and faster. A big enterprise very easily reaches thousands of data sources of any technology.
Dealing with this diversity is one of the objectives of a data mesh enterprise infrastructure. Being able to manage this diversity with no impact on the business is what makes Data Mesh effective to scale out. Building a new data product requires integration with many technologies and with limited resources.
This is where tools can help. Nevertheless, the approach is agnostic to any technologies and only dictates standardized interfaces.
Do not let technologies drive the Data Mesh initiative.
There are companies that embraced any sort of Data Mesh precursor much longer ago than its definition. This is because it is all about data management best practices and organizational issues that can be fixed through a strong cultural change.
Companies living this path from the beginning throughout all steps find themselves with a different culture about data management and a deep understanding of the value of their achievements going through the three cultural levels.
Set company’s objectives and select platform enabling technologies.
Data Mesh is about how to break cultural and data silos that reflect organizational limitations to scale up the business. It’s not a one-fit-all approach, neither theory nor magic.
It is a strategy that requires a company to embrace cultural change. Technologies must serve as enablers for domain-driven development at scale but should not be foundational to the platform's working principles.
How hard it is to evolve culture
In the course of our history, humans’ signs of progress have been impressive. We explored the infinitely small and the infinitely big by building theories, tools, and practices to handle both extremes.
At the same time, while science and technologies move very fast, human culture uses to move ahead a little slower. No one uses ’80s mobile phones, we are forced to use smartphones because technologies require us to renew our tools. Anyhow, flat earth believers still exist 4 centuries after Galileo Galilei’s The Dialogue Concerning the Two Chief World Systems and all of them use a smartphone.
In 2001, the Agile Manifesto seemed to kick off a new era. It was not an actual revolution. Ken Schwaber, Robert C. Martin, and the other signatories of the Agile Manifesto were expert developers and professionals that were talking about their experiences, not inventions.
Thus, we would expect that 21 years after that moment, given the pace of the technologies, everyone building complex software systems should rely on agile methodologies. Yet, this is not true. Many big companies have tried to adopt agile methodologies through a sterile approach.
Most of them failed, even if they claim they didn’t. The reason for them to fail resides in a cultural change where the exposed beliefs of the top management do not match their own behaviors and implicit values. The top management is mostly driven by business objectives and it often happens that adopting agile is part of the strategy to bring innovation and business value.
The missing part is the understanding of the value carried out by such a methodology. The top management does not embrace the change, rather it dictates the change. C-levels transliterate terms talking about backlogs and thinking about long terms plans.
Of course, it is not a matter of tools. JIRA is a great tool but you do not need it to understand why a backlog is different from a Gantt and why the backlog can manage complexity more realistically in modern software development and with business objectives in mind.
DevOps had the same fate. It has been one of the most abused IT terms ever. DevOps is all about breaking silos.
Silos are big chunks of an organization that usually work as a service for other departments but they do not have the right context to act at their best.
Most of the companies that claim they adopted DevOps either added another IT silo called DevOps or bought a bunch of automation tools. Running new silos dedicated to DevOps is proof that there is no match between implicit values and exposed behaviors.
The whole value chain risks big losses from those misunderstandings since those companies do not get the advantage of the cultural values. No culture, no party.
Again, it is not Jenkins that enables DevOps. It is a great tool, something that is surely worth learning but the understanding of the competitive advantage of shifting the organization from IT Silos to DevOps teams to improve the time to market is not a lesson that you learn from tools.
Our history bears witness to the failures due to the lack of effort to genuinely promote cultural change.
This article started with the framework of prof. Edgar Shein to analyze organizational culture and go deeper into issues that a company facing the Data Mesh challenge can meet.
I approached the intuition we have about our experience to what this framework provides to give the reader an articulated perspective of the problem that can be faced at the cultural level.
This contribution started with some other revolutions that we have lived in our lifespan: The Agile Manifesto, DDD, and DevOps movement. It’s impressive how they have in common with Data Mesh, how they are rooted in the same organizational problems (from different perspectives), and how they share some (if not all) principles: they are not about technologies, they want to break silos, they require a strong cultural change.
A Data Mesh initiative requires maturity in terms of those practices. Make sure your company identifies with common implicit values about what those practices convey. An honest and genuine understanding of your company’s culture will give the right perspective on the effectiveness of your initiative.
This article wants to open a discussion about how culture affects Data Mesh (and in general ICT transformation) initiatives in practice and give you a base for self-assessment based on a grounded theory.
If you recognize any of the dysfunctions that I have mentioned in this article or others that I missed, feel free to leave me a comment.
Business Unit Leader of Utilities. Ugo leads high-performing engineering teams and has a track record of successful digital transformation, DevOps and Big Data projects. He believes in servant and gentle leadership and cultivates architecture, organization, methodologies and agile practices.
Shaping your organization is one of the first steps of your Data Mesh journey. Find out whether you are on the right path ready by taking our Readiness Assessment!