A book to help you understand the Data Mesh revolution
If you are anything like me, you start from a healthy scepticism for new technology buzzwords, like Data Fabric or Data Mesh. All too often these have proven to be just marketing hype by IT suppliers who are rebranding old solutions. However, the risk of such a reaction is you can miss what is important new thinking.
I recently completed reading “Data Mesh: Delivering Data-Driven Value at Scale” by Zhamak Dehghni. It has convinced me that there is much more to Data Mesh than I realised and that it matters. Unlike all those over-hyped IT supplier pitches, it painstakingly explains both the need for a new ‘inflection point’ in our approach to managing data and how data mesh can help. She also clarifies exactly what data mesh means, both in theory and in detailed logical design.
So, although this is a much more technical text than I would usually review on this blog, I consider it important enough to include it. Understanding the principles of a data mesh approach to producing, maintaining, discovering & sharing the data needed within an organisation could revolutionise your approach. So, I encourage data leaders to get their heads around this new vision of what Zhamak describes as a sociotechnical approach to managing data. It has implications far beyond the hardware & software involved.
How this book shares what Data Mesh means in practice
It is amazing how much detail Zhamak manages to include within this book, for two reasons. First, it is patiently explained step by step, so even those whose specialism is not data management or data engineering can grasp all the concepts. Second, as she honestly highlights at several points, this is still an emerging field. Many of the theoretical components are not yet available “off the shelf”. This could of made this book feel like it had been written too soon or was trying to hit a moving target. Zhamak manages to avoid both pitfalls by both laying thorough theoretical foundations & by staying at a logical level of implementation design (but still sufficiently detailed to be helpful to practitioners).
So, what will readers discover in this book? The material is divided into 5 parts. Firstly, explaining the concept of data mesh and each of the key principles. Secondly, making the case for why data mesh and the benefits it can deliver after the inflection point of change. Thirdly, how to design the key components of a Data Mesh Architecture. Fourthly, how to design the key services of a Data Mesh Product Architecture. Lastly, how to get started in terms of both systems and organisational culture.
What makes this book so brilliant is how Zhamak manages to effectively communicate with multiple levels of readers. Data leaders will find parts 1 & 2 the most relevant. Once they are convinced they can share parts 3 & 4 with their teams to help them get to grips with the practical implications, especially in terms of architecture and design. Then they can both read and plan together using part 5. A recipe for a book that will not just be read once, but kept as a reference guide as this technology matures.
But what is a Data Mesh?
Put simply, it’s a different approach to storing, managing, sharing, using & deploying data to generate value. Within this book, Zhamak makes a convincing case for why we have reached the end of the centralised approach to data management. Relying a central warehouse or lake and data team is no longer viable. The speed of change & complexity of today’s organisations, couples with the proliferation of potential data sources and growth in expectations of use cases – all conspire to ask more than the old approach can deliver.
So, what is different with Data Mesh? At its simplest it comes down to the four principles that are explained in this book (both in theory & at a design level):
- Domain-Driven Design (business domains own data & responsibilities)
- Data Products (encapsulated data & code, supporting services to use that data properly)
- Self-Serve Data Platform (delivering a network of domains that effectively share data products)
- Federated Computational Governance (semi-automated solutions embedded into all the above)
It is only when you see the traditional problems that each element solves that you begin to grasp the brilliance of this solution. It is a radical change from traditional ways of working. But, as technology & organisations catch up it offers a far more capable vision of more adaptive data usage in businesses.
What are some of the key takeaways for data leaders?
The first I think is that things cannot continue as they are. Even if you are not ready to make a major change like the transition to a Data Mesh approach, I recommend reading the start of this book. That is because it effectively critiques why the current approach is doomed to failure. Zhamak shines a light on how & why a central data team is doomed to failure in the light of emerging needs. It is not possible for one team to understand and respond to the changing realities and needs of the rest of a business. Especially as the scope of data usage extends to most functions within each business domain.
The other key challenge is organisational redesign and accountability. Perhaps the most challenging aspect of migrating to such a federated approach is the responsibility it places on areas that have avoided it up till now. To achieve this approach requires data expertise & technology expertise within each domain. Domain leaders will have to go far beyond fine words to actually act as data owners in practice. It is needed, but it is also a significant culture change. As the author puts it “…executing data mesh needs a multifaceted organizational change. Iteratively and along with delivery of your data mesh thin slices, I encourage you to look at modifying all facets of organisational design decisions…”
Lastly, I would recommend reading this book to get ready for the future. As you work through the detail in the logical design chapters you realise how many components are not yet readily available. Pioneers will need to build key parts of this approach or work around limited systems. That will delay mass adoption, but also drive the development of future packaged solutions. As with all advances in technology, progress will probably be slower than we imagine now. However, the latter implications may well be larger than we can’t imagine now. So, I advise data leaders to get their heads around this sooner rather than later. Otherwise, they too may end up as outdated as data warehouses are beginning to look.