Some context.
Remember, the primal goal of any data effort is not about capturing and storing, it is very much about turning that data into Information that is useful, making it easily accessible to anyone who needs it to make sense of it and become knowledgeable.
Simple need indeed.
As they say, necessity is the mother of invention. One of the much talked about topics in the data management domain today – Data Mesh – sure fits into this realm of modern-day needs. It provides a new perspective on distributed architecture principles, that have risen out of increased digitalization and hence the proliferation of data across any modern digital enterprise.
As we all know, one of the most important derivatives of digital transformation is the accumulation of vast amounts of data across an organization. In its attempt to make data easily accessible and available to the business decision-makers, the fresh perspectives around Data Mesh attempts to solve the different components of the data management puzzle from Capture, Store, Analyze and Act in a different way. The ultimate goal is to drive the adoption of data management related technology investments and improve the ROIs and business outcomes. Alright, not let’s try understanding “what is data mesh?”.
What is Data Mesh?
Not to put the horse before the cart; before we get into the Why part of the concept around data mesh architecture, let’s try to get a little deeper into ‘What is Data Mesh?’, which might help better understand the Why.
In simple terms, a Data Mesh is characterized by:
A decentralized data architecture that organizes data by specific business domains. Wait, wait. Isn’t this how it usually is in any large enterprise? For example Customer, Product, Sales, and Revenue data are owned, governed, and managed by respective data owners. So, what is different?
The key is the desire to shift from centralized, monolithic, large data warehouses and management systems to a highly federated approach aimed at making data management much simpler, eliminating bottlenecks, and making data access simpler.
Simplicity is the ultimate sophistication!
Raw data is no longer a mere derivative of a digitized business process. It is about value addition. Data is now perceived as a product with opportunities for value addition.
The data mesh architecture attempts to make the data producers as the real subject matter experts and puts the onus of the value generation process on them. Being the primary data producers, they also become the primary source of the critical piece in the analytics self-service puzzle, namely meta data and also the quality of the data. The data producers become responsible for the semantic definition, data catalogue, processes to define and enhance quality and in setting the permissions on usage and governance. Having understood “what is data mesh?”, let’s move to the “why data mesh architecture?” part.
Why do we need Data Mesh Architecture?
While the modern data pipeline architectures have been in existence for long, the sad truth is that many of them have turned out to be monolithic, convoluted and expensive to maintain over a period of time, with increasing failure rates. A significantly large portion of investments in digital transformation efforts these days is spent in data integration; setting up complex data pipelines and polyglot storage systems. Managing such a large data infrastructure also becomes a challenge, leave alone the governance aspects of it and the responsibility that comes to be accountable for the ROI.
There has been an increasing trend of reports in the industry, talking more about the misgivings of such data platform initiatives.
How does Data Mesh Architecture work?
Anyone who has spent time building modern-day digital applications would easily be able to appreciate the resemblance of the Data Mesh approach to that of a Microservices architecture that leverages lightweight, loosely coupled services, organized around business capabilities, to build a customer-facing application. A data mesh attempts to extend similar benefits, to produce valuable data outcomes closer to where the data is created, where data from diverse business domains make more sense, and can be immediately consumed by users for analytics needs and decision-making.
If you ask me, the data mesh architecture is not such a paradigm shift in our thinking on data management. What goes around comes around with the change in our needs and growing complexity. Quite simply, the digital world has resulted in just too much of data exhaust, as we might like to call it, that it has become quite unmanageable and unsustainable to get any meaningful insights out of this data.
Data, data everywhere, not a byte that is useful when we need it the most.
Data mesh is just a more clarified and reimagined approach, in aligning the creation, management and consumption of the data across the business domains in an enterprise in manageable chunks. Data producers have the opportunity, from the value generation perspective, to aid the business users and in the process remove the IT middleman (read as the centralized data warehouse teams) in the scheme of things.
Domain-driven design, modelling, event-driven architectures and streaming design patterns to manage the flow of high velocity, high volume data, domain-oriented, federated ownership, governance and management – all form crucial elements of the design approach, as beautifully explained by the thought leader in this space.
In summary.
A data mesh architecture attempts to solve some of the most fundamental challenges in data usage and adoption, namely
- Value addition motivation and responsibility with the data producers
- Bring the consumption layer closer to the producer, thereby reducing the complexity of data transformation and improving data quality
- Move away from complex, monolithic warehouse architectures that are easy to maintain
- Better aligned and synergies with a microservices-based architecture for operational applications
- A federated approach for management, governance and control
- Above all, a simple goal towards making better sense of data
Data mesh related technologies are still in the early stages of validation and market maturity. Just like any new enterprise architecture principle, the challenge in adoption may not be the complexity of the underlying principles and technology, rather, the mindset change and in understanding the Why.
Necessity drives innovation. Period.
Good reads:
