The Other Reason why Data Mesh is Hard (And How You Can Solve It)

Abhishek Mishra
8 min readOct 1, 2024

--

I am making an assumption already- Data Mesh is hard (and hence exploring the “other reason”)
But if we think about it- should data mesh be hard?

Should de-centralizing data and federating control be hard? Should designing and maintaining data assets from the point of view of the actual use case be hard?
Let’s see what implementing data mesh looks like at a high level.

Should Data Mesh be Hard?

First of all, data mesh is always a choice. And it it not for everybody.

It’s not for everybody- so data mesh is only hard by choice.

You see, generally, you won’t have a nascent data team adopting data mesh.

There can be exceptions surely. But you will find that data teams which are struggling to scale due to centralization will be considering paradigms like data mesh. These teams have history, momentum, practices- the whole shebang that led to problems at scale, which in turn led to data mesh coming into consideration. Newer and smaller teams don’t adopt data mesh because the time and money it will take to set up the entire architecture and plumbing for data mesh won’t be worth the returns (at least not immediately).

That said once an experienced data team adopts a mesh strategy, they still have their work cut out for them. Not only do they have to adopt data mesh, they need to do it while continuing to deliver value to their existing data users, while maintaining the infrastructure, context and code that runs today all while they repurpose and/or build the things needed to mesh it up.

It’s not just changing the wheels of a moving car but re-configuring the engine of a rocket ship while it is cruising in space.

Data Initiatives at Scale is a Like Changing a Rocket’s Engine in Space

Hence, it needs change management from an established today (I am saying established in terms of stability and value to consumers) to a demanding but promising tomorrow. I will talk more about these change management complexities in another write up.

Today, I want to talk about something that is still a challenge even if you figure out all the change management stuff and have a great tool to accelerate you a couple or bunch of paces along the way.

The other reason why data mesh is hard is because of the self serve platform.

Did Someone Say Self Serve?

What the heck is the self serve platform?

The self serve platform enables different personas in an organization to work with data products.

It is one of the major parts of data mesh as per the original data mesh book by Zhamak Deghani. Some of the other components of data mesh are “data as a product”, “federated computational governance” etc.

To understand the self serve platform, think of this analogy. If data mesh is a big opera concert, the self serve platform is the conductor in the middle of it all- guiding and orchestrating the various moving parts to produce beautiful music.

The self serve platform empowers self serve journeys for key personas in the data world. Let us examine these journeys at a high level, for two key high level personas.

Data producers

You have data producers or data product owners who would be creating data products using the data platform. If we truly follow the vision for data, then the process of creating the data product should be via declarative resources and not with code. The closer you are to defining data products with configuration, the more aligned it is to the data mesh vision where you are enabling semi technical or analytical users to create products.

Why this is hard
Taking declarative syntax and creating a data product is hard. Why? Because doing this includes-

  1. Organizing a set of data assets and the code (transformation) that created those assets
  2. Including upstream assets that are behind the valuable data assets (valuable assets are also known as “output ports” in data product parlance)
  3. Enabling connecting to output ports programmatically (typically output ports being API endpoints, Kafka queues)
  4. Enabling a copy of the governance policies at play and the necessary configuration to run the data product independently (called as “the sidecar”)
  5. Creating this entire container (logical and physical) with just declarative syntax

Of course, there is existing technology to be able to do some of that, but it needs to be modified to be able to deploy this unit independently.

The other interesting thing that makes doing this a lot harder- this all needs to happen within a domain. At enterprise scale, every domain may have its own flavor and restriction of infrastructure. This means there could be one domain or one department that works on GCP, while there is another department that works on AWS.
You can understand that the scale and complexity grows when you need to support this kind of a multi cloud setup- since the declarative syntax that drives the creation of data products still needs to be agnostic.

That was a sneak peak into what the self serve platform needs to do for data producers, with some of the complexities. Let’s analyze the other important persona we have.

Data consumers

Now I personally feel that the data product consumers’ journey makes the self serve platform even more complex. As per the data mesh vision, the data product consumer should be able to utilize the self serve platform to be able to work with a product.

Let’s see what working with the product means.

  1. You can connect to the outputs.
  2. You can further create new data products.
  3. You can create other assets using the data like models etc.

So imagine, you access the self serve platform- maybe it has a data catalog that is hooked up to it already? Here is what your journey could look like-

  1. You use a catalog to perform discovery.
  2. You locate the products that look interesting to you.
  3. There needs to be technology that allows you to connect to the data product, interact with the physical layer
  4. You are able to work on multiple data products without worrying about the underlying connectivities and infrastructure.

The reason why it becomes harder- as you go enterprise you have to do all of this in a more restricted (read: governed) way.
So for example there may be a data access policy that says,
“if you belong to a certain geography, then you’re not permitted to use a certain class of data”. So whenever you are playing with data products you would want these to be enforced.
Hence, there is tech that needs to responsible for building of the policies and enforcing them at runtime. All of this needs to be in a way that does not hamper your experience i.e. seamlessness.

But the fun doesn’t end here- let’s add in the perspective of multiple domains and different datasets. Once you consider playing with data products belonging to different domains- you see that every domain may have a different set of policies, that may or may not play along nicely with each other.
Sometimes data sets can be combined to result in data which is more problematic while the individual datasets may be fine. This increases complexity while consuming data products- also a responsibility of the self serve platform.

So how do we solve it? Can we thin slice our way to value?

Before we get there, there is another important piece of the self serve platform puzzle- observation and traceability. If the data platform does not give confidence to the auditors or to the data governance officers that they really know what’s happening with data, then the initiative will never take off.

Alright, enough scary stories. It’s time to break the behemoth down 💪

Presenting the Data Mesh Thin Slice Strategy- Metadata Edition!

So the very first idea that you can work on is something that we do at Atlan, which is working more at a metadata layer. So there are two parts to data products.
There is the physical layer where you publish- where you create the actual data product. You actually create the infrastructure, the container etc. that is responsible for, you know hosting the data product.

But there is a logical element to it as well. The logical element is more about making the data product discoverable and governed.

The metadata-first strategy to implement data products
So one strategy to get at least enter into this world is to have start with a catalog experience as a first step towards data mesh. Taking an example, imagine you have 10,000 raw data assets that are around. You haven’t started creating data products out of any of these. There is no physical layer that exists today.
The question to ask is- can we at least have a logical segregation? A logical grouping of some of these assets that we start calling them as data products? Of course, I’m biased, but I believe Atlan has probably the most superior mark market offering here.

Why does this work
The reason why this thin slice works is because you logically organise the structure within the domain and it allows you to talk to the outputs of the product. Think of them as “planned products” if it makes it easier. This allows you to test whether your data product idea has value or not based on the feedback that people give you on the logical segregations. That will help you to validate whether people even need self serve capabilities for working on physical products without really committing to all the infrastructure madness.
That’s great! You have your “planned products” ready and your data consumers are telling you if they find value in them or not. They are able to connect to the physical assets too. The next question to ask is who do I get some observability and auditability in this thin slice? The metadata access will always be governed as many modern catalogs have governance capabilities. So your build time policies will be good- that should solve most of your troubles especially when you are in your initial days of data mesh.

It’s the runtime data access which can get challenging. But a trick here is knowing if you have enough of observability set in your existing warehouse platforms. It’s not a one-stop solution to start with, but it lets you keep your observability burden at bay.

It’s not a very tidy solution, but that’s a thin slice.

That’s that’s one strategy to skin the cat. There is yet another way- approaching it from the physical layer first- which I will include in my next write up in the interest of time and length :)

--

--

Abhishek Mishra

Product manager- building a home for data teams @ Atlan. Data & agile enthusiast. Ex- Thoughtworker. Wrote a book. Behind 9 products gone live!