I think consistency is a vital piece here and something that worries me about Data Mesh and its enticing goal of decentralized data. To some this means, completely autonomous units managing the data that they care about. In reality there is a lot of data that is common across domains and complete autonomy will take us back to silos and poor interoperability. I think a well designed data platform can support a Data Mesh program and provide that consistency across domains. And good data architecture and data modeling can make this happen!
You are completely bang on Jamie. The biggest problem I have seen with Data Mesh focused platforms is the lack of a foundational layer and communication channels to allow different decentralised teams to work together efficiently. Yes it enables the business users, but most Mesh architectures also mean you are starting from scratch every time you build something new...
Thanks Kaan! Yeah I'm always going to mention the people in it because it is such a crucial element that companies often forget when thinking about tech
I think data platform is more ambiguous than warehouse. I do plan on doing an article on warehouses vs. lakehouses vs. lake, etc. so I will take a look at the publish teh data bit you mentioned. Unfortunately nothing is really simple in the end!
Oomph... Data x philosophy? Love it. Of course Cartesian doubt was all about answering the question "do I even exist?" - therefore anything that makes sense of data CAN constitute a data platform I guess?
I think consistency is a vital piece here and something that worries me about Data Mesh and its enticing goal of decentralized data. To some this means, completely autonomous units managing the data that they care about. In reality there is a lot of data that is common across domains and complete autonomy will take us back to silos and poor interoperability. I think a well designed data platform can support a Data Mesh program and provide that consistency across domains. And good data architecture and data modeling can make this happen!
You are completely bang on Jamie. The biggest problem I have seen with Data Mesh focused platforms is the lack of a foundational layer and communication channels to allow different decentralised teams to work together efficiently. Yes it enables the business users, but most Mesh architectures also mean you are starting from scratch every time you build something new...
I think a pragmatic hybrid of decentralization with some central coordination can really work well. That foundational layer you mention is vital.
Solid and concise, awesome work Dylan! By the way, I love the 4th point (especially the one about the right people)
Thanks Kaan! Yeah I'm always going to mention the people in it because it is such a crucial element that companies often forget when thinking about tech
Excellent article! It all comes down to people, in the end.
Thanks Morgan! Totally, people are the ones driving the platform, so they have to underpin whatever the technology facilitates
So a “data platform” is a new term for “data warehouse”, perhaps? I like Dr. Kimball’s “publish the data” analogy.
I think data platform is more ambiguous than warehouse. I do plan on doing an article on warehouses vs. lakehouses vs. lake, etc. so I will take a look at the publish teh data bit you mentioned. Unfortunately nothing is really simple in the end!
I agree! I think a platform could be realized in many different ways including a warehouse, lake, virtualization platform, Data as a Service APIS, etc
Oomph... Data x philosophy? Love it. Of course Cartesian doubt was all about answering the question "do I even exist?" - therefore anything that makes sense of data CAN constitute a data platform I guess?
Haha data x philosophy is my jam! And I like the way you are going at it Johnny!