Fantastic article that encapsulates many of the challenges I see as I get more involved with Data Engineering. The investment in DE leaders is particularly acute, but goes hand in hand with lack of strategy.
A lot of the challenges in this space come from the lack of time and space C-Suite are being given to drive real change in organisations and the "football manager" approach to changing leadership when things don't go right.
This leads to DE teams having to respond to leaders expecting quick wins and not making space for crucial foundational level fixes that are often required.
Thanks Chris! I'm not an engineer so only work with them from the strategic high-level view, so its good to know that a lot of these challenges are valid.
And totally agree with the lack of time and expectations for quick wins. People don't understand that good engineering is good because its built on strong foundations
Excellent article and great content and such an important topic. As you might have anticipated there might be a few different in opinions. I have just one:
1. Introducing and set up of DataOps and Data Observability capabilities, tooling and their integration with other tech in the data platform stack is not the responsibility of data engineers. I agree and I have an eventually learnt to include in the Lead DataOps Engineer responsibility in the data platform team.
2. However, using their tools and making sure that DataOps practices are followed are very much responsibility of data engineers. I also see them practicing data observability practice in their pipelines and this introduces proactive data quality controls.
Fantastic article that encapsulates many of the challenges I see as I get more involved with Data Engineering. The investment in DE leaders is particularly acute, but goes hand in hand with lack of strategy.
A lot of the challenges in this space come from the lack of time and space C-Suite are being given to drive real change in organisations and the "football manager" approach to changing leadership when things don't go right.
This leads to DE teams having to respond to leaders expecting quick wins and not making space for crucial foundational level fixes that are often required.
Fantastic article that encapsulates many of the challenges I see as I get more involved with Data Engineering. The investment in DE leaders is particularly acute, but goes hand in hand with lack of strategy.
A lot of the challenges in this space come from the lack of time and space C-Suite are being given to drive real change in organisations and the "football manager" approach to changing leadership when things don't go right.
This leads to DE teams having to respond to leaders expecting quick wins and not making space for crucial foundational level fixes that are often required.
Fantastic article that encapsulates many of the challenges I see as I get more involved with Data Engineering. The investment in DE leaders is particularly acute, but goes hand in hand with lack of strategy.
A lot of the challenges in this space come from the lack of time and space C-Suite are being given to drive real change in organisations and the "football manager" approach to changing leadership when things don't go right.
This leads to DE teams having to respond to leaders expecting quick wins and not making space for crucial foundational level fixes that are often required.
Thanks Chris! I'm not an engineer so only work with them from the strategic high-level view, so its good to know that a lot of these challenges are valid.
And totally agree with the lack of time and expectations for quick wins. People don't understand that good engineering is good because its built on strong foundations
Excellent article and great content and such an important topic. As you might have anticipated there might be a few different in opinions. I have just one:
1. Introducing and set up of DataOps and Data Observability capabilities, tooling and their integration with other tech in the data platform stack is not the responsibility of data engineers. I agree and I have an eventually learnt to include in the Lead DataOps Engineer responsibility in the data platform team.
2. However, using their tools and making sure that DataOps practices are followed are very much responsibility of data engineers. I also see them practicing data observability practice in their pipelines and this introduces proactive data quality controls.
Just my two cents…
Fantastic article that encapsulates many of the challenges I see as I get more involved with Data Engineering. The investment in DE leaders is particularly acute, but goes hand in hand with lack of strategy.
A lot of the challenges in this space come from the lack of time and space C-Suite are being given to drive real change in organisations and the "football manager" approach to changing leadership when things don't go right.
This leads to DE teams having to respond to leaders expecting quick wins and not making space for crucial foundational level fixes that are often required.
Fantastic article that encapsulates many of the challenges I see as I get more involved with Data Engineering. The investment in DE leaders is particularly acute, but goes hand in hand with lack of strategy.
A lot of the challenges in this space come from the lack of time and space C-Suite are being given to drive real change in organisations and the "football manager" approach to changing leadership when things don't go right.
This leads to DE teams having to respond to leaders expecting quick wins and not making space for crucial foundational level fixes that are often required.