This week, I came across a few interesting ideas that made me think about how small actions and habits can lead to growth.

I’ve learned that being a well-rounded data engineer goes beyond technical skills. It’s about how you approach problems, communicate with teams, and deliver outcomes. Here are four ideas that have helped me work better and make a greater impact, with some examples

2024 was a busy year. A personal highlight is that I married my wife in the summer in Nice and had an amazing honeymoon in Santorini. I immersed myself in learning. I read and took notes on over 800 articles that spoke to me. They were on tech, science, data engineering, productivity, stoicism and many other topics. I simplified my workflows by cutting out complexity.

When I started out, I didn’t give much thought to how I stored my configurations. A JSON file here, a YAML file there, whatever came to my mind first. As projects grew more complex, the way I structured my configuration files began to matter a lot more.

After years of building data pipelines, from ones that lived in janky notebooks to ones that automate complex workflows in production environments, logging wasn’t something I paid a lot of attention to.

I believe we can all agree that writing tests can feel as mundane as doing the laundry as most of us would probably rather be coding new features or learning something new. We would also agree that testing is undeniably a critical component of data engineering, setting individuals apart by the skill it demands.

A data engineer doesn’t just code all day, they also design, plan, tinker, analyse, fix bugs, attend meetings and manage trade off of best principles all day. Being familiar with Terraform is a great skill to have to automate provisioning and management of infrastructure. As well as a cross-functional skill, it aligns data engineers with DevOps principles, fostering continuous integration, delivery, and deployment.

In my early days as an analyst, I was always eager to automate tasks, often finding myself repeating lines of code without scalable solutions in sight. As I delved deeper into the field of data engineering, I started tinkering with cloud, where I found myself hooked on the possibilities.

Why is it that data quality is always left for last or never given the attention?

Picture yourself immersed in a tricky data engineering task - juggling date manipulations and debugging errors.