Mage is not just any ordinary data orchestration tool, it could be the superhero of modern data engineering. It streamlines and optimises data processing in an effortless way and will make your other data pipeline tools look like rusty old relics.

In the ever-evolving field of data, many analytics roles offer growth, ownership and innovation opportunities. Data engineers play a pivotal role in designing, building, and maintaining systems that enable organisations to effectively leverage their data. They work closely with data scientists to ensure that data is properly stored, secured, and made available for analysis. Aside form the core concepts that a data engineer should be familiar with, there are also a plethora of tools out there to work with.

Version control systems are the backbone of collaborative development, ensuring that everyone is working on the same codebase and keeping track of changes. But when things go wrong, Git can be a daunting tool to navigate.

Software engineering is far more mature than the current state of data engineering, particularly when it comes to principles. For example, in software engineering, SOLID principles are a set of five design principles that help in writing maintainable and scalable code.

It seems that we are living in a time that will one day be recognised as the beginning of the AI age explosion. AI tools are becoming more intelligent and more accessible. With the release of ChatGPT, concerns have raised again about AI outperforming humans in our jobs, but I believe we have many years ahead of us before AI can fully design and develop our data products from start to finish.

Pandas is a popular data manipulation library in Python, but its syntax is often criticized for being confusing. There are multiple ways to achieve the same output, and these methods may have subtle differences. In this blog post, we will examine the most common Pandas methods and compare them, so you can choose the best approach for your data manipulation needs.

Getting your first data engineering role can be difficult depending on what kind of experience and skill level a company is looking for. The transferrable skills from software engineering and data science makes the transition easier than starting out from scratch and so more often than not, data engineering roles require some years of experience with programming languages like Python and SQL, data warehouse knowledge or similar experience from other data related roles.

In 2023 the demand for data engineers is showing no sign of slowing down. Here we’ll be going over the core skills a data engineer should have and the concepts that have survived the years, with some excellent resources to look at. First let’s take a step back and understand the state of data engineering today - what has changed and what has remained - from the early days of data practices to the modern data stack.

For most Python projects, the same foundational tools can speed up development of your project and remove inefficiencies. This article goes over some of my favourite tools for creating the perfect project.

Apache Airflow is a widely used open source tool in organisations with large amounts of data processing. Created by AirBnb in 2015, Airflow is highly extensible, supporting many use cases in data engineering. It is desgined to orchestrate your data pipelines which are defined by directed acyclic graphs(DAGs).