A data engineer builds and runs the plumbing that every team working with data depends on: analysts, data scientists, product teams, finance, operations. Their work goes largely unnoticed when everything runs smoothly, and very much noticed when a dashboard shows wrong figures or a model trains on yesterday’s data. It is a software engineering job applied to data, with its own constraints of volume, freshness, quality and cost. This article describes what a data engineer does, what sets the role apart from neighbouring jobs, the skills expected, the paths that lead to it and how to present this profile on a resume.
What a data engineer does day to day
The heart of the job fits in one word: pipelines. A data pipeline fetches information where it is created (an application, a production database, a sensor, a file sent by a partner, an API), transforms it and delivers it where it will be used (a data warehouse, a data lake, a visualisation tool, a model).
In practice, a data engineer’s week mixes several types of work.
- Building ingestions: connecting a new source, handling formats, schema changes and recovery after errors.
- Writing transformations: cleaning, deduplicating, joining, aggregating, most often in SQL, sometimes in Python or with Spark 🔗 when volumes require it.
- Orchestrating: scheduling jobs and managing their dependencies and failures with a tool such as Airflow 🔗, Dagster or the cloud providers’ orchestrators.
- Guaranteeing quality: writing tests on data, monitoring volumes and outliers, alerting the right people when a source drifts.
- Managing the platform and its costs: choosing how tables are partitioned, setting quotas, watching the warehouse bill, automating infrastructure deployment.
- Working with users: understanding what a team needs, documenting tables, explaining why a metric has changed.
The balance varies from one company to another. In a scale-up, the data engineer is often alone or one of two and covers the whole chain. In a large group, they may focus on real-time streaming ingestion, on the platform or on a business domain.
Data engineer, analytics engineer, data scientist
The titles overlap, and job ads mix them up. Three reference points help you find your way.
The data engineer is responsible for moving data and keeping it reliable: ingestion, streams, platform, performance, costs. They code a lot, in Python, SQL, sometimes Scala or Java, and work with software engineering tools (Git, tests, continuous integration, containers, infrastructure as code).
The analytics engineer is a more recent role, popularised by tools such as dbt 🔗. It sits between the data engineer and the analyst: starting from data already ingested into the warehouse, they build the clean, tested and documented models that analysts use. Their main tool is SQL, combined with software practices (version control, reviews, tests).
The data scientist uses data to answer a question or build a model: forecasting, segmentation, recommendation, experimentation. They consume the work of the first two. For them, statistics and modelling take precedence over plumbing.
In a small organisation, the same person may hold two of these roles. On a resume, however, you have to choose your centre of gravity. The data engineer resume template and the data scientist resume template show the difference clearly: the same languages, but bullets that tell completely different stories.
The skills expected
Data engineer job ads converge on a common foundation, which can be arranged in four blocks.
The software foundation
- Python for ingestions, scripts and tests, with good development practices: project structure, dependency management, unit tests;
- Git, code review and continuous integration;
- Docker, and often some infrastructure as code (Terraform) to deploy your own tools.
Data
- Advanced SQL: complex joins, window functions, query optimisation, reading an execution plan;
- modelling: star schema, fact and dimension tables, tracking historical changes;
- formats and storage: Parquet, Avro, partitioning, compression.
The platform
- a cloud warehouse: BigQuery, Snowflake, Redshift or Databricks;
- an orchestrator: Airflow, Dagster, Prefect;
- a transformation tool: dbt has become very widespread;
- real time for some roles: Kafka 🔗, Pub/Sub, Kinesis, Spark Structured Streaming.
Practices
- data quality: tests, schema contracts, observability;
- governance: catalogue, documentation, access management, compliance with personal data protection rules;
- cost control: every badly written query on a cloud warehouse has a price.
Nobody masters everything. A junior is expected to know Python, SQL, Git and an orchestrator; an experienced engineer to design robust pipelines and model data; a senior to handle platform architecture, quality and costs at scale.
The paths that lead there
There is no single data engineering degree. Profiles come in through several doors.
Engineering schools and master’s degrees in computer science or data science provide the most direct foundation: databases, distributed systems, software engineering. An internship or work-study placement in a data team then gives direct access to a junior role.
Backend developers often make a very good transition: they already master code, tests and production. What remains is to deepen analytical SQL, modelling and data tools. On a resume, they highlight the batch jobs, message queues and databases they have already worked with.
Data analysts who code frequently go through the analytics engineer role before becoming data engineers. Their asset is knowledge of business needs; their work in progress is the software foundation (structured Python, tests, Git, deployment).
Database administrators and operations engineers can also move into this job, drawing on their knowledge of performance and infrastructure.
Data-oriented certifications from cloud providers, such as Google Cloud Professional Data Engineer 🔗, AWS Certified Data Engineer – Associate 🔗 or the Databricks 🔗 one, give structure to learning a platform. They help with the first screening when the ad mentions the provider, without replacing a pipeline actually built.
Building a data engineer resume
A convincing data engineer resume tells the story of data flows, not tools. For each experience, it answers four questions: where the data comes from, how much of it flows, how quickly it is available, and for whom.
Before: "Developed data pipelines with Airflow, Spark and BigQuery."
After: "Designed the ingestion pipeline for website and store orders into BigQuery (Airflow, dbt): 3 million rows per day, data available to the pricing team before 7 am, freshness and volume tests with alerts in the team’s channel."
A few additional tips:
- show quality through an effect: fewer incidents reported by users, shorter time to fix, tests added on critical tables;
- mention costs if you reduced them, even modestly;
- group your tools by use (ingestion, orchestration, transformation, warehouse, infrastructure), as explained in our article on presenting your tech stack;
- prepare for the design interview: you will almost always be asked to draw a pipeline on a whiteboard. Our article on preparing for a technical interview offers a method for talking about a project in depth.
Content makes the difference; form keeps you from being rejected too early. The TechCV editor lays out your pipelines and tools on a plain page whose text screening software extracts correctly. You can start from the data engineer template and add your own volumes.
Frequently asked questions
Do you need to code to be a data engineer?
Yes. It is in fact what distinguishes the data engineer from many analysis roles. Python and SQL are essential, and software development practices (tests, code reviews, continuous integration) are increasingly expected from junior level onwards.
Data engineer or analytics engineer: which should I choose to start?
If you come from analysis and SQL is your strong point, the analytics engineer role is a natural entry point. If you come from development, or you enjoy infrastructure and real time, aim directly for data engineer roles.
Is Big Data still a term to put on a resume?
The term has aged. Recruiters prefer precise volumes and tools: "90 million events a day on Spark" says much more than "Big Data environment". Keep the word only if it appears as such in the ad.
Do I need to know a particular cloud?
It is better to know one in depth than three superficially. The concepts (object storage, warehouse, orchestration, access management) carry over from one provider to another, and recruiters know it.