Senior Data Engineer (Databricks)

About the role

A Senior Data Engineer is needed to build and maintain the data pipelines that turn raw data into stable, trusted datasets used to optimise performance. Working closely with Physics and Analytics Engineers, the role turns exploratory and lab-built datasets into production-grade, shared data. It suits someone who can go deep into technical detail while keeping the wider system in view, and who can explain their work clearly to technical and non-technical audiences.

Key responsibilities

• Design, build, maintain and optimise data pipelines that transform raw data into reliable datasets for performance optimisation.
• Work with Physics and Analytics Engineers to understand data needs and turn them into production solutions.
• Turn lab and exploratory datasets into trusted, documented, shared datasets.
• Translate scientific and analytical concepts into scalable, production-ready code.
• Gather and clarify requirements with stakeholders and confirm that delivered data meets their needs.
• Lead or support incident response, including root cause analysis, troubleshooting and debugging.
• Monitor pipeline health and data quality, and build dashboards and alerts.
• Present findings, designs and status updates to technical and non-technical audiences.

Required profile

• Strong data fundamentals: data modelling, data quality, ETL/ELT patterns and pipeline design.
• Advanced proficiency in Python and SQL.
• Hands-on experience with Spark and PySpark for large-scale data processing.
• Proven root cause analysis, troubleshooting and debugging skills in production environments.
• Ability to move between detailed technical analysis and broader system context.
• Strong communication and presentation skills.
• Preferred: experience with scientific, physics or engineering data; experience with Grafana for monitoring and observability.

Required skills

• Databricks
• Python and SQL
• Spark and PySpark
• Git version control
• ETL/ELT patterns
• Preferred: Grafana