Data Engineer / Data Analyst – Real‑time Databricks Lakehouse

About the role

Lumenalta is seeking an experienced Data Engineer / Data Analyst to support a high‑performance motorsport client building a real‑time Databricks Lakehouse platform. You will be responsible for profiling incoming high‑frequency telemetry, optical tracking, and timing/scoring data, and defining the data‑quality rules that ensure trustworthy information for engineers, analysts, and race officials.

Key responsibilities

• Profile raw sensor data in the Bronze layer, measuring distributions, null rates, cardinality, schema drift, duplicates, and out‑of‑order records.
• Collaborate with race engineers, analysts, and competition officials to translate domain knowledge into concrete data‑quality expectations.
• Define, document, and codify validity, completeness, uniqueness, timeliness, and range checks for Bronze‑to‑Silver promotion.
• Build and maintain transformation pipelines that enforce schema, cast types, deduplicate, standardize, and quarantine records that fail quality checks.
• Implement data‑quality enforcement using Databricks‑native tooling such as Delta Live Tables, Lakeflow Declarative Pipelines, Great Expectations, or Databricks DQX.
• Establish data contracts and maintain documentation of expected schemas and semantics.
• Monitor rule pass/fail rates over time and generate alerts for quality deviations.

Required profile

• Proven experience in data engineering or data analysis, preferably with real‑time streaming data.
• Strong understanding of data‑quality concepts and ability to translate business requirements into technical rules.
• Experience working closely with domain experts to define trustworthy data pipelines.

Required skills

• Databricks
• Delta Live Tables
• Lakeflow Declarative Pipelines
• Great Expectations
• Databricks DQX