Data Engineer

About the role

RedCloud is looking for a Data Engineer to join a client’s customer data team. The role focuses on building and optimizing large‑scale batch and streaming data pipelines while ensuring production reliability.

Key responsibilities

• Design, build and maintain scalable batch and streaming pipelines using Spark, Scala and Python.
• Optimize Spark workloads for performance, cost efficiency and reliability.
• Monitor and troubleshoot production pipelines, resolve data quality issues, and ensure SLAs/SLOs are met.
• Orchestrate workflows with Airflow or similar tools and process real‑time data using Kafka, Flink or Kafka Streams.
• Support data modeling, schema design, validation and observability across the platform.

Required profile

• Strong experience with Python, Scala and Apache Spark.
• Hands‑on experience building batch and streaming pipelines and optimizing Spark jobs.
• Familiarity with Kafka, Flink, Kafka Streams or similar streaming technologies.
• Experience with Airflow workflow orchestration.
• Ability to review and validate AI/LLM‑generated code for production readiness.

Required skills

• Python
• Scala
• Apache Spark
• Kafka / Kafka Streams
• Flink (preferred)
• Airflow
• NoSQL databases (Scylla, Cassandra, DynamoDB)
• Data modeling and schema design