Senior Data Engineer - AWS & RAG Pipelines

About the role

We are seeking a Senior Data Engineer to design, build, and operate the cloud data infrastructure that powers our AI initiatives. You will lead the creation of production‑scale data lakes on AWS, develop real‑time ingestion and observability pipelines, and manage the vector search and embedding layers that feed our Retrieval‑Augmented Generation (RAG) systems and autonomous agents.

Key responsibilities

• Architect and maintain multi‑tenant data lakes on AWS, including S3 partitioning, lifecycle policies, and columnar storage formats.
• Design and implement real‑time data ingestion, CDC from PostgreSQL, and observability pipelines using OpenTelemetry collectors.
• Build and optimise Elasticsearch/OpenSearch clusters for combined lexical and high‑dimensional vector search.
• Develop and integrate Amazon Bedrock generative AI pipelines for data enrichment, classification, and semantic parsing.
• Expose data‑lake utilities via APIs (C#, Java, Python, or Node.js) for AI agents and downstream services.
• Collaborate with data scientists to manage embeddings, vectorisation, and knowledge‑base synchronisation.

Required profile

• 7+ years of experience in Data Engineering, Distributed Systems, or Data Architecture.
• 4+ years designing production‑scale data lakes and event‑streaming solutions on AWS.
• 2+ years building Retrieval‑Augmented Generation (RAG) pipelines and managing embeddings.
• Strong background in cloud‑native data storage, real‑time observability, and log analytics.

Required skills

• AWS services: S3, Glue, Lake Formation, Athena, Redshift Spectrum, OpenSearch, Bedrock.
• Search technologies: Elasticsearch, OpenSearch, vector search, embeddings.
• Programming languages: C# (.NET Core), Java, Python, Node.js.
• Database change‑data‑capture: PostgreSQL, Debezium, AWS DMS.
• ETL/ELT pipelines for unstructured event data from SaaS APIs.

What we offer

• Fully remote work arrangement.
• 13 floating holidays and 15 vacation days per year.
• Supportive and inclusive working environment.
• Equal‑opportunity employment policy.