Lead Engineer, AI Platform Quality

About the role

Circle builds an AI-powered, all-in-one platform for digital businesses, and its AI Quality engineering team owns the foundation for measuring, diagnosing and improving the quality of AI-powered features. It focuses on the infrastructure for production AI systems, not on ML research or model training. The company is looking for a Lead Engineer to build evaluation frameworks, observability tooling and diagnostic infrastructure for its AI Agents, and to lead and manage the AI Quality engineering team. The role is hands-on and player-coach.

Key responsibilities

• Build and own the evaluation infrastructure: design CI/CD pipelines, scorers and datasets for AI agents (planners, tool-callers, sub-agents).
• Diagnose where quality breaks down, tracing failures across the agent pipeline (plan creation vs. execution, tool selection, tool trajectories in workflows, site builder and analytics).
• Turn findings into prototypes and grow the datasets used for evaluation, including annotation workflows and AI-generated and simulated conversations.
• Run structured experiments across prompts, models and the agent harness, evaluate new and open-source models against the production baseline, and build the framework for deciding when to shift models.
• Pursue cost and latency improvements through model swaps, caching and routing by plan complexity.
• Lead and grow the AI Quality engineering team, set technical direction and manage day-to-day priorities while staying hands-on in the code.
• Partner closely with AI Core engineering so engineers are confident their changes improve quality.

Required profile

• 7+ years of experience building and shipping production software, ideally with LLM-powered agents that take real actions in a product.
• Experience with complex, tool-using systems (multiple tools, planning or orchestration, sub-agents), not just simple single-turn assistants.
• Experience building evaluation or observability infrastructure for ML/AI systems, such as eval pipelines, scorers, dashboards or CI/CD for evals.
• Experience designing datasets, annotation workflows or labeling pipelines for ML/AI evaluation.
• Comfortable learning fast, running many experiments, and thriving in a fast-paced, ambiguous environment.
• Proficiency in English at CEFR Level C2 / ILR Level 5.

Required skills

• Ruby on Rails (production system) and Python (strong plus, especially for data and evaluation).
• Evaluation frameworks such as Braintrust, LangSmith or similar.

What we offer

• Compensation benchmarked to the U.S., equity with ongoing refresh grants, and 35 days of paid time off per year.
• Remote-only work with two company retreats a year.
• A benefits package supporting health, wellbeing and professional growth.