Lead QA Engineer - AI Agents & LLM Applications

About the role

RV Staffing is seeking an experienced Lead QA Engineer with hands-on experience testing AI agents, LLM-based applications and agentic workflows. The role focuses on the quality, reliability and consistency of AI-driven systems through structured testing, defect investigation and continuous improvement of QA processes. It is a full-time remote position.

The position requires flexible working hours to support biweekly releases starting around 9:00 PM Pacific Standard Time; availability during these release windows is required.

Key responsibilities

• Design and run test cases for AI agents, LLM apps and agentic workflows, covering expected behavior, edge cases, failures and result consistency.
• Develop and run manual and automated test cases for new features, functional requirements and regression testing.
• Identify gaps in test coverage, recommend process improvements and strengthen QA practices across the development lifecycle.
• Identify, document, prioritize and track defects with reproduction steps, expected versus actual results and severity levels.
• Maintain or improve automated test scripts using Playwright, Selenium WebDriver, Cypress or pytest.
• Perform manual and automated API testing and use database queries to validate data and troubleshoot defects.
• Collaborate with developers and product stakeholders and take part in biweekly release cycles, including evening release windows.

Required profile

• Bachelor's degree in Computer Science, Information Technology or a related field, or equivalent practical experience.
• 3+ years of professional software quality assurance experience.
• Demonstrated hands-on experience testing AI agents, LLM applications or agentic workflows.
• Understanding of AI-specific testing challenges such as non-deterministic outputs and response consistency.
• Strong understanding of manual testing, test case design, defect lifecycle and regression testing.
• Ability to work independently in a remote environment and communicate technical issues to technical and non-technical stakeholders.

Required skills

• Functional, integration, API and cross-browser testing.
• Jira and TestRail for bug tracking and test management.
• SQL, MongoDB and Databricks.
• Git and familiarity with CI/CD pipelines.
• At least one test automation framework: Playwright, Selenium WebDriver, Cypress or pytest.