Senior QA Automation Engineer - SIP/VOIP

About the role

idt is looking for an experienced Senior QA Automation Engineer to own quality for its AI voice virtual agent. The role suits someone who can reason about a large language model's behaviour and read a SIP trace, and who enjoys testing non-deterministic systems. Candidates must submit their CV in English, and only candidates from LATAM are accepted.

Key responsibilities

• Embed in a Scrum team, taking part in all ceremonies and maintaining the associated artifacts.
• Own the end-to-end test strategy across the LLM, speech and telephony layers.
• Design test scenarios for complex conversational behaviour, including multi-turn dialogues, interruptions, barge-in, ambiguous input, out-of-scope requests, human escalation and failure recovery.
• Build an evaluation framework for non-deterministic output using golden datasets, semantic/rubric-based assertions and automated regression detection.
• Adversarially test the agent against hallucinations, incorrect grounding, guardrail bypasses, prompt injections and data leakage.
• Provision SIP devices and endpoints and drive calls programmatically to automate call origination.
• Diagnose failures across the stack, isolating them to the LLM, ASR, TTS, media path or call routing layer.
• Integrate test suites into the CI/CD pipeline as release gates and track quality metrics such as intent accuracy, transcription accuracy and response latency.
• Maintain documented test coverage in the TMS with traceability to business requirements and escalate quality risks to stakeholders.

Required profile

• 5+ years in QA, with recent hands-on ownership of a complex, integrated product.
• Practical experience testing SIP/VoIP systems (call flows, SIP signalling, media handling, softphones, SIP trunking) and troubleshooting with SIP traces or packet captures.
• Creativity in test design, which will be a core part of the interview.
• Experience testing non-deterministic systems and defining assertions for variable outputs.
• Strong analytical skills and ability to define and justify quality metrics.
• Fluent English, with overlap with US Eastern business hours.

Required skills

• Python, enough to build and maintain automation frameworks.
• Hands-on experience with formal TMS tools and Jira.
• Preferred: LLM evaluation frameworks, ASR/TTS metrics (WER), RAG evaluation, WebRTC/media testing, IVR testing, multilingual testing and prompt engineering.

What we offer

• Remote work opportunity.
• B2B employment with competitive salary and annual performance review.
• Stable job with long-term growth, continuous learning and career growth opportunities.
• Good hardware, professional training, seminars and conferences, and company-supported English classes.
• Referral programme.