
Job Title: Senior QA & Evaluation Engineer
Key Skills: Test Automation, .NET, API Testing, CI/CD, Automated Testing, LLM Evaluation, AI Testing, Test Frameworks
Experience: 4+ years of experience.
Location: Costa Rica, Peru, Colombia, Bolivia.
Mode: Remote.
We at Coforge are hiring Mid-Senior QA & Evaluation Engineer (N-5) with the following skill set.
Key Responsibilities
· Design and maintain automated test suites covering unit, integration, API, and end-to-end testing.
· Integrate automated testing into CI/CD pipelines and establish automated suites as release quality gates.
· Build evaluation frameworks for AI capabilities, including scored rubrics, LLM-as-a-judge approaches, and normalized pass thresholds.
· Build and maintain held-out evaluation datasets, measure AI behavior against ground truth, and report defect escape and evaluation results as part of delivery metrics.
Required Skills & Qualifications
· 4+ years of experience in Quality Engineering with a strong focus on test automation.
· Hands-on experience with automated testing frameworks, preferably within the .NET ecosystem.
· Strong experience with API testing and automated test development.
· Experience integrating automated test suites into CI/CD pipelines.
· Understanding of testing strategies for non-deterministic and probabilistic AI systems.
· Ability to design evaluation approaches based on scored thresholds rather than exact-match assertions.
· Experience working with test data, ground truth datasets, and automated validation.
· Strong analytical and problem-solving skills.
· Strong working proficiency in English.
· Bachelor's degree in Computer Science, Software Engineering, Information Technology, or a related field, or equivalent professional experience.
Preferred Skills:
· Direct experience with LLM evaluation, including rubric design, LLM-as-a-judge, golden/held-out datasets, or tools such as Azure AI Foundry Evaluators, promptfoo, Ragas, or DeepEval.
· Experience testing information extraction or classification pipelines using accuracy-based measurements.
· Experience with regulated-industry QA environments where test evidence and results must support formal review.
· Familiarity with AI/ML testing methodologies and evaluation frameworks.
Posted On: 24-09-2026
At Coforge, we hire professionals based solely on their skills and do not discriminate based on age, disability, religion, gender, sexual orientation, socioeconomic status, or nationality.
You will be redirected to the company website to complete your application.





