AI Evaluation Engineer (Agentic Coding / Software Engineering)
Gramian Consulting Group
$0-$0 / yr
Salary
brazil
Region
ASAP
Start Date
About Gramian Consulting Group
Gramian Consultancy brings together the perspective of a software engineer, the knowledge of a technical recruiter, and the vision of a business builder. This unique experience is our signature advantage to delivering top quality services in the domain of recruiting, staff augmentation, and outsourcing.
About this Role.
About Us
Gramian Consultancy is a boutique consultancy specializing in IT professional services and engineering talent solutions. With a strong background in software engineering and leadership, we help companies build high-performing teams by matching them with professionals who truly fit their needs.
Role overview
We are looking for an AI Evaluation Engineer specialized in software engineering workflows to evaluate and improve datasets used for agentic coding models.
In this role, you will work on realistic coding tasks — reviewing model trajectories, validating outputs, and producing high-quality evaluations. This is a hands-on engineering role, requiring strong debugging skills, attention to detail, and the ability to assess correctness in real code scenarios.
Commitments Required: 8 hours per day with an overlap of 4 hours with PST.
Employment type: Contractor assignment (no medical/paid leave)
Duration of contract: 5 weeks+
Location: Bangladesh, Brazil, Colombia, Egypt, Ghana, India, Indonesia, Kenya, Nigeria,Turkey, Vietnam
Interview: take home assessment (60min)
Responsibilities
Execute coding tasks within agentic coding environments, maintaining strict evaluation protocols
Review and evaluate model-generated code trajectories for correctness and completeness
Validate outputs by reading code, running tests, analyzing logs, and inspecting artifacts
Perform targeted validation using scripts, tests, and manual checks
Write clear, evidence-based rationales for evaluations and rankings
Design realistic, multi-step coding tasks and workflows (offline work)
Create and refine evaluation rubrics and scoring criteria
Ensure consistency, quality, and compliance across evaluations
Identify issues in environments, instructions, or workflows and report with clear evidence
Requirements
5+ years of experience in software engineering, QA, developer tooling, or similar code-heavy roles
Strong proficiency in at least one programming ecosystem (e.g., Python, JavaScript/TypeScript, Java, C/C++, Rust, SQL)
Ability to read and understand unfamiliar codebases and implement/debug changes
Experience running and interpreting tests, scripts, and CLI tools
Strong debugging and problem-solving skills, including handling edge cases
Comfortable working in Linux/terminal environments
Familiarity with Git workflows and standard development tooling
Experience with AI coding tools or agentic coding environments (e.g., Cursor, Claude Code, or similar)
Strong attention to detail and ability to produce consistent, high-quality evaluations