AI QA Analyst

At Ryz Labs, we are looking for an AI QA Analyst to join one of our clients, an AI-first travel and hospitality company. In this role, you will act as the bridge between model development and user experience. Your primary goal is to ensure our generative models are accurate, safe, and contextually aware by implementing rigorous evaluation frameworks and data-driven insights. Qualifications: Ability to spot subtle logical inconsistencies or factual inaccuracies in AI-generated content. Experience managing, cleaning, or labeling datasets for machine learning or NLP projects. Familiarity with the AI lifecycle, Prompt Engineering, and tools like Google Sheets/Excel (SQL or Python is a significant plus). Key Responsibilities: Review and annotate complex conversation traces to rate response quality based on metrics such as helpfulness, honesty, and harmlessness (HHH). Build and maintain high-quality "Golden Datasets" and benchmarks to stress-test the model across various domains and edge cases. Conduct pre-deployment testing and A/B model comparisons to identify performance regressions or improvements. Categorize model failures (hallucinations, logic errors, tone drift) to provide actionable feedback to the Engineering and Research teams. Help define and refine the rubric for "what a good response looks like" as the product evolves.