Job Description
\n
We are seeking a Senior AI Evaluations Engineer to work alongside AWS Professional Services (ProServe) teams, designing, developing, and operating enterprise-grade AI evaluation frameworks for large enterprise customers, primarily within financial services.
\n
This role focuses on ensuring that Generative AI and agentic AI applications are accurate, reliable, secure, and production-ready. The ideal candidate brings advanced Python engineering expertise, hands-on experience with Amazon Bedrock, and a strong background in developing automated evaluation systems for LLM-powered applications and multi-agent workflows.
\n
The engineer will lead the design of LLM-as-Judge and Agent-as-Judge frameworks, implement quality rubrics and scoring architectures, and develop offline regression testing and continuous live-traffic evaluation capabilities.
\n
Key Responsibilities
\n
- \n
- AI Evaluation Frameworks: Design, develop, and operate end-to-end evaluation frameworks for Generative AI and agentic AI applications, including offline regression testing, online monitoring, continuous evaluation, and A/B testing.
- Python Engineering: Develop scalable, production-grade evaluation services, automation frameworks, APIs, and testing pipelines using advanced Python.
- LLM-as-Judge / Agent-as-Judge: Build automated evaluation pipelines to assess agent behavior, task completion, reasoning trajectories, tool-call accuracy, groundedness, and safety.
- Evaluation Rubrics and Scoring: Define structured scoring methodologies and quality standards at the session, response, and individual tool-call levels.
- Agentic AI Development: Develop and evaluate multi-agent systems using LangGraph, Strands, CrewAI, or similar agent orchestration frameworks.
- AWS AI Infrastructure: Build and integrate evaluation solutions using Amazon Bedrock, Bedrock Guardrails, AWS Lambda, Amazon S3, IAM, and CloudWatch.
- Evaluation Infrastructure: Design reusable evaluation components, testing frameworks, and automated benchmarking solutions supporting enterprise AI deployments.
- CI/CD Integration: Integrate AI evaluation pipelines into software delivery workflows, enabling automated regression testing, quality gates, and production release validation.
- AI Observability: Implement monitoring, distributed tracing, structured logging, and performance analytics to assess production AI behavior and reliability.
- RAG Evaluation: Evaluate retrieval-augmented generation applications using metrics such as faithfulness, groundedness, context precision, and recall.
- AI Governance and Security: Implement evaluation processes that address hallucinations, data security, safety, compliance, and auditability within regulated enterprise environments.
- Technical Leadership: Collaborate with AWS architects, software engineers, data scientists, and enterprise stakeholders to establish evaluation standards, reusable frameworks, and AI engineering best practices.
\n
\n
\n
\n
\n
\n
\n
\n
\n
\n
\n
\n
\n
Required Qualifications
\n
- \n
- Expert-level Generative AI engineering experience, including hands-on development and deployment of production-grade LLM and agentic AI applications.
- Advanced Python programming expertise, including building scalable backend services, APIs, automated testing frameworks, and production evaluation pipelines.
- Demonstrated experience developing LLM evaluation frameworks, including offline/on-demand regression testing and online/continuous evaluation of production AI systems.
- Hands-on experience with LLM-as-Judge, evaluation rubric development, scoring methodologies, and automated quality assessment.
- Experience evaluating agent performance across task completion, tool-call accuracy, trajectory quality, groundedness, and safety.
- Strong experience with LangGraph, Strands, CrewAI, or comparable agentic AI frameworks.
- Hands-on knowledge of Amazon Bedrock, Bedrock Guardrails, AWS Lambda, S3, IAM, and CloudWatch.
- Experience designing and operating scalable, production-grade applications and AI infrastructure.
- Familiarity with CI/CD automation, application monitoring, logging, and production troubleshooting.
- Strong communication skills and experience collaborating with enterprise engineering and architecture teams.
- Active AWS Certification (one required):
- AWS Certified Generative AI Developer – Professional
- AWS Certified Machine Learning Engineer – Associate
\n
\n
\n
\n
\n
\n
\n
\n
\n
\n
\n
\n
\n
\n
Preferred Qualifications
\n
- \n
- Experience with Amazon Bedrock AgentCore Evaluations, including on-demand evaluation runners, online monitoring, and A/B testing.
- Advanced knowledge of Agent-as-Judge frameworks, human-in-the-loop evaluation, and automated AI quality assurance.
- Experience with AI benchmarking frameworks such as tau-bench, SWE-bench, GAIA, and custom evaluation datasets.
- Experience evaluating RAG systems using faithfulness, context precision, context recall, and response quality metrics.
- Familiarity with distributed tracing, structured logging, observability platforms, and automated evaluation within CI/CD pipelines.
- Experience with enterprise legacy modernization, including applications built using Java/Spring, Sybase, DB2, or COBOL.
- Background delivering AI solutions within financial services, banking, capital markets, or other regulated industries.
- Knowledge of financial services compliance requirements, data governance, audit controls, and security standards.
- Experience designing reusable AI evaluation architectures and components across large enterprise application portfolios, including environments supporting 50+ applications.
\n
\n
\n
\n
\n
\n
\n
\n
\n
\n