Job Description
Job Description
Senior Engineer - Data Science
10-15 Years
San Francisco
Job description
Job Description: Senior Engineer with Data Science + Gen AI
Project Focus
Build production-grade Generative AI systems with emphasis on:
RAG pipelines leveraging LLMs
Prompt engineering (system prompts, tool prompts, function calling, versioning with evaluations)
Evaluation & observability, including:
Ground truth setup
Confusion metrics
Hallucination rate measurement
LLM-as-judge with human review
Cost & latency monitoring
Retrieval strategies & prompt patterns
Context management
Hallucination mitigation
Faithfulness & grounding validation
Agentic workflows
Multi-step reasoning
Tool orchestration
Agent success rate measurement
LangChain / LlamaIndex (or equivalent) proficiency
Cloud LLM providers
Azure OpenAI
AWS Bedrock
Vertex AI
Workflow orchestration
Airflow
Dagster
Security & privacy
PII handling
RBAC
Strong Python expertise with real production experience
Success Criteria
Production-ready prompt + RAG system
Measurable quality lift (accuracy, faithfulness, confusion metrics)
Reduced hallucinations
Repeatable evaluation harness
Observable system (quality, latency, cost, drift monitoring)
Skills
Large Language Models, AWS, Prompt Engineering, Python, Machine Learning, Gen AI, Confluence, Jira