Job Description
Role : Azure Databricks & Agentic AI Architect
\n
Location: Chicago, IL(Hybrid- 3 days/week in office)
\n
Contract role
\n
Looking for 16-20 Years minimum experience
\n
\n
Mandatory skills
\n
· Azure, DataBricks, Agentic AI, ETL, SQL, Data Engineering
\n
\n
Role Summary:
\n
We are seeking a visionary Azure Databricks & Agentic AI Architect to design and implement next-generation AI-powered data platforms. This role combines deep expertise in Azure Databricks, Lakehouse Architecture, Data Engineering, and Generative AI to build intelligent, autonomous, and self-optimizing data ecosystems.
\n
The ideal candidate will lead the adoption of Agentic AI within Data Engineering and AI-DLC, enabling autonomous data ingestion, transformation, quality management, lineage discovery, observability, optimization, testing, and governance.
\n
Key Responsibilities
\n
Agentic Data Engineering Leadership
\n
· Design and implement AI-powered Data Engineering platforms leveraging Azure Databricks and Lakehouse architecture.
\n
· Define autonomous workflows using AI Agents for:
\n
· Data ingestion
\n
· Data mapping
\n
· Schema evolution
\n
· Data quality remediation
\n
· Metadata Enrichment
\n
· Pipeline optimization
\n
· Root cause analysis
\n
· Establish frameworks for Human-in-the-Loop (HITL) decision-making and governance.
\n
AI-Driven Data Lifecycle (AI-DLC)
\n
· Lead architecture for AI-enabled Data Development Lifecycle across:
\n
· Requirement analysis
\n
· Data modeling
\n
· Pipeline generation
\n
· Automated testing
\n
· Code review
\n
· Documentation
\n
· Deployment
\n
· Monitoring
\n
· Implement AI copilots to accelerate developer productivity.
\n
· Enable automated lineage creation and intelligent impact analysis.
\n
Lakehouse & Data Platform Architecture
\n
· Design scalable Lakehouse platforms using:
\n
· Azure Databricks
\n
· Delta Lake
\n
· Unity Catalog
\n
· ADLS Gen2
\n
· Databricks Workflows
\n
· Delta Live Tables
\n
Enterprise GenAI Integration
\n
· Architect RAG-based solutions using enterprise data assets.
\n
· Design agent orchestration frameworks using:
\n
· Azure OpenAI
\n
· LangGraph
\n
· Semantic Kernel
\n
· AutoGen
\n
· MCP-enabled architectures
\n
· Build domain-specific AI agents supporting Data Engineering and Analytics teams.
\n
AI Governance & Responsible AI
\n
· Define guardrails for enterprise GenAI adoption.
\n
· Implement:
\n
· Prompt governance
\n
· Observability
\n
· Cost monitoring
\n
· Auditability
\n
· Explainability
\n
· Security controls
\n
· Establish governance models for autonomous AI agents.
\n
AI-Powered Platform Optimization
\n
· Design self-healing data pipelines.
\n
· Implement AI-driven:
\n
· Incident triage
\n
· Failure prediction
\n
· Capacity planning
\n
· Cost optimization
\n
· SLA monitoring
\n
· Enable intelligent workload placement and model routing.
\n
Technical Skills
\n
· Data Platform
\n
· Azure Databricks
\n
· Delta Lake
\n
· Unity Catalog
\n
· Azure Data Factory
\n
AI & Agentic Frameworks
\n
· Azure OpenAI
\n
· Knowledge Graph
\n
· RAG Architecture
\n
· LangChain
\n
· LangGraph
\n
· MCP Protocol
\n
· Vector Databases
\n
· AI Agent Orchestration
\n
Data Engineering
\n
· PySpark
\n
· Spark SQL
\n
· Python
\n
· SQL
\n
· ELT/ETL Modernization
\n
DevOps & AI-DLC
\n
· Azure DevOps
\n
· GitHub Actions
\n
· CI/CD
\n
· MLOps
\n
· LLMOps
\n
· Evaluation Frameworks
\n
· AI Testing Frameworks
\n
Leadership Expectations
\n
· Drive AI-First Data Engineering transformation.
\n
· Define enterprise patterns, accelerators, and reusable AI agents.
\n
· Mentor architects, data engineers, and AI engineers.
\n
· Lead executive conversations on AI adoption, ROI, and transformation roadmaps.