Databricks Architect - Hybrid
Technology
Job Description
Job Description
- Expert on Databricks and AWS, with minimum 10+ years of experience
- Provide guidance and support to teams and analysts.
- Databrick certification is a must
- Collaborate on data strategy with business and IT partners.
- Identify and address data issues.
- Lead architecture, design, and implementation of scalable and efficient data pipelines and analytics solutions on Databricks.
- Work closely with data engineering, platform, and analytics teams to integrate Databricks with AWS S3 Data Lake, Redshift, Talend, SAP BW/HANA, Oracle, and other tools.
- Architect solutions supporting batch and real-time data processing using Delta Lake, Spark, and ML Flow.
- Collaborate with business stakeholders to understand data requirements, assess current state, and provide strategic direction for future-state architecture.
- Ensure best practices for data quality, governance, and security are implemented.
- Own design, development, and delivery of pipelines and models as per the project plan.
- Ensure best practices are followed in performance, security, and governance.
- Provide project documentation and conduct KT sessions
- Ensure following acceptance criteria is met, by working along with current development and pool partners
- Successful ingestion and transformation of in-scope objects.
- Validated and reconciled transformed data within defined thresholds.
- Fully operational Snowflake warehouse with business-validated datasets.
- Documented operational playbooks and KT completed.
- No critical issues post go-live during hyper-care window