Data Architect - Databricks

Published: 6/14/2022

Technology

Job Description

Job Title: Data Architect – Databricks
Department: Data & AI / Technology Consulting
Reports To: Head of Data & AI / Enterprise Architecture Lead
Location: [City, State / Remote / Hybrid]
Employment Type: Full-Time

Position Overview

We are seeking an experienced Data Architect – Databricks to design and implement scalable, secure, and high-performance data architectures using the Databricks platform. The ideal candidate will have deep expertise in modern data architecture, lakehouse design, cloud data platforms, data engineering, enterprise data integration, and data governance.

The Data Architect will work closely with business stakeholders, enterprise architects, data engineers, analytics teams, AI/ML engineers, and cloud platform teams to translate business requirements into robust technical architectures. This role will lead architecture decisions for data modernization, cloud migration, enterprise analytics, and AI initiatives, ensuring that data platforms are scalable, governed, cost-effective, and aligned with enterprise standards.

Key Responsibilities

1. Data Architecture & Solution Design

Design end-to-end enterprise data architectures leveraging the Databricks Data Intelligence Platform and lakehouse architecture.
Define architectural patterns for batch processing, streaming data, data ingestion, transformation, storage, and consumption.
Develop conceptual, logical, and physical data models aligned with business and technical requirements.
Establish architecture standards, design principles, reference architectures, and reusable data engineering patterns.
Evaluate data platform requirements for scalability, performance, availability, reliability, and cost optimization.
Integrate Databricks with enterprise applications, operational databases, cloud storage, APIs, and downstream analytics platforms.

2. Databricks Platform Architecture

Design and implement solutions using Delta Lake, Unity Catalog, Databricks SQL, and Lakeflow capabilities, as appropriate to the solution.
Define medallion architecture patterns, including Bronze, Silver, and Gold data layers, where suitable.
Architect data ingestion and transformation pipelines using Spark, PySpark, SQL, and supported orchestration tools.
Design secure data access, cataloging, lineage, discovery, and governance using Unity Catalog.
Establish standards for Delta tables, data sharing, schema evolution, data quality, and lifecycle management.
Evaluate Databricks platform capabilities and recommend appropriate services and configurations for enterprise use cases.

3. Cloud Data Platform Integration

Design and implement Databricks solutions on Microsoft Azure, AWS, or Google Cloud Platform.
Integrate cloud object storage, identity and access management, networking, key management, and monitoring services.
Define secure connectivity and data integration patterns across cloud and on-premises environments.
Collaborate with cloud architects and DevOps teams to establish infrastructure and deployment standards.
Ensure compliance with enterprise security requirements and cloud architecture best practices.

4. Data Engineering & Pipeline Architecture

Define scalable data ingestion, transformation, and processing frameworks for structured, semi-structured, and unstructured data.
Guide engineering teams in building reliable ETL/ELT pipelines using PySpark, Spark SQL, SQL, and applicable orchestration frameworks.
Establish reusable frameworks for incremental processing, change data capture, error handling, data reconciliation, and recovery.
Design real-time and near-real-time data processing solutions using streaming technologies where required.
Define engineering standards for code quality, testing, CI/CD, observability, and production support.

5. Data Governance, Security & Quality

Establish enterprise data governance and security architecture using Unity Catalog and related platform capabilities.
Define data access policies, role-based access controls, data classification, masking, and privacy controls.
Develop standards for data lineage, metadata management, auditability, and regulatory compliance.
Establish data quality frameworks, validation rules, monitoring, and remediation processes.
Collaborate with information security, privacy, compliance, and business data owners to ensure appropriate controls.

6. Analytics, AI & Machine Learning Enablement

Design data architectures that support business intelligence, advanced analytics, machine learning, and generative AI use cases.
Enable trusted, well-governed data products for analytics and AI consumption.
Collaborate with AI/ML engineers on feature engineering, model data pipelines, and model lifecycle requirements.
Evaluate architecture patterns for vector search, retrieval-augmented generation (RAG), and other enterprise AI applications where appropriate.
Ensure that data quality, access controls, lineage, and governance requirements are incorporated into AI solutions.

7. Performance Optimization & Cost Management

Analyze and optimize Spark workloads, SQL queries, storage layouts, and data processing pipelines.
Establish best practices for workload management, compute sizing, autoscaling, and resource utilization.
Identify opportunities to reduce cloud and Databricks platform costs while maintaining performance and reliability.
Define monitoring and operational metrics for platform health, pipeline performance, and service-level objectives.
Conduct architecture reviews and recommend improvements to existing data platforms.

8. Technical Leadership & Stakeholder Collaboration

Lead architecture workshops, technical discovery sessions, and solution design reviews.
Translate business requirements into technical architecture documents, diagrams, and implementation roadmaps.
Mentor data engineers and technical teams on Databricks architecture and engineering best practices.
Provide technical guidance during development, testing, deployment, and production rollout.
Communicate architecture decisions, risks, trade-offs, and recommendations to technical and business stakeholders.

Required Qualifications

Bachelor's degree in Computer Science, Information Technology, Engineering, or a related field; equivalent professional experience may be considered.
7+ years of experience in data engineering, data architecture, cloud data platforms, or enterprise data solutions.
3+ years of hands-on experience designing or implementing solutions on Databricks, or equivalent depth of experience with the platform.
Strong understanding of lakehouse architecture, distributed data processing, data modeling, and modern data platform design.
Hands-on expertise with Apache Spark, PySpark, Spark SQL, and SQL.
Experience with Delta Lake, Unity Catalog, and Databricks platform architecture.
Experience designing ETL/ELT pipelines and integrating data from multiple enterprise sources.
Experience with at least one major cloud platform: Microsoft Azure, AWS, or Google Cloud.
Strong understanding of data security, governance, data quality, metadata, and lineage.
Experience with architecture documentation, design reviews, and technical stakeholder management.
Strong analytical, communication, problem-solving, and technical leadership skills.

Preferred Qualifications

Databricks Certified Data Engineer Professional, Databricks Certified Data Engineer Associate, or relevant Databricks certification.
Databricks Certified Data Architect or other relevant architecture credentials, if applicable and currently available.
Experience with enterprise data modernization and migration from legacy data warehouses or data lakes.
Familiarity with Lakeflow, Delta Sharing, Databricks SQL, Genie Agents, and applicable serverless or platform administration capabilities.
Experience with Azure Data Factory, Azure Data Lake Storage, AWS S3, AWS Glue, Google Cloud Storage, or comparable cloud services.
Knowledge of data modeling techniques, including dimensional modeling, Data Vault, and domain-oriented data products.
Experience with Terraform, infrastructure as code, CI/CD, Git, and automated deployment practices.
Familiarity with data observability, data quality tools, and enterprise metadata management platforms.
Exposure to MLflow, feature engineering, vector search, RAG, and enterprise generative AI architectures.
Experience in consulting or client-facing solution architecture roles.

Key Deliverables

Enterprise and solution-level data architecture blueprints.
Databricks lakehouse architecture and implementation patterns.
Data ingestion, transformation, and integration design specifications.
Data governance, security, and access control frameworks.
Data modeling standards and data quality implementation patterns.
Performance optimization and platform cost-management recommendations.
Architecture decision records, technical design documents, and implementation roadmaps.
Production readiness, deployment, and operational architecture standards.

Key Performance Indicators (KPIs)

Success in this role will be measured by:

Quality, scalability, security, and maintainability of implemented data architectures.
Performance and reliability of data pipelines and platform workloads.
Compliance with enterprise architecture, governance, and security standards.
Reduction in data processing costs and improvement in workload efficiency.
Data quality, availability, and successful integration of enterprise data sources.
Adoption of reusable architecture patterns and engineering standards.
Successful delivery of data modernization, analytics, and AI enablement initiatives.
Stakeholder satisfaction and effective collaboration with engineering and business teams.

Core Competencies

Enterprise data architecture
Databricks lakehouse architecture
Apache Spark, PySpark, and SQL
Data modeling and data engineering
Cloud architecture and integration
Data governance and security
Data quality and metadata management
Performance tuning and cost optimization
AI-ready data architecture
Technical leadership and stakeholder communication

Why Join Us?

Join a growing Data & AI organization delivering modern data platforms, enterprise analytics, and AI solutions. You will have the opportunity to shape enterprise data architecture, lead complex Databricks implementations, establish modern engineering practices, and enable customers to unlock business value from their data and AI investments.

Equal Opportunity Statement: [Company Name] is an equal opportunity employer and considers qualified applicants without regard to protected characteristics under applicable law.

Recommended hiring profile

For recruiting purposes, I would target a candidate with:

Experience: 7–12 years in data engineering and architecture, including substantial Databricks project experience.

Technical depth: Strong PySpark, SQL, Delta Lake, Unity Catalog, and cloud architecture skills.

Architecture ownership: Experience designing end-to-end platforms rather than only developing pipelines.

Business skills: Ability to lead architecture workshops and explain technical decisions to clients and executives.

Certifications: Relevant Databricks and cloud certifications preferred, but demonstrated implementation experience should carry more weight.

One important consideration: if this is a client-facing consulting role, emphasize architecture assessments, solution proposals, migration strategy, and technical leadership. If it is an internal enterprise architecture role, place greater emphasis on governance, architecture standards, platform strategy, and long-term operating models.

\nCompany Description

About Us: Bizmetric is a leading tech company specializing in innovative software solutions We have been providing strategic IT consultation with rich domain expertise to organizations since 2011. Our customers are typically large enterprises like Oil and Gas Majors, Beverages and other Retail Giants, Manufacturing Houses. We operate Globally.