Search

Lead Data Engineer - Databricks

PublishedPublished: 6/14/2022
Technology
|

Job Description

Job Title: Lead Data Engineer

\n

Location: Grand Rapid, MI

\n

Work Mode: Hybrid

\n

Job Type: Contract / Full-Time

\n

Certification: Databricks Certification preferred

\n


\n

Job Summary

\n


\n

We are seeking an experienced Lead Data Engineer to design and implement scalable, secure, and high-performance data lakehouse solutions. The ideal candidate should have strong expertise in Databricks, Medallion Architecture, Azure/AWS cloud platforms, SQL, PySpark, Python, and Databricks Auto Loader.

\n


\n

Key Responsibilities

\n


\n

    \n
  • Design and implement enterprise-scale data architectures using Databricks.
  • \n

  • Develop scalable data pipelines using SQL, PySpark, and Python.
  • \n

  • Implement Medallion Architecture, including Bronze, Silver, and Gold layers.
  • \n

  • Build incremental and streaming data ingestion pipelines using Databricks Auto Loader.
  • \n

  • Integrate Databricks with Azure or AWS cloud services.
  • \n

  • Optimize Spark workloads, SQL queries, cluster configurations, and data processing performance.
  • \n

  • Implement Delta Lake solutions for reliable data storage and processing.
  • \n

  • Establish data governance, security, and access control best practices.
  • \n

  • Provide technical leadership and architectural guidance to data engineering teams.
  • \n

  • Support production deployments, troubleshooting, and performance optimization.
  • \n

\n


\n

Required Skills and Qualifications

\n


\n

    \n
  • Strong experience with Databricks architecture and Lakehouse solutions.
  • \n

  • Hands-on expertise in Medallion Architecture.
  • \n

  • Proficiency in SQL, PySpark, and Python.
  • \n

  • Practical experience with Databricks Auto Loader.
  • \n

  • Experience with Azure Databricks or AWS Databricks.
  • \n

  • Strong understanding of Apache Spark, Delta Lake, and ETL/ELT pipelines.
  • \n

  • Experience in data pipeline optimization and production-grade data engineering.
  • \n

  • Knowledge of data governance, security, and scalable architecture design.
  • \n

  • Strong communication, analytical, and problem-solving skills.
  • \n

\n


\n

Preferred Qualifications

\n


\n

    \n
  • Databricks certification preferred.
  • \n

  • Experience with Unity Catalog and data governance.
  • \n

  • Knowledge of Delta Live Tables (DLT) or Lakeflow Declarative Pipelines.
  • \n

  • Experience with CI/CD, Git, and infrastructure-as-code tools.
  • \n

  • Experience with cloud migration and enterprise data platform modernization.
  • \n

  • Previous experience leading technical teams or defining data architecture standards.
  • \n

\n


\n

Ideal Candidate

\n

The ideal candidate will be a hands-on technical person who can design end-to-end Databricks solutions, guide engineering teams, and deliver scalable data platforms using modern cloud and lakehouse technologies.

Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...