Job Description
Job Description
Position will be 3 days remote with 2 days (Mondays and Thursdays) required to be onsite
Core Duties & Responsibilities:
• Design, develop, and optimize scalable data solutions on Databricks using PySpark or Scala for large-scale data processing.
• Build and maintain ingestion pipelines, Lakeflow Declarative Pipelines (DLT), and Medallion Architecture (Bronze, Silver, Gold) to support enterprise analytics and reporting.
• Develop robust data models, implement data quality, validation, and governance frameworks.
• Create dynamic dashboards, Databricks Apps, and analytical solutions to deliver actionable business insights.
• Optimize workloads, monitoring, and operational processes to ensure scalability, security, and cost efficiency.
Minimum Required Skills (with years):
• IT Solution Design & Deployment: 8+ years
• Databricks & Apache Spark ETL/ELT Pipelines: 8+ years
• Data Warehousing & Dimensional Data Modeling (Star/Snowflake): 8+ years
• SQL and Python (or Scala) for Data Processing: 8+ years
• Databricks Native Dashboards & Apps: 8+ years
• Data Governance, Quality, & Security Practices: 8+ years
• Lakeflow Declarative Pipelines (DLT): 8+ years
• Delta Lake, Medallion Architecture, & Job Orchestration (Lakeflow Jobs/Airflow): 8+ years
• Written and Verbal Communication Skills: 8+ years
Preferred Skills (Lookout for):
• Public Sector or State Government Environment Experience: 1+ year
• Databricks Certification (Associate/Professional Data Engineer): 1+ year
• CI/CD Practices for Data Pipelines (DevOps, Git): 1+ year
II. WORKER SKILLS AND QUALIFICATIONS
Please fill in the "Actual Years Experience" column. Candidates that do not meet or exceed the minimum stated requirements (skills/experience) will be displayed to customers but may not be chosen for this opportunity.
Actual Years Experience
Years Experience Needed
Required / Preferred
Skills / Experience
8
Required
Experience in IT, supporting the design, development, deployment, or delivery of technology solutions.
8
Required
Experience with Databricks, including building and optimizing ETL/ELT data pipelines using Apache Spark.
8
Required
Experience in data warehousing and dimensional data modeling (star/snowflake schemas).
8
Required
Proficiency in SQL and Python (or Scala) for large-scale data processing.
8
Required
Experience designing and developing dashboards and applications natively within Databricks (e.g., Databricks SQL dashboards, Databricks Apps).
8
Required
Experience implementing data governance, data quality, and data security practices.
8
Required
Experience implementing Lakeflow Declarative Pipelines (formerly Delta Live Tables/DLT) for building and managing production data pipelines.
8
Required
Experience with Delta Lake, medallion architecture (bronze/silver/gold layers), data lakehouse design, and creating and scheduling offline jobs using Lakeflow Jobs (formerly Databricks Workflows) or similar orchestration tools (e.g., Airflow).
8
Required
Excellent communication skills, both verbal and written, including presenting insights to technical and business stakeholders.
1
Preferred
Experience working in public sector or state government environments.
1
Preferred
Databricks certification (e.g., Databricks Certified Data Engineer Associate/Professional).
1
Preferred
Experience with CI/CD practices for data pipelines (DevOps, Git-based workflows).