Data Engineer - Databricks, AWS, Python

Job Description

Job DescriptionData Engineer – Databricks, AWS, Python

100% Remote / MUST interview on-site: Phila., PA 19103

$125,000 - $135,000/yrRole OverviewWe are seeking a Data Engineer to support the development, modernization, and delivery of data engineering assets, and the Yield Looker-to-Tableau migration, and Supply/Yield Data Engineering initiatives through the remainder of 2026. This role will focus on building reliable ETL pipelines, enhancing legacy jobs, supporting migration work, and delivering core data assets that feed analytics and reporting in Tableau Cloud.Key Responsibilities

• Develop, test, and maintain ETL and data engineering jobs that bring data into Databricks and support downstream analytics and reporting.
• Work with business and technical stakeholders to understand requirements and translate them into scalable, production-ready data pipelines.
• Build and maintain pipelines across AWS, Databricks, and Snowflake environments.
• Enhance existing legacy data engineering jobs as part of broader modernization and migration efforts.
• Create and maintain technical documentation, technical specifications, test plans, and operational support materials.
• Perform testing and quality assurance to validate data accuracy, pipeline reliability, and readiness for production deployment.
• Use Terraform to provision, configure, and manage cloud infrastructure supporting data engineering workloads.
• Set up and maintain CI/CD pipelines using Concourse, GitHub Actions, or similar tooling to enable smooth, repeatable deployments.
• Troubleshoot pipeline issues, resolve defects quickly, and help ensure data jobs run reliably within expected service levels.
• Provide post-delivery support, including issue investigation, defect resolution, and operational handoff.
• Identify and mitigate risks, including single points of failure, reliability gaps, and operational dependencies.
• Collaborate with engineers, analysts, and reporting teams to share knowledge, strengthen best practices, and improve delivery quality.

Project ScopeThe Data Engineer will augment capacity on the Supply/Yield Data Engineering team and support priority work related to INVENTORY ART, including migration activities and the development of core data pipelines and assets. These assets will support analytics and reporting used by Steve S.’s teams in Tableau Cloud. The role will also provide additional support for the Yield Looker-to-Tableau migration and related modernization efforts.Required Skills and Experience

• Hands-on experience developing ETL, ELT, or data engineering pipelines in production environments.
• Strong experience with Databricks, Spark-based processing, and modern data pipeline development.
• Experience working with AWS cloud services and Snowflake data platforms.
• Proficiency with infrastructure-as-code practices, preferably Terraform.
• Experience implementing CI/CD practices using Concourse, GitHub Actions, or comparable deployment automation tools.
• Ability to write clear technical specifications, documentation, and support materials.
• Strong testing, quality assurance, troubleshooting, and operational support skills.
• Ability to work collaboratively with cross-functional teams and translate requirements into working technical solutions.

Preferred Qualifications

• Experience supporting analytics or BI migrations, especially from Looker to Tableau.
• Experience developing data assets for Tableau Cloud reporting and analytics use cases.
• Familiarity with supply, yield, inventory, or advertising-related data domains.
• Experience modernizing or enhancing legacy data engineering jobs.
• Strong understanding of operational reliability, risk mitigation, and single-point-of-failure reduction.

#ZR
#INDGEN