Search

Data Pipeline Technical Lead

PublishedPublished: 6/14/2022
Technology
|

Job Description

Job Description

Data Pipeline Technical Lead

\n

Lead an engineering team delivering production data pipelines on AWS for IRS data modernization. Set the technical direction for Databricks and Spark, plan the transition from legacy and mainframe sources, and coordinate delivery with federal stakeholders.

\n

Key Responsibilities

\n

    \n
  • Set architecture and engineering standards for ingestion, transformation, orchestration, data quality, and monitoring across Databricks, Delta Lake, and AWS data services.
  • \n

  • Guide engineers through design and code reviews, resolve technical dependencies, and address pipeline performance and reliability issues.
  • \n

  • Plan migration waves and define reconciliation, cutover, and rollback criteria with source-system owners and downstream data consumers.
  • \n

  • Manage the Agile backlog and release plan in Jira. Coordinate priorities, acceptance criteria, and release readiness with federal stakeholders.
  • \n

  • Partner with deployment and production support teams to document pipeline dependencies, validate operational readiness, and hand off releases.
  • \n

\n

Required Qualifications

\n

    \n
  • 12+ years in data engineering or data platform delivery, including 5+ years leading teams working with Spark or Databricks.
  • \n

  • Experience delivering production ETL/ELT pipelines on AWS using Databricks, Delta Lake, Amazon Redshift, Amazon RDS for PostgreSQL, and AWS Database Migration Service (DMS).
  • \n

  • Experience leading a migration or modernization from legacy or mainframe sources to a cloud data platform.
  • \n

  • Experience managing an Agile backlog and releases in Jira for a federal customer.
  • \n

  • Bachelor’s degree in computer science, engineering, or a related field, or equivalent experience.
  • \n

  • Ability to pass an IRS background investigation.
  • \n

\n

Preferred Qualifications

\n

    \n
  • Databricks Certified Data Engineer Professional certification. AWS Certified Solutions Architect or AWS Certified Data Engineer certification.
  • \n

  • IRS or Treasury experience, especially with EDP, CADE 2, or Individual Master File (IMF) modernization.
  • \n

  • Experience with Informatica and enterprise business intelligence tools such as Tableau, SAP BusinessObjects, or Power BI.
  • \n

Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...