Senior Platform Data Engineer

Job Description

Senior Platform Data Engineer

\n

\n

We are seeking a Senior Platform Data Engineer. This is a production engineering role focused on building and operating a centralized, enterprise-grade data platform that institutionalizes proprietary knowledge and delivers data as a product — not a side effect.

\n

\n

You will design, build, and operate models, pipelines, integrations, and infrastructure across the full medallion architecture (Bronze → Silver → Gold). This role requires true end-to-end ownership: design, build, test, deploy, monitor, and continuously improve.

\n

\n

What You’ll Do

\n

\n
• Build and maintain ingestion pipelines from raw Bronze landing through curated Gold domain models

\n

• Develop transformation models using dbt with testing, Slim CI, and exposure tracking

\n

• Author and manage production-grade Airflow DAGs with SLA monitoring and backfill strategies

\n

• Design and implement streaming and event-driven pipelines (Kafka, Glue Streaming, Delta Live Tables)

\n

• Collaborate with data scientists, platform engineers, and product teams to ship reliable data products

\n

• Implement data quality frameworks and enforce governance standards

\n

• Containerize workloads and deploy to Kubernetes (EKS)

\n

• Contribute to infrastructure as code (Terraform provisioning for S3, IAM, Glue, etc.)

\n

• Document lineage, metadata, ownership, and sensitivity classification

\n

\n

\n

This is not a “pipeline-only” role — it is platform ownership at scale.

\n

\n

Core Technical Skills

\n

\n

Python

\n

\n
• Production-level standards

\n

\n

SQL

\n

\n
• Advanced analytical queries, window functions, optimization

\n

• Experience with Redshift, Snowflake, or Databricks SQL

\n

\n

dbt

\n

\n
• Gold-layer modeling and testing

\n

• Slim CI, exposures, meta configurations

\n

\n

Orchestration

\n

\n
• Apache Airflow (DAG authoring, task dependencies, SLAs, backfills)

\n

• Prefect or Dagster is a plus

\n

\n

Streaming & Event Pipelines

\n

\n
• Kafka topic consumption

\n

• Bronze landing jobs

\n

\n

Cloud Data Platforms (hands-on with at least one)

\n

\n
• Databricks (Delta Lake, Unity Catalog, Auto Loader, DLT)

\n

• AWS (S3, Glue, Lake Formation, Redshift)

\n

\n

Data Quality

\n

\n
• Great Expectations, dbt tests, or equivalent

\n

\n

Docker & Kubernetes

\n

\n
• Containerized jobs

\n

• Kubernetes Jobs and CronJobs

\n

• EKS deployments

\n

\n

Infrastructure as Code

\n

\n
• Terraform familiarity

\n

\n

Data Modeling

\n

\n
• Dimensional modeling

\n

• Slowly Changing Dimensions

\n

• Medallion architecture design

\n

\n

\n

Data Platform-Specific Experience

\n

\n
• SaaS ingestion via Fivetran, Airbyte, or custom connectors (REST, SFTP, JDBC)

\n

• Change Data Capture (Debezium or connector-based CDC modes)

\n

• Handling semi-structured and unstructured data (PDF, Excel, JSON)

\n

• Designing Bronze schemas for fidelity and downstream flexibility

\n

• Vector embedding pipelines (chunking strategies, embeddings, pgvector/OpenSearch)

\n

• Data lineage tooling (OpenLineage, DataHub, Unity Catalog)

\n

• Governance standards: PII handling, sensitivity classification, column masking, partition-level access

\n

\n

\n

Ways of Working

\n

\n
• Agile sprint participation (planning, standups, retros)

\n

• Treat pipelines as production software (tests, runbooks, SLAs, alerts)

\n

• Migration-safe schema changes (expand/contract pattern)

\n

• Proactive data quality ownership

\n

• Rigorous peer code reviews

\n

• Close collaboration with data science and platform engineering teams

\n

\n

\n

Nice to Have

\n

\n
• Migration of legacy data assets (Excel, Access, shared drives)

\n

• ML feature engineering and point-in-time correct pipelines

\n

• Open-source data tooling contributions

\n

\n

\n

Qualifications

\n

\n
• 5–8+ years of experience developing data products and platforms

\n

• Strong background in Agile software development

\n

• Demonstrated success in prior engineering roles

\n

• Strong analytical and problem-solving capabilities

\n

\n

\n

What We’re Looking For

\n

\n
• Clear communicator who can document complex data flows

\n

• Results-driven and impact-focused

\n

• Proactive and self-directed

\n

• Comfortable interfacing across technical and business stakeholders

\n

• Entrepreneurial mindset with a willingness to innovate

\n

\n

\n

If you’re a production-minded data engineer who thrives on building scalable platforms and owning pipelines end-to-end, we’d love to connect.