Job Description
\n
\n
Our healthcare client is looking for a Data Engineer with 7+ years experience to own the multi-stage data pipeline layer that ingests from external sources (BigQuery direct queries, vendor file feeds, APIs) into a governed lake/warehouse. Deliver scaled event-driven integrations; managing ETL for downstream integrations, an event-bus for user engagement services, and API / query access for billions of data points.
\n
What you'll do in this role:
\n
Ship a production-ready, like-for-like ETL replacement with zero data loss and validated parity against legacy outputs.
\n
Establish lake → warehouse pipelines with schemas, contracts, and idempotent restatement for corrections.
\n
Implement rules evaluation and event context enrichment to support downstream filtering and event-response integrations.
\n
Stand up observability for pipelines and jobs (SLOs, error/retry rates, latency dashboards, lineage) and on-call runbooks.
\n
Provision reproducible infrastructure with Terraform; define CI/CD and cost/performance guardrails across environments.
\n
Develop and support agentic workflows (data access, context management, & modern AI development practices in your own work) across the organization.
\n
Key Responsibilities
\n
Design and implement ingestion/connectors (BigQuery direct, CSV→JSON, REST) and normalization into standardized data models.
\n
Build event-driven jobs and services (Python/Node) that enrich, dedupe, and apply rules; ensure idempotency and safe replays.
\n
Define data contracts, schema evolution, backfill strategy, and cutover plans; partner with stakeholders on acceptance criteria.
\n
Operate AWS workloads (EC2, Lambda, AppRunner, RDS, Redshift) with Terraform; secure secrets, roles, and least-privilege access.
\n
Optimize SQL for MPP systems (Redshift, Snowflake, or similar); profile queries, partition/cluster, and tune materializations.
\n
Implement observability (logs, metrics, tracing, lineage) and incident response; drive postmortems and remediation.
\n
Maintain concise documentation of architecture, workflows, standards, and governance.
\n
Qualifications
\n
7–10+ years backend/data engineering with production ownership of large event-driven data systems.
\n
Proficient in Python, Node, and similar toolsets for ETL job implementation; strong testing and reliability toolset.
\n
Deep AWS experience and Terraform-based IaC; CI/CD for data and application deployments.
\n
Expert SQL and performance tuning on Redshift, Snowflake, or equivalent.
\n
Experience delivering idempotent pipelines, restaging/reconciliation, and parity validation against legacy systems.
\n
Experience in highly-governed environments (HIPAA, GLBA, PCI, etc.).
\n
Solid security and compliance practices, supporting internal and external audits (SOC 2 ISO 27001, etc.) and remediation.
\n
Preferred
\n
Orchestration experience (Airflow or similar), streaming (Kinesis/Kafka/SQS), and dbt or warehouse-centric modeling.
\n
Data quality frameworks, lineage/metadata tooling, and SLA/SLO design.
\n
Exposure to real-time enrichment and rules engines; familiarity with warehouse-native features (tasks/streams).
\n
Things you should be able to do:
\n
Diagram a multi-stage lake → warehouse pipeline with cutover, backfill, restaging for corrections, and SLOs.
\n
Discuss agentic workflow support from a data/platform perspective.
\n
Demonstrate experience (positive and lessons-learned moments) for at-scale data/platform implementations.