Full Time position /W2 Contract position - Sr. Data Engineer (Databricks) -- Onsite in Charlotte, NC
Job Description
Sr. Data Engineer (Databricks)
\n
Location: Charlotte, NC (Hybrid - 2 days Onsite)
\n
Contract Type: C2H in 6 months
\n
Duration: 12+ months
\n
Interview Type:
\n
\n
\n
Job Description:
\n
1. Advanced Spark Optimization - Optimizing DataFrame transformations to avoid shuffles and wide dependencies, Adaptive Query Execution (AQE) tuning, Handling skew, Caching & checkpointing strategies, Cluster sizing & autoscaling strategies
\n
2. Delta Lake Internals & Performance Tuning - Optimizing Delta performance, Understanding Delta logs & transaction protocol, Time travel & schema evolution best practices, CDC (Change Data Capture) patterns, Multi-hop architecture (Bronze/Silver/Gold)
\n
3. Databricks Workflows & Orchestration - Orchestrating ETL/ELT with Jobs & Workflows, Multi-task job pipelines (with task dependencies), Job clusters vs all-purpose clusters, Error handling & retries, CI/CD deployment using repos
\n
4. Unity Catalog & Enterprise Data Governance - Fine-grained access control (schemas, tables, columns, views), Data lineage tracking, Secure data sharing across workspaces, Managing tokens, service principals & permissions models
\n
5. Databricks SQL & Lakehouse Warehouses - Writing high-performance queries on the Lakehouse, Materialized views, SQL dashboards, Understanding Photon execution engine for high-speed queries
\n
7. Streaming & Real‑Time Pipelines - Structured Streaming internals, Auto Loader for incremental ingest, Trigger types & watermarking, Handling late-arriving data, Stateful stream processing
\n
8. Advanced Cluster & Compute Management - Spot vs on-demand clusters, Photon runtime vs standard, Cluster policies for cost control, SQL warehouses vs all-purpose compute, Monitoring with Ganglia / metrics dashboards
\n
9. Best Practices in Lakehouse Architecture - Medallion Architecture patterns, Modular, reusable ETL code patterns, Cost optimization strategies, Data quality frameworks (e.g., expectations, constraints)
\n
10. DevOps & CI/CD Integration - Git integration via Databricks Repos, Promoting code across dev/test/prod, Using the Databricks CLI, Automated deployments using Azure DevOps/GitHub Actions
\n
\n
Nice to have:
\n
11. Azure Data Factory - Understanding of Pipelines, Activities, and Datasets, Linked Services configuration, Integration Runtime types (Auto‑IR, Self‑Hosted IR, Azure IR), Source and Sink concepts in ADF
\n
ETL/ELT Data Integration - Building data ingestion pipelines (batch + incremental loads), Designing end‑to‑end data transformation