Technical Data Lead

Job Description

Technical Data Lead

\n

Location – Fort Mill, SC

\n

\n

Responsibilities

\n

\n
• Design, develop, and implement data pipelines using AWS Glue and Lambda to support data ingestion and transformation.

\n

• Utilize PySpark for data processing and analysis, ensuring efficient data handling and performance optimization.

\n

• Implement Infrastructure as Code (IAC) using Terraform (TF) to automate deployment and management of cloud resources.

\n

• Develop and maintain SQL queries for data extraction, transformation, and loading (ETL) processes.

\n

• Manage containerized applications using Docker and orchestrate them with Amazon EKS.

\n

• Design and optimize data warehouse solutions using Snowflake (SF) or Redshift (RS) to support analytics and reporting.

\n

• Work with data lake technologies such as Iceberg, Lake Formation, and Delta tables to enhance data accessibility and governance.

\n

• Collaborate with data scientists, analysts, and other stakeholders to understand data requirements and deliver solutions that meet business needs.

\n

• Ensure data quality, integrity, and security throughout the data lifecycle.

\n

• Stay updated with the latest industry trends and technologies to continuously improve data engineering practices.

\n

\n

Mandatory Skills

\n

\n
• Strong experience with AWS services, particularly Glue and Lambda.

\n

• Proficiency in PySpark for data processing.

\n

• Expertise in Infrastructure as Code (IAC) using Terraform.

\n

• Solid SQL skills for data manipulation and querying.

\n

• Experience with containerization using Docker and orchestration with EKS.

\n

• Hands-on experience with data warehousing solutions like Snowflake or Redshift.

\n

• Familiarity with data lake technologies such as Iceberg, Lake Formation, and Delta tables.

\n

• Proven ability to work in a cloud environment.

\n

\n

Preferred Skills

\n

\n
• Experience with data governance and data quality frameworks.

\n

• Knowledge of data modeling and database design principles.

\n

• Familiarity with Agile methodologies and project management tools.

\n

• Strong analytical and problem-solving skills.

\n

• Excellent communication and collaboration abilities.

\n

\n

Qualifications

\n

\n
• Bachelor's degree in Computer Science, Information Technology, or a related field.

\n

• 8-12 years of experience in data engineering or a related role.

\n

• Proven track record of delivering data solutions in a cloud environment.

\n

• Strong understanding of data architecture and data management best practices.

\n

\n

\n

\n

\n