Job Description
Job Description
Technical Stack & Requirements:
- Core Competencies: Python, Apache Spark (PySpark), AWS Glue ETL, Data Lake concepts (Medallion Architecture).
- Infrastructure: CloudFormation/Terraform, CI/CD with GitHub Actions.
- Core Responsibilities: Orchestrate data extraction from diverse legacy and modern sources, engineer high-performance scalable pipelines, and provide expert-level production support to ensure system reliability.
Qualifications:
- Bachelor’s degree in Computer Science or related field.
- Strong hands-on experience with Apache Spark and Glue.
- Proven ability to work with Python-based data pipelines.
- Experience with Git and GitHub Actions is mandatory.
Required Qualifications
- Bachelor's degree in Computer Science, Engineering, Information Systems, or related discipline.
- 7 years of enterprise Data Engineering experience.
- Strong Python development experience.
- Advanced SQL programming skills.
- Extensive experience with Apache Spark and PySpark.
- Hands-on experience with AWS data services including:
- S3
- AWS Glue
- Athena
- Lambda
- Step Functions
- EventBridge
- Experience building enterprise ETL and ELT pipelines.
- Strong understanding of distributed data processing.
- Experience working with large-scale cloud data platforms.
- Strong knowledge of dimensional data modeling.
- Experience implementing CI/CD pipelines.
- Familiarity with Agile/Scrum methodologies.
- Excellent communication and stakeholder management skills.
Preferred Qualifications
- Financial Services or Asset Management industry experience.
- Experience with Snowflake or Databricks.
- Knowledge of Apache Iceberg, Delta Lake, or Hudi.
- Experience with Airflow.
- Knowledge of Kafka or Kinesis streaming.
- Experience supporting AI/ML and advanced analytics platforms.
- AWS Professional or Specialty Certifications.