Job Title: Data Engineering Lead
Location: Remote/NCR
Type: Full-Time
Experience Level: Senior
About the Role
We are looking for a Data Engineering Lead with deep expertise in AWS, Python, and modern data engineering practices to architect and deliver scalable, secure, and efficient data solutions. In this role, you’ll lead the design and development of ETL pipelines, implement robust data warehousing solutions, and support machine learning workflows across various business domains.
Your strong understanding of ETL and Data Warehousing concepts, including Slowly Changing Dimensions (SCD) and Medallion Architecture, along with proficiency in SQL and AWS-native tools, will be key to success.
Key Responsibilities
Data Pipeline & Architecture
- Design, develop, and maintain scalable ETL pipelines leveraging AWS services such as Glue, S3, Lambda, and Athena.
- Hands on Python and Pyspark coding experience.
- Implement Medallion Architecture (bronze, silver, gold layers) to organize and optimize data for both operational and analytical use cases.
- Apply best practices in data modeling and data warehousing, including handling SCD Type 1 and Type 2 transformations.
Data Management & Quality
- Ensure data accuracy, completeness, and reliability across batch and near-real-time workflows.
- Create robust error handling, logging, and monitoring processes for pipeline operations.
- Write and optimize complex SQL queries for data transformations, validations, and reporting use cases.
Collaboration & Delivery
- Partner with data engineers, analysts, and product teams to enable data access and support analytical models and dashboards.
- Participate in code reviews, contribute to best practices, and ensure high engineering standards.
- Troubleshoot and resolve data pipeline and infrastructure issues in production environments.
Must-Have Qualifications
- 5+ years of hands-on experience in data engineering, particularly with Python, PySpark, and SQL.
- Proven experience building ETL pipelines using AWS services (Glue, Lambda, S3, Athena, Step Functions).
- Strong knowledge of Data Warehousing concepts, SCD Types (Type 1, Type 2), and Medallion Architecture.
- Solid understanding of data modeling, partitioning, and performance optimization in big data environments.
- Excellent communication skills with the ability to explain technical topics to non-technical stakeholders.
Nice-to-Have Skills
- Experience with DBT for data transformation and testing.
- Hands-on exposure to Databricks and the Lakehouse architecture.
- Familiarity with CI/CD for data pipelines and infrastructure-as-code using tools like Terraform or CloudFormation.
Read Full Description