Corteva

We are seeking a Data Engineer to design, develop, and optimize scalable data pipelines supporting advanced analytics and machine learning solutions in a cloud-based environment. The ideal candidate has hands-on experience with Azure Data Services and Databricks, a strong background in data pipeline orchestration, proven expertise in data quality management and process automation, and experience in Procurement or Supply Chain.

Key Responsibilities:

1. Data Pipeline Architecture & Development:

• Design, develop, and maintain robust ETL/ELT pipelines to handle large-scale data ingestion, transformation, and integration.
• Build and optimize data workflows using Azure Data Factory, Databricks (PySpark, Spark SQL), and Azure Synapse Analytics.
• Ensure pipeline scalability, fault tolerance, and efficiency across diverse data sources, primarily structured (tabular) datasets.
• Implement incremental loads, change data capture (CDC), and other advanced data ingestion strategies.

2. Automation & Process Optimization:

• Develop and maintain automated data pipelines with a focus on performance optimization and cost-efficiency in the Azure environment.
• Implement CI/CD pipelines for seamless deployment of data solutions, leveraging DevOps tools and Databricks Workflows.
• Collaborate with cloud architects to optimize resource usage and adhere to cloud governance best practices.

3. Data Management & Quality Assurance:

• Lead the design and implementation of data quality frameworks to ensure data integrity, consistency, and compliance across systems.
• Develop monitoring solutions for pipeline health, data freshness, and anomaly detection.
• Maintain comprehensive documentation covering data models, transformation logic, and operational procedures.

4. Cross-functional Collaboration & Stakeholder Engagement:

• Partner with Data Scientists, Analysts, and Business Stakeholders to understand data needs and translate them into effective solutions.
• Facilitate integration of machine learning models into production data pipelines.
• Provide technical mentorship to junior data engineers and contribute to team knowledge-sharing initiatives.

Required Skills & Qualifications:

Education: Bachelor’s degree in Computer Science, Data Engineering, Analytics, Statistics, Mathematics, or a related field. (Master’s degree is a plus.)

Experience:

• 3+ years of hands-on experience in data engineering or a related discipline.
• Proven experience designing and deploying end-to-end data pipelines in Azure and Databricks environments.

Language: Proficiency in English (written and spoken) is required, with strong English skills being prioritized.

Technical Skills:

Programming & Data Processing:

• Advanced proficiency in SQL and Python for data manipulation, transformation, and analysis.
• Extensive experience with PySpark and Spark SQL for big data processing in Databricks.

Cloud & Data Services (Azure):

• In-depth knowledge of Azure services, including:
• Azure Data Factory (ADF) for pipeline orchestration
• Azure Data Lake Storage (ADLS) for data storage and management
• Azure SQL Database for relational data management
• Experience with Azure Functions and event-driven architectures is a plus

Automation & DevOps:

• Hands-on experience implementing CI/CD pipelines using tools like Azure DevOps, GitHub Actions, or similar.
• Familiarity with infrastructure-as-code (IaC) tools such as Terraform or ARM templates.
• Experience with Databricks Workflows and job orchestration tools.

Data Management & Warehousing:

• Strong understanding of data lakehouse architectures and data warehousing solutions (e.g., SQL Server, Redshift, BigQuery).
• Experience designing and maintaining data models and schema designs for analytical use cases.
• Familiarity with data governance, security best practices, and compliance standards.

Machine Learning Integration (Preferred):

• Experience supporting machine learning workflows and integrating models into production pipelines.
• Understanding of MLOps practices is a plus.

Preferred Qualifications:

• Experience with real-time data processing (e.g., Apache Kafka, Azure Stream Analytics).
• Familiarity with Power BI data connections and reporting structures.
• Hands-on experience with Databricks Workflows for complex pipeline orchestration.