We are seeking a highly skilled and experienced BigData/PySpark Engineer to join our dynamic Big Data Analytics team. This role is pivotal in designing, developing, and optimizing robust, scalable data pipelines for large-scale data processing and analytics.
Key Responsibilities:
- Design & Development: Create and optimize scalable ETL (Extraction, Transformation, Loading) pipelines using PySpark for massive datasets.
- Coding & Engineering: Write clean, efficient, well-documented code primarily in Python (PySpark) often leveraging frameworks/tools.
- Collaboration: Work with cross-functional teams (senior developers, data engineers, analysts, business partners) to understand data requirements and ensure seamless solution integration.
- Troubleshooting & Optimization: Debug and resolve data processing issues and performance bottlenecks in Spark applications and other big data technologies.
- Full SDLC Involvement: Participate in the entire software development lifecycle, from requirements analysis and design to testing, deployment, and operations.
- Data Integrity: Ensure high data quality and integrity throughout the data lifecycle.
This candidate possesses 4-8 years of experience in developing and managing Enterprise Applications, demonstrating a robust foundation in Big Data technologies and a strong grasp of software development principles.
Key Experience & Expertise:
- Enterprise Application Development: 4-8 years in developing and managing enterprise-grade applications.
- Object-Oriented Programming (OOP): Solid foundation in OOP concepts.
- Big Data Development: Expertise in PySpark, HDFS, Hive, Sqoop, and Hadoop for Big Data environments.
- Database Technologies: Good exposure to SQL Server and ORACLE databases. Experience with query writing for data validation/manipulation
- Scripting & Automation: Proficient in Shell Scripting and experience with job scheduling tools like Autosys.
- BI Reporting Tools: Some exposure to BI tools, specifically Tableau.
- Tools & Practices: Proficient with Git; experience with JIRA, Confluence. Familiarity with DevOps and CI/CD pipelines.
Job Family Group:
Technology
Job Family:
Applications Development
Time Type:
Full time
Most Relevant Skills
Please see the requirements listed above.
Other Relevant Skills
For complementary skills, please see above and/or contact the recruiter.
Citi is an equal opportunity employer, and qualified candidates will receive consideration without regard to their race, color, religion, sex, sexual orientation, gender identity, national origin, disability, status as a protected veteran, or any other characteristic protected by law.
If you are a person with a disability and need a reasonable accommodation to use our search tools and/or apply for a career opportunity review Accessibility at Citi.
View Citi’s EEO Policy Statement and the Know Your Rights poster.
Read Full Description