Senior Hadoop Infrastructure Engineer - Data Platform


DaaS (Data as a Service) team at Paytm Labs operates a charter of capturing, storing and processing data reliably at scale. The DaaS Team makes this data available for a large set of products that are used for internal and external services. The core infrastructure that powers this platform operates at a scale of speed, performance, and complexity that few others can claim. The issues we face with large-scale data storage, low-latency retrievals, high volume requests and high availability are common yet complex. To help us solve these challenges, we are looking for the best of the best engineering talent to come and join our cool & rewarding environment! Paytm Labs is in a hyper growth phase, and this is a stellar opportunity to make an impact. As Senior Hadoop Engineer, you will be a core contributor in the Data Platform team and help us deliver the world-class Data Platform Paytm is gearing towards for data products growth.

Your responsibilities:

• Get stuff done: A problem partially solved today is better than a perfect solution next year. Have an idea during the night? Discuss it with your team in the morning, code it, push it at noon, test it in the afternoon and deploy it the next morning.

•  You will participate in and build tools to diagnose and fix complex distributed systems handling petabytes of data & drive opportunities to automate infrastructure, deployments, and observability of data services.

• You will test, monitor, administer, optimize and operate multiple Hadoop / Spark clusters across cloud providers - AWS, GCP, Aliyun and on premise data centers, primarily in Python , Java and Scala.

•  Investigate emerging technologies in Hadoop ecosystem that relate to our needs and implement those technologies.

•  Partner with Hadoop developers in building best practices for Data Warehouse and analytics environment. Share an on-call rotation and handle service incidents.

• Working with Big Data tools and building high performance, high throughput, and distributed data pipeline and big data platform with Hadoop, Spark, Kakfa, Hive, and Presto.


•  Minimum 3+ years of handling services in a large scale distributed systems environment, preferably Hadoop.

• Hands on experience with Hadoop (or similar) ecosystem - Yarn, Hive, HDFS, Spark, Presto, Parquet, HBase

•  Experience with workflow management (Airflow, Oozie, Azkaban)

Familiarity with systems management tools (Puppet, Chef, Capistrano, etc)

•  Knowledge of Linux operating system internals, file systems, disk/storage technologies and storage protocols and networking stack.

•  Proven knowledge of systems programming (bash and shell tools) and/or at least one scripting language (Python, Ruby, Perl).

•  Be adaptable and able to focus on the simplest, most efficient & reliable solutions.

What We Offer:

• A collaborative, open work environment that fosters ownership, creativity, and urgency.

• Group Health Benefits plan and enrollment right from the day you start working, no waiting period.

• 100 Day Plan: We build a plan of action every 100-days. Each plan has a theme, a mission so to speak. As we grow, so have our plans and our ambitions.

• We believe in hard work, fun, innovation, and team building. Every other month, we plan an activity to celebrate that. So far, we've been biking, axe-throwing, bowling, rocking climbing, and go-karting.

• Chocolates & Snacks: Our "Chief Chocolate Officer" is in charge of stocking high-quality chocolates for all.

• Birthdays are celebrated with cakes, and special office events have beer and whiskey.

• We also get a weekly delivery of groceries to our office.

• Ping Pong: A chance to be Paytm Labs table tennis champion!

*Selected candidates will be given a technical challenge*

Paytm Labs is committed to meeting the accessibility needs of all individuals in accordance with the Accessibility for Ontarians with Disabilities Act (AODA) and the Ontario Human Rights Code (OHRC). Should you require accommodations during the recruitment and selection process, please let us know. Paytm Labs is an equal opportunity employer.

We thank all applicants, however, only those selected for an interview will be contacted.

Read Full Description
Confirmed 20 hours ago. Posted 30+ days ago.

Discover Similar Jobs

Suggested Articles