Area(s) of responsibility

Job description

As a Platform Admin, will play a crucial role in designing, implementing, and maintaining our data platform using Databricks on the Microsoft Azure cloud. You’ll collaborate with cross-functional teams to ensure the scalability, reliability, and performance of our analytics and data solutions.

Experience Level:Experience Level: Level 2 (Overall 5 to 7 yrs)

Must Have Skills:

Must Have Skills: 2 to 4 years of exp in the below skill set

Snowflake, Databricks, SQL, Python, Terraform, Azure Devops

Good to Have Skills:

exp in Azure Cloud, Database services, DevOps Operation

Job Qualification / Requirements:

Candidate working hours would be Chicago time zone (18:30 IST to 2:30 IST).
Bachelor’s degree in Computer Science, Engineering, or related field.
Proven experience as a Databricks administrator or similar role.
Strong expertise in Databricks platform and its components, including workspaces, clusters, and jobs.
Experience in configuring and optimizing Databricks clusters for big data processing.
Familiarity with security measures and access controls within the Databricks platform.
Understanding of data governance principles and experience with data cataloging tools like Unity Catalog.
Experience with Infrastructure as Code (IaC) and automation tools, preferably Terraform.
Knowledge of data catalog tools (e.g., Microsoft Purview).
Excellent problem-solving skills and ability to work in a collaborative environment.
Relevant certifications in Databricks or related technologies are a plus.

Job Responsibilities:

Manage and administer the Databricks platform, including the Unity Catalog, to support data engineering and data science workflows.
Work with Unity Catalog to organize, discover, and govern data assets within the Databricks platform.
Maintain the data quality and consistency within the Unity Catalog.
Collaborate with data engineers and data scientists to understand their requirements and configure the Databricks platform accordingly.
Create and manage workspaces, clusters, and jobs in the Databricks environment.
Develop and maintain backup and disaster recovery strategies to ensure data integrity and availability.
Monitor and manage platform costs, including optimizing resource utilization to control expenses.
Implement security measures and access controls to ensure data privacy and protection within the Databricks platform.
Configure and optimize Databricks clusters for efficient and scalable data processing and analytics.
Stay up-to-date with security best practices and compliance requirements.
Monitor the performance and health of the Databricks platform and troubleshoot any issues that arise.
Monitor and manage platform costs, including optimizing resource utilization to control expenses.
Setup Databricks MLOps
Collaborate with data architects and data stewards to ensure data quality and consistency within the Unity Catalog.
Maintain detailed documentation of configurations, procedures, and best practices.
Design Implement and maintain Iac solutions using Terraform to provision and manage Azure Cloud resources like clusters and SQL Warehouses in Databricks.
Build, Configure and maintain CICD pipelines using GitHub and Azure Devops for the deployment of application and infrastructure on Databricks.