We are looking for a Senior AI Engineer with Databricks expertise to design, deploy and maintain scalable machine learning pipelines using the Databricks platform. In this role, you will deliver production-ready ML pipelines, automated training and retraining workflows, deployed models, monitoring dashboards and CI/CD pipelines for ML systems.
Unlock the potential of remote work in Kazakhstan, giving you the flexibility to work from home or access our offices in Astana, Almaty or Karaganda.
Responsibilities
- Design, implement and maintain end-to-end ML pipelines on Databricks
- Build workflows for data ingestion, preprocessing, feature engineering, training and inference
- Leverage PySpark, Spark ML and Databricks notebooks/jobs
- Manage model versioning, experiment tracking and reproducibility using MLflow
- Package and deploy models for batch and real-time inference
- Monitor model performance, drift and retraining cycles
- Develop scalable ETL/ELT pipelines using Databricks Delta Lake
- Optimize data storage and access patterns through partitioning, Z-ordering and caching
- Integrate with data sources such as Azure Data Lake, S3, APIs and databases
- Implement CI/CD pipelines for ML workflows using Azure DevOps, GitHub Actions and Databricks Repos and Jobs API
- Configure clusters, autoscaling and cost optimization while applying Infrastructure as Code with Terraform, ARM and Bicep
- Implement logging, alerting and observability to ensure high availability and fault tolerance of ML systems
Requirements
- 3+ years of experience in machine learning engineering or related roles
- Expertise in the Databricks platform including workspaces, jobs and clusters
- Proficiency in Apache Spark, PySpark and Python with pandas and scikit-learn
- Skills in MLflow for tracking, registry and deployment
- Competency in CI/CD pipelines, Docker containerization and REST APIs for model serving
- Familiarity with version control using Git
- Background in Azure including Azure Databricks, ADLS, ACR and AML
- Knowledge of data preprocessing, feature engineering and model training and evaluation
- Understanding of libraries such as XGBoost, LightGBM and CatBoost
- English proficiency at B2 level or higher
Nice to have
- Familiarity with AWS including S3, EMR and SageMaker
- Skills in streaming pipelines with Spark Structured Streaming and Databricks Feature Store
- Knowledge of Kubernetes
- Competency in monitoring tools such as Prometheus and Grafana
- Experience with large-scale production systems
We offer/Benefits
We connect like-minded people:
- Delivering innovative solutions to industry leaders, making a global impact
- Enjoyable working environment, whether it is the vibrant office or the comfort of your own home
- Opportunity to work abroad for up to two months per year
- Relocation opportunities within our offices in 55+ countries
- Corporate and social events
We invest in your growth:
- Leadership development, career advising, soft skills and well-being programs
- Certifications, including GCP, Azure and AWS
- Unlimited access to LinkedIn Learning and Udemy
- Free English classes with certified teachers
- Discounts in local language schools, including online courses for the Kazakh language
We cover it all:
- Participation in the Employee Stock Purchase Plan
- Monetary bonuses for engaging in the referral program
- Medical & family care package
- Six trust days per year (sick leave without a medical certificate)
- Coverage of psychology sessions of your choice
- Benefits package (sports activities, a variety of stores and services)
EPAM is a team of technologists and innovators united by a passion for technology. In Kazakhstan, we operate across all cities with offices in Astana, Almaty, and Karaganda and work with the world's leading companies from different industries. In 2023, EPAM received the Export Excellence Award at the esteemed Digital Bridge Awards, showcasing our commitment to excellence and innovation.
Read Full Description