Equifax is seeking a highly motivated Senior Python Developer to spearhead the development, maintenance, and optimization of Python-based web scraping solutions. This position mandates the advanced application of web scraping tools and libraries like Beautiful Soup, Scrapy, requests, Playwright, and Selenium for the efficient collection, cleansing, and structuring of data from a variety of sources.
The incumbent will be responsible for ensuring the utmost accuracy and completeness of collected data for critical analytical and automation initiatives. Essential to this role is proven expertise in extracting, cleaning, and processing data from online sources, effectively navigating challenges such as dynamic content and anti-bot countermeasures.
The ideal candidate will possess a robust foundation in Python, extensive experience with web scraping tools, a demonstrated history of managing large and complex datasets, and a comprehensive understanding of HTML, CSS, and browser behavior. Furthermore, this senior-level role encompasses the mentorship of junior team members and active contribution to data strategy and collection reliability.
What you'll do
- Optimize web scrapers for high throughput and scalability to handle millions of records daily/monthly.
- Implement robust data validation and quality checks to ensure the accuracy and completeness of extracted information, aligning with business data quality standards.
- Help define and implement the organization's data acquisition strategy, ensuring alignment with business goals.
- Document scraper logic, configurations, and data extraction processes for maintainability and knowledge transfer.
- Experience with advanced anti-bot techniques and proxy management solutions (e.g., rotating proxies, residential proxies, CAPTCHA solving services).
- Solid understanding of legal and ethical considerations related to web scraping, including data privacy (e.g., GDPR, CCPA) and website terms of service.
- Collaborating with data analysts, product managers, and other developers to understand data requirements and deliver high-quality results.
What experience do you need
- Bachelor’s degree in Computer Science, Software Engineering, Information Technology, or a related subject.
- 5+ years of demonstrated expertise in Python coding.
- 2+ years of practical experience in building and deploying web scrapers, including handling dynamic content, pagination, and anti-scraping measures.
- 1+ years of expertise in reverse engineering, obfuscation, unpacking, distributed systems, data analysis, and data mining.
- English proficiency B2 or above.
What could set you apart
- Experience with advanced anti-bot techniques and proxy management solutions (e.g., rotating proxies, residential proxies, CAPTCHA solving services).
- Familiarity with cloud platforms (e.g., AWS, Google Cloud, Azure) for deploying and managing scraping infrastructure.
- Familiarity with CI/CD pipelines and tools like Airflow or Jenkins for automated testing and deployment of scraping solutions.
- Practical experience implementing data quality frameworks in a cloud environment.
Primary Location:
CRI-Sabana
Function:
Function - Tech Dev and Client Services
Schedule:
Full time
Read Full Description