MeetMe has flagged the Data Pipeline Engineer job as unavailable. Let’s keep looking.

Headquartered at Carnegie Mellon University, the Delphi research group was founded in 2012 to advance the theory and practice of epidemic forecasting. Our long-term mission is to make epidemic forecasting as widely accepted and useful as weather forecasting is today.

Delphi is in a critical growth phase. Pre-pandemic our focus was seasonal influenza; during the critical phase of the pandemic, we shifted our focus to COVID-19. Our signals are used regularly by public health officials, government agencies, data journalists, healthcare companies, financial firms, and fellow modelers. We are a CDC-designated Center of Excellence (one of two nationally) as well as a Center of innovation (one of five nationally), and work closely with CDC COVID, forecasting, and flu teams. We now are setting our sights beyond COVID and influenza, developing the technology and building the infrastructure needed for tracking and forecasting any fast-moving epidemic that presents a significant burden to public health. 

As a crucial part of that mission, Delphi's data pipelines run daily to produce the nation's largest repository of diverse, geographically-detailed, real-time indicators of infectious disease activity in the U.S. Public health data systems are, at core, made out of people, and the errors that are normal, human, and rare at the individual facility level can rapidly accumulate at the national level to create frequent outages. 

We seek an addition to our data pipeline operations team who can leverage their skills in systems administration, programming, and data science to prevent, identify, diagnose, and rectify problems or outages that occur in our data processing pipelines. Is this you? Interested in learning more? Apply today!

Duties:

  • Monitor and maintain the health and efficiency of data pipelines.
  • Troubleshoot and perform root cause analysis for data discrepancies and pipeline issues.
  • Communicate with data providers to understand data discrepancies and manage changes in data delivery.
  • Implement fixes and enhancements to improve data quality and pipeline performance.
  • Collaborate with data scientists and analysts to understand data needs and implement effective data solutions.
  • Develop strategies for data validation and quality assurance.
  • Optimize data flow and collection to improve system efficiency.
  • Document and manage data pipeline architectures, including maintenance and update protocols.
  • Use tools such as SQL, version control and CI/CD, containerization, task schedulers, python frameworks, and cloud services for data pipeline management.
  • Ensure compliance with data governance and security standards.

Skills and Experience:

  • Proficiency in data pipeline tools
  • Experience with Python and SQL
  • Strong understanding of data structures, algorithms, and system design
  • Familiarity with data storage solutions (e.g., databases, data lakes)
  • Excellent problem-solving skills and attention to detail
  • Effective communication skills, both synchronous and asynchronous

Nice to Have:

  • Experience in data engineering, particularly in data pipeline development and management
  • Experience in epidemiological data analysis or related fields
  • Advanced degree in a relevant field
  • Familiarity with machine learning concepts and tools

Flexibility, excellence, and passion are vital qualities within Delphi and the Machine Learning Department. Inclusion, collaboration, and cultural sensitivity are valued competencies across all CMU organizations. We are looking for someone who shares our values and who will support the mission of the group and the university through their work.

Minimum Qualifications:

  • Bachelor’s degree in Computer Science or related field.
  • 1-3 years’ work experience
  • Or a combination of education and relevant experience from which comparable knowledge is demonstrated.

Requirements:

  • Successful background check

The coming years are an exciting time of growth for us. Come join us to help make our vision a reality!

Location

Pittsburgh, PA

Job Function

Software/Applications Development/Engineering

Position Type

Staff – Regular

Full Time/Part time

Full time

Pay Basis

Salary

More Information: 

  • Please visit Why Carnegie Mellonto learn more about becoming part of an institution inspiring innovations that change the world. 
  • Click here to view a listing of employee benefits
  • Carnegie Mellon University is an Equal Opportunity Employer/Disability/Veteran
  • Statement of Assurance
Read Full Description
Confirmed 12 hours ago. Posted 30+ days ago.

Discover Similar Jobs

Suggested Articles