AI Product Management and Operation (Evaluation) - Flow

ByteDance

Responsibilities

About the Team

Flow is ByteDance's AI-native innovation business team, focusing on developing products in emerging fields such as LLM. Our product portfolio includes Doubao (China) and Cici (Global). The Model Evaluation Team plays a key role within Flow, providing accurate and timely assessment signals for product iterations. We are responsible for the entire evaluation process, from designing methodology to executing evaluations, identifying issues, and proposing feasible improvement suggestions.

Join us to explore innovative opportunities in AI, grow in a dynamic team environment, and create real value for users!

Responsibilities

  • Evaluation Framework Establishment: Design evaluation frameworks for the Cici App including standards, test cases, and workflows. Create metrics that measure performance across key dimensions and enable data-driven decisions in existing and upcoming use scenarios
  • Human-Machine Evaluation Integration: Build workflows combining human expert and automated evaluation. Ensure automated processes align with human judgment, identify performance gaps, and increase testing coverage to supplement human evaluation
  • Evaluation Results Analysis: Interpret and analyze evaluation findings with depth, transforming nuanced case analysis into actionable insights. Develop data production and product design strategies aimed at enhancing model capabilities and/or user experience
  • Cross-functional Collaboration: Align product, engineering and evaluation teams across cultural and professional boundaries. Facilitate communication, resolve differences, and build consensus around overall evaluation methodologies
  • Evaluation Methodology Research: Stay at the forefront of LLM evaluation by researching industry-leading assessment methodologies and reports. Integrate insights with practical applications to continuously improve our evaluation framework

Qualifications

Minimum Qualifications

  • Bachelor's degree or above in Computer Science, Data Science, Business Analytics, or related fields
  • 3+ years of experience as a Product Manager, Business Analyst, or Operations Specialist in technology companies or other highly dynamic environment
  • Strong data analysis skills with experience in designing metrics for product performance
  • Project management capabilities with the ability to balance priorities and handle ambiguous requirements
  • Quick adaptation to new tools, technologies and concepts, with demonstrated learning agility
  • Genuine passion for AI/LLM technologies

Preferred Qualifications

  • Experience in AI/ML product management, particularly with conversational AI
  • Hands-on experience with LLM evaluation methodologies
  • Background in developing metrics frameworks for measuring product performance
  • Excellent stakeholder management and communication skills
Read Full Description
Confirmed 9 hours ago. Posted 6 days ago.

Discover Similar Jobs

Suggested Articles