Director, Research - Evaluation & Training

4w4 weeks ago

Snorkel AI

San Francisco, US · Full-time · $220,000 – $300,000

About this role

At Snorkel AI, we believe meaningful AI doesn't start with the model, it starts with the data. We are looking for a manager to lead a team of researchers focused on data evaluation, error analysis, and data valuation methods to predict model performance.

You and your team will be responsible for Snorkel's data design flywheel by analyzing model failures, finding capability and skill gaps in current models, and suggesting the next benchmarks to invest in. You will synthesize trends from model-failure analysis into recommendations that drive the company's data strategy.

This role acts as the primary bridge between the team's findings and Product, GTM, and our customers. You will lead and grow a team of researchers, setting a high bar for quality, rigor, and speed of execution while turning research into business outcomes.

Snorkel AI came out of the Stanford AI lab and works with some of the world's largest organizations. You will have the opportunity to help redefine how AI is built by empowering experts to build custom AI with their data faster than ever before.

Requirements

7+ years in applied AI, ML, or research roles, with 4+ years managing technical teams
A leader who has repeatedly turned research and analysis into business outcomes
Strong business and market judgment in the AI/ML space, understanding the competitive and frontier-lab landscape
Technically conversant in LLM evaluation, benchmarking, and model behavior analysis to set direction and judge experimental quality
A nose for trends: able to extract signal from evaluation results and failure cases to drive what gets built next
Excellent communication and storytelling skills to make technical results legible to non-research audiences
Familiarity with data valuation or data attribution research is a strong plus
Bonus: experience working with frontier labs, public benchmarks, or commercial AI data/eval products

Responsibilities

Own a multi-quarter roadmap centered on novel evaluation, error analysis, and data valuation techniques
Synthesize and share trends from model-failure analysis and benchmarking into recommendations on datasets the community should focus on
Focus on data valuation techniques that quantify how Snorkel data meaningfully improves model performance
Lead and grow a team of researchers, setting a high bar for quality, rigor and speed of execution
Act as the primary bridge between the team's findings and Product, GTM, and our customers
Analyze model failures to find capability and skill gaps in current models and suggest next benchmarks to invest in
Prove the value of Snorkel's data for customers through rigorous evaluation and error analysis

Benefits

Work with some of the world's largest organizations to build custom AI
Be part of a team pioneering data-centric methods that power sophisticated AI implementations
Join a company that started as a research project in the Stanford AI Lab
Remote and hybrid work options available