Member of Technical Staff (Data Scientist, Evals)
Skills & Technologies
Requirements
- Experience
- 4+ years
- Education
- PhD
Joblaze summary
In this role, the individual will focus on developing and maintaining automated evaluation pipelines to enhance the quality of answers provided by Perplexity's search engine. Key skills include proficiency in Python and SQL, along with experience in data science and machine learning, particularly in cloud environments like AWS and Databricks. This position is ideal for someone with a strong technical background and at least four years of relevant experience, especially those familiar with large language models and consumer-facing applications. The team operates in a dynamic environment where their work directly influences product improvements.
Joblaze insights
- Listed about a month ago on Joblaze — tracked directly from Perplexity's career page.
- Perplexity has 59 other open roles including Customer Success & Enterprise Support Lead, APAC , Sales Systems Engineer, Enterprise Operations, Scaled Enterprise Customer Success Manager.
- 2358 active Python roles on Joblaze right now.
- 1250 active AWS roles on Joblaze right now.
- 1036 active SQL roles on Joblaze right now.
Quick facts
- How much experience is required?
- At least 4 years of relevant experience for this Member of Technical Staff (Data Scientist, Evals) role.
- What's the tech stack?
- Joblaze extracted these technologies from the posting: AWS, Databricks, SQL, Python.
- What seniority level is this role?
- Perplexity targets staff-level candidates for this position.
- Is this full-time or contract?
- Full-time for this Member of Technical Staff (Data Scientist, Evals) role at Perplexity.
From the original posting
Perplexity serves tens of millions of users daily with reliable, high-quality answers grounded in an LLM-first search engine and our specialized data sources. We aim to use the latest models as they are released, but the intelligence frontier is a jagged one, and popular benchmarks do not effectively cover our use cases. In this role, you will build specialized evals to improve answer quality across Perplexity, covering search-based LLM answers and other scenarios popular with our users.
Responsibilities
Architect and maintain automated evaluation pipelines to assess answer quality across Perplexity's products, ensuring high standards for accuracy and helpfulness
Design evaluation sets and methods specifically to measure the impact of tool calls (particularly web search retrieval) on the final answer's quality
Develop VLM-based solutions to programmatically evaluate how final answers render visually across different platforms and devices
Continuously review public benchmarks and academic evaluations for their applicability to the Perplexity product, adapting and incorporating them into our regular performance measurements
Operate within a small, high-impact team where your evaluation metrics directly shape product changes, collaborating closely with technical leadership to measure and improve Answer Quality
Qualifications
PhD or MS in a technical field or equivalent experience
4+ years of experience in data science or machine learning
Strong proficiency in Python and SQL (expected to write production-grade code)
Experience building within a modern cloud data stack, specifically AWS and Databricks
Comfortable with agentic coding workflows and using AI-assisted development tools to iterate faster
Preferred Qualifications
1+ years of experience working with LLMs at scale, specifically with LLM-as-a-judge setups
Prior experience working on customer-facing web products or consumer apps, with real user traffic at scale
A strong research background, with experience applying research methods to real-world ML problems
Experience defining evaluation metrics (e.g., factual consistency, hallucination rate, retrieval precision) and building ground truth datasets