Join Profound as an Evals Engineer to build evaluation systems for AI marketing tools in a fast-paced startup environment.
Posted by employer 1 week ago
First seen on Joblaze 1 week ago
Last verified on the company career page 1 week ago
What you'll build
- Build Profound’s eval system
- Write evals for every pipeline behind the product
- Define what a great AI Marketer run looks like
- Study how answer engines behave
- Build tooling for debugging model and agent behavior
Must have
- 3+ years building evaluation or measurement systems
- Strong Python and data skills
- Deep familiarity with LLM evaluation
Nice to have
- Experience with search, retrieval, citation, or fact-verification systems
- Publications at venues like NeurIPS, ICML, ICLR, ACL, or COLM
AI in the day-to-day
Profound's AI Marketer uses data to find opportunities and automates reporting and content drafting.
Requirements
- Experience
- 3+ years
Not disclosed in this posting: visa sponsorship.
Benefits
Joblaze summary
In the role of Evals Engineer at Profound, the individual is responsible for developing and maintaining the evaluation systems that ensure the accuracy and reliability of the company's AI marketing tools. Key skills include strong Python programming, data analysis, and a deep understanding of LLM evaluation methodologies. This position is ideal for someone with over three years of experience in measurement systems, particularly in AI, who thrives in a fast-paced startup environment. The role significantly influences product decisions by integrating evaluation metrics into the development process.
Joblaze insights
- Listed 9 days ago — first seen on Joblaze October 31, 2026. Last confirmed on Profound's careers page November 2, 2026.
- Salary band is above the typical range for AI/ML roles (median ~$190,000).
- Starts above 89% of 95 comparable mid ai/ml roles in United States that list AI/ML we track (median $150,000 across 39 companies). See AI/ML salary trends
- AI/ML appears in 53.4% of 472 comparable mid ai/ml roles in United States; Python appears in 48.9% of 472 comparable mid ai/ml roles in United States.
Quick facts
- Is the Member of Technical Staff, Evals role remote?
- No — this is an on-site role in New York, New York.
- What's the salary range?
- Profound lists $215,000–$300,000 for this role.
- How much experience is required?
- At least 3 years of relevant experience for this Member of Technical Staff, Evals role.
- Where is the role based?
- Profound is hiring for this position in New York, New York.
- What's the tech stack?
- Joblaze extracted these technologies from the posting: AI/ML, Python.
- What seniority level is this role?
- Profound targets mid-level candidates for this position.
- Is this full-time or contract?
- Full-time for this Member of Technical Staff, Evals role at Profound.
From the original posting
As an Evals Engineer, you will own how Profound knows its AI is right. At Profound, measurement is the product: when a Fortune 500 CMO sees their brand’s visibility drop in ChatGPT, they act on that number. No public benchmark measures whether a model attributed a citation correctly or ran a competitive analysis well, so we build our own. You’ll decide what gets measured, why, and how results shape what we ship. Every product runs through the evals you design, so your impact compounds across all of them.
What you’ll do
Build Profound’s eval system: golden datasets, scorers and LLM judges, programmatic verifiers, regression alerts, and dashboards
Write evals for every pipeline behind the product: brand extraction, visibility scoring, sentiment, FactCheck, and citation classification
Define what a great, mediocre, and broken AI Marketer run looks like, and measure it across multi-step tool use, analytical correctness, and content quality
Study how answer engines behave across runs, platforms, regions, and personas, and design the sampling and statistics that keep customer-facing numbers trustworthy
Close the loop from real usage: collect customer signals, cluster failure modes, and turn regressions into permanent guardrails
Build tooling for debugging model and agent behavior, plus the labeling workflows and annotation guidelines behind our ground truth
Make evals part of how we decide things: wired into CI, launch reviews, and model selection
Who you are
3+ years building evaluation or measurement systems, whether AI evals, experimentation, ranking, or search quality
Strong Python and data skills, with a track record of shipping production systems
Deep familiarity with LLM evaluation: LLM-as-judge, benchmark design, agent harnesses, calibration, and significance testing
Sound experimental design, and the instinct to separate signal from noise in non-deterministic systems
Strong opinions about how models and agents behave, and you keep up with current research
You believe a good eval is clear, reliable, and hard to game
Work that shows how you think: shipped benchmarks, datasets, open-source tools, or publications
Comfortable operating in a fast-paced startup environment with rapid iteration cycles
Nice to have (optional):
Experience with search, retrieval, citation, or fact-verification systems
Publications at venues like NeurIPS, ICML, ICLR, ACL, or COLM
Location
This is an on-site role in our NYC or SF office, designed for builders who thrive on speed, iteration, and meaningful impact.
For this role, the expected base salary range is $215,000 to $300,000. Profound’s total compensation package is designed to be competitive and includes base salary, equity, and a full range of benefits and perks. Final compensation will depend on factors such as your skills, experience, qualifications, and location, and will be determined during the interview process. Our recruiting team will share more details about the full compensation package and benefits as you move through hiring.
#LI-PRO
Standard company text repeated across Profound's postings is omitted here.
Similar positions
Explore more