Research Engineer, Benchmarks & Evaluation

The problems are high-stakes, data-dense, and unforgiving.

About Ramp Labs

Ramp Labs is Ramp's AI research group. We post-train models on finance work, build the environments and evals they learn from, and ship them into the products 70,000+ companies use to move money. We think the defensible part of an AI product is the loop: proprietary environments, evals that track what customers care about, and models trained against both.

About the Role

Every model decision at Ramp Labs runs through a benchmark. Which model to route to. Whether a post-training run worked. Whether an agent is ready to post a corrective journal entry to a customer's ledger without a human in the loop.

You'll own Ramp's benchmarking effort. You'll turn real finance work (expense policy review, invoice coding, month-end close, spreadsheet Q&A) into benchmarks and RL environments, build the graders, and tell the team what is actually improving.

What You’ll Do

• Build benchmarks from real workflows: turn agent traces, customer edge cases and expert labels into versioned benchmark suites for long-horizon agents that run for hours across tools and systems, plus tool use, multi-step reasoning and financial judgment

• Own the benchmarking effort: decide what we measure, run benchmark suites across models and agent versions, and report results the team makes decisions on

• Design graders: rubrics, programmatic checks and llm-as-judge, calibrated against human agreement, with a clear view of where each one breaks

• Close the loop with post-training: turn evals into reward functions and RL environments, and catch reward hacking and contamination before they ship

• Publish external benchmarks and write-ups on how models perform at real finance work

What You Need

• Hands-on experience building LLM benchmarks or evals that people used to make real decisions

• Strong Python and software engineering; your eval code is reproducible and fast

• Statistical judgment: variance, confidence intervals, sample size, inter-rater agreement, and knowing when a delta is noise

• Working knowledge of post-training (SFT, RL with verifiable rewards) and how evals feed into it

• Comfort with SQL and large datasets; you'll work directly from production traces

• Taste for what is worth measuring

Nice-to-Have

• Industry experience on an eval, benchmarking or post-training team

• Built RL environments or verifiers for agentic tasks

• Published benchmark or evaluation research (NeurIPS Datasets & Benchmarks, ICML, ACL)

• Built or contributed to eval frameworks (verifiers, lm-evaluation-harness, HELM or similar)

• Familiarity with accounting, finance or other expert-domain work

• Public work we can look at: a benchmark, an eval suite, a failure-analysis write-up

• Flexible PTO

• Centralized home-office equipment ordering

• Health and wellness stipend

• Budget for intra-office travel

• Weekly coffee stipend

United States

• 100% medical, dental & vision insurance coverage for you, with partial coverage for dependents

• One Medical annual membership

• Fertility HRA (up to $10,000 per year)

• Pet insurance

• In-office perks: lunch, snacks, drinks, and more

Canada

• Group medical, dental, and vision coverage through Sun Life

• Life, AD&D, and disability coverage

• Fertility drug coverage (up to $4,000 lifetime)

United Kingdom

• Private medical insurance through Freedom Elite

Ramp Applicant Privacy Notice