Member of Technical Staff - Pre-Training Evals Infra
Build evaluation infrastructure for AI models at Reflection AI, enhancing model capabilities and training processes.
Posted by employer 3 days ago
First seen on Joblaze 2 days ago
Last verified on the company career page 4 hours ago
What you'll build
- Build evaluation infrastructure
- Develop distributed systems for evaluating model checkpoints
- Optimize evaluation and inference performance
- Establish evaluation correctness and reproducibility
- Create intuitive interfaces and dashboards
Must have
- Strong proficiency in Python, C++, or Rust
- Experience designing and operating distributed systems
- Working knowledge of model inference and evaluation
Nice to have
- Ability to investigate complex problems
- Strong product judgment
- Attention to detail
Requirements
- Visa
- Sponsorship available
Not disclosed in this posting: compensation, years of experience, work arrangement.
Benefits
Joblaze summary
In this role, the individual will develop and maintain the evaluation infrastructure that supports researchers in assessing model capabilities and guiding training decisions. Key skills include proficiency in Python, C++, or Rust, along with experience in building reliable distributed systems and understanding model evaluation processes. This position is well-suited for someone with a strong technical background who can navigate complex problems and prioritize research needs effectively. Reflection AI emphasizes collaboration with researchers to create tools that enhance evaluation workflows.
Joblaze insights
Quick facts
- What's the tech stack?
- Joblaze extracted these technologies from the posting: C++, Python, Rust.
- Does Reflection AI sponsor work visas for this role?
- Yes — the posting indicates visa sponsorship is available for the right candidate.
- What seniority level is this role?
- Reflection AI targets staff-level candidates for this position.
- Is this full-time or contract?
- Full-time for this Member of Technical Staff - Pre-Training Evals Infra role at Reflection AI.
From the original posting
About the Role
You will build the evaluation infrastructure that researchers use to understand model capabilities, guide training decisions, and assess readiness for release. The role spans evaluation authoring, distributed execution, and results analysis, with a particular focus on pre-training. You will work directly with researchers to make evaluations faster, more reliable, and easier to use, ensuring that the results accurately reflect the models being evaluated.
Design and evolve evaluation abstractions. Build clear APIs and data models for evaluation tasks, datasets, configurations, model outputs, and scores, making it straightforward to add new evaluations and extend existing workflows.
Build scalable, reliable execution infrastructure. Develop distributed systems for evaluating model checkpoints throughout training and ahead of release, with robust orchestration, observability, and recovery from failures.
Optimize evaluation and inference performance. Profile inference servers and distributed execution to improve throughput, GPU utilization, and time to results.
Establish evaluation correctness and reproducibility. Build validation and regression tests, version datasets and configurations, and preserve the provenance needed to reproduce results and investigate discrepancies.
Build researcher-facing tools. Create intuitive interfaces and dashboards for inspecting model outputs, comparing checkpoints and experiments, and debugging unexpected behavior.
Operate and monitor evaluations as training progresses. Track evaluation workloads and results, triage failures, and develop alerts and diagnostics that help distinguish changes in model quality from inference or evaluation issues.
Turn research needs into reusable systems. Partner directly with researchers to identify bottlenecks, prioritize high-impact improvements, and turn bespoke workflows into self-service tools that work across teams.
About You
Strong proficiency in Python, C++, or Rust, with the ability to reason deeply about concurrency, memory behavior, and performance.
Experience designing, building, and operating reliable, maintainable distributed systems, with sound judgment about interfaces, failure modes, and architectural tradeoffs.
Working knowledge of model inference and evaluation, including how implementation and configuration choices can affect measured model quality.
Ability to investigate complex problems across system boundaries, follow evidence, and identify root causes rather than just address symptoms.
Strong product judgment and experience translating the needs of researchers or other technical users into clear APIs and usable tools.
Comfort owning ambiguous problems end to end, prioritizing competing research needs, and balancing rapid delivery with long-term reliability and maintainability.
Attention to detail and the persistence to investigate subtle discrepancies in model outputs, evaluation results, and system behavior.
Top-tier compensation: Salary and equity structured to recognize and retain our talent globally.
Standard company text repeated across Reflection AI's postings is omitted here.