Staff ML Engineer, Frontier AI

The Role:

As a Staff Machine Learning Engineer at Ambience, you will help set the technical direction for the AI systems that power our clinical products. You’ll identify the highest-impact opportunities to improve model behavior, evaluation, post-training, and agentic systems, and lead the design and execution of cross-cutting initiatives.
This is a highly hands-on role with broad technical influence. You’ll work closely with clinicians, product managers, researchers, and engineers to translate cutting-edge research into reliable, production-grade AI systems.
Our engineering roles are hybrid — working onsite at our San Francisco office three days per week.

What You’ll Own:

• Define the AI Quality and Evaluation Strategy: Establish how we measure production AI quality across LLM and agentic systems, including automated graders, regression testing, human evaluation, failure taxonomies, and offline-to-online validation.

• Drive Production Model Improvement: Identify high-impact failure modes across products and lead improvements across prompting, retrieval, context, routing, data, and post-training. Design experiments that determine the right intervention and prove that improvements hold in production.

• Advance Post-Training Capabilities: Shape how Ambience uses techniques such as supervised fine-tuning, preference optimization, reinforcement learning, distillation, and synthetic data to improve model behavior.

• Architect Agentic AI Systems: Set technical direction for production systems involving tool use, retrieval, context and state management, routing, orchestration, tracing, and failure recovery.

• Build Self-Improving AI Loops: Design the systems that turn production failures and user feedback into better datasets, evaluations, and model behavior, creating scalable improvement flywheels across products.

• Shape the AI Roadmap: Co-develop the 12-month technical roadmap, balancing near-term product wins with longer-term research and platform investments.

• Raise the Technical Bar: Lead technical reviews, mentor engineers, establish best practices, and create reusable systems and frameworks that improve the effectiveness of the broader AI team.

• Stay at the Cutting Edge: Distill insights from recent research in LLMs, agents, post-training, NLP, speech, and multimodal AI, and drive experiments that keep Ambience at the forefront of clinical AI.

Who You Are:

• Expert in Production AI Systems
  7+ years in production ML, research engineering, or applied AI.
  Deep expertise with modern LLMs, transformers, and complex production AI systems.
  Have led or materially shaped consequential AI systems with real production impact.
  Able to reason across models, data, orchestration, evaluation, serving, and product behavior.

• Deep Evaluation Rigor
  Significant experience designing evaluations for LLMs, agents, or other complex AI systems.
  Can turn ambiguous product-quality problems into robust metrics, datasets, and experiments.
  Experienced with grader bias, leakage, contamination, misleading aggregate metrics, regression detection, and offline-online mismatch.
  Knows how to establish whether a model or system improvement actually translates into better user outcomes.

• Agentic Systems Expertise
  Have built or architected production systems involving multiple models, tools, retrieval, context and state management, routing, orchestration, tracing, and failure recovery.
  Understand how to evaluate and debug complex agent behavior across model, tool, and system boundaries.
  Comfortable making architectural decisions that span multiple products or teams.

• Post-Training and Model Improvement Expertise
  Have materially improved model behavior through post-training or model adaptation techniques such as supervised fine-tuning, preference optimization, reinforcement learning, distillation, synthetic data, or related approaches.
  Understand how data quality, objective design, evaluation, and post-training choices affect downstream model behavior.
  Know when post-training is the right intervention versus prompting, retrieval, context optimization, routing, or changes to the surrounding system.
  Have taken model improvements from experimentation through rigorous evaluation and production deployment.

• Production-Grade Software Engineer
  Proficient in Python and modern ML frameworks; PyTorch preferred.
  Comfortable with modern MLOps, CI/CD, observability, and containerized deployments.
  Remains deeply hands-on: writes code, inspects traces, analyzes failures, and prototypes solutions.

• Data-Centric AI Developer
  Skilled at building large, high-quality datasets and feedback loops.
  Experienced using production failures, user feedback, and active learning to improve model and system quality.
  Able to design data and evaluation infrastructure that creates leverage across multiple teams.

• Effective Technical Leader
  Able to work closely with clinicians, product managers, researchers, and engineers.
  Strong communicator who can simplify complex AI concepts and build alignment around technical decisions.
  Comfortable operating in ambiguity, identifying the right problems to solve, and driving execution across multiple workstreams.

• Mission-Aligned
  Passion for healthcare or other mission-driven industries.
  Thrives in a fast-paced, early-stage environment.
  Takes broad ownership of technical outcomes and raises the effectiveness of the people around them.

  Nice-to-Haves

• Experience with realtime voice, conversational AI, or multimodal systems.

• Prior work in healthcare, clinical AI, or other regulated, high-stakes industries.

• Experience interviewing or hiring ML engineers.

• Open-source contributions to ML, agent, evaluation, or post-training tooling.

Why Here:

Our products power specialty-specific note generation, chart-aware diagnosis prediction, and real-time clinical decision support in real clinical settings. The work is deeply technical, but the goal is simple: help clinicians do great work with less friction.
To keep improving, we can't wait around for bigger models. The fastest path is building intelligence that gets better with every encounter — learning from how clinicians actually use the products, what they change, and what outcomes follow.
You will own the hardest model quality problems across our clinical AI suite: coding models that navigate a proprietary million-term ontology with multi-objective precision, a scribe that learns from edit signals without introducing regressions, long-context chart understanding that stays faithful under real clinical complexity, and population-level reasoning that surfaces patterns across patients in a way that's auditable and actionable.
This is not applied ML on clean benchmarks — it's research-grade model work with production stakes, where your improvements directly shape what clinicians experience every day.

• Work on mission-critical AI technology that directly improves clinicians’ day-to-day lives and health system financial health across some of the most complex, high-stakes workflows in the world.

• Comprehensive medical, dental, and vision coverage for you and your dependents

• Parental leave to support your family needs