Site Reliability Engineer - Fleet Command & Control

Job Summary

You'll be one of Etched's first Site Reliability Engineers, building the command-and-control layer for our fleet of frontier inference systems: the alerting, automated recovery and fleet data that keep clusters serving, and the routing that gets each problem to the right expert. Our Supercomputing and Infrastructure teams build the fleet management, rollout and telemetry backbone; but you'll decide which signals define a healthy system, set how we alert and respond, and build agentic hardware health workflows on top of it.

Your job is to design and run a scalable system. Fleet SREs own detection, first response and handoff; domain SREs embedded in each team own the deep product fixes. Over time, most problems should be caught by an alert that measures what users experience, which is then triaged by agents, and sent to an engineer as needed, with the logs, dashboards and recent changes already attached.

Key Responsibilities

• Define what a healthy inference system looks like. In collaboration with software teams building out management and telemetry systems, identify the signals that most clearly show an unhealthy site, rack, node or chip, keep refining them as the fleet changes, and decide which actions (drain, cordon, restart, reroute, roll back) run on their own and which page a person.

• Set a common alerting standard. Define SLOs with each team (latency such as time to first token, error rate, availability), alert on user-facing symptoms, and verify that the metrics behind each alert measure what the alert assumes.

• Cut alert noise. Suppress expected alerts during deploys and maintenance automatically, correlate related alerts into one incident, and retire pages nobody acts on.

• Automate recovery. Turn runbooks into agentic hardware health recovery: service restarts, node drain and traffic shift, spare swaps. Build it with guardrails (scoped permissions, rate limits, blast-radius checks, an audit trail)..

• Route every event to the right engineer. Classify events by domain, page the right domain SRE with context attached, and dispatch on-site technicians for physical work such as tray swaps, reseats and leak response.

• Shape the fleet data. Work with Supercomputing and Infrastructure, who build the telemetry pipeline, so it captures the key signals. Set the tracing standard (OpenTelemetry) so every service can be followed end to end, and report uptime and reliability trends that drive engineering and capacity decisions.

• Run operationally excellent incident management. Own severity levels, incident channels, communications, postmortems and corrective-action tracking. Design on-call rotations, handoffs and onboarding so on-call is sustainable.

• Raise the reliability bar for services. Review service designs with their owners for failure modes and graceful degradation before they ship, and partner with Supercomputing on canaries, rolling node-by-node deploys and graceful drain, so updates stop causing downtime.

• Share on-call in a follow-the-sun rotation with SREs in other time zones.

Hands-on debug

• Be the first line of defense. Acknowledge, size the user impact, open the incident, and decide fast between restart, drain, rollback and escalation.

• Find where requests are stuck. Trace a latency or throughput regression through the serving stack (queueing, prefill, decode, handoffs, cache) to the stage that's backing up.

• Triage hardware on live racks. Work through errors: chips dropping off PCIe, firmware panic resets, chip-to-chip link errors, HBM ECC errors and power faults; run recovery tools and decide when a node comes out.

• Correlate across layers. Follow a request end to end with distributed traces (OpenTelemetry), then line up deploys, firmware events, host and BMC logs, network data and facilities telemetry to tell a software bug from a hardware fault from a coolant or power event.

• Check the alert, not just the system. When a page and a dashboard disagree, work out which one is wrong: a capped metric, a stale exporter or a failed scrape.

• Direct hands on site. Walk on-site technicians through reseats, PSU and tray swaps, and leak response with exact steps.

• Hand off cleanly. Leave a timeline, evidence and current hypothesis so the domain SRE or next shift picks up without starting over.

• Close the loop. Turn every manual fix into a runbook, then into tasks that can be executed by agents, so the same page doesn't wake anyone twice.

You may be a good fit if you have (Must-have qualifications)

• 2+ years in site reliability, production engineering or infrastructure for large-scale production systems (8+ years for the San Jose lead seat).

• You've defined alerting standards that several teams adopted, using Prometheus, Grafana or similar, and you know SLOs, error budgets and alert design well enough to teach them.

• Real on-call experience for services with external users, including leading incidents and writing blameless postmortems.

• A track record of debugging production incidents in systems you didn't write, under time pressure, from logs, metrics, traces and core dumps.

• Experience with application and scripting languages like python, golang, rust, etc.

• Hands-on with Linux, containers, Kubernetes and infrastructure as code (Terraform, Ansible or similar).

• Fluency with Linux debugging tools (perf, strace, gdb, dmesg, journalctl, lspci, tcpdump) for when the dashboards don't have the answer.

• Comfort with hardware in the loop: devices dropping off PCIe, ECC errors, power and thermal events, and failures that cross software and physical layers.

• Sound judgment about when automation should act on its own and when it should page a person.

• Clear writing, and the ability to get five or more teams to agree on shared standards.

• Operated GPU, accelerator or HPC fleets.

• LLM inference serving: prefill and decode, KV cache, batching, and latency metrics such as time to first token.

• Network debugging from switch to edge: tracing problems across switches, load balancers and Cloudflare, and using edge controls like rate limits to shed or shift traffic during an incident.

• Out-of-band management (BMC, Redfish, IPMI) and fleet-wide firmware rollouts.

• Data center telemetry: power distribution, liquid cooling and CDUs, DCIM.

• PagerDuty or similar incident tooling, including event correlation and automated remediation.

• Working with colocation providers and remote-hands technicians.

• Founding or scaling an SRE function at a startup.

Benefits

• Medical, dental, and vision packages with generous premium coverage

• Housing subsidy of $2,500 per month for those living within walking distance of the office

• Daily lunch and dinner in our office

• Unlimited compute budget subject to ROI justification

How we’re different

Etched believes in the Bitter Lesson. We are the first inference-focused frontier AI system, betting early on transformer and transformer-like architectures and on increasing model sizes. Our addressable market is the entirety of inference, unlike many of our competitors.