Supercomputing Engineer, Fleet Software

Job Summary

Fleet Software enables deployments at scale through configuration management, safe software and firmware rollouts, and software recovery, while partnering with other engineering teams to drive product improvements as we ramp production at unprecedented speed. The team works closely with Node Systems, which builds the single-rack software, clients, and tuning recipes that Fleet deploys and manages across many racks.

The team also builds the monitoring and analysis systems that connect signals across the rack. Engineers and their agents use this data to investigate failures, spot patterns, and continually improve the performance and reliability of the inference fleet. We're looking for a hands-on engineer who has built infrastructure and owned it through real deployments in the field.

Key Responsibilities

• Build the fleet management system that provisions, configures, updates, and recovers Etched-owned hardware across labs and customer deployments

• Implement safe software and firmware rollouts, including validation, staged deployment, and handling of partial failures

• Build telemetry and log pipelines that collect rack metrics and logs into a central store, including decisions on storage, retention, and query performance

• Develop monitoring, alerting, and analysis tools that help engineers, their agents, and customers understand fleet behavior and investigate failures

• Build secure, least-privilege access to BMCs and servers across lab, office, and data center networks

• Give lab operators simple controls for managed hardware, such as draining and powering down for repair, and rolling out validated versions

• Partner with Node Systems to define the interfaces, clients, and validated configurations needed to manage hardware across the fleet

• Work with Customer Deployments, Infrastructure, Lab Operations, and Factory Testing, including support for customer environments, local monitoring, and customer telemetry ingress and analysis

• Act as first-line triage for issues on Etched-owned systems, find root causes in the management system, and escalate to Infrastructure when needed

• Turn lessons from deployments and incidents into engineering priorities, and help decide where to adopt existing tools versus building our own
  

You may be a good fit if you have

• Experience building infrastructure software and owning it in production, including deployments, incidents, and the changes that followed

• Background in fleet management, provisioning, configuration management, or observability for compute infrastructure or connected devices

• Solid distributed systems and Linux fundamentals. You understand how systems behave under load, during upgrades, and when components fail

• Hands-on experience with Kubernetes and container orchestration, including running it on bare metal

• Experience with Slurm or similar HPC and GPU cluster schedulers and resource management

• Experience with network boot and bare metal provisioning (PXE, HTTP boot, iPXE, DHCP/TFTP) to bring hosts up and re-image them at scale

• Strong proficiency in at least one of Go, Python, Rust, or C/C++

• Strong debugging skills using logs, metrics, and tracing

• Strong communication skills and comfort collaborating across hardware, firmware, and software teams
  

Strong candidates may also have experience with (Nice-to-have qualifications)

• Telemetry and log pipelines, including storage, retention, and query performance

• Bare metal servers, accelerator clusters, HPC, robotics, or other hardware fleets

• Firmware updates, hardware diagnostics, and automated recovery

• BMC, Redfish, or IPMI

• Software that runs in customer environments as well as internal infrastructure

• Docker and container tooling

• Familiarity with kernel, drivers, PCIe, or networking

• Designing test harnesses and failure-injection frameworks

Benefits

• Medical, dental, and vision packages with generous premium coverage

  • $500 per month credit for waiving medical benefits

• Daily lunch + dinner in our office

• Unlimited compute budget subject to ROI justification