Join fal as a senior software engineer to build large-scale distributed systems for AI products.
Posted by employer 7 months ago
First seen on Joblaze 1 month ago
Last verified on the company career page 1 day ago
AI in the day-to-day
Leverage AI to automate the mundane parts of building complex but reliable systems.
Requirements
Not disclosed in this posting: compensation, work arrangement, visa sponsorship.
Benefits
Joblaze summary
In this role, the software engineer focuses on developing and enhancing fal's core platform for distributed systems, tackling challenges like request routing and AI workload orchestration. Proficiency in Python or Rust is essential, along with a solid grasp of distributed systems principles such as fault tolerance and capacity planning. This position is ideal for seasoned engineers with a strong background in building scalable systems under production loads. The team is engaged in cutting-edge generative media technology, offering ample opportunities for professional growth.
Joblaze insights
Quick facts
From the original posting
You are an experienced software engineer who is passionate about building large-scale computing platforms. You have experience building systems that remain reliable under high traffic, partial failures, and changing capacity. You know how to deliver reliability, performance, and scale with minimum operational load.
Take ownership of one or more of the Rust/Python systems that power our AI inference and large-scale GPU computing platform
Design for 100x growth in workloads and GPU capacity, with reliable, low-latency execution across the globe
Use AI aggressively to accelerate development and automate repetitive operations, alerting, and recovery
You will specialize in one or more of these critical areas:
Request routing and queuing on a global scale
App deployment and autoscaling
Worker orchestration and placement
GPU fleet autoscaling and capacity management
GPU fleet lifecycle automation: provisioning, diagnostics & recovery, upgrades, tracking
Global file storage, caching and distribution
Global IP backbone: private networking across datacenters, elastic public IPs, ACLs/firewalls, private connectivity
Workload execution: containerization, globally distributed filesystems, RAM/VRAM snapshotting
3+ years building and operating large-scale production systems, with a track record of reliability and scale
Strong Rust and/or Python skills
Strong systems fundamentals, with technical depth relevant to your specialization: distributed coordination, scheduling, fault tolerance, Linux, networking, storage, or capacity management
Experience building and using observability to drive performance and reliability decisions
Clear communication, sound technical judgment, and the initiative to move quickly, drive decisions across teams, and own systems from design through production
Multi-tenant compute platforms, AI inference or training infrastructure, GPU workload scheduling
High-performance systems programming: async runtimes, zero-copy, memory-safe concurrency
Linux systems engineering and fleet automation: configuration management, kernel and driver debugging, safe upgrades, and automated recovery
Distributed filesystems like JuiceFS and Lustre
Experience with Nvidia/AMD GPU infrastructure; DCGM, NVLink, RDMA, InfiniBand/RoCEv2
Global networking and routing: BGP, ECMP, tunnels, eBPF, Geneve
Interesting and challenging work
A lot of learning and growth opportunities
Regular team events and offsites
Standard company text repeated across Fal's postings is omitted here.