Performance Engineer

The Role

Due to the rapid scaling of our compute requirements, we are seeking a Performance Engineer to serve as the vital bridge between our frontier AI models and our high-performance computing infrastructure.

Your Impact

In this role you will be building the optimization layers and profiling infrastructure that maximizes the efficiency of our large-scale training and inference workloads, which is critical for accelerating our materials discovery pipeline and ensuring our researchers can iterate at the speed of thought.

At CuspAI, compute is our most precious resource. Your main focus initially will be profiling our current training runs to identify bottlenecks in Model FLOP Utilization (MFU). Over time you are likely to get involved in designing custom kernels and low-level optimizations that allow us to train larger, more complex models for chemistry and physics that were previously computationally prohibitive.

What You Will Do

Training & Inference Optimization

• Analyze model architectures to optimize how mathematical operations map onto hardware, specifically focusing on NVIDIA GPU kernels and collective communication layers (NCCL).

• Implement state-of-the-art efficiency techniques such as quantization, model pruning, and distillation to lower latency for our materials “search engine” APIs.

• Streamline data pipelines to ensure high-performance accelerators are never “starved” for data during massive training runs.

Profiling, Observability & Cost Management

• Use advanced observability tools to trace end-to-end execution of training jobs, pinpointing exact locations of processing clogs or interconnect contention.

• Build and maintain real-time dashboards tracking vital system metrics, including MFU, memory bandwidth utilization, and cluster uptime.

• Monitor and justify the utilization of our AI accelerators (e.g. H100 clusters), preventing idle time and memory leaks to ensure every dollar of compute spend translates into scientific progress.

• Develop scaling laws and internal tools to predict system behavior across compute, memory, and networking as our models grow in parameter count.

Interdisciplinary Collaboration

• Work closely with AI Researchers to adapt model architectures for better hardware compatibility without sacrificing scientific accuracy.

• Partner with the Infrastructure team to influence the design of our future compute clusters based on performance data and profiling results.

Must Have Skills and Qualifications:

• You are someone who gets excited about the opportunity to enable scientists to work on world changing challenges in this domain, with a personal interest in the potential applications of the technology that Cusp is building.

• You have 3+ years of experience in performance engineering, site reliability engineering, or low-level software development, specifically focused on high-performance computing (HPC) or AI workloads.

• Expert-level knowledge of GPU architecture and experience optimizing CUDA kernels or using libraries like Triton.

• Deep understanding of distributed training frameworks (e.g. PyTorch Distributed, DeepSpeed, or Megatron-LM) and networking protocols.

• Proficiency in profiling tools such as Nsight Systems, Nsight Compute, or Weights & Biases for performance tracking.

• A "builder" mindset with the ability to work autonomously in an early-stage, fast-paced startup environment.

Bonus Points (But Not Critical):

• Experience working with specialized AI hardware such as TPUs or alternative accelerators.

• An academic background or personal interest in Materials Science, Chemistry, or Physics.

• Knowledge of MLOps practices, specifically regarding the deployment of models at scale in cloud environments (AWS/GCP/Azure).

• Contributions to open-source performance optimization or machine learning libraries.
  

Additional Considerations

This role could be based in our Cambridge, London, Amsterdam or Berlin offices, with the expectation of being in the office three days per week. Additionally, there may be regular travel required to other locations for collaboration and project work.

As part of your onboarding, you will be expected to spend at least a few weeks in our Cambridge/London office learning about our platform and getting to know us in person.

Additionally, there may be regular travel required to Europe and other regional and global locations for collaboration and project work.

What We Offer

• A competitive salary: We value and reward impact and growth