Join Mirelo AI as a Training Infrastructure Engineer to optimize and design scalable systems for training generative AI models.
Posted by employer 10 months ago
First seen on Joblaze 1 month ago
Last verified on the company career page 5 hours ago
AI in the day-to-day
We develop generative AI models that create custom audio for video content.
Not disclosed in this posting: compensation, years of experience, work arrangement, visa sponsorship.
Benefits
Joblaze summary
The Training Infrastructure Engineer at Mirelo AI is responsible for optimizing the training stack, focusing on GPU profiling, debugging pipelines, and enhancing throughput for generative AI models. Key skills include expertise in GPU memory management, parallelism strategies, and experience with PyTorch, particularly in optimizing training and inference workloads. This role is suited for candidates with a strong technical background in machine learning infrastructure and a hands-on approach to implementing advanced optimization techniques. Mirelo AI is in a growth phase, having recently secured significant funding, which allows for impactful contributions to its innovative projects.
Joblaze insights
Quick facts
- What's the tech stack?
- Joblaze extracted these technologies from the posting: GPU, ML, PyTorch, Slurm.
- What seniority level is this role?
- Mirelo AI targets senior candidates for this position.
- Is this full-time or contract?
- Full-time for this Training Infrastructure Engineer role at Mirelo AI.
From the original posting
In this role, you’ll focus on the full training stack - profiling GPU behavior, debugging training pipelines, improving throughput, choosing the right parallelism strategies, and designing the infrastructure that lets us train models efficiently at scale. You’ll work across cluster management, model training, efficient data pipelines for video and audio, inference and optimizing pytorch code. Your work will shape the foundation on which all of our generative models are built and iterated.
Key Responsibilities
Find ideal training strategies (parallelism approaches, precision trade-offs) for a variety of model sizes and compute loads
Profile, debug, and optimize single and multi-GPU operations using tools like Nsight and stack trace viewers to understand what's actually happening at the hardware level
Analyze and improve the whole training pipeline from start to end (efficient data storage, data loading, distributed training, checkpoint/artifact saving, logging, …)
Set up scalable systems for experiment tracking, data/model versioning, experiment insights.
Design, deploy and maintain large-scale ML training clusters running SLURM for distributed workload orchestration
Ideal Candidate Profile
Familiarity with the latest and most effective techniques in optimizing training and inference workloads—not from reading papers, but from implementing them
Deep understanding of GPU memory hierarchy and computation capabilities—knowing what the hardware can do theoretically and what prevents us from achieving it
Experience optimizing for both memory-bound and compute-bound operations and understanding when each constraint matters
Expertise with efficient attention algorithms and their performance characteristics at different scales
Nice to Have
Experience in implementing custom GPU kernels and integrating them into PyTorch.
Experience with diffusion and autoregressive models and understanding of their specific optimization challenges
Familiarity with high-performance storage solutions (VAST, blob storage) and understanding of their performance characteristics for ML workloads
Experience with managing SLURM clusters at scale
Why Join?
Join at a pivotal moment. We've secured fresh funding and are gaining traction - now is when your contributions can make a real difference to our success.
Standard company text repeated across Mirelo AI's postings is omitted here.