ML Performance Engineer, Stability AI
Deep Learning
Open source generative AI company
Job no longer available
Open source generative AI company
101-200 employees
Job no longer available
101-200 employees
Company mission
To activate humanity's potential through generative AI.
Role
Who you are
- At least 8+ years of relevant experience
- Applied programming experience in Python, C, and/or C++
- Experience with libraries and tools like PyTorch and CUDA
- Experience in building, productizing and monitoring orchestration pipelines for AI and Machine Learning pipelines
- Experience with training frameworks like Megatrong, NVIDIA or similar frameworks
- Experience in leading more junior engineers
- Experience with AWS and/or GCP
- Experience/exposure to CI tools infra tools is a nice to have (Kubernetes)
- Experience with Linux-based environments and scripting (Shell Scripting, Python, Powershell)
- Ability to work well as an individual contributor as well as within a multidisciplinary team environment
- Strong communicator with excellent interpersonal skills and can-do attitude to work and thrive in a fast-paced team environment
What the job involves
- We are looking for a talented ML Performance Engineer with a focus on Deep Learning and High-Performance Computing that will work with a growing multidisciplinary team of talented research scientists and machine learning engineers to improve and scale the efficiency within our computing capacity
- Optimizing Deep Learning Workflows:
- Monitor reports and dashboards and detect low utilization jobs, projects, users
- Partner with researchers to check their workflow when they lack performance
- Identify bottlenecks and suggest scripting optimisations
- For high-scale jobs, introduce AWS proprietary profiler and libraries to boost performance
- Scale-up gating process: check the scripts performance and vet requests to scale up
- Build a knowledge base / best practices documentation for all researchers
- Implement and monitor CPU usage levels for our CPU clusters; identify users that need assistance in properly coding to maximize usage of CPU’s
- Train researchers on best practices on how to implement automatization strategies to minimize human oversight on jobs
- Develop and Test Strategies for Future Workloads:
- Benchmark new systems capabilities and identify strategies to properly utilize them (H100, TRN2, TPUv5, Intel Gaudi)
- Define the minimum needs for storage speeds and find better data loading strategies to support high processing demands of the new accelerators
- High-Performance Computing (below responsibilities are optional; only the above are required) :
- Maintain HPC cluster operations
- Monitor dead nodes and recover them; document dead nodes and their fixes
- Monitor shared volumes health, usage, and clean-up needs
- Monitor the HPC Help Center and solve user problems
- Assist users in properly launching their jobs
- Monitor all CPU clusters for users
- Create and maintain processes around authentication, authorization and accounting for clusters usage
- Develop processes around security aspects of the HPC clusters, including tools to in case of security risks are identified (globally, by user, by team, by location, etc)
- Convert and deploy SLURM scheduling for all clouds and all resource types; integrate TPUs into our larger enterprise approach when SLURM becomes available
- Maintain AWS resources associated with the HPC clusters (login nodes, S3 buckets, FSx volumes, VPCs, subnets, NAT Gateways, S3 VPC Endpoints, routing tables)
Salary benchmarks
Our take
Stability AI is an AI-driven visual art company that designs and implements an open AI tool that creates images based on text input. Through collective intelligence and augmented technology, Stability AI is helping research communities develop cutting-edge AI models for image, language, audio, video, 3D content, biotech and other scientific research.
Working in partnership with Electronic Arts (EA), Stability AI is co-developing next-generation generative AI model and workflows designed to empower artists, designers, and developers across interactive entertainment. By embedding its research teams directly alongside EA's creative talent, Stability AI is advancing image-to-3D, text-to-3D, and volumetric media tools that accelerate world-building, rapid prototyping, and visual storytelling. This collaboration highlights Stability AI's growing leadership i multimodal and 3D generative AI, and its mission to put powerful, creator-centric tools in the hands of global creative industries.
The company is also focused on expanding its creative sector offerings, hiring legendary filmmaker James Cameron to bring an artistic perspective to its suite of storytelling tools. This exciting development will further help Stability AI push the boundaries of AI-driven media and help on its mission to transform visual storytelling.
Freddie
Company Specialist at Welcome to the Jungle
Company
Funding (last 2 of 3 rounds)
Jun 2024
$80m
EARLY VC
Oct 2023
$50m
CONVERTIBLE
Total funding: $231m
Company HQ
Notting Hill, London, UK
Share this job
View 8 more jobs at Stability AI