Machine Learning Intern, d-Matrix

• Currently pursuing a degree in Computer Science, Electrical Engineering, Machine Learning, or a related field
• Familiarity with PyTorch and deep learning concepts, particularly regarding model optimization and memory management
• Understanding of CUDA programming and hardware-accelerated computation (experience with CUDA is a plus)
• Strong programming skills in Python, with experience in PyTorch
• Analytical mindset with the ability to approach problems creatively
• This role is ideal for a self-motivated intern interested in applying advanced memory management techniques in the context of large-scale machine learning inference. If you’re passionate about optimizing machine learning models and are excited to explore cutting-edge solutions in model inference, we encourage you to apply

Desirable

• We are seeking a motivated and innovative Machine Learning Intern to join our team
• The intern will work on developing a dynamic Key-Value (KV) cache solution for Large Language Model (LLM) inference, aimed at enhancing memory utilization and execution efficiency on D-Matrix hardware
• This project will involve modeling at the PyTorch graph level to enable efficient, torch-native support for KV-Cache, addressing limitations in current solutions
• Research and analyze existing KV-Cache implementations used in LLM inference, particularly those utilizing lists of past-key-values PyTorch tensors
• Investigate “Paged Attention” mechanisms that leverage dedicated CUDA data structures to optimize memory for variable sequence lengths
• Design and implement a torch-native dynamic KV-Cache model that can be integrated seamlessly within PyTorch
• Model KV-Cache behavior within the PyTorch compute graph to improve compatibility with torch.compile and facilitate the export of the compute graph
• Conduct experiments to evaluate memory utilization and inference efficiency on D-Matrix hardware
• Develop an efficient support system for KV-Cache on D-Matrix hardware
• Create a torch-level modeling framework for dynamic KV-Cache
• Ensure compatibility of the KV-Cache model with torch.compile and other PyTorch features for optimized graph export