Machine Learning Operations Engineer, Proton

• We are looking for humble, mission-driven, systems-thinking people who want to make a big impact in a startup environment
• Your expertise in debugging and problem-solving will be essential in addressing issues that arise in production environments
• Strong expertise in infrastructure automation tools (Puppet, Terraform, OpenNebula, Kubernetes)
• Proficient in scripting languages (Bash, Python)
• Experience in managing and scaling data storage solutions, particularly MySQL, Kafka, and ElasticSearch
• Familiarity with Dask and Airflow for data processing and workflow orchestration
• Solid experience with CI/CD pipelines for automated deployments
• Hands-on experience with Docker for containerization
• Ability to operate in a fast-paced environment with a global team and participate in on-call rotations
• Excellent communication skills in English, both written and spoken

Desirable

• MSA is a highly multi-disciplinary group started in 2019 to tackle difficult mail delivery, spam, abuse, and account security problems that impact the Proton ecosystem
• We built sophisticated systems from scratch that combine human intelligence and machine learning to make tens of millions of realtime or asynchronous decisions each day
• As of late 2023, our areas of focus are 1) mail delivery and spam, 2) abuse and account security, 3) site reliability of Proton, and 4) a new product involving blockchains
• Over the past few years, our custom systems have reduced spam filter misclassifications by over 60%, blocked millions of abusive bulk signups, protected hundreds of thousands of users from account compromise by attackers, and mitigated hundreds of DDoS attacks
• In order to move fast with limited resources, we are organized into autonomous teams responsible for the whole system, with expertise across infrastructure, data storage, backend, web and mobile apps, machine learning, security and operational excellence
• In 3 years, we grew from 2 engineers to over 30 engineers and analysts, spread across 3 continents for 24/7 coverage
• As an ML Operations Engineer at Proton, you will be at the forefront of maintaining the operational excellence of our machine learning and anti-abuse systems
• These critical services must run reliably 24/7 and scale with increasing traffic and complexity
• Your responsibilities will include the research, development, stress testing, and rollout of new services and infrastructure
• You will collaborate with Software and ML Engineers to ensure that systems have comprehensive logging, monitoring, alerting, and on-call capabilities
• Ensure the operational excellence of machine learning and anti-abuse systems running 24/7
• Research, develop, stress test, and roll out new services and infrastructure
• Collaborate with engineers to implement logging, monitoring, alerting, and on-call capabilities
• Debug and resolve production issues efficiently
• Manage and monitor production systems involving in-house developed solutions
• Deploy, run, and maintain machine learning systems in production environments
• Utilize Docker for containerization and efficient deployment
• Implement and manage Continuous Integration/Continuous Deployment (CI/CD) pipelines