Lead/Staff Backend Engineer, Lighthouse

• Master’s or Bachelor’s degree in a STEM field
• Experience:
• 3+ years Rust, including mastery of advanced Rust concepts and async Rust, or
• 1+ years Rust and 3+ years in any other systems programming language (e.g. C++)
• High level understanding of networking protocols
• Experience in cloud environment and designing infrastructure architecture
• Eagerness to learn and explore the world of web scraping
• Strong problem-solving and analytical skills, with the ability to identify and troubleshoot complex technical issues
• Excellent communication and collaboration skills, with the ability to work effectively in a team environment
• Fluent in English

Desirable

• We are in search of a Lead/Staff Backend Engineer with a deep fascination for the inner workings of the internet and a desire to impart this passion to our team
• You will play a critical role in our data extraction process, ensuring that we can extract and process billions of data points every day
• Drive the optimization and expansion of our crawling infrastructure, ensuring both high scalability and superior performance, while maintaining effective web scraping capabilities
• As part of this role, you will be responsible for driving developments on our man-in-the-middle proxy, written in Rust. This component is the heart of our crawler infrastructure, as it is responsible for monitoring, upstream proxy routing and request processing for all our data extraction traffic
• Introduce innovative methods for enhancing our monitoring and troubleshooting processes and tools
• Engineer scraping solutions by applying findings from our R&D
• Actively collaborate with diverse engineering teams to address challenges and advance initiatives related to backend infrastructure
• Serve as a subject matter expert, providing mentorship and support to team members while fostering an environment of continuous learning and development
• Our team is at the helm of our sophisticated data extraction infrastructure. Tasked with the responsibility of acquiring pricing data from over 200 unique data sources, we process nearly 200 million requests each day
• The data we collect is the lifeblood of our products. We are committed to ensuring its reliable and timely delivery, thereby powering our diverse product portfolio