Distributed Systems Software Engineer, Voltron Data
$160-220k
+ Equity
Enterprise support for Apache Arrow
Job no longer available
Enterprise support for Apache Arrow
101-200 employees
Job no longer available
$160-220k
+ Equity
101-200 employees
Company mission
To bring the best of open-source together and build bridges across the data analytics ecosystem with Apache Arrow.
Role
Who you are
- You have built large-scale distributed systems or networking protocols that handle throughput greater than 10-100GB/s
- You have worked with batch processing and stream processing systems. You are opinionated about data platform evolution and a strong thought leader
- You have good intuitions about distributed system challenges and failure characteristics
- You have experience planning capacity and estimations in cloud-native or on-premise data centers
- You love building at a scale that most companies don’t even imagine
- You have experience operating large-scale production systems and supporting your customers
- You have experience with hardware like GPUs, DPUs, FPGAs, and associated software
What the job involves
- We are looking for a highly motivated Distributed Systems Software Engineer to play a pivotal role in realizing Voltron Data’s streaming data vision within the composable data ecosystem
- As a critical team member, you will be entrusted with meticulous component designs and driving executions to completion
- You will collaborate alongside highly competent product engineering experts
- You will also have the opportunity to work across open-source stacks (e.g. , Ibis, Arrow, Substrait) and proprietary implementations
- Your contributions will directly impact the advancement of our unified data processing capabilities, drive innovation, and set new industry benchmarks. Join us in shaping the future of data analytics
- Upon joining:
- Spending time learning about the Ibis and Apache Arrow, the open-source software we support as critical building blocks in the composable data ecosystem
- Understand stream processing technologies that power data platform concepts such as Data Movement, Data Mesh, etc
- Getting exposed to accelerated computing concepts leveraging hardware such as GPUs and DPUs
- Learning and embracing the software development culture at Voltron Data
- Within a month:
- Engage actively in owning components from design to implementation in a native language like C++, Rust, or similar
- Contribute to technical discussions and technical design documents
- Work closely with open-source tools such as Ibis, Arrow, Kafka, DuckDb, Flink, and RAPIDS/cuDF
- Contribute to bug-fixing efforts and propose areas for improvement in conjunction with the team
- Within 6 months:
- Developing a comprehensive set of distributed stateless streaming operators and Arrow-based networking protocols to address the industry-wide data movement bottleneck
- You will work with the team to implement an end-to-end system that significantly beats the current cost-efficiency benchmark
- Leveraging columnar memory format, GPU compression, and stream processing techniques to transform an existing proof of concept into a more crafted offering
- Identifying and building reusable components across the accelerated data movement code base
- Within 12 months:
- Continuously profiling and analyzing throughput in a distributed system to identify inefficiencies, and designing solutions to solve them
- Incrementally evolving the streaming operators to support more complex stateful use cases in a unified compute engine
- Mature the technology to be ready for broader adoption
- Become one of the company's streaming go-to experts
Our take
Voltron Data offers enterprise support to companies using the increasingly popular open source Apache Arrow software development platform. Its flagship service is designed specifically for companies that wish to leverage the power of Apache Arrow, but are unable or unwilling to handle the intricacies of Arrow.
The strong links to Apache Arrow are easy to explain, as co-founders Wes McKinney and Joshua Patterson were heavily involved in the creation and rise in popularity of the Apache Arrow project. In a way, a major part of Voltron Data’s remit is to promote the software development platform, while making it more palatable and user-friendly for customers.
With the likes of Google and Microsoft now adopting Apache Arrow into their own projects, Voltron Data has been able to attract incredible amounts of investment, despite only launching its first commercial product in early 2022. Voltron Data has focused on growing all aspects of its team and acquired real-time AI platform startup Claypot AI in 2024, indicating that it's maintaining its momentum.
Kirsty
Company Specialist at Welcome to the Jungle
Insights
1% employee growth in 12 months
Company
Funding (2 rounds)
Feb 2022
$88m
SERIES A
Feb 2022
$22m
SEED
Total funding: $110m
Company benefits
- Work from Anywhere - Payroll and Benefits in 150+ Countries
- Unlimited PTO
- Medical, Dental, and Vision
- Retirement [USA Only]
- Home Office Budget
- Continuing Education Budget
Company HQ
Old Mountain View, Mountain View, CA
Leadership
Joshua Patterson
(CEO)Former Data Science Principal at Accenture Technology Labs, former Senior Director for Rapids.ai, an end to end platform for accelerating data science on NVIDIA GPU.
Wes McKinney
(Senior Advisor)Before co-founding Voltron Data, they were a part of the Project Management Committee for the creation of the Apache Arrow project. Creator of the Python pandas project, which made Python the go-to language for data science, machine learning and AI.
Share this job