Software Engineer, Anyscale
Ray Data
$170.1-237k
+ Stock Options
3+ days a week in office (Palo Alto, CA)
Open-source framework for AI software development
Be an early applicant
Open-source framework for AI software development
201-500 employees
Be an early applicant
$170.1-237k
+ Stock Options
3+ days a week in office (Palo Alto, CA)
201-500 employees
Company mission
To remove distributed systems expertise from the critical path of realizing the business potential of AI.
Role
Who you are
- At least 2 year of relevant work experience
- Solid background in algorithms, data structures, system design
- Experience in building scalable and fault-tolerant distributed systems
- Experience with data processing, database internals including Spark or Dask (streaming is a plus)
What the job involves
- Ray aims to provide a universal API for building distributed applications (e.g. a machine learning pipeline of feature engineering, model training, and evaluation)
- Data is usually a core element connecting these different stages, and therefore plays a critical role in Ray’s usability, performance, and stability
- We are looking for strong engineers to build, optimize, and scale Ray’s Datasets library and data processing capabilities in general
- The Ray Data team currently develops and maintains the Ray Datasets library, which is already powering critical production use cases (e.g. large scale data compaction at Amazon, and ML pipeline at Alibaba)
- Ray Datasets is a Python library built on top of Apache Arrow and Ray Core (Ray’s C++ backend), and the Ray Data team interacts closely with Ray Core components including the scheduler and the memory & I/O subsystems
- The Ray Data team also works closely with Ray’s ML libraries including Train, RLlib, and Serve
- A snapshot of projects you will work on:
- Performance of Ray Datasets at large scale (leveraging Arrow primitives, optimizing Ray object manager, etc.)
- Integration with ML training and data sources
- Stability and stress testing infrastructure
- Lead future work integrating streaming workloads into Ray such as Beam on Ray
- Differentiate Data operations in Anyscale hosted Ray service
- Develop high quality open source software to simplify distributed programming (Ray)
- Identify, implement, and evaluate architectural improvements to Ray core and Datasets
- Improve the testing process for Ray to make releases as smooth as possible
- Communicate your work to a broader audience through talks, tutorials, and blog posts
Our take
AnyScale is a platform facilitating the development of distributed applications with high computing demands on its Ray platform. It answers an increasing requirement for higher-level tech strategies among companies which are incorporating AI and machine learning into their products, and a similarly broad and growing demand for easier access to cloud programming.
AnyScale launched its first commercial offering in 2021, and intends to continue developing products that will lower the skill and resource threshold for cloud programming. In 2023 it released Aviary, a project to simplify open-source large language model (LLM) deployment. What is promising, and lucrative, about its proposition is that it offers a general-use distributed system, this sidesteps the cumbersome process of stitching together disparate distributed systems, previously a requirement for novel applications. It is currently finding use in a range of applications such as supply chain, environmental restoration and retail by organisations including Amazon and Uber.
Anyscape has raised considerable funding led by Andreessen Horowitz and Addition. This is being used to scale its team and further develop the Ray platform. As the demand for AI apps rapidly increases, Anyscape is poised to disrupt a $13 trillion market.
Kirsty
Company Specialist at Welcome to the Jungle
Insights
133% employee growth in 12 months
Company
Funding (last 2 of 4 rounds)
Aug 2022
$99m
SERIES C
Dec 2021
$100m
SERIES C
Total funding: $259.6m
Company benefits
- We offer a wide range of health, dental, and vision coverage options for you and your family — including many that are 99% covered by Anyscale
- Whether you’re hopping on public transit or driving in yourself, Anyscale covers a portion of your commuting costs each month
- Lunch is served every day in our San Francisco office. Dinner, too, if you’re ever working late. And did we mention daily boba runs?
- Give back to the causes and communities you love with paid time off specifically for volunteer work
- Anyscalers can take advantage of 12 weeks of paid leave following the arrival of a new little one
- Paid time off at Anyscale is flexible and unlimited. We encourage all Anyscalers to rest up and recharge when needed
Company HQ
Yerba Buena, San Francisco, CA
Leadership
Robert Nishihara
(CEO)Former research intern at Facebook and Software engineer intern at Google. Phd in Computer Science at Berkley.
Phillip Moritz
(CTO)Former employee at Ambi Robotics with PHD in Computer Science from Berkley.
Salary benchmarks
We don't have enough data yet to provide salary benchmarks for this role.
Submit your salary to help other candidates with crowdsourced salary estimates.
Share this job
View 23 more jobs at Anyscale