Data Engineer, Pathway

Stream & Distributed Data Processing

Salary not provided

SQL

AWS

Kubernetes

Python

Kafka

Airflow

Rust

Azure

Spark

Snowflake

Flink

dbt

Entry and Junior level

San Francisco Bay Area

Remote from US

Real-time data analysis

Job no longer available

Real-time data analysis

21-100 employees

B2BArtificial IntelligenceEnterpriseDeep TechData Analysis

Job no longer available

Salary not provided

SQL

AWS

Kubernetes

Python

Kafka

Airflow

Rust

Azure

Spark

Snowflake

Flink

dbt

Entry and Junior level

San Francisco Bay Area

Remote from US

21-100 employees

B2BArtificial IntelligenceEnterpriseDeep TechData Analysis

Company mission

To help software engineers to focus on developing code logic, without having to worry about the way data changes over time.

Job

Company

Role

Who you are

Inside-out understanding of at least one major distributed data processing framework (Spark, Dask, Ray,...)
6 months+ experience working with a streaming dataflow framework (e.g.: Flink, Kafka Streams or ksqldb, Spark in streaming mode, Beam/Dataflow)
Ability to set up distributed dataflows independently
Experience with data streams: message queues, message brokers (Kafka), CDC
Working familiarity with data schema and schema versioning concepts; Avro, Protobuf, or others
Familiarities with Kubernetes
Familiarity with deployments in both Azure and AWS clouds
Good working knowledge of Python
Good working knowledge of SQL
Experienced in working for an innovative tech company (SaaS, IT infrastructure or similar preferred), with a long-term vision
Warmly disposed towards open-source and open-core software, but pragmatic about licensing

Desirable

Know the ways of developers in a corporate environment
Passionate about trends in data
Proficiency in Rust
Experience with Machine Learning pipelines or MLOps
Familiarity with any modern data transformation workflow tooling (dbt, Airflow, Dagster, Prefect,...)
Familiarity with Databricks Data Lakehouse architecture
Familiarity with Snowflake's data product vision (2022+)
Experience in a startup environment

What the job involves

You will be working closely with our CTO, Head of Product, as well as key developers
Implement the flow of data from their location in client's warehouses up to Pathway's ingress
Set up CDC interfaces for change streams between client data stores and i/o data processed by Pathway; ensuring data persistence for Pathway outputs
Design ETL pipelines within Pathway
Contribute to benchmark framework design (throughput / latency / memory footprint; consistency), including in a distributed system setup
Contribute to building open-source test frameworks for simulated streaming data scenarios on public datasets

Our take

Machine learning has immense implications for reporting when used on a company's data, but despite its speed, accuracy and thoroughness, it can't currently be applied to incoming data. Put simply, the infrastructure to make this possible is missing, and without this, it's impossible to analyse data in real-time.

Pathway seeks to fix this, providing a streaming data processing layer that transforms complex, raw event data streams in real-time, allowing enterprises to derive insights in real-time. Reducing latency by over 900x compared to traditional means of data cleansing and transformation, it is currently used in logistics operations by companies such as DB Schenker and La Poste, but has applications in financial fraud detection, marketing and IoT usage tracking, among many others.

Founded by women, the company is something of a rarity in the male-dominated European deep tech space, and the Pathway's founders are something of a role model for female engineers. This has only strengthened the founders' resolve to succeed, and with an innovative product meeting a pressing business need, Pathway is poised to achieve this.

Steph

Company Specialist at Welcome to the Jungle

Company

Funding (2 rounds)

Nov 2024

$10m

SEED

Dec 2022

$4.5m

SEED

Total funding: $14.5m

Company benefits

Intellectually stimulating work environment. Be a pioneer: you get to work with a new type of data processing framework
Work in one of the hottest data/AI startups in France, with exciting career prospects
Responsibilities and ability to make significant contribution to the company’ success
Inclusive workplace culture

Company HQ

Downtown North, Palo Alto, CA

Leadership

Zuzanna Stamirowska

(CEO)

Previously Co-Chair of the Logistics and Supply Chain Working Group for AIOTI, a spin-off of the European Commission. Before that, they were a PhD Researcher at Université Panthéon-Sorbonne and a Senior Sales Manager for Sphere Research Labs.

Claire Nouet

(Co-Founder & COO)

Previously an Analyst at Veolia, a Parliamentary Assistant for the French National Assembly, and a Legal Assistant with Thales.

Salary benchmarks

We don't have enough data yet to provide salary benchmarks for this role.

Submit your salary to help other candidates with crowdsourced salary estimates.

Share this job

View 7 more jobs at Pathway