Data Engineer, Vortexa

• Experienced in building and deploying distributed scalable backend data processing pipelines that can go through terabytes of data daily using AWS, K8s, and Airflow
• With solid software engineering fundamentals, fluent in both Java and Python (with Rust good to have)
• Knowledgeable about data lake systems like Athena, and big data storage formats like Parquet, HDF5, ORC, with a focus on data ingestion
• Driven by working in an intellectually engaging environment with the top minds in the industry, where constructive and friendly challenges and debates are encouraged, not avoided
• Excited about working in a start-up environment: not afraid of challenges, excited to bring new ideas to production, and a positive can-do will-do person, not afraid to push the boundaries of your job role
• Passionate about coaching developers, helping them improve their skills and grow their careers
• Deep experience of the full software development life cycle (SDLC), including technical design, coding standards, code review, source control, build, test, deploy, and operations

Desirable

• The Data Production Team is responsible for all of Vortexa’s data. It ranges from mixing raw satellite data from 600,000 vessels with rich but incomplete text data, to generating high-value forecasts such as the vessel destination, cargo onboard, ship-to-ship transfer detection, dark vessels, congestion, future prices, etc
• You’ll be instrumental in designing and building infrastructure and applications to propel the design, deployment, and benchmarking of existing and new pipelines and ML models
• Working with software and data engineers, data scientists and market analysts, you’ll help bridge the gap between scientific experiments and commercial products by ensuring 100% uptime and bulletproof fault-tolerance of every component of the team's data pipelines