Data Engineer, Vortexa

• Experienced in building and deploying distributed scalable backend data processing pipelines that can go through terabytes of data daily using AWS, K8s, and Airflow
• With solid software engineering fundamentals, fluent in both Java and Python (with Rust good to have)
• Knowledgeable about data lake systems like Athena, and big data storage formats like Parquet, HDF5, ORC, with a focus on data ingestion
• Driven by working in an intellectually engaging environment with the top minds in the industry, where constructive and friendly challenges and debates are encouraged, not avoided
• Excited about working in a start-up environment: not afraid of challenges, excited to bring new ideas to production, and a positive can-do will-do person, not afraid to push the boundaries of your job role
• Passionate about coaching developers, helping them improve their skills and grow their careers
• Deep experience of the full software development life cycle (SDLC), including technical design, coding standards, code review, source control, build, test, deploy, and operations

Desirable

• Processing thousands of rich data points per second from many and vastly different external sources, moving terabytes of data while processing it in real-time, running complex prediction and forecasting AI models while coupling their output into a hybrid human-machine data refinement process and presenting the result through a nimble low-latency SaaS solution used by customers around the globe is no small feat of science and engineering
• This processing requires models that can survive the scrutiny of industry experts, data analysts and traders, with the performance, stability, latency and agility a fast-moving startup influencing multi-$m transactions requires
• The Data Production Team is responsible for all of Vortexa’s data. It ranges from mixing raw satellite data from 600,000 vessels with rich but incomplete text data, to generating high-value forecasts such as the vessel destination, cargo onboard, ship-to-ship transfer detection, dark vessels, congestion, future prices, etc
• The team has built a variety of procedural, statistical and machine learning models that enabled us to provide the most accurate and comprehensive view of energy flows. We take pride in applying cutting-edge research to real-world problems in a robust, long-lasting and maintainable way
• The quality of our data is continuously benchmarked and assessed by experienced in-house market and data analysts to ensure the quality of our predictions
• You’ll be instrumental in designing and building infrastructure and applications to propel the design, deployment, and benchmarking of existing and new pipelines and ML models
• Working with software and data engineers, data scientists and market analysts, you’ll help bridge the gap between scientific experiments and commercial products by ensuring 100% uptime and bulletproof fault-tolerance of every component of the team's data pipelines