Senior Data Engineer, Rad AI

• 4+ years relevant experience in data engineering
• Expertise in designing and developing distributed data pipelines using big data technologies on large scale data sets
• Deep and hands-on experience designing, planning, productionizing, maintaining and documenting reliable and scalable data infrastructure and data products in complex environments
• Solid experience with big data processing and analytics on AWS, using services such as Amazon EMR and AWS Batch
• Experience in large scale data processing technologies such as Spark
• Expertise in orchestrating workflows using tools like Metaflow
• Experience with various database technologies including SQL, NoSQL databases (e.g., AWS DynamoDB, ElasticSearch, Postgresql)
• Hands-on experience with containerization technologies, such as Docker and Kubernetes
• Prior Software Engineering experience is a big plus

Desirable

• We are seeking a Senior Data Engineer to join our engineering team
• Given our large client growth and projected movement in the year ahead, we're seeking to expand this team and add a strong resource on board who will play an instrumental role in the construction and maintenance of data collection systems, pipelines and management tools
• Design and implement the data architecture, ensuring scalability, flexibility, and efficiency using pipeline authoring tools like Metaflow and large-scale data processing technologies like Spark
• Define and extend our internal standards for style, maintenance, and best practices for a high-scale data platform
• Collaborate with researchers and other stakeholders to understand their data needs including model training and production monitoring systems and develop solutions that meet those requirements
• Take ownership of key data engineering projects and work independently to design, develop, and maintain high-quality data solutions
• Ensure data quality, integrity, and security by implementing robust data validation, monitoring, and access controls
• Evaluate and recommend data technologies and tools to improve the efficiency and effectiveness of the data engineering process
• Continuously monitor, maintain, and improve the performance and stability of the data infrastructure