Senior Data Engineer, Spotter

• Bachelor’s degree, preferably in Computer Science or Computer Information Systems
• 7+ years of Data Engineering experience. Ideally also have 2-3 years software engineering
• 4+ years experience with Apache Spark or Apache Flink
• 3+ years of experience running software and services in the cloud
• Proficiency in working with DataFrame APIs (Pandas and Spark) for parallel and single node processing
• Proficiency using advanced languages and techniques with Python, Scala, etc. with modern data optimized file formats such as Parquet and Avro
• Proficiency with SQL on RDBMS and data warehouse solutions like Redshift

Desirable

• Develop and maintain scalable data pipelines, including:
• ETL pipelines, both single and multi-node solutions
• Build data quality assurance steps for new and existing pipelines
• Create derived datasets with augmented properties
• Work on analytics ready datasets to power internal and creator facing tools
• Troubleshoot issues when they arise, working directly with internal data consumers
• Automate pipeline runs with scheduling and orchestration tools
• Work with large scale datasets
• Work with/use various external APIs to enhance data
• Setup database tables for analytics users to consume the data collected by the Data Engineering team
• Work with big data technologies to improve data availability and data quality in the cloud (AWS)
• Mentor more junior members of the team