Staff Software Engineer, ASAPP

• 12+ years of experience in general software development and/or dev-ops, sre roles in AWS
• 5+ years experience in data engineering, data systems, pipeline and stream processing
• Expertise in at least one flavor of SQL, e.g. Redshift, Postgres, MySQL, Presto/Trino, Spark SQL, Hive
• Proficiency in a high-level programming language(s). We use Python, Scala, Java, Kotlin, and Go
• Experience with CI/CD (continuous integration and deployment)
• Experience with workflow management systems such as Airflow, Oozie, Luigi, and Azkaban
• Experience implementing data governance, i.e. access management policies, data retention, IAM, etc
• Confidence operating in a devops-like capacity working with AWS, Kubernetes, Jenkins, Terraform, etc. thinking about automation, alerting, monitoring, and security and other declarative infrastructure

Desirable

• The Data Engineering team at ASAPP designs, builds and maintains our mission-critical core data infrastructure and analytics platform.
• Accurate, easy-to-access, and secure data is critical to our natural language processing (NLP) customer interaction platform which interacts with tens of millions of end-users in real-time
• We’re looking to hire a Staff Data Engineer with the knack for building out data infrastructure systems that can handle our ever-growing volumes of data and the demands we want to make of it.
• Automation is a key part of our workflow, so you’ll help design and build highly-available data processing pipelines that self-monitor and report anomalies.
• You’ll need to be an expert in ETL processes and know the in’s and out’s of various data stores that serve data rapidly and securely to all internal and external stakeholders.
• As part of our fast-growing data engineering team, this role will also play an integral role in shaping the future of data infrastructure as it applies to improving our existing metric-driven development and machine learning capabilities
• Applicants with all or some relevant combination of the requirements listed below are encouraged to apply. We are able to consider remote and hybrid candidates for this role
• Design and deploy improvements to our mission-critical production data pipeline, data warehouses, data systems
• Recognize data flow patterns and generalizations to automate as much as possible to drive productivity gains
• Expand our logging and monitoring processes to discover and resolve anomalies and issues before they become problems
• Develop state-of-the-art automation and data solutions in Python, Spark and Flink
• Maintain, Manage, Monitor our infrastructure related including Kafka, Kubernetes, Spark, Flink, Jenkins, general OLAP and RDBMS databases, S3 objects buckets, permissions
• Increase the efficiency, accuracy, and repeatability of our ETL processes
• Know how to make the tradeoffs required to ship without compromising quality