Senior Data Engineer, Affinity

• 5+ years of experience as a Data Engineer or Data Platform Engineer, working on complex, sometimes ambiguous engineering projects across team boundaries
• Proficiency in data modeling, data warehousing, and ETL pipeline development is essential
• Proven hands-on experience building scalable data platforms and reliable data pipelines using Spark and Databricks, and familiarity with Hadoop, AWS SQS, AWS Kinesis, Kafka, or similar technologies
• Comfortable working with large datasets and high-scale data ingestion, transformation, and distributed processing tools such as Apache Spark (Scala or Python)
• Strong proficiency in SQL
• Familiar with industry-standard databases and analytics technologies, including Data Warehousing and Data Lakes
• Experience with cloud platforms such as AWS, Databricks, GCP, Azure or related technologies
• Familiar with CI/CD processes and test frameworks
• Comfortable partnering with product and machine learning teams on large, strategic data projects

Desirable

• Affinity stitches together billions of data points from massive datasets to create a powerful, accurate representation of the world's professional relationship graph
• Based on this data, we offer our users the insights and visibility they need to nurture and tap into their team's network of opportunities
• This role is part of the AI Insights team, which owns the services that power Affinity's industry-leading relationship intelligence platform
• Our team extracts and retrieves information from billions of structured and unstructured data points to deliver insights to our customers
• As a Senior Data Engineer, you will collaborate with machine learning engineers, software engineers, and product managers to shape the future of private capital's leading CRM platform
• This involves designing and building scalable, efficient data extraction, load, and transform (ELT) solutions, monitoring and managing data quality, and ensuring data security and best practices
• Design scalable and reliable data pipelines to consume, integrate and analyze large volumes of complex data from different sources, supporting the evolving needs of our business
• Help define our data roadmap. You'll collaborate with our team of machine learning engineers, software engineers, product, and business leaders to use data to shape product development
• Build and maintain frameworks for measuring and monitoring data quality and integrity
• Establish and optimize CI/CD processes, test frameworks, and infrastructure-as-code tooling
• Build and implement robust data solutions using Spark, Python, Databricks, Kafka, and the AWS ecosystem (including S3, Redshift, EMR, Athena, Glue)
• Identify skill and process gaps within the team, and develop processes to drive team effectiveness and success
• Articulate the trade-offs of different approaches to building ETL pipelines and storage solutions, providing clear recommendations aligned with product and business requirements