Data Engineering Lead, YipitData

• 6+ years of professional experience in Data Engineering or equivalent technical roles (e.g., data architecture, big data development, or ETL engineering)
• 2+ years of managerial experience, including mentoring, team leadership, and supporting delivery
• Strong expertise in SQL and distributed data systems
• Proficiency with PySpark and Databricks for processing and scaling large datasets
• Hands-on experience with Airflow for pipeline orchestration (Dagster/dbt a plus)
• Proven track record of delivering in fast-paced, deadline-driven environments with minimal oversight
• Strong problem-solving skills and ability to translate business needs into scalable technical solutions
• Excellent communication and collaboration skills with both technical and non-technical stakeholders

Desirable

• We are looking for a highly skilled Data Engineering Manager to join our Data Engineering team and play a central role in building and scaling our vendor universe—the database of companies we track and analyze
• Reporting directly to the Senior Director of Data Engineering, this role demands technical leadership, independence, and the ability to deliver under tight deadlines in a fast-paced environment
• As a Data Engineering Manager, you will design and maintain large-scale data pipelines, define best practices for ETL, and integrate emerging technologies to keep our platform on the cutting edge
• You’ll work cross-functionally to deliver reliable, efficient, and scalable data solutions while contributing hands-on expertise in Databricks, Airflow, PySpark, and SQL
• Own the design, build, and optimization of end-to-end data pipelines that power our vendor universe
• Establish and enforce best practices in data modeling, orchestration, and system reliability
• Collaborate with product, engineering, and business stakeholders to translate requirements into robust, scalable data solutions
• Work extensively with Databricks and Airflow for large-scale data processing and orchestration
• Troubleshoot and resolve complex pipeline issues to ensure reliability and performance
• Contribute to the team’s technical strategy, helping drive improvements in scalability, performance, and efficiency
• Lead, mentor, and support engineers through challenges, code reviews, and project execution
• Working with product team to bring an idea to reality
• Leveraging cutting edge technologies to solve complex data problems
• Helping to lead a team through different data models and architectural decisions