Staff Data Engineer, DISQO

• 8+ years of experience building, architecting, and supporting complex production data pipelines, distributed systems, and backend infrastructure
• Expert-Level Scala & Spark: Deep, hands-on expertise in Scala and Apache Spark. You must understand Spark internals, query plans, memory management, and advanced performance tuning for massive-scale batch processing
• Applied Generative AI Experience: Proven experience integrating Gen AI / LLMs (e.g., OpenAI APIs, Anthropic, Bedrock) into data products or data engineering workflows. Hands on experience developing with AI dev tools such as Claude code, etc
• Strong Python Skills: Proficiency in Python specifically to interface with modern AI ecosystems, data APIs, and orchestration tools
• Cloud Mastery: Extensive architectural experience within the AWS ecosystem (EMR, Glue, Athena, S3, Bedrock, etc.)
• Core Data Foundations: Deep understanding of advanced ETL/ELT concepts, complex data modeling, and performance-tuning SQL
• Orchestration: Expert-level experience with workflow orchestration tools such as Airflow
• Leadership: Proven track record of leading technical initiatives, making architectural decisions, and mentoring teams in an agile, fast-moving environment

Desirable

• This is a great opportunity to join a fun, highly motivated team and lead the development of intelligent data products that directly power how brands measure advertising effectiveness
• At DISQO, we use modern cloud infrastructure, Generative AI, and expert-level data engineering to solve complex, real-world problems at scale
• We are looking for a visionary technical leader who is a master of distributed data processing (Scala/Spark) and passionate about the intersection of data engineering and Artificial Intelligence
• You’ll serve as a force multiplier, working closely with engineering leadership, product managers, and analysts in a collaborative environment where rapid innovation and systemic impact matter
• We believe the best software is built by highly aligned, autonomous teams that take ownership and move quickly
• We use agile development practices, modern tooling, and strong engineering discipline to deliver early and often
• We care deeply about architectural excellence, data correctness, system reliability, and building intelligent systems the right way
• As a Staff Data Engineer, you will set the technical direction for DISQO’s ad measurement platform
• You will architect, build, and scale our most complex data pipelines while spearheading the integration of Generative AI capabilities directly into our core data infrastructure and products
• You will tackle our hardest scalability challenges, utilizing expert-level Spark and Scala to process massive datasets, while leveraging LLMs to unlock new value from unstructured and structured data
• Operating with a high degree of autonomy, you will lead cross-functional technical initiatives, drive architectural decisions, and pioneer how we use AI to enrich data, automate pipelines, and improve data quality
• You will mentor senior and mid-level engineers, raising the technical bar for the entire team while expanding DISQO's technical depth across big data systems, cloud infrastructure, and applied AI
• Architect and Lead: Design, build, and maintain highly scalable, fault-tolerant data pipelines using expert-level Scala and Apache Spark
• Gen AI Integration: Pioneer the use of Generative AI within our data ecosystem—incorporating LLMs to enrich datasets, extract value from unstructured data, automate metadata generation, and build intelligent data products
• Cross-Functional Strategy: Partner with Product and Engineering leadership to translate complex business requirements into forward-looking data and AI-augmented architectures
• Optimize Systems: Architect and aggressively optimize large-scale ETL/ELT workflows. Dive deep into Spark internals to resolve complex performance bottlenecks, memory issues, and data skew
• Modern AI Tooling: Implement and manage infrastructure to support AI integration, including vector databases, embeddings pipelines, and Retrieval-Augmented Generation (RAG) architectures
• Set the Standard: Write clean, highly optimized, and maintainable code, while establishing standards for code quality, testing, and system architecture across the organization
• Ensure Operational Excellence: Champion data quality, observability, and system health to consistently meet enterprise SLAs and customer commitments
• Mentorship: Actively mentor engineers, lead technical design reviews, and foster a culture of continuous learning and technical rigor