Staff Data Engineer, Blackbird AI

• 8+ years of software engineering experience with 5+ years focused on data platforms or data engineering
• Deep expertise with Databricks, Apache Spark, and data lakehouse architectures
• Strong experience building and operating data pipelines at scale (handling TBs+ of data)
• Experience integrating AI/ML capabilities into data pipelines (clustering, LLM APIs, classification, summarization)
• Proficiency in Python, DBT, and SQL for data processing and pipeline development
• Experience with both batch and streaming large scale data processing patterns
• Strong understanding of cloud platforms (AWS, Azure)
• Excellent communication skills and ability to mentor engineers
• Technical Excellence: You write clean, maintainable code and make thoughtful architectural decisions
• Pragmatism: You balance perfection with shipping and know when to optimize vs. when "good enough" is sufficient
• Ownership: You take end-to-end responsibility for your systems and their reliability
• Collaboration: You elevate those around you and thrive in a team environment
• Impact Orientation: You focus on outcomes and business value, not just technical elegance
• Learning Mindset: You stay current with evolving technologies and continuously improve your craft
• We've outlined specific skills, experience, and requirements for this position, but don't stress if you don't meet every single one
• Our Talent Team is dedicated to discovering exceptional individuals, and they might identify a relevant aspect of your background that suits this role or another opportunity within Blackbird.AI
• If you have passion for the role, please still apply

Desirable

• As a Staff Data Engineer, you will play a critical role in architecting and scaling our data platform and AI/ML processing infrastructure
• You'll be a technical leader responsible for our entire data ecosystem—from ingestion pipelines that process diverse data sources to the lakehouse architecture that powers our narrative analysis capabilities
• You'll architect systems that seamlessly support batch and streaming data patterns while building real time alerting on generated insights
• You'll work at the intersection of data engineering, AI-powered data transformation, and platform engineering, making architectural decisions that will shape our ability to detect misinformation, disinformation, and narrative attacks at scale while managing costs effectively
• A key aspect of this role involves building intelligent pipelines that use traditional AI and generative AI to cluster, enrich, classify, and extract insights from data as it flows through our system
• Design and implement scalable data platform architecture on Databricks, supporting both batch and streaming ingestion
• Build robust, fault-tolerant data ingestion pipelines that integrate with multiple third-party APIs and data providers
• Design and implement AI-powered enrichment stages within pipelines—applying ML clustering, generative AI summarization, classification, and entity extraction to transform raw data into actionable intelligence
• Build analytical systems with full-text search capabilities using Elasticsearch for rapid querying and analysis of enriched data
• Work with AI/ML researchers to implement, integrate and scaling AI processing
• Expose data platform capabilities as APIs and other interfaces for downstream consumption by applications and services
• Optimize data lake and lakehouse architecture for performance, cost-efficiency, and scalability
• Design and implement data quality frameworks, monitoring, and alerting systems
• Design efficient architectures for calling external AI APIs and managing rate limits, costs, and reliability
• Architect solutions with cost-efficiency as a first-class concern, implementing monitoring and optimization strategies for compute and storage
• Make critical build-vs-buy decisions and establish architectural standards for the data organization
• Mentor engineers and elevate the team's technical capabilities through code reviews, design discussions, and knowledge sharing