Senior / Principal Software Engineer, Blackbird AI

• Bachelor's degree in Computer Science or a related field
• Minimum of 5 years of experience in data engineering and building data platforms
• Minimum of 2 years of professional experience in Machine Learning or a closely related field, demonstrating a track record of successfully delivering novel research and/or product offerings
• Demonstrated success in deploying cloud and SaaS model products, with expertise in building optimized processing pipelines for streaming analytics applications
• Proficiency in databases and query optimization (PostgresSQL, ElasticSearch, MongoDB, Redis, Druid), including experience with NoSQL and graph databases
• Experience in horizontally scaling databases
• Expertise in Kafka and Airflow, with a strong understanding of runtime profiling tools
• Experience in big data processing systems similar to Apache Spark, Flink, Beam or similar
• Strong skills in build automation, continuous integration, and deployment (CI/CD) tools
• Expert-level Python coding skills
• Project management skills and the ability to lead development iniatives

Desirable

• As a Senior/Principal Software Engineer reporting to the Head of Engineering, you will not only optimize data pipelines for our advanced real-time streaming cloud-hosted analytics platform but also lead the development of new features and capabilities
• Your role will be pivotal in shaping the future of our platform, offering the chance to solve some of the most challenging data problems in the industry
• You will be the key driver of our collection, analysis, and visualization processes and will play a pivotal role in shaping the future of our platform
• This Engineer will also work with our AI research and development team and be responsible for a broad range of cutting-edge applications in machine learning, addressing critical challenges impacting businesses, governments, and society
• This Engineer’s responsibilities will also encompass writing ETL processes, designing database systems, and developing tools for query and analytic processing, all focusing on real-time streaming applications
• Additionally, you will elevate performance tradeoffs, own the database architecture, and lead build automation, continuous integration, and deployment efforts while ensuring adherence to security requirements
• Design and implement real time distributed data processing systems analyzing public data and detecting emergent threats
• Focus on research and development in information extraction, multilingual NLP, automated summarization, graph network analysis, Computer visions, and LLMs (Large Language Models)
• Oversee the gathering and annotating of large custom datasets for tasks like text or vision classification, unsupervised pre-training, translation, tagging, and related challenges
• Apply your knowledge of machine learning performance evaluation techniques, recognizing failure modes and limitations in the process
• Lead and manage the development and optimization of ETL processes for various data formats from social media, news, and web sources
• Design and implement robust database systems and develop tools for query and analytic processing, focusing on real-time streaming applications
• Conduct performance analysis and empirical studies, making strategic decisions on tradeoffs (e.g., cost vs. throughput/latency)
• Develop, manage, and oversee the database architecture for our real-time streaming cloud-hosted analytics platform
• Spearhead build automation, continuous integration, deployment, and performance optimization efforts, upholding our strict security requirements
• Drive the project management, leading the development of new features and capabilities
• Design test suites and implement inline instrumentation to ensure data correctness
• Design secure and scalable REST APIs