Staff Platform Engineer, DISCO

• 10+ years of relevant experience in backend engineering, with a substantial focus on big data platforms
• 7+ years of professional experience with each of the following: Service Oriented Architecture, Docker-based services, micro-services, API and Data Model Design or Implementation
• Experience with Large-Scale Systems: Involvement in developing, monitoring and maintaining large-scale, high-availability software systems
• Experience designing and developing from a Security Perspective
• Demonstrated experience in leading or playing key roles in significant software projects, showing an ability to work effectively both independently and as part of a team
• Demonstrated expertise in designing, implementing, and maintaining (through operational observability) high-availability, high-performance, distributed data processing systems
• Experience with gRPC and Protocol Buffers for efficient, language-agnostic service-to-service communication
• Proven ability to deliver well-crafted, tested, and maintainable code solutions to complex technical challenges
• Experience with a variety of software stacks including but not limited to, JVM-focused tech stacks, particularly server-side Kotlin and gRPC
• Deep expertise with both relational and non-relational data stores, Elasticsearch, Cassandra, MySQL, and PostgreSQL
• Experience delivering data platform software solutions (including designing, implementing, and operating) in the cloud
• Knowledge of microservices architecture principles, including decentralized data management, service discovery, and bounded contexts. Skilled in microservices technology stack, ability to optimize performance, and ensure security and scalability across the system
• Skilled in containerization technologies such as Docker and orchestration tools like Kubernetes
• Experience in creating, deploying, and managing containers. Skilled in implementing RESTful APIs for outward-facing services and using gRPC for efficient internal service-to-service communication
• Ability to build backend systems that can support future UI/UX initiatives and performance requirements
• Knowledge of system architecture, design patterns, and best practices in software development
• Skilled at designing and operating software in a Cloud Provider, such as AWS, Azure, or GCP
• Knowledge of both relational and non-relational data stores, Elasticsearch, Cassandra, MySQL, PostgreSQL
• Skilled at analytical abilities and problem-solving, capable of addressing complex technical challenges
• Skilled in programming languages and frameworks commonly used in SaaS development. In particular, Java/Kotlin, Python, C#, as well as a deep understanding of software engineering principles
• Familiarity with server management, network configurations, and an understanding of cloud services (like AWS, Azure, or Google Cloud)
• Knowledge of various stages of the SDLC, including design, development, testing, and deployment
• Knowledge of async message or event based dataflow architectures (Kafka, etc)
• Knowledge of design, implementation, and operation of data-intensive, distributed systems

Desirable

• The Staff Software Engineer (Platform) leads large-scale projects and is a key contributor to their group's success.
• They are involved in long-term strategic planning, mentor senior engineers, and are influential in technical decision-making across multiple teams or a department. They focus on implementation and system design, as well as maintaining and evolving DISCO’s distributed data platform
• Technical Leadership and Mentorship: Provides technical leadership in project teams, guiding engineers, and contributing to technical decision-making. Sets technical direction for large projects or teams, driving innovation, and solving organization-wide problems
• Performance Optimization: Focuses on designing systems for high performance, scalability, and reliability
• Ownership: Proactively identifies areas of products and systems that currently have problems or will develop problems as well as makes plans to address those problems
• Stakeholder Engagement: Works with stakeholders, including product managers, designers,developers, and executives, to align architectural designs with business needs and customer expectations
• Software Development: Designs, develops, and maintains sophisticated, scalable, high-performance data software solutions, ensuring they are efficient, scalable, and meet business requirements with architectural excellence for our big data platform at petabyte scale. Responsible for the boundary identification, system design and grooming for one or more sub-systems
• Code Quality and Optimization: Upholds and enhances code quality through rigorous review processes, optimization techniques, and adherence to best practices
• Technical Problem Solving: Addresses intricate technical issues through innovative problem-solving strategies
• Collaboration and Communication: Works effectively within the engineering team and cross-functional teams, collaborating with other engineers, product managers, and stakeholders, contributing to project planning and execution
• Continuous Learning and Adaptation: Stays updated with the latest technology trends and advancements, continually enhancing skills and knowledge, bringing innovative ideas to the team
• Technical Documentation: Creates and maintains comprehensive technical documentation, ensuring clarity and accessibility of information
• Data Platform Development: Builds and maintains scalable and robust data platforms to support analytics, machine learning, and business intelligence
• Data Integration and ETL: Develops and optimizes ETL (Extract, Transform, Load) processes and data integration pipelines. Ensures efficient data flow across various sources and systems
• Advanced File Processing: Develops extensible file processing pipelines that offer flexible, self-service artifact generation. Researches and evaluates file processing software based on fidelity, reliability, and performance criteria. Models file processing outcomes within an observability framework to provide insights for engineering teams and business leaders. Develops engineering systems that facilitate rapid and automatic evaluation of file processors
• Data Management: Crafts and orchestrates scaleout data pipelines that optimize serverless and containerized compute to balance cost, latency, and duration for big-data solutions. Contributes to algorithmically intensive data engines operating on streaming, large, or multi-tenant datasets. Troubleshoots systems, identify root causes, and rectify bugs