Senior Software Engineer, HOLOPLOT

• Several years of professional software engineering experience, most of it spent designing and shipping production systems
• Fluency in software engineering as a craft: concurrency, error handling, testing, and designing APIs that other teams build on
• Working knowledge of the Kubernetes control plane and the operator/controller pattern, including how clusters behave when they are unhealthy
• Hands-on experience with AWS, and a firm belief in infrastructure as code. Terraform and Ansible are what we use today
• Comfort across several languages, and the ability to pick up a new one when the problem calls for it. Our platform work is mostly Go and Python, with Rust and C++ elsewhere in the stack
• Experience building distributed systems, and a clear understanding of how they fail: partitions, partial rollouts, clock skew, retries, and reconciliation
• A good sense for software architecture and maintainability, and the habit of writing code that colleagues can pick up and change
• A strong command of Linux, scripting, and automation
• Experience instrumenting software for telemetry and using it to answer real questions about running systems
• Good judgment about what to automate and what to leave alone
• Strong communication skills in English. You can explain a design or a failure clearly to engineers, product stakeholders, and non-specialists
• Willingness to take part in compensated on-call and weekend work when installations need it
• A can-do mentality, an eye for detail, and the courage to take on problems outside your comfort zone
• Experience in using AI for engineering

Desirable

• We are looking for a Senior Software Engineer who builds the platform our sound systems run on, and who takes responsibility for how it behaves in production
• This is a software engineering role first. You will spend your time writing software, either with your own team or embedded with one of the other HOLOPLOT teams: services, controllers, agents, and tooling that provision, configure, observe, and update our systems
• Our infrastructure is defined as code and evolves through the same review and release process as the rest of our software. You design the operational side, you build it, and you are on call when it misbehaves in the field
• You will join the team that owns the platform underneath our installations and connected services, working alongside backend, embedded, and frontend engineers who are organized in cross-functional product teams
• Architect and implement services to provision, configure, and manage server fleets across AWS and local clusters
• Create control-plane logic for venue systems to ensure consistency, health monitoring, rollout management, and recovery, even during uplink disruptions
• Establish end-to-end provisioning workflows for embedded Linux hardware, spanning from initial manufacturing to customer deployment
• Enhance update processes to streamline software deployments for active customer installations
• Develop APIs and developer tooling that empower engineering teams to independently deploy, observe, and troubleshoot their services
• Embed telemetry capabilities into the platform to facilitate remote diagnosis of venue system behavior
• Own the AWS infrastructure behind our connected services: remote access, access control, telemetry ingestion, remote backup, secret management, and monitoring. All of it is infrastructure as code
• Operate and evolve the on-premises Kubernetes footprint at venues, including air-gapped and intermittently connected installations
• Monitor large environments, investigate problems across the software/hardware boundary, and turn each incident into a fix in the code
• Be part of a compensated on-call rotation
• Advise and collaborate with product teams on building distributed systems that are scalable, secure, and observable, and review their designs with production reality in mind
• Research and introduce tools and technologies where they measurably improve how we build and operate
• Contribute back to the open-source projects we depend on, through issues and fixes