Staff Software Engineer, Grafana Labs

• You may not meet every requirement, and that’s okay. If this role excites you, we’d love you to raise your hand for what could be a truly career-defining opportunity
• You have a passion for observability and like to share your knowledge by writing documentation and blog posts
• You love to engage with customers and help them out
• You have excellent communication skills
• You have relevant open source experience, ideally in the observability domain
• You are willing to become an active member of the OpenTelemetry and Prometheus communities
• You’re curious and you enjoy learning new programming languages and frameworks, setting up examples, and figuring out how things work
• You have a good understanding of typical production environments. Ideally you have been responsible for operating production services and organizing on-call
• You actively mentor other team members, identifying areas for focus and improvement
• Strong 8+ years of experience with at least one programming language - any major language (Python, .NET, Java, Go, Rust, etc) is acceptable
• Demonstrated working experience in operating high-scale production systems running on Kubernetes and monitoring it, including on-call participation, incident response, and postmortem practices
• Familiarity with observability tooling (e.g. Grafana)
• Strong understanding of time-series data, metrics cardinality challenges, and cost/performance tradeoffs/optimizations in observability systems
• Experience in a hands-on technical leadership role - setting technical direction, leading project teams, and influencing architectural decisions beyond your immediate team
• Deep understanding of distributed systems concepts including scalability, consistency, high availability, and failure modes in large-scale systems
• Experience writing clean, maintainable, robust, and performant software
• Experience with delivering projects from start to finish in a self-driven manner
• Excellent problem-solving and debugging skills
• Strong mentoring and leadership skills

Desirable

• Grafana Cloud is our composable observability platform that integrates metrics, logs, and traces with Grafana
• It allows our customers to leverage the best open source observability software – including Prometheus, Mimir, Loki, and Tempo – without the overhead of installing, maintaining and scaling their own observability stack
• The Observability department is focused on enabling developers to understand the health and performance of their applications and infrastructure in any environment by providing tools to instrument their code, ingest observability data into Grafana Cloud and visualize and explore it
• In this role, you will be part of the team that builds our Cloud Observability stack that allows customers to collect and visualize metrics from various systems and applications
• We build and maintain the backend of opinionated applications such as Cloud Provider Observability, Database Observability, and Kubernetes Monitoring
• This includes the dashboards, alerts, documentation, and infrastructure while working closely with other teams to ensure seamless experiences
• We also strive to incorporate OSS contributions in our work by contributing to projects such as Alloy, Prometheus, OpenTelemetry, and Beyla
• The Observability department provides a core building block for customers using Grafana Cloud
• As a company we are remote-first and global, we embrace people of different experiences and backgrounds to build diverse teams where every person brings a unique perspective to the software
• We are looking for Engineers that are passionate about communicating with data and providing seamless experiences for our customers to join our growing team
• Engineers at Grafana also have the opportunity to contribute to Open Source communities
• At Grafana Labs, our engineers have a dedicated career path and do not have to become managers to progress in their career
• Staff Software Engineers at Grafana have a large amount of experience across multiple areas
• They are able to estimate, plan, coordinate and deliver large tasks spanning multiple systems
• They actively coach and mentor other team members in their team and are able to identify and resolve issues with technology and product processes
• You will bring your passion for observability and software engineering expertise to help us take our infrastructure monitoring capabilities within Grafana Cloud to the next level
• This will include working with our Kubernetes monitoring solution
• Design and implement high-quality, scalable integrations for various infrastructure components, applications, and data ingestion pipelines
• Create middleware components and libraries that simplify development and maintenance of observability solutions
• When necessary, represent Grafana Labs in open source forums, working groups, and events
• Work with product teams, in addition to design and docs, to develop features that align with wider product strategy and customer needs
• Lead the technical direction and vision of the team, contributing to strategic discussions and future development of observability solutions
• Work with other departments including Sales, Product, and Support teams to deliver a holistic product experience
• Take ownership of the services you’re running by deploying well tested clean code
• Embrace our open-source culture and contribute to other projects that may not directly fall within your team’s scope