Staff Software Engineer, Grafana Labs

• You have a passion for observability and like to share your knowledge by writing documentation and blog posts
• You love to engage with customers and help them out
• You have excellent communication skills
• You have relevant open source experience, ideally in the observability domain
• You are willing to become an active member of the OpenTelemetry and Prometheus communities
• You’re curious and you enjoy learning new programming languages and frameworks, setting up examples, and figuring out how things work
• You have a good understanding of typical production environments. Ideally you have been responsible for operating production services and organizing on-call
• You actively mentor other team members, identifying areas for focus and improvement
• Strong 8+ years of experience with at least one programming language - any major language (Python, .NET, Java, Go, Rust, etc) is acceptable
• Demonstrated working experience in operating high-scale production systems running on Kubernetes and monitoring it, including on-call participation, incident response, and postmortem practices
• Familiarity with observability tooling (e.g. Grafana)
• Strong understanding of time-series data, metrics cardinality challenges, and cost/performance tradeoffs/optimizations in observability systems
• Experience in a hands-on technical leadership role - setting technical direction, leading project teams, and influencing architectural decisions beyond your immediate team
• Deep understanding of distributed systems concepts including scalability, consistency, high availability, and failure modes in large-scale systems
• Experience writing clean, maintainable, robust, and performant software
• Experience with delivering projects from start to finish in a self-driven manner
• Excellent problem-solving and debugging skills
• Strong mentoring and leadership skills

Desirable

• In this role, you will be part of the team that builds our Cloud Observability stack that allows customers to collect and visualize metrics from various systems and applications
• We build and maintain the backend of opinionated applications such as Cloud Provider Observability, Database Observability, and Kubernetes Monitoring
• This includes the dashboards, alerts, documentation, and infrastructure while working closely with other teams to ensure seamless experiences
• We also strive to incorporate OSS contributions in our work by contributing to projects such as Alloy, Prometheus, OpenTelemetry, and Beyla
• The Observability department provides a core building block for customers using Grafana Cloud
• At Grafana Labs, our engineers have a dedicated career path and do not have to become managers to progress in their career
• Staff Software Engineers at Grafana have a large amount of experience across multiple areas
• They are able to estimate, plan, coordinate and deliver large tasks spanning multiple systems
• They actively coach and mentor other team members in their team and are able to identify and resolve issues with technology and product processes
• You will bring your passion for observability and software engineering expertise to help us take our infrastructure monitoring capabilities within Grafana Cloud to the next level
• This will include working with our Kubernetes monitoring solution
• Design and implement high-quality, scalable integrations for various infrastructure components, applications, and data ingestion pipelines
• Create middleware components and libraries that simplify development and maintenance of observability solutions
• When necessary, represent Grafana Labs in open source forums, working groups, and events
• Work with product teams, in addition to design and docs, to develop features that align with wider product strategy and customer needs
• Lead the technical direction and vision of the team, contributing to strategic discussions and future development of observability solutions
• Work with other departments including Sales, Product, and Support teams to deliver a holistic product experience
• Take ownership of the services you’re running by deploying well tested clean code
• Embrace our open-source culture and contribute to other projects that may not directly fall within your team’s scope