Senior AI Engineer, Grafana Labs
GenAI & ML Evaluation Frameworks, Grafana Ops, AI/ML
CA$164.5-197.4k
+ Equity
The open and composable observability platform
Job no longer available
The open and composable observability platform
1001+ employees
Job no longer available
CA$164.5-197.4k
+ Equity
1001+ employees
Company mission
To unite data, no matter where it lives, and empower its users to analyze, take action, and make smart decisions.
Role
Who you are
- Experience designing and implementing evaluation frameworks for AI/ML systems
- Familiarity with prompt engineering, structured output evaluation, and context-window management in LLM systems
- High autonomy to collaborate and translate team goals into clear, testable criteria supported by effective tooling
Desirable
- Experience working in environments with rapid iteration and experimental development
- A pragmatic mindset that values reproducibility, developer experience, and thoughtful trade-offs when scaling GenAI systems
- A passion for minimizing human toil and building AI systems that actively support engineers
What the job involves
- At Grafana, we build observability tools that help users understand, respond to, and improve their systems – regardless of scale, complexity, or tech stack
- The Grafana AI teams play a key role in this mission by helping users make sense of complex observability data through AI-driven features
- These capabilities reduce toil, lower the barrier of domain expertise, and surface meaningful signals from noisy environments
- We are looking for an experienced engineer with expertise in evaluating Generative AI systems, particularly Large Language Models (LLMs), to help us build and evolve our internal evaluation frameworks, and/or integrate existing best-of-breed tools
- This role involves designing and scaling automated evaluation pipelines, integrating them into CI/CD workflows, and defining metrics that reflect both product goals and model behavior
- As the team matures, there’s a broad opportunity to expand or redefine this role based on impact and initiative
- Design and implement robust evaluation frameworks for GenAI and LLM-based systems, including golden test sets, regression tracking, LLM-as-judge methods, and structured output verification
- Develop tooling to enable automated, low-friction evaluation of model outputs, prompts, and agent behaviors
- Define and refine metrics for both structure and semantics, ensuring alignment with realistic use cases and operational constraints
- Lead the development of dataset management processes and guide teams across Grafana in best practices for GenAI evaluation
Our take
Observability plays a crucial role in software development, where dashboards are pivotal for monitoring the health of IT infrastructure across an organization. However, for developers, the frustration of spending unnecessary time navigating between different dashboards to find the latest information detracts from valuable programming time.
Grafana collaborates closely with its clients to create customizable observability platforms, striving to enhance efficiency in observability. Its offerings include cloud-based data structures, dashboards, API plugins, and collaboration tools. Currently, Grafana boasts over 25M users globally, with a customer base exceeding 5,000, including prominent names like Bloomberg, PJ Morgan Chase, eBay, PayPal, and Sony.
While Grafana provides free plans with a simplified tool package, the Grafana Enterprise Stack offers customized observability solutions on a monthly subscription basis. The company's ongoing strategy revolves around continually improving Grafana to meet users' demands, enhancing functionality with each iteration, and fostering community engagement by enabling users to share their Grafana dashboards with the broader open-source community.
Kirsty
Company Specialist at Welcome to the Jungle
Insights
49% employee growth in 12 months
Company
Funding (last 2 of 6 rounds)
Aug 2024
$270m
SERIES D
Apr 2022
$240m
SERIES D
Total funding: $805.2m
Company benefits
- Vacation: Balance is key. Our team enjoys 30 days of paid vacation each year on top of national holidays, parental leave, and sick leave. We also take a breather on a number of Grafana Shutdown Days each year
- Healthcare: We’re proud to provide health coverage or stipends for our colleagues in the US, UK, Canada, the Netherlands, Sweden, Singapore, and India
- Retirement planning: There’s no time like the present to start saving for your future. We make employer contributions into the pension pots of our team members in the US, UK, Canada, the Netherlands, Sweden, and Germany
- Professional development: On top of a $1,500 annual learning and development stipend, Grafanistas have thousands of on-demand courses at their fingertips to help them grow professionally. Want to attend a conference or training? Go ahead. Just pass on what you learned
- Work location: Vast majority of our roles are fully remote, focused on hiring the best talent and allowing you to perform from the comfort of your home. If you fancy a change of scene, we’ll also reimburse you up to $175 a month for a personal co-working space
- Choice of tech: There’s no one-size-fits-all when it comes to the tech required to do your job. Choose the laptop and accessories you need when you join us, and we’ll refresh them every three years
- Mindfulness: When you join the team, you can sign up for a complimentary subscription to Headspace to take advantage of the benefits of mindfulness and meditation. Our wellbeing resource group also organize sessions run by fellow Grafanistas or external trainers
- Paid parental leave: Grafana offers paid parental leave to all eligible new parents. This offers Grafanistas time to bond with and care for their children in the first year after birth or adoption
- Global Employee Assistance Program: We offer all team members a 100% confidential support service with 24/7 365 access to professionally qualified counsellors and specialists
Company HQ
Financial District, New York, NY
Leadership
Raj Dutt
(Co-Founder & CEO)Currently a Board Member at NSONE. Previously founded Voxel where they served as the CEO for 12 years. They also served as the Senior Vice President of Technology at Internap Network Services.
Torkel Ödegaard
(Co-Founder)Graduated from Mälardalen University with a Master's in Computer Science. Founder of Coding Instinct. Previously worked as a Consultant at H&M, Avega, and Ebay.
Anthony Wood
(Co-Founder)Previously worked at Visa and Voxel as a Senior Systems Engineer. They also have Systems Engineering experience at PalVision and iiNet.
Share this job
View 34 more jobs at Grafana Labs