Artificial Intelligence Engineer, Fieldguide
Quality
$170-220k
Software for assurance & advisory firms
Job no longer available
Software for assurance & advisory firms
101-200 employees
Job no longer available
$170-220k
101-200 employees
Company mission
Fieldguide's mission is to increase trust in commerce and capital markets.
Role
Who you are
- You are an engineer who believes that evaluations are foundational to building reliable AI systems, not a nice-to-have
- The following operating principles should resonate with you:
- Evaluation-first mindset: You understand that for an AI company, not being able to evaluate a new model quickly is unacceptable
- AI-native instincts: You treat LLMs, agents, and automation as fundamental building blocks and parts of the craft of engineering
- Data-driven rigor: You make decisions based on metrics and are obsessed with measuring what matters
- Production-oriented: You understand that evaluations must work on real production behavior, not just offline datasets
- Strong product judgment: You can decide what matters and why, without waiting for guidance, not just how to implement it
- Bias to building: You move fast and build working systems rather than perfect specifications
- Multiple years of experience shipping production software in complex, real-world systems
- Experience with TypeScript, React, Python, and Postgres
- Built and deployed LLM-powered features serving production traffic
- Implemented evaluation frameworks for model outputs and agent behaviors
- Designed observability or tracing infrastructure for AI/ML systems
- Worked with vector databases, embedding models, and RAG architectures
- Experience with evaluation platforms (LangSmith, Langfuse, or similar)
- Comfort operating in ambiguity and taking responsibility for outcomes
- Deep empathy for professional-grade, mission-critical software (experience with audit and accounting workflows are not required)
What the job involves
- As an AI Engineer, Quality, you will own the evaluation infrastructure that ensures our AI agents perform reliably at enterprise scale
- This role is 100% focused on making evaluations a first-class engineering capability: building the unified platform, automated pipelines, and production feedback loops that let us evaluate any new model against all critical workflows within hours
- You'll work at the intersection of ML engineering, observability, and quality assurance to ensure our agents meet the rigorous standards our customers demand
- Design and build a unified evaluation platform that serves as the single source of truth for all of our agentic systems and audit workflows
- Build observability systems that surface agent behavior, trace execution, and failure modes in production, and feedback loops that turn production failures into first-class evaluation cases
- Own the evaluation infrastructure stack including integration with LangSmith and LangGraph
- Translate customer problems into concrete agent behaviors and workflows
- Integrate and orchestrate LLMs, tools, retrieval systems, and logic into cohesive, reliable agent experiences
- Build automated pipelines that evaluate new models against all critical workflows within hours of release
- Design evaluation harnesses for our most complex Agentic systems and workflows
- Implement comparison frameworks that measure effectiveness, consistency, latency, and cost across model versions
- Design guardrails and monitoring systems that catch quality regressions before they reach customers
- Use AI as core leverage in how you design, build, test, and iterate
- Prototype quickly to resolve uncertainty, then harden systems for enterprise-grade reliability
- Build evaluations, feedback mechanisms, and guardrails so agents improve over time
- Work with SMEs and ML Engineers to create evaluation datasets by curating production traces
- Design prompts, retrieval pipelines, and agent orchestration systems that perform reliably at scale
- Define and document evaluation standards, best practices, and processes for the engineering organization
- Advocate for evaluation-driven development and make it easy for the team to write and run evals
- Partner with product and ML engineers to integrate evaluation requirements into agent development from day one
- Take full ownership of large product areas rather than executing on narrow tasks
Our take
Fieldguide offers a cloud-native collaboration and automation platform for accounting and advisory practices. The idea is to make workflow and collaboration more streamlined, and automate parts of it altogether, for teams working on cybersecurity, risk, and ESG audits.
Fieldguide has launched with a focus on three high-demand and high-growth areas of business. With cybersecurity and ESG in particular seeing increased corporate investment, thanks to the steep costs of error or regulatory non-compliance, demand for tech that can help manage this influx of auditing work is likely to grow fast. Indeed, Fieldguide received an impressive funding injection in 2024 that will help expand the kinds of audits its product applies to, as well as grow its team.
Fieldguide has already netted some key audit and advisory firms as clients, suggesting that the young company is onto something good. Internal productivity tools are booming across multiple industries, with automation also seeing a significant uptake for enterprise businesses. If Fieldguide can successfully manage its scale-up, it could become an important software player.
Kirsty
Company Specialist at Welcome to the Jungle
Insights
55% employee growth in 12 months
Company
Funding (last 2 of 5 rounds)
Feb 2026
$75m
SERIES C
Mar 2024
$30m
SERIES B
Total funding: $122.4m
Company benefits
- Health
- Dental
- PTO
Company HQ
Eureka Valley, San Francisco, CA
Leadership
Chris Szymansky
(Co-Founder & CTO)Previously co-founded live video streaming platform Vivo, before moving on to work in software development at Fireman Creative. Subsequently held various high level positions, including CTO at JazzHR, and VP of Engineering, Product, and Design at Atrium.
Jin Chang
(Co-Founder & CEO)Originally a Management Consultant at EY, before co-founding blockchain aggregation API company, Bitbutter. Moved on to serve as Chief of Staff at Atrium.
Salary benchmarks
We don't have enough data yet to provide salary benchmarks for this role.
Submit your salary to help other candidates with crowdsourced salary estimates.
Share this job
View 29 more jobs at Fieldguide