LLM Development Analyst, Leadtech

• Proven experience building applications with Large Language Models (LLMs)
• Strong expertise in prompt engineering, RAG architectures, and agent frameworks (e.g., LangChain, LlamaIndex)
• Experience with Databricks/Snowflake/GCP, including PySpark and SQL
• Strong proficiency in Python
• Solid understanding of analytics use cases such as A/B testing, marketing performance, or data workflows
• Experience implementing LLM evaluation, monitoring, and guardrails
• Strong problem-solving skills with the ability to translate business needs into technical solutions
• A builder mindset with a passion for applied AI and automation
• Ability to operate in ambiguous, fast-paced environments
• Strong ownership and accountability for end-to-end delivery
• Curiosity to explore and implement cutting-edge AI capabilities
• A pragmatic approach to balancing innovation with reliability

Desirable

• We are looking for a highly skilled LLM Dev Analyst to design, build, and scale intelligent agents within our Databricks ecosystem, redefining how data is accessed, interpreted, and activated across the organization
• In this role, you will operate at the intersection of AI engineering, analytics, and data platforms, developing domain-specific agents that automate and augment decision-making across key business areas including A/B testing, marketing performance, data engineering workflows, and MarTech processes
• You will translate complex business challenges into agent-driven solutions powered by Large Language Models (LLMs), leveraging both structured and unstructured data to deliver real-time insights and automation at scale
• As a core contributor to our next-generation data stack, you will define and implement best practices for agent architecture, prompt engineering, evaluation frameworks, and orchestration within Databricks, ensuring production-grade reliability and business impact
• 1. Agent Design & Development
• Design, build, and deploy LLM-powered agents for multiple business domains (A/B testing, marketing analytics, data engineering automation, MarTech workflows)
• Develop multi-step reasoning agents that integrate with internal data systems, APIs, and tools
• Implement RAG architectures to enable agents to leverage enterprise data effectively
• 2. Databricks & Data Integration
• Integrate LLM agents within the Databricks lakehouse architecture
• Build scalable pipelines using PySpark, SQL, and Databricks workflows
• Enable seamless interaction between agents and data warehouses, event streams, and APIs
• 3. Prompt Engineering & Evaluation
• Design and optimize prompts, tools, and agent workflows for accuracy and performance
• Develop evaluation frameworks to measure agent quality, reliability, and business impact
• Implement strategies to reduce hallucinations and improve response consistency
• 4. Automation & Use Case Delivery
• Build agents that automate:
• Experimentation analysis (A/B testing insights)
• Marketing performance reporting and optimization
• Data Engineering workflows and monitoring
• MarTech processes and campaign operations
• Deliver solutions that drive measurable efficiency gains and decision velocity
• 5. Productionization & Reliability
• Productionize agent systems with monitoring, logging, and observability
• Implement guardrails, security controls, and governance frameworks
• Ensure scalability, latency optimization, and cost efficiency
• 6. Cross-Functional Collaboration
• Partner with Data Engineering, Analytics, Marketing, and Product teams
• Translate business requirements into scalable AI solutions
• Communicate insights and capabilities to both technical and non-technical stakeholders