Staff Software Engineer, Temporal Technologies

• 5 or more years of experience as an "Arranger" and/or "Builder/Enhancer" of highly scalable distributed systems. see HERE for more info re: "Arranger" and/or "Builder/Enhancer"
• Strong computer science fundamentals in distributed systems, including concurrency, consistency models, and failure modes
• Significant experience writing and operating concurrent production systems in Go, Java, or similar languages, at a high-end intermediate to expert level
• Experience writing concurrent code in production with languages like Go or Java or other applicable languages with skill level as "high end of Intermediate" and/or "Advanced" or "Expert" levels. see HERE for more info re: "high end of Intermediate" and/or "Advanced" or "Expert levels"
• Hands-on experience designing, operating, and tuning ClickHouse and/or Elasticsearch, ideally in self-hosted environments (managed services are a strong plus)
• Experience building and running services on AWS. Bonus: Azure and/or GCP experience
• Demonstrated ability to lead large, multi-quarter technical initiatives, especially those involving core data infrastructure and live data migrations

Desirable

• As a Staff Engineer on CDS, you will take a leading role in a ground-up rewrite of the Visibility persistence layer, with the goal of making it dramatically more performant, scalable, and operable
• This includes evaluating and driving deep integrations with modern analytical and search data stores such as ClickHouse and Elasticsearch, with a strong preference for self-hosted operation (while supporting managed offerings where appropriate)
• You will design, build, and operate significant portions of our backend for highly scalable, multi-tenant services, and own large technical initiatives end-to-end — from initial design through live data migration, rollout, and long-term operational stewardship
• Lead the design and implementation of a new persistence layer for Temporal Visibility, informed by its real-world access patterns (high-volume writes, time-based queries, filtering, sorting, and pagination across long-running workflows)
• Evaluate and select the most appropriate storage technologies (e.g., ClickHouse, Elasticsearch, or complementary systems), clearly articulating tradeoffs around indexing models, consistency, cost, latency, and operational complexity
• Design schemas, APIs, and query models that make Visibility both powerful and intuitive for customers
• Plan and execute online migrations of live Visibility data from existing persistence stores to new backends, at scale and without customer downtime
• Design dual-write, backfill, validation, and cutover strategies that prioritize correctness, observability, and rollback safety
• Build tooling and automation to validate data integrity and performance throughout the migration lifecycle
• Define and own SLOs for Visibility storage and query paths
• Profile hot paths, design benchmarks, and lead systematic performance tuning efforts
• Build operational playbooks, dashboards, and alerting that make the system understandable and debuggable for on-call engineers
• Lead incident reviews and reliability improvements related to persistence and indexing systems
• Break down large, ambiguous roadmap initiatives into concrete, executable phases
• Author and steward design docs and RFCs through review with peers and stakeholders
• Mentor and unblock other engineers working in the persistence and storage domain
• Partner closely with Server, Cloud, and Developer Experience teams to land features end-to-end