Principal Site Reliability & Forward Deployed Engineer, Abacus Insights

• 10+ years of experience in software engineering, SRE, sustaining engineering, or production operations
• Deep hands-on experience operating production systems in AWS
• Strong experience troubleshooting Databricks and large-scale data platforms
• Proficiency in Python and experience building production services or tooling
• Strong understanding of:
• Distributed systems
• Incident management and RCA practices
• Monitoring, alerting, and observability
• CI/CD Pipelines that leverage Infrastructure as Code
• Proven ability to own problems end-to-end, from detection to permanent resolution
• Excellent communication skills, especially during incidents and customer escalations
• Ability to work backward from customer impact to root cause across systems and codebases, delivering fixes in environments with minimal documentation
• Strong instinct for operational risk, with the ability to proactively identify failure modes and harden systems before they impact customers

Desirable

• The Principal Sustaining and Forward Deployed Engineer is a senior individual contributor responsible for production operations, incident response, and post‑launch system reliability across Abacus Insights’ platform
• This role blends:
• SRE and production operations
• SWAT-style deep technical problem solving
• Forward‑deployed, customer‑facing technical work
• Hands‑on software development and automation
• You will own the most complex, ambiguous, and high‑impact production issues, particularly those involving AWS infrastructure, Databricks workloads, and large‑scale data pipelines. You will work directly with customers on escalations and deployments, while ensuring learnings are translated back into durable product and platform improvements
• This is a hands-on role. You are expected to lead technically during incidents, debug live systems, and write production code where it meaningfully improves reliability and operability
• Act as a senior technical escalation point during production incidents
• Lead real-time incident triage, mitigation, and recovery efforts
• Drive root cause analysis (RCA) with a focus on systemic, long-term fixes
• Identify recurring failure patterns and push for architectural or operational improvements
• Partner with Customer Success and Engineering to manage customer impact during incidents
• Own post-launch reliability, stability, and operational quality of core systems
• Investigate and resolve complex field issues and production defects
• Ensure fixes developed during incidents or customer escalations are up streamed into the core product
• Improve operational readiness of services through better runbooks, monitoring, and alerting
• Reduce operational toil by converting repeated manual work into automation
• Engage directly with strategic customers to solve real-world, production-grade technical challenges
• Support complex deployments, integrations, and escalations in customer environments
• Act as a trusted technical partner to customers during high-impact issues
• Translate customer learnings into concrete product, platform, and operational improvements
• Contribute to reusable tools, playbooks, and best practices that accelerate future deployments
• Serve as a subject matter expert for AWS-hosted production systems
• Troubleshoot and resolve issues across:
• AWS compute, storage, networking, IAM, and security
• Databricks jobs, clusters, and Spark-based data pipelines
• Debug performance degradation, scalability issues, job failures, and data correctness problems
• Partner with platform and data teams to harden systems for reliability, scale, and operability
• Write production-quality code to:
• Automate operational workflows
• Improve reliability and observability
• Eliminate manual intervention and reduce incident frequency
• Contribute primarily in Python, with exposure to JVM-based systems as needed
• Review code with a strong emphasis on operability, resiliency, and maintainability
• Advocate for “build it so it can be operated” engineering standards
• Provide technical leadership without formal authority, influencing design and operational decisions
• Mentor engineers through pairing, reviews, and incident leadership
• Collaborate closely with Product, Engineering, Data, and Customer teams
• Operate effectively in high-pressure, ambiguous environments, especially during customer-impacting incidents