Senior Data Platform Engineer, Upstart

• Bachelor’s degree in Computer Science or a related field, or equivalent practical experience
• 5+ years of experience in infrastructure engineering, DevOps, or data platform engineering
• Hands-on experience operating a lakehouse or big data platform (e.g., Databricks, Snowflake, Apache Spark, Delta Lake, Apache Iceberg)
• Hands-on experience with AWS (e.g., IAM, S3, EC2, EKS), Python, infrastructure as code (Terraform or CDK), CI/CD, and Kubernetes

Desirable

• The Data Platform Engineering team (Part of Upstart's Decisioning & Data) provides developer tools, frameworks, and scalable data infrastructure as shared services across Upstart
• The team's primary objective is to provide Data Analysts, ML scientists, and Software Engineers access to high-quality data and developer tools to create business metrics and training data for respective product verticals
• You will join the Data Production Experience (PX) team, which owns Upstart’s Lakehouse (Databricks) infrastructure
• This is the platform that Machine Learning, Analytics, and Software Engineering teams use to build the training data and metrics behind our lending, fraud, and pricing models—enabling growth and reducing risk across our business verticals
• As a Senior Data Platform Engineer joining this team, you will focus on:
• Lakehouse Infrastructure and Security: Own the Databricks platform across multiple environments, including a newly launched regulated environment (Upstart Bank). Responsibilities include expanding infrastructure as code for new Lakehouse features, enforcing a no-click-ops policy, establishing disaster recovery, implementing least-privilege and attribute-based access controls (e.g., based on data classification), ensuring environment isolation, and resolving other known security risks
• Compliance: Close control gaps and automate evidence gathering for OCC, SOX and SOC audits across access management, change management, and data quality
• Cost and usage efficiency: Reduce Databricks and AWS spend through accurate cost attribution, waste detection, and platform-level optimizations (e.g., compute rightsizing, improving pruning, migrating to Graviton and spot instances, Delta Lake Archival)
• Evaluate and adopt new technologies, including open-source lakehouse alternatives, to expand platform capabilities
• Improve developer efficiency and infrastructure reliability by working with cross-functional stakeholders like Data Engineering, Machine Learning, Analytics, Compliance, and Security
• Improve Lakehouse security, usage efficiency, and observability by collaborating closely with the Data Engineering and Security teams
• Evangelize good coding and engineering best practices while uplifting fellow engineers and mentoring junior team members
• Help break down complex projects and requirements into quarterly commitments and sprints
• Own Lakehouse infrastructure components end to end, from technical requirements and design through rollout and production support