Senior Data Engineer, Vantage Data Centers

• Bachelor’s degree in Engineering, Computer Science, Data Analytics, or a related field, or equivalent experience
• Minimum of 5–8 years of experience in data engineering, analytics engineering, or a closely related technical data role
• Proficiency in Python for building and maintaining data pipelines, automation, data processing workflows, and PySpark-based transformations
• Proficiency in SQL for querying, transformation, analytical data processing, model validation, and data quality checks
• Solid understanding of ETL/ELT pipelines, data transformation patterns, data integration concepts, incremental processing, and production support practices
• Experience analyzing enterprise data sources to identify data relationships, transformations, business rules, grain, ownership, and quality constraints
• Experience building solutions on the Microsoft Azure platform with exposure to Azure Data Factory, Azure Synapse, Azure Data Lake Storage Gen2, Microsoft Fabric / Lakehouse patterns, and related analytics services
• Working knowledge of data modeling fundamentals, including fact and dimension tables, semantic models, reusable data products, and analytics-ready structures
• Experience supporting governed data products, including metadata, lineage, issue documentation, access-control awareness, data-quality validation, and operational runbooks
• Experience working with source control and CI/CD workflows using tools such as GitHub or Azure DevOps
• Strong communication and interpersonal skills with the ability to collaborate across IT Global, business SMEs, data governance partners, platform teams, and operations stakeholders in a fast-paced environment
• Experience working in Agile development environments and using collaboration or project tracking tools such as Jira or similar tools
• Travel required is expected to be up to 10% but may increase over time as the business evolves

Desirable

• Within Operational Excellence, the Data Analytics & Intelligence function enables Operations to move from reactive reporting to proactive, insight-driven execution
• The team is responsible for building trusted data foundations, governed KPI frameworks, operational intelligence products, and AI-ready data assets that support performance visibility, decision-making, predictive insights, and scalable operational excellence across North America
• This work directly supports Operational Excellence's mission of embedding delivery rigor, process discipline, and data intelligence into how Operations plans, executes, predicts, and continuously improves
• Vantage Data Centers is seeking a Sr Data Engineer to help build, operate, and scale the governed data foundation for Operations, North America
• This role is designed for an engineer who can independently deliver production-ready pipelines, curated datasets, semantic-model inputs, and AI-ready data products that support reporting, Executive reporting insight preparation, and the emerging AI Insight Solution
• As part of the Data Engineering & Business Intelligence team, you will be responsible for delivering reliable data products that support analytics, AI data agents, operational intelligence, reporting, and an emerging AI-enabled platform
• You will work closely with IT Global, solution build teams, business SMEs, and data governance partners to ensure data products are secure, reusable, explainable, and aligned with enterprise AI / Fabric direction
• Success in this position requires comfort with ambiguity, strong execution discipline, and accountability for building trusted data assets that can be reused across analytics, operational intelligence, and AI-enabled use cases
• Design, build, and maintain reliable, scalable data pipelines using Python and PySpark on the Microsoft Azure data platform
• Develop and operate batch and incremental data pipelines leveraging Azure Data Factory for orchestration and Azure Data Lake Storage Gen2 as the primary data store
• Build and maintain curated lakehouse / gold-layer datasets and semantic-model inputs that support governed operational insights and AI-enabled consumption
• Independently implement SQL- and Spark-based transformations to produce curated datasets that support enterprise reporting, analytics, AI-enabled insight preparation, and downstream consumption
• Take ownership of assigned data pipelines and datasets, including monitoring, troubleshooting, performance optimization, documentation, and production support
• Work with Azure Synapse, Microsoft Fabric / Lakehouse patterns where applicable, and related Azure analytics services to support analytical workloads and data consumption patterns
• Prepare structured operational data for AI-enabled use cases by documenting business rules, source lineage, data reliability constraints, known quality limitations, and data dictionary definitions
• Support source visibility, confidence context, and Data Reliability & Trust Indicator integration where applicable so downstream analytics and AI outputs can be understood and trusted
• Contribute to ontology, taxonomy, semantic model, and data dictionary alignment needed to connect operational context, KPIs, incidents, work orders, and other enterprise data domains
• Collaborate with business analysts, operations SMEs, data stewards, IT Global, and cross-functional stakeholders to translate requirements into practical, working data solutions
• Apply established data governance, security, access-control, data classification, and engineering standards to ensure compliant, maintainable, and scalable solutions
• Identify, document, and route data-quality issues to accountable owners, helping improve source correction rather than masking defects downstream
• Participate in code reviews, technical discussions, sprint planning, and platform improvement initiatives as an active contributor
• Proactively identify data quality issues, pipeline risks, platform dependencies, and improvement opportunities, and communicate them clearly in a fast-paced environment
• Develop and maintain PySpark notebooks and jobs to ingest, transform, validate, and curate data within the enterprise data platform
• Build and modify Azure Data Factory pipelines for batch and incremental data ingestion
• Implement Spark-based transformations that write curated datasets to Azure Data Lake Storage Gen2 and/or Fabric Lakehouse patterns using established folder structures, naming conventions, and governance standards
• Create and maintain SQL views, tables, lakehouse objects, and semantic-model inputs to support analytics, operational intelligence, and AI-enabled consumption patterns
• Prepare datasets for Fabric Data Agent / AI agent use cases by documenting business rules, joins, grain, quality limitations, source lineage, and operational definitions
• Respond to pipeline failures, data validation issues, operational alerts, and data-quality escalations with clear root-cause analysis and practical remediation steps
• Perform performance tuning of Spark jobs and SQL workloads, including partitioning, filtering, incremental logic, query optimization, and resource-aware design within established architectural patterns
• Validate data outputs with business partners, operations SMEs, and data stewards, and address defects or discrepancies through documented correction paths
• Support observability, logging, and auditability practices for data pipelines and AI-consumable datasets where applicable
• Commit code using Git, follow branching standards, participate in pull request reviews, and support CI/CD ways of working using GitHub, Azure DevOps, or similar tools
• Update documentation for pipelines, datasets, data contracts, data dictionaries, business rules, and operational runbooks as changes are made
• Execute assigned backlog items within sprint timelines and raise risks, dependencies, or blockers early
• Additional duties as assigned by management