Principal Data Engineer, Mastercard

• The number of years of experience is flexible if all other requirements are met
• Proven expertise in leading and driving real-time data pipelines and large-scale distributed systems
• Hands-on experience with tools such as Hadoop, Apache Spark, Kafka, NiFi, and Big Data ecosystems
• Strong knowledge of data security, compliance, and data governance standards
• Proficiency in programming languages such as Java, Python, or Scala
• Experience with microservices architecture and messaging frameworks (e.g., Kafka, RabbitMQ)
• Extensive familiarity with cloud platforms (AWS) and container orchestration tools (Kubernetes, Docker)
• Background in designing and managing high-performance, scalable and secure financial systems with global-scale architecture

Desirable

• Our decency quotient, or DQ, drives our culture and everything we do inside and outside of our company. With connections across more than 210 countries and territories, we are building a sustainable world that unlocks priceless possibilities for all
• Design & Develop Scalable Retrieval Infrastructure:
• Architect and implement real-time (RT) and near-real-time (NRT) data engineering pipelines between Mastercard systems to support AI deployments
• Enhance data engineering architecture to ensure reliability, maintainability, and scalability by improving deployment, monitoring, and fault tolerance
• Develop monitoring solutions to assess the health and performance of AI observability systems
• Lead technical decision-making, ensuring alignment with business goals and AI objectives
• Lead and Drive scalable and efficient ETL/ELT processes to transform raw data into actionable insights and AI-driven velocities
• Ensure Data Security, Privacy, and Compliance:
• Implement robust data security and compliance mechanisms, including anonymization and encryption, to handle sensitive data in retrieval systems
• Collaborate with compliance teams to align retrieval infrastructure with Mastercard's data governance policies
• Ensure the quality, accuracy, and integrity of datasets across the organization
• Establish monitoring frameworks to proactively address data pipeline issues and detect ML model drifts
• Implement strategies for data versioning, lineage tracking, and auditability
• Innovate in Data Solution Technologies:
• Ensure pipeline performance, monitoring and optimization balancing business and technology trade-offs
• Explore and integrate emerging data strategies, AI model enhancements, and distributed systems for improved AI application performance
• Drive the adoption of memory-based retrieval systems (e.g., Redis) and advanced AI infrastructure to enhance system scalability
• Architect solutions utilizing platform-as-a-service (PaaS), containerization technologies (e.g., Kubernetes, PCF), and cloud-native architectures
• Contribute to designing and implementing high-transaction volume financial systems with global scalability and extreme uptime requirements
• Design and develop global-scale back-end microservices using Java, Kafka, RabbitMQ, and related technologies
• Lead and Drive scalable data pipelines with Big Data technologies (e.g., Hadoop, Apache Spark, Spark SQL, Kafka, NiFi) for batch and incremental data loads
• Deploy AI/ML pipelines in production environments using MLOps best practices
• Continuously improve data systems to meet the evolving needs of AI applications and business objectives
• Collaborate with AI and Product Engineering Teams:
• Work closely with data scientists, analysts, and engineers to ensure seamless data accessibility and usability for AI applications
• Partner with business stakeholders, product managers, and leadership teams to translate business requirements into technical solutions
• Collaborate with software engineers and architects to integrate data platforms with Mastercard’s enterprise systems

• End Date: March 18, 2025