Data Engineer, Alliants

• 5+ years of experience in database administration and data engineering roles
• Proficiency in SQL and experience with various languages including Rails, ORM (Active Record) and Node.js
• Strong knowledge of data modelling, ETL processes, and data warehousing concepts
• Experience with cloud-based data solutions (AWS and Azure)
• Understanding of AWS services for data storage, management, transfer, indexing and caching, such as RDS, Glue, Kinesis, Redshift, Athena, ElastiCache, OpenSearch, etc
• Familiarity with data governance principles and practices
• Knowledge of data security best practices and compliance requirements
• Proven track record in performance tuning and query optimization
• This position requires a versatile professional who can bridge the gap between traditional database administration and modern data engineering practices while also implementing robust data governance frameworks

Desirable

• To design, manage, optimise, and improve data architecture across the business, including AXP (SaaS) and services projects
• This role will encompass database administration, data engineering, and data governance responsibilities to ensure data integrity, performance, security, and compliance
• The position will focus on designing and implementing robust data pipelines, ETL processes, and data integration solutions to support both internal analytics and client-facing data products and service projects
• The role will drive the development of scalable, efficient, cost-effective and reliable data infrastructure that enables data-driven decision-making and enhances the value of our data assets
• Data Integration and ETL: Design and implement data integration solutions and ETL processes to support both internal analytics and client-facing data products
• Data Architecture: Design and implement scalable data architectures. Select and configure databases for optimal performance, security and cost-effectiveness across various environments
• Cross-functional Collaboration: Work closely with development, data science, and DevOps teams to align data architecture with application and analytical requirements
• Database Performance and Security: Ensure database performance, security, and reliability for both the AXP application and client projects. Work with developers and data scientists to optimise data structures
• Performance Tuning: Monitor and optimise database queries and data pipelines. Utilise Rails' Active Record, custom SQL queries, and other data engineering techniques to ensure efficient data retrieval and manipulation
• Data Security and Governance: Implement security measures to protect sensitive data, including encryption and access controls. Develop and maintain data governance frameworks to ensure data quality and compliance
• Schema Management: Manage database schema changes efficiently using Rails migrations and other tools. Ensure smooth migrations without affecting application availability
• Backup, Recovery, and Disaster Planning: Establish comprehensive strategies for data backup, recovery, and disaster planning across all data systems
• Capacity Planning and Scaling: Assess and plan for future data growth. Develop strategies for scaling data infrastructure to handle increased load and complexity while minimising cost