Principal Data Platforms Engineer, Simple Machines
Salary not provided
SQL
MongoDB
AWS
Docker
Kubernetes
Python
Tableau
Looker
Scala
Java
Kafka
Airflow
Cassandra
Terraform
MySQL
Rust
Azure
Redshift
Spark
Snowflake
Flink
Kinesis
BigQuery
Sagemaker
Parquet
JSON
Golang
dbt
Pulumi
Expert level
London
1-5 days a week in office
Data-driven platforms & applications
Job no longer available
Data-driven platforms & applications
21-100 employees
B2BArtificial IntelligenceBig dataDeep TechMachine LearningData Integration
Job no longer available
Salary not provided
SQL
MongoDB
AWS
Docker
Kubernetes
Python
Tableau
Looker
Scala
Java
Kafka
Airflow
Cassandra
Terraform
MySQL
Rust
Azure
Redshift
Spark
Snowflake
Flink
Kinesis
BigQuery
Sagemaker
Parquet
JSON
Golang
dbt
Pulumi
Expert level
London
1-5 days a week in office
21-100 employees
B2BArtificial IntelligenceBig dataDeep TechMachine LearningData Integration
Company mission
To architect and engineer highly performant, distributed, data driven platforms and machine learning applications that perform at massive scale.
Role
Who you are
- Core Data Engineering Tools & Technologies: Proficient with SQL, Spark, and platforms like Databricks, Snowflake, along with various storage technologies such as S3 (AWS), BigQuery (Google Cloud), Cassandra, Mongo, Neo4J, and HDFS. Highly skilled in pipeline orchestration tools like Glue (AWS), Airflow, dbt, and streaming technologies such as Kafka, Kinesis (AWS), Pub/Sub (Google Cloud), and Event Hubs (Azure)
- Presentation Layer Tools: Experience with presentation layer tools including Superset, PowerBI, Tableau, and Looker
- Data Science Tools: Exposure to, or an understanding of data science/MLOps tools like Sagemaker (AWS), Dataiku, MLflow
- Data Storage Expertise: Experienced with data warehousing technologies like BigQuery, Snowflake, Databricks, and proficient in handling various data storage formats including Parquet, Delta, ORC, Avro, and JSON, ensuring optimal data storage and retrieval strategies
- Building and Managing Large-scale Data Systems: Developed and managed large-scale data pipelines and data-intensive applications in a production environment
- Site Reliability Engineering: Demonstrated ability to build robust platforms and applications, leveraging SRE principles to define and measure goals and commitments with stakeholders
- Data Modelling Expertise: Experienced in data modelling with a deep understanding of the trade-offs of various approaches
- Infrastructure Configuration for Data Systems: Skilled in configuring infrastructure for data systems, with a preference for infrastructure-as-code practices using tools like Terraform and Pulumi
- Programming Languages: Proficient in programming languages such as Python, SQL and exposure to other languages such as Java, Scala, GoLang, or Rust a bonus
- Containerised Solutions Expertise: Knowledgeable in containerised solutions using Docker and Kubernetes, enhancing the deployment and scalability of applications
- CI/CD Implementation: Familiar with CI/CD tools such as GitHub Actions and ArgoCD, streamlining development processes and ensuring high-quality software delivery
- Testing Tools and Frameworks: Experience with testing tools and frameworks like DBT, Great Expectations, and Soda to maintain high data quality and reliability in complex data systems
- Commercial Application of Data Engineering Expertise: Demonstrated track record of expertise in Data Engineering applied across a range of industries and organisations in a commercial setting
- Agile Delivery and Project Management: Experienced in building solutions using agile, scrum or kanban delivery methods
- Consulting and Advisory Skills: Experience working in a professional services firm or technology consultancy, providing expert advice and tailored solutions to clients. Skilled in stakeholder engagement, understanding client needs, and translating these into actionable data engineering strategies
- Leadership in Project and Team Management: Lead teams and projects within consultancy settings, ensuring alignment of technical solutions with business objectives, and mentoring team members in best practices
- Professional Experience: At least 10+ years of data engineering or equivalent experience in a commercial, enterprise, or start-up environment. Consulting experience within a technology consultancy or professional services firm is highly beneficial
- Educational Background: Degree or equivalent experience in computer science or a related field
What the job involves
- A Principal Data Platforms Engineer at Simple Machines embodies a blend of deep technical expertise and consulting acumen, essential for leading advanced data projects across diverse industries.
- This role involves architecting, designing and implementing impactful data-driven solutions using a variety of tools and platforms such as Databricks, Snowflake, Google Cloud, and AWS.
- As a key consultant, the Principal Data Platforms Engineer advises clients on optimising their data architecture, aligns technical solutions with business goals, and leads teams in delivering high-impact data-driven outcomes
- Developing Data Solutions: Architect, design and implement data-driven solutions that integrate with clients' existing systems, using tools such as Databricks, Snowflake, Google Cloud and AWS. Incorporate modern data thinking like data products, data contracts and data mesh to foster a decentralised and consumer-oriented approach
- Data Pipeline Development: Develop robust data pipelines using tools like Spark, Flink, Airflow, and Kafka to enable real-time and batch data processing. Implement frameworks that support scalable and maintainable data flows, aligning with principles of data as a product
- Database and Storage Optimisation: Optimise and manage a variety of database technologies, including relational (e.g., PostgreSQL, MySQL) and NoSQL databases (e.g., MongoDB, Cassandra), ensuring efficient data storage and retrieval. Focus on achieving optimal data accessibility and quality
- Big Data Technologies: Leverage big data technologies like Spark and Flink to manage large-scale data processing and analysis. Employ approaches that promote data democratisation and accessibility within the organisation
- Cloud Data Management: Implement and manage cloud-specific data services such as AWS Redshift, S3, Google BigQuery, and Google Cloud Storage. Utilise cloud architectures that enhance data sharing and collaboration across business units
- Security and Compliance: Ensure compliance with data security policies and regulations, implementing secure data practices across various platforms. Incorporate security by design in data infrastructure and solutions
- Client Advisory: Advise clients on the best data practices and technologies to meet their business needs and project objectives. Provide expertise in selecting the appropriate tools and designing scalable data architectures
- Project Leadership and Management: Lead project teams, ensuring timely delivery of services. Manage project scope, timeline, and resources effectively, often involving an agile or Kanban based approach
- Business Needs Analysis: Work closely with clients to understand their operational needs and strategic goals, translating business requirements into technical solutions
- Stakeholder Engagement: Communicate effectively with client stakeholders to align technology solutions with business strategies, ensuring clear understanding and agreement on project deliverables
- Training and Empowerment: Train client teams on new technologies and data management practices, empowering them to effectively use and manage the implemented systems
- Contributions to Business Development: Assist in business development efforts by contributing to proposals, statements of work, and by providing technical insights during client meetings and presentations
- Thought Leadership: Serve as a thought leader by staying abreast of industry trends and emerging technologies, incorporating innovative solutions into client projects
Salary benchmarks
Our take
Many organisations struggle to unlock the true potential of their data. This data, often vast and complex, can hold valuable insights but requires the right tools and expertise to be harnessed effectively.
Simple Machines step in to address this challenge. Its team of global specialists, combining creative and expert technologists, unlock the true potential of data. It designs and builds custom data platforms, intelligent systems, and unique software products tailored to each client's specific needs. This allows organisations to "engineer data to life", transforming raw information into actionable insights that drive real-world impact.
Simple Machines leverages its expertise with leading technology providers like Google Cloud Platform (GCP), Amazon Web Services (AWS), and Azure to build robust and scalable solutions. This, coupled with its diverse partnerships with enterprises, governments, and technology companies, allows the company to stay at the forefront of the ever-evolving data landscape.
Freddie
Company Specialist at Welcome to the Jungle
Company
Company benefits
- Generous parental leave: Three months’ fully paid primary carer leave and one month fully paid secondary carer leave, subject to a minimum 12 month tenure
- Flexible working: Want to work from home or leave early to pick up the kids? Need to start early some days, later on others? No problem!
- "Your day": One day per year, in addition to annual leave days that you can take off with no explanation required
- Employee assistance programme
- Upskill - Learning and development: We all love continually learning and have a generous budget for you to upskill, join conferences, get certified and qualified
Company HQ
Darlinghurst, NSW
Leadership
Daniel Pritchard
(CEO)Previously Founder and Director of BangTango, Managing Director at Jive.
Share this job
View 2 more jobs at Simple Machines