Senior Data Engineer, Simple Machines
Salary not provided
SQL
MongoDB
AWS
GCP
Python
Scala
Java
Kafka
Postgres
Airflow
Cassandra
Terraform
MySQL
Rust
Azure
Redshift
Spark
Snowflake
Flink
Kinesis
BigQuery
Parquet
JSON
Golang
dbt
Pulumi
Senior and Expert level
London
1-5 days a week in office
Data-driven platforms & applications
Be an early applicant
Data-driven platforms & applications
21-100 employees
B2BArtificial IntelligenceBig dataDeep TechMachine LearningData Integration
Be an early applicant
Salary not provided
SQL
MongoDB
AWS
GCP
Python
Scala
Java
Kafka
Postgres
Airflow
Cassandra
Terraform
MySQL
Rust
Azure
Redshift
Spark
Snowflake
Flink
Kinesis
BigQuery
Parquet
JSON
Golang
dbt
Pulumi
Senior and Expert level
London
1-5 days a week in office
21-100 employees
B2BArtificial IntelligenceBig dataDeep TechMachine LearningData Integration
Company mission
To architect and engineer highly performant, distributed, data driven platforms and machine learning applications that perform at massive scale.
Role
Who you are
- Professional Experience: At least 5+ years of data engineering or equivalent experience in a commercial, enterprise, or start-up environment. Consulting experience within a technology consultancy or professional services firm is highly beneficial
- Educational Background: Degree or equivalent experience in computer science or a related field
What the job involves
- The Senior Data Engineer at Simple Machines is a dynamic, hands-on role focused on building real-time data pipelines and implementing data mesh architectures to enhance client data interactions. This position blends deep technical expertise in modern data engineering methods with a client-facing consulting approach, enabling clients to effectively manage and utilise their data. Within a team of top-tier engineers, the role involves developing greenfield data solutions that deliver tangible business outcomes across various environments
- Developing Data Solutions: Implement and enhance data-driven solutions integrating with clients' systems using state-of-the-art tools such as Databricks, Snowflake, Google Cloud, and AWS. Embrace modern data architecture philosophies including data products, data contracts, and data mesh to ensure a decentralized and consumer-oriented approach to data management
- Data Pipeline Development: Develop and optimise high-performance, batch and real-time data pipelines employing advanced streaming technologies like Kafka, and Flink. Utilise workflow orchestration tools such as Dataflow and Airflow
- Database and Storage Optimization: Optimize and manage a broad array of database technologies, from traditional relational databases (e.g., PostgreSQL, MySQL) to modern NoSQL solutions (e.g., MongoDB, Cassandra). Focus on strategies that enhance data accessibility, integrity, and performance
- Big Data Processing & Analytics: Utilise big data frameworks such as Apache Spark and Apache Flink to address challenges associated with large-scale data processing and analysis. These technologies are crucial for managing vast datasets and performing complex data transformations and aggregations
- Cloud Data Management: Implement and oversee cloud-specific data services including AWS Redshift, S3, Google BigQuery, and Google Cloud Storage. Leverage cloud architectures to improve data sharing and interoperability across different business units
- Security and Compliance: Ensure all data practices comply with security policies and regulations, embedding security by design in the data infrastructure. Incorporate tools and methodologies recommended for data security and compliance, ensuring robust protection and governance of data assets
- Consulting Responsibilities:
- Client Advisory: Provide expert advice to clients on optimal data practices that align with their business requirements and project goals
- Training and Empowerment: Educate client teams on the latest technologies and data management strategies, enabling them to efficiently utilise and maintain the solutions we have developed
- Professional Development: Keep up with the latest industry trends and technological advancements, continually upgrading skills and achieving certifications in the technologies Simple Machines implements across its client base
- Core Data Engineering Tools & Technologies: Demonstrates proficiency in SQL and Spark, and familiarity with platforms such as Databricks and Snowflake. Well-versed in various storage technologies including AWS S3, Google Cloud BigQuery, Cassandra, MongoDB, Neo4J, and HDFS. Adept in pipeline orchestration tools like AWS Glue, Apache Airflow, and dbt, as well as streaming technologies like Kafka, AWS Kinesis, Google Cloud Pub/Sub, and Azure Event Hubs
- Data Storage Expertise: Knowledgeable in data warehousing technologies like BigQuery, Snowflake, and Databricks, proficient in managing various data storage formats including Parquet, Delta, ORC, Avro, and JSON to optimise data storage and retrieval
- Building and Managing Large-scale Data Systems: Experienced in developing and overseeing large-scale data pipelines and data-intensive applications within production environments
- Data Modelling Expertise: Proficient in data modelling, understanding the implications and trade-offs of various methodologies and approaches
- Infrastructure Configuration for Data Systems: Competent in setting up data system infrastructures, favouring infrastructure-as-code practices using tools such as Terraform and Pulumi
- Programming Languages: Proficient in Python and SQL, with additional experience in programming languages like Java, Scala, GoLang, and Rust considered advantageous
- CI/CD Implementation: Knowledgeable about continuous integration and continuous deployment practices using tools like GitHub Actions and ArgoCD, enhancing software development and quality assurance
- Testing Tools and Frameworks: Experienced with data quality and testing frameworks such as DBT, Great Expectations, and Soda, ensuring the reliability of complex data systems
- Commercial Application of Data Engineering Expertise: Demonstrated experience in applying data engineering skills across various industries and organisations in a commercial context
- Agile Delivery and Project Management: Skilled in agile, scrum, and kanban project delivery methods, ensuring efficient and effective solution development
- Consulting and Advisory Skills: Experienced in a consultancy or professional services setting, offering expert advice and crafting customised solutions that address client needs. Effective in engaging stakeholders and translating business requirements into practical data engineering strategies
Salary benchmarks
Our take
Many organisations struggle to unlock the true potential of their data. This data, often vast and complex, can hold valuable insights but requires the right tools and expertise to be harnessed effectively.
Simple Machines step in to address this challenge. Its team of global specialists, combining creative and expert technologists, unlock the true potential of data. It designs and builds custom data platforms, intelligent systems, and unique software products tailored to each client's specific needs. This allows organisations to "engineer data to life", transforming raw information into actionable insights that drive real-world impact.
Simple Machines leverages its expertise with leading technology providers like Google Cloud Platform (GCP), Amazon Web Services (AWS), and Azure to build robust and scalable solutions. This, coupled with its diverse partnerships with enterprises, governments, and technology companies, allows the company to stay at the forefront of the ever-evolving data landscape.
Freddie
Company Specialist at Welcome to the Jungle
Company
Company benefits
- Generous parental leave: Three months’ fully paid primary carer leave and one month fully paid secondary carer leave, subject to a minimum 12 month tenure
- Flexible working: Want to work from home or leave early to pick up the kids? Need to start early some days, later on others? No problem!
- "Your day": One day per year, in addition to annual leave days that you can take off with no explanation required
- Employee assistance programme
- Upskill - Learning and development: We all love continually learning and have a generous budget for you to upskill, join conferences, get certified and qualified
Company HQ
Darlinghurst, NSW
Leadership
Daniel Pritchard
(CEO)Previously Founder and Director of BangTango, Managing Director at Jive.
Share this job
View 1 more job at Simple Machines