Data Integration Engineer, O'Reilly Media

• The ideal candidate will have a deep understanding of modern data engineering concepts and will have shipped or supported code and infrastructure with a user base in the millions and datasets with billions of records
• 6+ years of professional data engineering experience (equivalent education and/or experience may be considered)
• Strong experience with Talend Data Integration for designing and optimizing ETL pipelines
• Experience working with JSON and ARRAY data structures in BigQuery, including nesting and unnesting
• Experience in integrating and optimizing streaming data pipelines in a cloud environment
• Experience with Python
• Experience with Git/ Github
• Experience with deployment tools such as Jenkins to build automated CI/CD pipelines
• Hands-on experience with Google Cloud Storage, Pub/Sub, Dataflow, and Dataprep for ETL and real-time data processing
• Proficient in building and managing real-time data pipelines with Google Pub/Sub and Dataflow
• Proficient in BigQuery, including dataset management, advanced SQL, partitioning, clustering, and federated queries
• Solid understanding of PostgreSQL, including PL/pgSQL, query optimization, and advanced functions
• Familiarity with optimizing BigQuery performance through reservations, slots, and cost-effective query techniques
• Excellent problem-solving skills and ability to optimize high-volume data workflows
• Strong communication skills to collaborate effectively with cross-functional teams
• Strong drive to experiment, learn and improve your skills
• Respect for the craft—you write self-documenting code with modern techniques
• Great written communication skills—we do a lot of work asynchronously in Slack and Google Docs
• Empathy for our users—a willingness to spend time understanding their needs and difficulties is central to the team
• Desire to be part of a compact, fun, and hard-working team

Desirable

• We are seeking an experienced and detail-oriented Data Integration Engineer to contribute to the development and expansion of a suite of systems and tools, with a primary focus on ETL processes
• The candidate will be routinely implementing features, fixing bugs, performing maintenance, consulting with product managers, and troubleshooting problems
• Changes you make will be accompanied by tests to confirm desired behavior
• Code reviews, in the form of pull requests reviewed by peers, are a regular and expected part of the job as well
• Architect and build complex ETL pipelines in Talend Data Integration, ensuring scalability, reusability, and maintainability of workflows
• Implement sophisticated data transformations, including lookups, joins, aggregates, and custom routines using Talend’s tMap, tJavaRow, tSQLROW and JSON components
• Automate data ingestion from REST APIs, FTP servers, cloud platforms, and relational databases into cloud or on-premises storage
• Leverage Talend's integration with BigQuery for seamless data flow into analytical systems, employing native connectors
• Familiarity with Talend’s debugging tools, logs, and monitoring dashboards to troubleshoot and resolve job execution issues
• Optimize Talend jobs by using efficient memory settings, parallelization, and dependency injection for high-volume data processing
• Integrate Talend with Google Cloud Storage, Pub/Sub, and Dataflow to create hybrid workflows combining batch and real-time data processing
• Manage Talend deployments using Talend Management Console (TMC) for scheduling, monitoring, and lifecycle management
• Build high-performance BigQuery datasets, implementing advanced partitioning (DATE, RANGE) and clustering for cost-effective queries
• Proficient in working with JSON and ARRAY data structures, with expertise in leveraging BigQuery to efficiently nest and unnest objects as required for complex data transformations and analysis
• Write advanced SQL queries for analytics, employing techniques like window functions, CTEs, and array operations for complex transformations
• Implement BigQuery federated queries to integrate external datasets from Cloud Storage or other data warehouses
• Fundamental understanding of Designing and managing BigQuery reservations and slots involves allocating compute resources effectively to balance performance, cost, and workload demands across various teams and projects
• Implement Pub/Sub topics and subscriptions to manage real-time data ingestion pipelines effectively
• Integrate Pub/Sub with Talend for real-time ETL workflows, ensuring low-latency data delivery
• Implement dynamic windowing and triggers for efficient aggregation and event handling
• Optimize streaming pipelines by fine-tuning autoscaling policies, worker counts, and resource configurations
• Be able to enhance, modify existing PostgreSQL queries and functions
• Write advanced PL/pgSQL functions and triggers for procedural data logic
• As needed develop materialized views and indexed expressions to speed up query execution for large datasets
• Monitor and optimize queries through EXPLAIN/ANALYZE