Data Operations Engineer

Job Description

The DataOps Platform Team’s mission is to ensure a reliable service level across every technological component of the EFG data platform (Databricks, Kafka, Spark, Elasticsearch, Apache MQ, Stardust Stack). We build and develop the CI/CD platform that supports data pipeline deployments and work closely with project teams to get releases across the line.

Main tasks

As our platform and data landscape continue to evolve, we’d like to strengthen our team with an experienced CI/CD engineer who specializes in data pipeline implementations. Expectations for data quality are constantly rising, along with the need for extremely stable service delivery. We must ensure that our delivery processes are controlled and transparent to our stakeholders. - We are particularly interested in someone who is familiar with Kafka, Spark, and Databricks operations using Python. - By joining our team, you will make a significant contribution to the continuity and quality of our operations. - Take on operational responsibility for specific data services (change management, quality assurance management, SLAs, SOPs). - Monitor and ensure that critical data pipelines run smoothly during Asian business hours by serving as the primary point of contact for critical issues in a “follow-the-sun” support model. - Design, implement, and optimize data pipelines in batch and streaming modes. - Follow and apply CI/CD practices to automate and streamline deployments and integrations. - Create, maintain, and improve documentation and operational procedures for which operational responsibility is assigned. - Collaborate cross-functionally to improve and refine governance around integration operations. - Maintain high standards for data quality, reliability, and performance. - Stay up to date with advancements in data engineering and DataOps technology to support continuous improvement.

Requirements and qualifications

- At least 5 years of hands-on experience in data engineering and data pipeline deployment, grounded in DevOps practices. - Experience with Python, SQL, PySpark, and Databricks. - Solid experience with leading CI/CD platforms and processes in hybrid environments (e.g., Azure DevOps, GitLab CI, GitHub Actions). - Familiarity with microservices architecture, API management, and observability is a plus. - Experience with Azure Databricks. - Strong communication and collaboration skills. - Rigorous and methodical in documentation and process creation. - Strong interest in convergence, standardization, and reusability. - Curious, enthusiastic, and proactive. - Passionate about data processing technologies, with a strong desire to learn and grow in a challenging environment. - Fluent in spoken and written English; French is a plus.

Our values

- Accountability: Taking responsibility for tasks and challenges, and striving for continuous improvement. - Hands-on: Proactive in delivering high-quality results quickly. - Passionate: Committed and striving for excellence. - Solution-oriented: Focus on customer outcomes and fair treatment of customers with a risk-aware mindset. - Partnership-oriented: Promoting collaboration and teamwork. Collaborating with an entrepreneurial spirit.