Principal - Cloud Data Platform Engineer

About the role

• As a Principal Cloud Data Platform Engineer at Northey Partners, you will be responsible for designing and implementing scalable, secure, and cost-effective data platforms and pipelines that support our mission-critical health research, data science, analytics, and business operations to drive innovation and deliver actionable insights.
• We are looking for an experienced and innovative cloud data engineer with expertise in cloud technologies to join our team and play a key role in further developing our data strategy.

Tasks

• Platform Implementation & Deployment:
  • Design, deployment, and maintenance of production and non-production systems.
  • Develop robust and scalable data pipelines that capture, transform, and process structured, semi-structured, and unstructured data.
  • Establish and maintain data quality practices to ensure data accuracy, validity, and completeness.
  • Prioritizing, building, and validating data platforms using a product-oriented mindset that informs key decisions.
  • Implementation of solutions designed for scalability, security, and cost-efficiency.
  • Supporting and promoting the adoption of data platforms.
• Data Integration & Governance:
  • Collaborate with data engineers on technology infrastructure and engineering teams across all organizational units to integrate data from various systems.
  • Implementation and enforcement of data governance policies and standards.
  • Support and promote data literacy throughout the organization. Collaborate with DevSecOps and SREs.
• Operations & Scalability:
  • Measuring and tracking KPIs for the cloud data platform, including monitoring, alerting, and troubleshooting.
  • Collaborate with cross-functional teams to ensure the reliability and availability of data platforms.
  • Supporting the data platform's capacity and future needs.
  • Supporting, mentoring, and training other team members in data platform roles.
• Metrics:
  • Identification, architecture, and reporting on success scenarios for the data platform.
  • Establishing and reducing incident response times.
  • Measuring platform productivity through monthly business reviews and feedback.
  • An understanding of what success means.

Requirements

• Qualifications:
  • At least 10 years of practical experience in technology infrastructure.
  • 5+ years of experience in cloud engineering on at least one leading cloud platform.
  • 10+ years of experience with AWS, Azure, or GCP.
  • 3+ years of experience as a cloud engineer managing data pipeline platforms.
  • 5+ years of experience with big data pipeline platforms, such as Spark, Hadoop, and Hazel.
  • Previous experience with big data platforms such as Apache Spark or Horton Data Platform.
• Skills & Competencies:
  • Practical architecture and experience with Snowflake, Databricks, and Data Lake.
  • Experience with pipeline environments (Azure, Apache Spark).
  • Proficient in developing ETL/ELT pipelines using Azure Data Factory, Azure Functions, DBT, and Azure Data Lake.
  • A deep understanding of distributed data systems and pipeline design.
  • Strong