Senior AI Test Automation Engineer

Tasks

• Define and further develop the technical test automation approach and the test framework architecture for the ML & AI ART
• Designing reusable, scalable test automation patterns (page objects, API clients, test data builders)
• Analyzing and evaluating requirements, features, and user stories in terms of testability
• Deriving test cases from technical and risk analyses of functional and non-functional requirements
• Automating test cases using Python-based frameworks (Playwright/Python, requests, pytest, Behave, or pytest-bdd)
• Designing and Implementing A/M-Specific Test Cases: Evaluation Pipelines for LLM Outputs
• Setting up and maintaining contact tests (e.g., PaRT) for platform APIs and microservice boundaries
• Integrating and Orchestrating Automated Test Data in GitLab CI/CD Pipelines
• Planning, scheduling, and triggering automated test runs across environments
• Monitoring execution health, investigating, and quarantining flaky test cases
• Prioritizing execution results, reporting errors in Jira with supporting evidence, and communicating quality signals
• Generating execution evidence (run metadata, artifacts, reports) for audit and release governance
• Actively contribute to PI planning, system design, Inspect & Adapt, and other SAFE ceremonies
• Ensuring end-to-end traceability from Jira features and stories to automated tests, defects, and test results
• Collaborate with product owners, Scrum Masters, ML engineers, MLOps engineers, and data engineers across squads

Requirements

• Proven expertise in Python-based test automation: Playwright/Python (UI), Behave, or pytest-bdd (BDD) for requests + pytest (API)/service validation
• Proven ability to design and maintain test automation frameworks, not just write test scripts
• Practical experience with integrating and running automated tests in CI/CD pipelines, ideally using GitLab or equivalent enterprise platforms
• Experience with test execution, parameterization, flaky test management, and modern reporting tools (Allure, pytest-html, or equivalent)
• Solid understanding of ETL and version control workflows, experience with clean code principles and code review practices
• Practical knowledge of Docker, familiarity with Kubernetes fundamentals (pods, namespaces)
• Experience with testing AI/ML systems, strong motivation to develop this expertise
• Experience with the integration, execution, and automation of layers; understanding of AI/ML design, microservices, event-driven architectures, and authentication layers
• Understanding of SATE and DevOps principles; experience with Agile Release Trains is a plus
• Experience with Jira for story/feature tracking and test management integration (Xray, Agile Hive)
• Demonstrable end-to-end thinking, linking user journeys