Senior AI Test Automation Engineer

Tasks

• Define and further develop the technical test automation approach and test framework architecture for ML and AI-ART
• Designing reusable, scalable test automation patterns (page objects, API clients, test data builders)
• Analyzing and evaluating requirements, functions, and user stories in terms of testability
• Deriving test cases from technical and risk analyses of functional and non-functional requirements
• Automating test cases using Python-based frameworks (Playwright+Python, requests+pytest, Behave, or pytest-bdd)
• Designing and Implementing A/ML-Specific Test Cases: Evaluation Pipelines for LLM Outputs
• Setting up and maintaining context-based tests (e.g., Pact) for platform APIs and microservice boundaries
• Integrating and Orchestrating Automated Tests in GitLab CI/CD Pipelines
• Planning, scheduling, and triggering automated test runs across environments
• Monitoring execution health, investigating, and quarantining flaky test cases
• Prioritizing execution results, reporting errors in Jira with supporting evidence, and communicating quality signals
• Generating execution evidence (run metadata, artifacts, reports) for audit and release governance
• Contribute to PI planning, system design, Inspect & Adapt, and other SAFe ceremonies
• Ensuring end-to-end traceability from Jira features and stories to automated tests, defects, and test results
• Help create and maintain BDD scenarios in Gherkin
• Collaborate with product owners, Scrum Masters, ML engineers, MLOps engineers, and data engineers
• Align with the bank's testing strategy and reporting requirements

Requirements

• Proven expertise in Python-based test automation: Playwright+Python (UI), Behave or pytest-bdd (APIs), pytest (API)
• Proven ability to design and maintain test automation frameworks, not just write test scripts
• Practical experience with integrating and running automated tests in CI/CD pipelines
• Experience with test-driven development, parameterization, flaky test management, and modern reporting tools (Allure, pytest-html)
• Solid knowledge of Git and version control workflows
• Practical experience with Docker and the basics of Kubernetes (jobs, namespaces)
• Experience with AI/ML systems through testing, development, or applied projects is a strong plus
• Ability to develop and maintain in-house test automation solutions
• Fluent English; Spanish is a plus

We offer