Senior AI Test Automation Engineer

Tasks

• Continuously evolve our technical test automation approach and framework architecture for the ML & AI ART, in line with the Bank's Test Strategy and Test Policy
• Design reusable, scalable test automation patterns (page objects, API clients, test data builders) that other engineers across squads can adopt, ensuring technical consistency in test automation across the ART
• Analyze and evaluate requirements, features, and user stories for testability during PI Planning, Backlog Refinement, and Iteration Planning
• Derive test cases from technical and risk analyses of functional and non-functional requirements (reliability, performance, safety, usability, robustness) and select appropriate testing techniques and the scope of automation based on risk, coverage goals, and ROI
• Automate identified test cases using Python-based frameworks—Playwright+Pytest for UI, requests + pytest for APIs, Behave or pytest-bdd for BDD/Gherkin—while applying clean code principles, ensuring reusability, readability, and stability
• Design and Implementation of A/M-Specific Test Cases: Evaluation Pipelines for LLM Outputs
• Create and maintain context tests (e.g., Pytest) for platform APIs and microservice boundaries
• Integrate and orchestrate automated tests in CI/CD pipelines, including merge requests (GitLab), GitLab CI/CD pipelines, Selenium Grid, and BrowserStack
• Use Docker and Kubernetes to provide isolated, reproducible test environments
• Plan, schedule, and trigger automated test runs, including regression suites, smoke tests, release runs, and on-demand runs linked to merge requests and ML releases
• Monitor execution health, investigate and isolate flaky test cases, and maintain a low false-positive rate to ensure that quality signals remain reliable
• Prioritize execution results, report bugs in Jira with supporting evidence (logs, traces, screenshots, videos), and communicate quality signals to the squad, product owner, and test manager
• Produce evidence of execution (run metadata, artifacts, reports) that is suitable for audit and release governance, in accordance with the bank's testing policy and retention requirements
• Actively participate in PI planning, system design, Inspect & Adapt, and other SAFE ceremonies as part of the ML & AI ART
• Ensure end-to-end traceability from Jira features and stories → automated tests → defects → test results, using the bank's test management integration
• Write and maintain BDD scenarios in Gherkin that are linked to acceptance criteria for stories
• Work closely with product owners, Scrum Masters, ML engineers, MLOps engineers, and data engineers from various squads
• Align work with the bank's test strategy and reporting requirements, and support audit-ready engineers from various squads

Requirements

• Proven expertise in Python-based test automation: Playwright+Python (UI), Behave or pytest-bdd (APIs), pytest (API)/service validation—practically equivalent to REST-Assured
• Proven ability to design and maintain test automation frameworks