Senior Test Manager & Quality Engineer, Full-Time (f/m/d)

Tasks

• Ensure that the quality of deliveries and the proper execution of tests for AI use cases are guaranteed
• Define and further develop the technical testing approach and framework architecture for ML and AI-ART, ensuring they are aligned with the bank's testing strategy and testing guidelines
• Design reusable, scalable test patterns (page objects, API clients, test data builders) that other engineers in the squads can adopt to ensure the technical consistency of tests across the ART
• Analyze and evaluate requirements, features, and user stories in terms of their testability during PI planning, backlog refinement, and iteration planning
• Derive test cases from technical and risk-based analyses of both functional and non-functional requirements (reliability, performance, security, usability, robustness), and select appropriate test techniques and levels of automation based on risk, coverage goals, and ROI
• Automate identified test cases using Python-based frameworks—Playwright/Python for user interfaces and requests, and Pytest or Pytest-BDD for BDD/Gherkin—while ensuring clean code, reusability, readability, and stability
• Design and implement AI/ML-specific test cases: evaluation pipelines for LLM outputs
• Integrate and orchestrate automated tests into GitLab CI/CD pipelines, including merge request pipelines and GitLab Runners
• Plan, schedule, and trigger automated test runs in various environments (DEV, INT, UAT, pre-PROD), including regression suites, smoke tests, release runs, and on-demand runs linked to merge requests and PI milestones
• Track the execution results, document defects in JIRA with supporting evidence (logs, traces, screenshots, videos), and communicate quality signals to the squad and the product owner
• Actively contribute to PI planning, system demos, Inspect & Adapt, and other SAFe ceremonies as part of the ML and AI ART
• Prepare test data, and ensure that synthetic or anonymized data is used whenever possible to meet confidentiality expectations

Requirements

• Practical experience in integrating and executing tests for AI solutions and/or large-scale data projects
• A solid understanding of Git and version control workflows, clean code principles, and code review culture
• Practical experience with Docker; familiarity with the basics of Kubernetes (jobs, namespaces)
• Exposure to testing AI/ML systems or a strong motivation to develop this expertise: evaluating LLM outputs, handling non-deterministic responses, challenges for RAG, and agent-based workflows
• Understanding of API design, microservices, event-driven architectures, and authentication layers
• A collaborative team player with a strong sense of personal responsibility who can handle automation challenges—from analysis through implementation to resolution—with minimal supervision