Senior Automation Engineer

1w1 week ago

Resillion

Glasgow, GB · Full-time · £55,000 – £75,000

About this role

We are seeking a Senior Automation Engineer / SDET with strong Python skills and hands-on experience testing AI systems, including LLM-driven applications and agentic AI workflows. You will design and build scalable automation frameworks, implement modern AI testing strategies, and ensure quality of next-generation conversational and agent-based AI products. This role suits those passionate about automation and AI/LLM behavior in real-world scenarios.

Develop and maintain a Python-based automation framework for validating AI agents like chatbots, voicebots, and agentic workflows. Design and execute automated tests for LLM responses, conversational flows, retrieval-augmented generation (RAG), and end-to-end agentic scenarios. Apply evaluation methods such as DeepEval or RAGAS to modernize AI testing.

Integrate automated tests into CI/CD pipelines using Jenkins, GitHub Actions, or Azure Repos. Build batch executions with Docker, Kubernetes, or cron jobs, and manage Git repositories with branching strategies and collaborative workflows. Collaborate closely with engineering, AI/ML, and product teams for high-quality AI feature delivery.

Join a culture built on “if you see something, say something,” taking responsibility and embracing change. Resillion invests in developing employee skills to lead the industry, driven by passion and commitment to the bigger picture. This role offers a chance to grow with a forward-thinking organization in AI.

Requirements

Strong Python development skills, particularly for automation (PyTest preferred)
Solid background in test automation and framework design
Experience testing or validating AI/LLM-based systems (agentic workflows, conversational AI, RAG pipelines, etc.)
Understanding of AI evaluation concepts — familiarity with tools like DeepEval or RAGAS is a bonus
Experience implementing CI/CD pipelines and automated batch executions
Strong understanding of source code management, Git workflows, and collaborative engineering practices
Ability to design structured, scalable, and maintainable automation solutions

Responsibilities

Develop and maintain a Python-based automation framework for validating AI agents (chatbots, voicebots, agentic workflows)
Design and execute automated tests for LLM responses, conversational flows, retrieval-augmented generation (RAG), and end-to-end agentic scenarios
Apply modern AI testing strategies and evaluation methods (e.g., DeepEval, RAGAS)
Integrate automated tests into CI/CD pipelines (Jenkins, GitHub Actions, Azure Repos)
Build and manage batch and scheduled executions using Docker, Kubernetes, cron jobs, or similar tooling
Manage code repositories using Git, including branching strategies, version control, and collaborative workflows
Collaborate closely with engineering, AI/ML, and product teams to ensure high-quality delivery of AI-powered features