Bruno LegeardChief AI Officer @Smartesting

AI Agents: Testing them? Using them for Testing?

This presentation is based on over a year of experience creating a virtual tester AI agent to execute manual test cases autonomously. This virtual tester functions like a subcontractor for test execution, akin to those in offshore or nearshore contexts. Like human testers, this AI agent autonomously executes test cases described in natural language. An AI agent is an AI system leveraging Large Language Models (LLMs) to automate processes autonomously or semi-autonomously. These agents interpret natural language instructions, reason through tasks, and execute actions independently, making them highly valuable for process automation. By combining advanced language comprehension with decision-making capabilities, AI agents streamline workflows, enhance efficiency, and minimize human intervention in routine tasks.

The rapid rise of AI agents presents two critical questions for the software testing profession:

1. How do we test them?

2. Can they be used for software testing?

Testing AI Agents: Trust in LLM-powered agents is crucial for their adoption, and trust is built through rigorous testing. While developing the virtual tester AI agent, we addressed this issue by creating specific, automatable functional metrics and designing a representative benchmark to measure performance, ensure progress, and test for non-regression. This approach enabled us to establish a confidence score for test executions, laying the groundwork for trust in AI agent performance.

Using AI Agents for Software Testing: Unlike LLM-based AI assistants, AI agents can act upon their environment, making them particularly suited for testing tasks that require action. However, their effectiveness hinges on the level of trust we place in them. Using our virtual tester AI agent as an example, we will demonstrate how agents can contribute to software testing activities and illustrate the collaborative potential between testers and AI agents to optimize software testing processes.

Bruno is the head of Smartesting’s AI Lab and has 20 years of experience in model-based testing and AI for testing. He is also active within the ISTQB as Product Owner for the Model-Based Testing syllabus, and Technical Lead for the Testing with Generative AI syllabus currently under development.