Stuart ReidCTO @ STA Consulting

Testing AI: A New Testing Specialism

Artificial intelligence (AI) is now a mainstream technology that touches the lives of most of us, with a global market predicted to reach US$500 billion within two years.

AI-based systems, and machine-learning (ML) systems in particular, are different from traditional IT systems, as the underlying code is not written by a programmer but is inferred from patterns in data.  This major change to classic software development has two significant knock-on effects.  It has created the attractive new role of AI specialist.  However, it has also created several new sources of specialised defects, such as the training data, the deliverable ML model and the ML development framework.

This talk proposes a new software testing specialism that is focused on addressing the new challenges introduced by AI and ML.  To be able to test a ML system, testers need to understand the underlying technology and how it can go wrong.  They also need to be able to address those system attributes, such as ethics and trust, which are so important with AI-based systems.

From now on, as AI-based systems grow in complexity and criticality, achieving acceptable levels of quality and trust will only be possible by employing independent, AI test specialists.  This talk will explain why they are needed, what specialist tasks they will do, and suggest how we should handle the transition to a future where the AI test specialist is a recognized role.

Dr Stuart Reid is Chief Technology Officer at STA Consulting in Seoul with nearly 40 years’ experience in the IT industry, working in development, testing, and education. While currently concentrating on the testing of AI, application areas range from safety-critical to financial and media.

Stuart supports the worldwide testing community in a number of roles. He is convener of the ISO Software Testing Working Group, which has published the ISO/IEC/IEEE 29119 series of software testing standards and is the co-convener of the ISO Joint Working Group on Testing AI.  Stuart previously led the ISO project on autonomous systems for software and systems engineering.  He also founded the International Software Testing Qualifications Board (ISTQB) to promote software testing qualifications globally and he was one of the authors of the new ISTQB certification on the testing of AI-based systems.