Matthias was a managing consultant at Sogeti Deutschland GmbH until he retired in September 2019. He specialised in test analysis, test management and test process improvement. He continues to be involved with the German Testing Board (GTB) and the International Software Testing Qualifications Board (ISTQB®) on a voluntary basis. He has served as the chair of the Glossary Working Group and editor of the ISTQB Glossary of Testing Terms for the past ten years. He also remains engaged as the product owner of the Glossary app. Additionally, he leads the task force for the ISTQB® Advanced Level Test Analyst Syllabus. Through these roles, he remains an active practitioner of testing and enjoys piloting innovative methods.
Keynote Speech: Evidence-Based Software Quality Assurance: Do We Know What We’re Doing?
Medicine spent decades confidently prescribing treatments that later turned out to be ineffective or harmful. The introduction of evidence-based medicine in the 1990s forced the field to ask an uncomfortable question: where is the proof that what we do actually works? Software testing faces a strikingly similar situation. We select test design techniques based on intuition, convention, and training curricula that rarely disclose their evidence base – and the research community has spent 25 years producing results that remain fragmented, hard to aggregate, and largely disconnected from industrial practice.
This keynote examines the empirical foundations of software quality assurance through three lenses. The first is evidence-based medicine itself: what controlled studies actually tell us about the effectiveness of test design techniques, and whether the dominant research instrument – mutation analysis – measures what we think it measures. The second is Fred Brooks’ seminal essay “No Silver Bullet”, which celebrates its 40th anniversary this year: his distinction between essential and accidental difficulties turns out to be remarkably revealing when applied to testing, and forces an uncomfortable question about where the industry has actually been making progress. The third lens is test management: even well-chosen techniques fail when organizational dynamics, schedule pressure, and the absence of shared industry data undermine their application.
The talk closes with a constructive agenda: what the software testing industry would need to build – shared defect taxonomies, open datasets, explicitly graded guidelines, and organizational structures that protect testing as a risk management function – to move toward a genuinely evidence-based practice, and why this matters for the one metric that counts: the overall cost of quality.
