Anastasia Engelhardt is a QA Engineer at Luxoft DXC Technology, with a primary focus on exploratory testing. Her work centers around understanding system behavior, identifying risks, and testing complex scenarios, including AI-driven applications.Anastasia is also a Women Techmakers Ambassador at Technovation, where she actively promotes women in tech and supports community initiatives. In addition, she is an IT blogger and a frequent speaker at tech events, sharing practical insights on QA, career development, and entering the IT industry.
Speech: How to Build a Manual Evaluation Framework for AI RAG Chatbots Without Code Access
You have an AI chatbot in production. It sounds confident and users trust it — but is it correct and safe?
In many enterprise environments, QA engineers must test AI systems without access to code, documentation, or internal architecture. This talk presents a practical approach to testing a corporate AI RAG chatbot from a black-box perspective, using only the system interface.
I will show how to build a manual evaluation framework from scratch: defining test strategy, creating a golden dataset, and evaluating responses using scoring metrics such as accuracy, relevance, robustness, and context handling. The session also covers RAG-specific evaluation, including context precision, recall, and groundedness.
Special focus is given to security testing, including prompt injection, data leakage, and access control scenarios, illustrated with a real example discovered during testing.
Finally, I will outline how this manual approach can evolve into automated evaluation, and why automation without a solid foundation often fails.
