Chatbot QA Suite
Automated regression checks for an LLM-backed chatbot API — a system that
answers differently every call, where assertEqual has nothing to
compare against. Three layers: a deterministic response contract, statistical
checks over repeated sampling (factual answers must agree with each other;
sensitive requests must be refused every time), and counterfactual pairs that
substitute one irrelevant detail to see whether it changes the answer. Every
threshold is a named constant, because requiring four passes out of five is a
specification and retrying until green is not.
Python · pytest · pydantic · GitHub Actions · source