Paying a consultancy a fortune does not buy defect-free software. If someone promised that, keep the slide deck; it may become useful evidence later.
The better question is whether the consultancy is materially improving your ability to find important problems, understand risk and release safely. Expensive testing can be excellent. It can also be an impressive mechanism for renting people at a markup. We can independently review what you are actually getting for the money.
What did you actually buy?
Some consultancies sell senior expertise during procurement and then deliver a rotating cast of junior people following the client’s existing process. There is nothing wrong with junior testers. There is something wrong with charging for transformation and delivering additional keyboard capacity.
Ask what the consultancy has changed for the better. Has coverage improved? Are serious defects being found earlier? Is automation reducing useful effort rather than simply increasing the automation percentage? Are environments, data and reporting better? Are your own people learning anything?
Defects in production are not the only measure
Some defects will escape. The important thing is what kind, how serious and whether the same classes of failure keep repeating. If production keeps exposing basic gaps the consultancy claims to cover, that deserves attention. Start by understanding what drives the price of testing consultancy rather than treating the contract total as proof of value.
On the other hand, if you are judging a complex programme as a failure because three low-impact defects reached production, you may be demanding magic rather than testing.
Watch for consultancy-shaped work
Be suspicious when the visible output is mostly maturity assessments, target operating models, roadmaps and beautiful presentations while actual testing remains obviously weak. Those things can be useful. They are not useful simply because they have gradients and arrows.
A good consultancy should be able to explain what it would stop doing as readily as what it wants to add. If every recommendation requires more people, more tooling and another workstream, the commercial model may have accidentally influenced the diagnosis.
Look for knowledge that stays after they leave
One sign of genuine consultancy value is whether your organisation becomes stronger. Do your own people understand the test approach better? Are useful tools, automation, data and practices being left behind? Can the programme continue without immediately buying the same number of consultants again?
There are engagements where long-term external capacity is exactly what you need. Fine. But if the work was sold as capability building or transformation, the client should eventually own more than a folder full of decks and a dependency on the next statement of work.
If you want us to review whether your testing consultancy is earning its keep, get onto us. We can look at delivery evidence, coverage, reporting, capability and contract outcomes and tell you whether you bought expertise or just an expensive amount of activity.