Ignore the size of the team for a moment. Ignore the glossy methodology. Ignore the slide showing 14 global delivery centres. Ask what risk the supplier has actually helped you understand or reduce.
This is exactly the sort of supplier review where independence matters. The people selling you the service are rarely the best people to tell you whether you are getting value from it.
Look past activity
Suppliers are very good at reporting things they can count: test cases written, cases executed, automation scripts, defects raised, utilisation. Those numbers may be useful operationally. They are not proof of good testing.
Look at production escapes. Look at whether serious risks were identified early. Look at the quality of defects. Look at whether the team challenges requirements and designs or simply waits for something to execute. Look at whether automation finds meaningful problems or exists because the contract promised an automation percentage.
Check who is actually doing the work
Some consultancies sell you the impressive people in the pitch and deliver a rotating cast of juniors afterwards. Juniors are not the problem; pretending the capability is something else is. For a more formal challenge, see what an independent supplier review should examine rather than accepting a supplier’s own scorecard.
Ask who owns the thinking. Who can explain the strategy without opening the deck? Who challenges delivery? Who understands the business risk? Who will tell you when the agreed scope is nonsense?
Beware the presentation factory
If every concern produces another workstream, maturity assessment and six-week discovery phase, you may be buying consultancy rather than improvement. The best advice is sometimes ‘stop doing that’. Unfortunately that fits badly across twelve billable slides.
If you want an independent supplier review, get onto us. We can review the contract, outputs, coverage, reporting, team capability and recent failures and tell you whether you are getting assurance or merely a very well-presented volume of testing.
A five-point supplier review you can actually run
Pick two recent releases, including one that caused trouble in production. Ask the supplier to walk you through the underlying evidence rather than preparing another presentation. Score each area clear, uncertain or concerning and write down why.
- Risk coverage: Which three customer or business failures would hurt most? Show the tests or exploratory evidence addressing them, and explain what was not covered.
- Escaped defects: For two serious production incidents, show whether the risk was tested, missed, accepted or made untestable by the environment. What changed afterwards?
- Quality of findings: Inspect five randomly selected defects. Could another engineer reproduce them? Did they explain business consequence or merely announce ‘page broken’?
- Team capability: Ask the people doing the work, rather than their account director, to describe a critical user journey and the decisions they made about testing it.
- Useful reporting: Trace one known high-risk defect from the test evidence through the weekly report to the client’s release decision. Did its importance survive the reporting process?
For each weak area, agree a specific change, a named owner and evidence of improvement on the next release. If the supplier refuses to show working evidence, that is itself useful evidence. Avoid creating a fake maturity score out of these questions. Their job is to produce decisions.
Test the supplier’s advice with an awkward question
Ask them what they think you should stop paying them to do. A supplier that understands your service should be able to identify low-value work, obsolete coverage or activity that should move into engineering. If every recommendation expands scope, headcount or tooling, you have learned something about the business model.
Also inspect staff continuity. Testing complex systems relies on accumulated knowledge. Constantly rotating people may keep a utilisation spreadsheet healthy while forcing your organisation to repeatedly fund the same learning curve.
Finally, check whether the supplier escalates bad news early. A good partner risks an uncomfortable conversation to protect the client. A weak one discovers optimism whenever contract renewal approaches.