It should look at the testing. This sounds obvious until you see supplier reviews that spend most of their time scoring governance templates and whether the weekly report was submitted on Tuesday.
Governance matters, but the point is to work out whether the supplier is helping you understand and reduce software risk. We do this kind of review, and the useful work starts with evidence rather than a maturity score.
Look at what they test and why
Take important journeys and serious risks and trace how the supplier covers them. Do not rely on statements such as “100% requirements coverage”. Requirements can be completely covered while the service still has enormous gaps nobody wrote down.
Look at exploratory work, integration coverage, data, permissions, failure handling, performance where relevant, and areas that have caused trouble before.
Look at what gets reported upwards
Compare raw evidence with the senior report. Have serious concerns been softened? Are untested areas visible? Do pass rates hide important failures? Does green mean risk is controlled or simply that the supplier met its milestones? That review complements the day-to-day question of whether your testing supplier is any good.
Look at capability, not CV decoration
Talk to the people doing the work. Can they explain the system, risks and why they are testing particular areas? Or are they mainly executing scripts somebody else wrote? Qualifications are fine. A wall of certification logos does not answer whether the team understands your product.
Review real releases, not just declared process
A supplier can have an excellent documented approach and execute it badly. Sample recent releases. Pick one that went well and one that did not. Trace what the supplier actually did, what it found, how quickly important problems moved and whether final reporting matched the evidence.
And look at the client’s contribution too. Clients routinely give suppliers unusable environments, late builds, vague ownership and impossible timelines, then ask why testing underperformed. A fair review needs to say when the client is causing the problem.
Do not score everything just because you can
Supplier reviews love scoring models because numbers make judgement look objective. You end up with 4.2 for governance, 3.7 for automation and a spider chart that appears scientific enough to frighten nobody.
Use ratings only where they help compare or prioritise. The review should be prepared to say the uncomfortable thing plainly: this team understands the product but is blocked by the client; this reporting is misleading; this coverage is weak; this contractual measure is encouraging nonsense. Precision is not the same as insight.
If you want us to review a testing supplier, get onto us. We can look at the contract, evidence, coverage, team capability and reporting and tell you where the supplier is strong, where it is weak and where your own organisation is making the situation worse.