Dashboards are useful until people start managing the dashboard instead of the risk.
A pass rate is not good news by itself. Ninety-nine routine checks can pass while the one failed test proves customers cannot pay. The percentage is technically accurate and practically ridiculous.
If the dashboard is reassuring everyone while customers are suffering, we can tell pretty quickly whether the measures are wrong, the reporting is filtered or the testing itself is missing the point.
Ask what sits underneath the number
For every big headline metric, ask what it hides. Test completion: completion of what? Defects: how serious? Automation coverage: coverage of which risks? Green status: who decided the threshold?
Be especially suspicious when metrics improve near a deadline without the software changing much. Defects get reclassified. Tests get descoped. Blocked becomes ‘not run’. Red becomes amber because ‘the issue is understood’. The system remains exactly as broken but the PowerPoint has made a remarkable recovery.
Audit one green metric all the way back to evidence
Pick one headline number and trace it. If the dashboard says 96% passed, find the denominator. Were failed tests removed from scope? Are blocked tests counted? Can a test be ‘passed with defects’? If automation coverage is 80%, ask 80% of what: code, requirements, test cases, journeys or somebody’s spreadsheet?
Then ask three people what would make that metric turn red. If you get three different answers, the number is not a control. It is decoration with arithmetic.
Management needs decisions, not decoration
A useful dashboard tells leaders what matters now: critical coverage, serious failures, major untested areas, meaningful trends, limitations in the evidence and decisions needed. It should make uncomfortable risk easier to see, not easier to hide.
If your test dashboard is colourful but nobody can explain whether you should release, get onto us. We can review the measures, trace them back to the underlying evidence and tell you what should stay, what should go and what information management is currently not seeing.
Use fewer metrics with sharper definitions
Most test dashboards would improve if half the numbers disappeared. Pick measures that expose decisions: critical journey coverage, serious open defects, blocked or untested risk, escaped-defect trends and anything specific to the service such as performance or recovery confidence.
Define them properly too. If one team marks a test ‘passed with defects’ and another marks it failed, the portfolio percentage is decorative arithmetic. If a test can be removed from scope after it fails without changing the denominator, congratulations: you have invented a machine that improves itself under pressure.
The point of reporting is not to make the organisation feel calm. It is to make the organisation appropriately informed.