Test metrics

Life Cycle Costs

The costs of releasing a product is something interesting to look at. The following schema demonstrates several things:
  • The cost of maintenance is huge in comparison to development (design + implementation)
  • The cost of maintenance is huge in comparison to testing
  • The cost of maintenance is the one where we can save a lot of money

Investing a bit more in QA could drastically drop down the cost of maintenance. The ROI (Return On Investment) of the operation is predictively huge then.


life cycle cost


Cost of fixing bugs

The same way, if we look at the cost of fixing bugs, the same kind of metrics are showing up again. This confirms the idea that working on automation is an extremely rentable activity. And this for different reasons:
  • It's easier to pass more often/regularly non-regression tests
  • It saves a tremendous amount of time:
    • On test execution
    • On test operator
  • This improves testing coverage
  • It's easier to quantify testing coverage
  • It is simpler to get precise metrics and statistics about the test campaigns
  • This helps keeping QA team motivated (it's much more interesting to develop/maintain automation code than passing some manual tests)



cost of fixing bugs


Testing graph

This diagram shows how different types of testing may interact together. In particular:
  • When they might be executed
  • How many bugs they may allow to discover
This is based on my own experience but obviously may differ a lot depending on the System Under Test:


testing graph