Measuring the efficiency of test automation in software projects

What is a “bad” test?

Whenever an automated test fails, it needs to be triaged. An engineer’s time is needed to dig in to the test failure to make a determination why the test failed. For an automated test, it could be for any of the following reasons:

  • The software product has a bug and is doing the wrong thing.
  • The test code has a logic error in it and is doing the wrong thing.
  • The product behaviour evolved, but the test code was not updated to reflect it.

How to identify low reliability automated tests?

Depending on the size of the team and number of automated tests, the test/automation engineers triaging the failures may have a good gut feeling for what the low reliability test suites are, since they are frequently fixing the same suites. As the number of tests scales up, it can be easier for low reliability test suites to fly undetected from the “gut feeling” radar.

Gleaming information from this data

Automated tests with high numbers of automation bugs associated with them represent the low reliability tests with high upkeep overhead. Since the entire point of automating tests is to save the time it would take to run them manually, once an automated test crosses the threshold where it takes considerably more effort to maintain it than it would to run it manually, then the automated test has outlived its usefulness.

What to do with the low reliability tests?

The answer could depend on what is causing elevated automation bugs in these test suites.

Lessons learned

Since the product we test has an end-user facing web-based UI, this part of the product is very frequently being updated to meet ever-evolving web usability standards and popular design patterns. While excellent automation libraries exist out there to help make UI testing easy (e.g. Selenium), our team discovered the hard way that investing in automating the testing of this UI was simply not worth the effort. Since we were constantly up against a moving target, we were spending a significant amount of time trying to keep up with the changes, and to make matters worse these test cases were extremely quick and easy to run manually.



Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store