Skip to content Contact us

Contact form

Exove by Anastasios Daskalopoulos,

Automated Regression Testing: High-Level Points To Remember

It should not come as a great surprise that the issue of software quality has recently become a significant topic now that development to release times have become shorter. Developing software with both speed and high quality has now become and will continue to be a big challenge in software production, and this mandates the necessity that both Development and Quality Assurance Engineers have to team up to solve the problem of speed vs quality.

Since the time of manual test execution against large sets of test cases that have been written, reviewed and approved has become a thing of the past, the QA Team must use a new set of skills, namely software development and scripting, along with the ability to install and put to use a set of tools that can execute a large number of tests. In addition to only executing tests, automation tools such as Robot Framework, Nightwatchjs and Jenkins can make both the tests themselves and the test results visible to not only QA, but also to the Developers, Project Managers, and other stakeholders for whom the quality of the software is important.

Continuous Development and Continuous Testing

Many buzzwords fly around now regarding software development, and of late there is DevOps, Agile, and Continuous Development. Along with the concept of Continuous Development, there must also be Continuous Testing. The idea behind this is simple: after a new build for a software feature, there must also be a set of tests to confirm the expected functionality of the just-built software. These tests can be written and executed in a number of test frameworks, the most common at Exove being Robot Framework and Nightwatchjs, and the tests are executed automatically as scheduled after being uploaded into Jenkins into a specific project.

The timeliness of execution after a new build is important because code for recent software features can interfere with previously-tested and verified software to create regression bugs. Regression testing aims at verifying that older, confirmed software functionality still operates as expected. As opposed to functional testing, in which the code is executed manually in one or sometimes in a second pass, regression testing is executed in Jenkins or similar tool such as Hudson, after having been written as a script on an automated testing framework. Automated regression testing is done frequently, that is daily or even multiple times a day, to verify the continued reliability of older software. Quality Assurance is informed if a test in the automated regression test set fails, and investigation can quickly begin accordingly, to examine if the failure was an issue in the test itself or if the regression test set has found a bug in the software.

Test Creation Vs. Test Maintenance

It has been said that the key to successful test automation owes more to maintaining and updating tests just as much if not more than to the creation of the test scripts themselves. It is inevitable that, at some point during the regression test set execution, there will be a failure in one or more tests. it is at this point important for the QA Engineer in charge of test automation to be made aware as soon as possible that a failure has occurred, examine the tests to verify that the failure has been caused by either an error in the script or if a bug has been found. After this, the QA Engineer will either fix the script so the test runs successfully to verify software stability, or report the bug.

The term “test automation” can be misleading, because after script creation and successful execution on the Continuous Integration system, the QA Engineer must be aware always of the pass/fail status of each test of the regression set. A lot more awareness and tester action is needed that what the term “test automation” implies. Fortunately, there are tools available to inform by email, for example, if a failure has occurred in an automated test run, but QA should always review automated test execution results to know not just what might be failing, but also to know what tests are always passing and know how to update each test if any changes are required. In addition, more tests can always be added to examine the software in greater detail, and tests that are no longer relevant can be removed to make the set manageable and efficient. An inevitability of regression testing is the regression test set that has become so bloated with old and no longer relevant tests will cause failures. The large number of tests for often old and no longer relevant features makes investigation difficult, so to prevent this, timely test editing is necessary to maintain a manageable test set, the execution of which can inform of continued software reliability as well as quickly inform if bugs are found.

In conclusion, test automation, if tests are maintained and updated to reflect current functionality, can speed up the testing time and provide the team with information of the software’s stability and reliability. The tests should always be updated to check where previously undetected bugs may be lurking, and to provide additional tests that will show the system’s present stability.

See all posts

Read next

See all posts

More from the blog

See all careers

Related open positions

Current selection of careers

Our offices