Software Test Automation

Automation is the buzzword in the entire QA spectrum. Firms in India and abroad have got a new outlook on how the QA efforts can be saved by automation. Some companies have begun to mature in this field and utilizing automation in most effective ways. For a list of myths related to Automation, read http://www.satisfice.com/articles/test_automation_snake_oil.pdf



It is important to educate the senior management about the expectations from Automation. Identifying what to automate and when to do it is as important as it is to plan the entire effort. Managing an automation project is similar to managing a software development project. The most critical part is to write maintainable scripts – scripts which can be reused across multiple releases of the same product with slight modification. That requires defining a set of guidelines before automation begins. Here are a few tips on how to get started with an Automation project:

The Manager and the entire team should attend trainings on the product and become familiar with the product functionality to the maximum. One of the reasons for the automation efforts going waste is the scriptor’s lack of knowledge of the component which is being automated.
Next the team should conduct brainstorming sessions on what can be automated and when. The outcome of all such sessions would be a detailed Automation Master Test Plan
By the time the planning activities are in progress, the team members should attend all the meetings associated with the component for which they are assigned – SRS Review meeting, DDS review meeting or any other meetings which could be with Manual testing team. This gives an insight on what can be automated and in which stages
Another activity which goes in parallel would be designing the Automation framework. The framework is one task which would be allocated a majority of effort, since it forms the crux of entire automation effort in future. The framework should be designed in such a way that it could handle all the ways in which an application could fail during execution of scripts and still recover smoothly to execute the next set of testcases. This is known as the Recovery System. At the same time, the framework should be flexible enough to accommodate any changes into the application functionality. Since Automation scripts would always need to be up-to-date with the application’s functionality.
Next comes the automation of scripts – how to get started? A senior member of the team should automate a few testcases for a particular area and demonstrate the same to entire team. The team should ensure that the automation has been done in accordance with the guidelines defined for automation of scripts.
Once every member is involved in writing automated scripts, two steps need to be performed:
Peer review of scripts every week or fortnight.
A check-in mechanism needs to be defined using some Version Control System such as VSS or CVS, so that scripts are always up-to-date. One member in the team should be assigned this responsibility of Check-in Administrator. Before every check-in the Admin has to ensure that the check-in does not break the existing framework.
Be in regular contact with the Manual Test Engineer and get the scripts reviewed from him. It is important to gain their trust.
It is not important to automate everything, but it is most important to ensure that whatever is automated should be in working condition on the current build. Due to this reason, the maintenance of scripts is a constantly on-going task.
There are many times when the scripts fail due to UI/functionality changes and not due to bug in the code. Those are trying times for Automation Engineers and they need to cope with all such changes. The Manager should ensure the motivation levels of the team are always up in such cases and a failure due to a bug in the software should never go unnoticed
To keep the automation efforts visible to management, it is important to produce some deliverables at different stages during the SDLC. The deliverables could be in the following order:
Automated set of Smoke testcases – a set of high-level testcases which verify that the basic functionality of the product is correct. Once the smoke tests are automated, they should be executed on every weekly build (some places it is also done on Daily Builds to identify the side-effects of any check-ins). Such tests could include – UI tests, Command-line. For tests which cannot be automated, maintain such a list separately so that they could be executed manually on every build. The results of execution should be published to all concerned teams – including Dev, PMG and QA
Automated set of command-line tests – Such tests one automated are resistant to change. Since the functionality is command-line which is not expected to change frequently, maintenance of such testcases requires minimal effort
Automated set of Functionality tests – This set should be automated based on the priority. For example – we associate a rotation with every testcase – R1, R2, R4, R8, R12, R16. R1- smoke test (Every 1 week); R2,R4,R8 – High priority functionality tests; R12,R16 – Low priority, low risk functionality tests.
Automated set of Regression tests – This requires keeping a track of all critical defects. A critical defect could be a showstopper (P0), P1 or even a P2. I always recommend to automate all P1s, selected P2s and selected P0s. This kind of Automation would always give a high visibility, since no manual tester would regress his fixed defects again. An execution report of such testcases could be published every fortnight or so.
Automation of workflows, shortcut keys, localization tests, performance tests