Search This Blog

Welcome to Machers Blog

Blogging the world of Technology and Testing which help people to build their career.

Thursday, August 7, 2008

Test LifeCycle Management Using VSTS

Test Life Cycle Using VSTS for Software Testers and TFS
Prepared for
Microsoft Global Services India (MGSI)
Wednesday, 19 September 2007
Version 0.3
Prepared by
Mahantesh Pattan (Monty)
Senior Test Consultant
mapattan@microsoft.com
Contributors
Rajagopalan Ganapati (rajagg@microsoft.com) – Test Consultant
Prepared for MGSI
Page ii
Test Life Cycle Management Using VSTS Test and TFS for, Version 0.3
Prepared by mapattan
Table of Contents
1 Introduction ......................................................................................................................................... 1
1.1 Abstract .......................................................................................................................................... 1
1.2 Scope ............................................................................................................................................. 1
2 VSTS – TFS Test Life Cycle Management ......................................................................................... 2
2.1 Test Project Planning with Microsoft Project & TFS Integration ..................................................... 3
2.2 Test Management (Test Case & Results) ..................................................................................... 3
2.3 Defect Management ...................................................................................................................... 7
2.4 Build Validation Integration with TFS Test Manager ...................................................................... 8
2.5 Functional Test Automation ............................................................................................................ 9
2.6 Performance Testing .................................................................................................................... 10
2.7 Test Project Tracking using TFS Reporting Services ................................................................... 12
Prepared for MGSI
Page 1
Test Life Cycle Management Using VSTS Test and TFS for, Version 0.3
Prepared by mapattan
1 INTRODUCTION
1.1 Abstract
Visual Studio 2005 Team Edition for Software Testers (VSTS) & Team Foundation Server comes with a set
of tools and features that enable test management in various ways for Test Case Management, Result
Management, Test Automation, Performance Testing, Test Progress tracking etc.
This document is created to give some guidance around best practices for using VSTS-TFS suite of tools for
overall test life cycle management to whatever extent supported & feasible as of now in VSTS 2005.
1.2 Scope
The scope of this document would be restricted to the best practices followed in MGSI for test life cycle
management. There could be lot of features and ways around to overcome limitations of out of the box
features of VSTS 2005 - TFS for Test Life Cycle Management but this paper talks about the only features
and methodologies followed in MGSI. This document can be treated as a quick starter guide to get a high
level understanding of Test Life Cycle Management using VSTS-TFS tools, however detailed
implementation guidelines would be out of scope in the current context.
Prepared for MGSI
Page 2
Test Life Cycle Management Using VSTS Test and TFS for, Version 0.3
Prepared by mapattan
2 VSTS – TFS TEST LIFE CYCLE MANAGEMENT
The above diagram illustrates the implementation of VSTS-TFS Test Suite of Client Server tools to cover various
phases and activities of Test Life Cycle Management.
Using VSTS for Software Testers in integration with Team Foundation Server and couple of other tools like MS
Project, Process Template Editor for TFS along with some customizations we can cover most of the critical
phases of Test Life Cycle like,
1. Test Project Planning (WBS Test Task Planning)
2. Test Case & Results Management
3. Defect Management
4. Build Validation Tests Integration with TFS Manager
5. Functional Test Automation
6. Performance Testing
7. Test Project Tracking using TFS Reporting Services
Prepared for MGSI
Page 3
Test Life Cycle Management Using VSTS Test and TFS for, Version 0.3
Prepared by mapattan
In overall TLC (Test Life Cycle), however all the activities described here are not supported by VSTS-TFS out of
the box, thus these limitations can be overcome using some customization flexibility of these tools with best
practices around the methodologies and specific processes.
2.1 Test Project Planning with Microsoft Project & TFS Integration
In TFS, work items are created for every activity which needs tracking for Project Management. Thus TFS
Work Item tracking can be used for Test Task Allocation & Progress Tracking.
Test Project Planning for Tasks, Schedules can be created using Microsoft Project or Microsoft Excel. TFS
Provides a facility to import all the tasks defined in an excel document or a MS Project MPP file into TFS by
which the tasks defined in excel or MPP are converted into work Items which helps Work Item Creation
Process easier.
2.2 Test Management (Test Case & Results)
Team Foundation Server comes up two types of Process Templates (MSF & CMMi) and each Process
Template is composed of different types of work item types such as Task, Defect, Risk, Scenario, Quality of
Service Requirement etc but in order to track test cases, a new Work Item type needs to be created with
type name as Test Case which will help Testers in creating and managing Test Cases.
Each Work Item type definition with Work Item Fields, Rules of workflow, security constraints is all defined in
an XML file. Thus to add a new work item type like Test Case and Test Results, one needs to create XML
files for these two work item types and import them in TFS Project. To create these work item types and
editing the TFS Process Template (MSF / CMMi), a Power tool “Process Template Editor” can be used
which can be downloaded from MSDN site. Once the Test Case & Test Results are created as Work Item
types in the form of XML, they are imported in TFS Project using the Power Tool.
Work items of types Test Tasks, Test Case, Test Results can be linked to work items used for requirements
management which gives a robust traceability as shown in below diagrams. This is one of the best practice
which can be followed to have better tracking of the quality of the solution at different stages.
Prepared for MGSI
Page 4
Test Life Cycle Management Using VSTS Test and TFS for, Version 0.3
Prepared by mapattan
In this way Visual Studio Team System with Team Foundation Server can be used as the test case & results
management tool. The default test type for test case management is Manual Test, provided by Visual Studio
Team System. VSTS does not impose a specific structure to the test case and test results tracking for
manual tests & is not similar to querying work items based on different parameters. These limitations are
overcome using two custom work item types, namely, Test Case and Test Result.
How to add a new test case?
Pre-requisites
_ Team Project is created in the Team Foundation Server
_ Custom work item type files corresponding to Test Case and Test Result are applied to the project as
mentioned in the above description
Steps
1. Open Visual Studio Team System
2. Click File > Open > Team Project
3. Select the project and click OK
4. Once the team project opens in Team Explorer, right click on the folder Project Name > Work Items
5. Click Add Work Item > Test Case as shown below
Figure 1: Add Test Case
6. A New Test Case work item, as shown below, is displayed
Prepared for MGSI
Page 5
Test Life Cycle Management Using VSTS Test and TFS for, Version 0.3
Prepared by mapattan
Figure 2: New Test Case
7. Save the Test Case after entering the test environment details and steps for test execution
8. Test Case is created with a unique id
How to add a test result?
Pre-requisites
_ A test case is present in the project
Steps
1. Open a test case by right-clicking on the folder Project Name > Work Items and selecting Go to Work
Item as shown below
Prepared for MGSI
Page 6
Test Life Cycle Management Using VSTS Test and TFS for, Version 0.3
Prepared by mapattan
Figure 3: Go to Work Item
2. Enter the test case id in the displayed pop-up and click OK
3. Once the test steps are executed, right click on the test case work item and select Add Related Work
Item > Test Result as shown below
Figure 4: Add Test Result
4. A new Test Result work item is displayed as shown below
Prepared for MGSI
Page 7
Test Life Cycle Management Using VSTS Test and TFS for, Version 0.3
Prepared by mapattan
Figure 5: New Test Result
5. Save the Test Result after entering the Value (Pass or Fail)
6. Test Result is created with a unique id and associated to the corresponding test case
Test cases will be maintained in Visual Studio Team System and test results will be tracked based on build
nos.
2.3 Defect Management
In Team Foundation Server, Defect Management is achieved by usage of work items of type Defect.
Again as described in the above sections, Defects can also be imported from an excel document into
TFS as work items. TFS Work Items of type defects can be linked to the development task work items,
test task work items or test cases to have better tracking but unless there is a streamlined process
defined and followed across development & test life cycle, it becomes a challenge to manage the work
item level links.
Test Life Cycle Management
Prepared by mapattan
There is no specific best practice around this except the above mentioned since here TFS is utilized as
a Defect tracking system similar to any other defect tracking tool except the advantage of having
facility to create integration between development, bu
enabling email notifications, testers who submitted defects
but this too needs manual configuration and is followed as a best practice. TFS helps in generating
good amount of reports on Defects as shown above to help understand the quality state of the solution
at a given stage enabling better decision making.
2.4 Build Validation Integration with TFS Test Manager
In case of automated builds, VSTS
along with the build process such that for every build, BVT is run automatically to know the health
status of the given build enabling in getting information if there are some issues with the build.
In this case, TFS Test Manager can be used which helps testers in creating test lists as shown below,
Testers can manage a large number of test cases using the Test Manager, group and sort tests using
attributes or categorize into lists to manage as a
with the build process such that Unit, BVT, Basic Performance Tests can directly be run in the build process
Using VSTS Test and TFS for, Version 0.3
build and test work items for better tracking. By
can be notified once the defects are fixed
VSTS–TFS can be used for automating the build validation tests as well
single artifact as Test List. This test list can be associated
Prepared for MGSI
Page 8
ild omating
Prepared for MGSI
Page 9
Test Life Cycle Management Using VSTS Test and TFS for, Version 0.3
Prepared by mapattan
such that test results captured with each build can help in tracking the Quality of the Solution over the period
of time.
2.5 Functional Test Automation
Visual Studio Team Edition for Testers provides couple of functional tests like Web Test, Ordered Test,
Generic Test, and Custom Tests. In these types of tests VSTS provides record/playback feature for Web
test and also provides automated C# script code generation called Coded Tests for the recorded web tests
such that testers can expand the scope of the test script and make it more robust and extensible. Using this
feature, testers can create scripts for web service testing as well.
VSTS Web test (as shown above) and Ordered Test will be used for automating only functional tests. Reusability
of functional tests as regression suite will be a primary objective.
Automation suite will be structured in such a fashion that the test scenarios will be broken down in to
multiple steps and each step is independently automated using VSTS Web Test and these steps will be
aggregated using VSTS Ordered Test to generate end-to-end scenarios. This incremental approach will
provide the flexibility for re-using Web Tests and reducing test suite maintenance overhead, since fixing any
errors or modifications required to the automated tests can be easily isolated to web tests that perform a
small number of user actions.
Parameterization and data-driven tests will be used as a guiding principle for developing automated tests.
VSTS Web Test records user actions at http level and therefore, parameterization of test variables, such as
url, port no. etc will enable tests to be run in different test environments by changing only the configuration
entry.
Data-driven tests will enable a single test to run with different set of input data. This will help maximize
automated test coverage when the input data selected is based on boundary-value analysis and
equivalence class partitioning.
Test Life Cycle Management
Prepared by mapattan
Test team will strive to automate as many as tests as possible with the goal of automating 40
tests.
2.6 Performance Testing
VSTS web tests can be used for functional as well as
in VSTS Load tests. In VSTS load tests, VSTS gets integrated with Performance Monitor enabling
Performance testers to add the counters as required from different servers (Database Servers, Web App
Servers etc). This tool is used to simulate production loads to diagnose performance issues quickly in
testing labs and pre-production environments. The inbuilt prescriptive guidance provides pre
counters, pre-set thresholds, pre-built views of load statis
built-in knowledge of server behavior.
The best practice in this case would be to decide the right set of monitors for system resources, Process
Counters for app specific processes of MOSS, IIS, SQL
generating custom reports for Load summary which can be used for root cause analysis of the failures
observed in the load tests.
VSTS Load test with web test provides validation and extraction rules which are criti
robust since the extraction rules help testers define rules to parameterize the dynamically generated data
avoiding failures of tests for applications which needs latest values of dynamic data and validation rules help
in setting validations at the end just the way any other test tool would do.
Using VSTS Test and TFS for, Version 0.3
for Performance Testing by adding the web tests
statistics and their effect on the servers under test and
Server etc. VSTS-TFS Integration helps in
lidations Prepared for MGSI
Page 10
40-50% of the
pre-packaged
tics critical to make the scripts
Prepared for MGSI
Page 11
Test Life Cycle Management Using VSTS Test and TFS for, Version 0.3
Prepared by mapattan
The above diagram illustrates a scenario of remote load testing which is typically done to create a real world
scenario where in requests will be coming to the server from different geographic locations. To do so in
VSTS Load test, testers needs to install multiple load agent tools in different machines with a supportive
load test agent controller in one box for administrating the load test. In this case the load test will reside in
the VSTS Load test client where the tester will configure and initiate the test such that the load agents will
generate load from different locations on the web application. All the test results are stored in TFS Data
Warehouse facilitating the testers to create detailed load test summary reports for Performance Test Results
Analysis.
In case Developers wants to do a code level profiling and understand the memory management and
performance of the code at unit level, it is recommended to create load tests by adding unit tests into load
tests and enable the VSTS Performance Analyzer which will help in doing detailed profiling of code at unit
level.
Prepared for MGSI
Page 12
Test Life Cycle Management Using VSTS Test and TFS for, Version 0.3
Prepared by mapattan
2.7 Test Project Tracking using TFS Reporting Services
TFS Reporting service provide couple of reports out of the box as shown below and also allows users to
create custom reports connecting to the TFS Data Warehouse for Project Tracking and Reporting. This
applies the same to Test Project tracking as well since Test Managers can derive analytical reports on Test
Task Progress, Defect Trends, Traceability between tests and defect, tests/defects and requirements based
on work items.
Below are few sample reports showcasing the best practices around developing custom reports in TFS for
Test Project & Defect Tracking,
Prepared for MGSI
Page 13
Test Life Cycle Management Using VSTS Test and TFS for, Version 0.3
Prepared by mapattan
TFS Reports
Requirements / Test Status Tracking
Prepared for MGSI
Page 14
Test Life Cycle Management Using VSTS Test and TFS for, Version 0.3
Prepared by mapattan
Task Tracking / Work Progress Reports
Where Do We Need To Shift
Resources?
Bulge in work in process (in testing)
indicates inadequate resources or
inadequate incoming quality
Test Effectiveness Report
Defect Statistical Reports
Prepared for MGSI
Page 15
Test Life Cycle Management Using VSTS Test and TFS for, Version 0.3
Prepared by mapattan
Prepared for MGSI
Page 16
Test Life Cycle Management Using VSTS Test and TFS for, Version 0.3
Prepared by mapattan
Issue Management Report

No comments: