Use functional and regression testing to validate SOA solutions

David W. Johnson



The service-orientated architecture (SOA) paradigm endeavors to address both the time-to-market challenge faced by business and the need for IT to develop, test, and deploy ever-evolving, complex solutions.







The tester must understand the relationships between the services as the business event unfolds and how to validate those services.












SOA breaks complex solutions into component parts (services) that present a simple interface to the application landscape while encapsulating both data and process. These services can be provided in-house, by business partners, and by commercial services. The most significant challenges presented by SOA, from a testing perspective, are that SOA application landscapes are "always on," continuously changing, loosely coupled, and usually involve multiple providers (in-house, business partners, and commercial services).

Finally, the quality of the individual services does not necessarily translate into the overall quality of the business solution. It is the quality of the whole that truly matters.

The focus of functional and regression testing is to validate that the SOA solution delivers the business functionality required.

SOA -– Example
In our previous article "SOA Unit and Integration Testing,"we presented a simple SOA solution: We looked at a simple and rather "coarse" (large complex services) example of an SOA application landscape. The SOA solution addresses the need to sell digital media online. Service layers consist of a Web-enabled presentation layer, customer account service, catalogue service, cart service, digital fulfillment service, customer history service, and an accounting service that interfaces to a standard financial services database. The following figure illustrates this SOA solution."




We will continue to use this as our basis for discussing SOA functional and regression testing.

We will address SOA functional and regression testing using a single business thread/event "Existing Customer Purchase". This event involves an existing customer logging on to the site, selecting articles from the catalogue, verifying the contents of the cart, purchasing the contents of the cart, and receiving the digital media.

SOA -– Functional and regression testing
Using keywords we can describe the "Existing Customer Purchase" event sequence and its supporting services:

Invoke presentation layer
Presentation layer
Customer logon
Presentation layer
Customer account service
Customer history service
Select catalogue articles
Presentation layer
Catalogue service
Cart service
Customer history service
Verify cart
Presentation layer
Cart service
Catalogue service
Customer history service
Purchase cart contents
Presentation layer
Cart service
Accounting service
Financial database
Fulfillment service
Customer history service
Verify digital fulfillment
Presentation layer
7. Even this simple sequence of events leads to questions. For example, why does "verify cart" require the "catalogue service"? Because the catalogue service provides the details that the presentation layer needs to present the articles in the cart to the user while the cart service tracks what is actually being selected for purchase. That means the tester must understand the relationships between the services as the business event unfolds and how to validate those services.

8. SOA -– "Black box" or user-centric approach
9. One approach to function testing SOA applications is to treat the entire landscape as a "black box" -- taking an interface-centric (user-centric) approach to testing. There are several advantages to this approach, not the least of which is its simplicity from a test planning, test design, test execution, and test automation perspective. As long as the services eventually feed back to a discernable user event, the event can be tested. The danger, of course, is that the service is not being tested, only "exercised" based on its and other services relationship with the user interface event.

10. The risks associated with that approach are directly related to how much of the SOA application landscape is in-house and the effectiveness of earlier unit and integration testing. If most of the services, especially the mission-critical ones, are developed in-house and are tested thoroughly during unit and integration testing, then the risk is minimal. On the other hand, if any significant portion of the SOA application landscape is not in-house, or the application development team is geographically (or politically) separated, then the potential for undetected defects to reach production increases dramatically.

11.SOA –- "White box" or request/response-centric approach
12. Functional testing of an SOA application service should entail providing requests to the service and then analyzing the responses received by the requestor to determine if they are correct. There are several challenges to triggering and then validating the request-response chain across an entire business event, not the least of which is the fact that testing tools have not yet fully evolved to meet this need. They have evolved to meet the integration testing challenge, but this is not the same challenge.

13. Testing at this level currently involves setting up manual "traps" across the request-response chain and then validating the process, message content, and data transformations that occur as the business event flows through the supporting services. As the description alludes to, this is an extremely time-consuming and technical activity that tends to defeat one of the most attractive aspects of SOA-driven business solutions -– time to market.

14.SOA -– Requirements-centric approach
15. This is really an extension of the "black box" or user-centric approach to testing SOA-driven applications. This approach leverages a requirements-based approach to function testing that minimizes the traditional risks of pure "black box" or user-centric approaches. The foundation being a set of well-articulated functional requirements from which both positive and negative test cases can be designed.

16. The function test must determine if each business event performs in accordance to the specifications, responds correctly to all conditions that may be presented by incoming events/data, and transports data correctly from one business event to the next (including data stores). It also must make sure that business events are initiated in the order required to meet the business objectives of the system.

17.SOA -– A blended approach
18. Currently the most effective approach to testing SOA application landscapes is to take a blended approach that leverages all three approaches. This requires an effective and ever-evolving test plan that leverages each approach appropriately -– the decision point being the risks associated with each service from both a business and architectural perspective.

19. The traditional risk-driven approaches to testing deal effectively with the business risks associated with each service (or service area). New to the mix are the architectural risks associated with any given service. Architectural risks have traditionally fallen into the purview of performance and load testing. The new challenge with service-orientated solutions is really "How does it fail?" This now becomes critical during function test because the failure or absence of even the most trivial non-critical service could cause the business event to fail.

20. As the SOA paradigm evolves, testing techniques and technologies will evolve to meet it. For now, rigorous adherence to development and testing standards combined with risk-driven test planning is our best road to success.