Validation Design – a more comprehensive approach

Habit #2 of Stephen Covey’s, “The 7 Habits of Highly Effective People” says,

Begin with the end in mind

The end of a well written test case is the validation.  And in my experience, beginning test development with the validation helps inform the development of the other phases of the test case — initialization and execution.

The most common practice in manual testing is to write a separate test case for verification of a specific requirement.  Each “passed” test case represents alignment of the application with a very specific requirement.  For manual testing, this approach makes sense, ensuring accurate and comprehensive accounting of each requirement.

However, this approach fails to leverage the computers ability to quickly and accurately verify any number of requirements, without the risk of overlooking a single one.  Though there is benefit to the speed at which the computer runs tests written in a manual style, automating tests using this approach will never unleash the full potential of the computer.

It is a widely held belief that a well written test verifies one and only one requirement.  To fully realize the potential of the computer, this definition must be expanded for automated testing: a well written test validates that the current state of a specific entity meets all related requirements.  Examples of entities are: web page, database record, or browser cookie.

Consider it this way: Given the related systems in an initial state, When a path is executed through the application to a specific destination, Then the state of a specific entity can be validated.  Combined with a modular test design, this approach supports well agile approaches like Business Driven Development (BDD).

Rather than numerous test cases using equivalent test data, navigating the same path through the application, with each verifying a very specific requirement, instead a single test can use one set of test data in an initial state, execute the necessary path, and validate any number of requirements for an entity.  The validation of an entity can be described using an individual verification for each requirement, making it easier to use the requirements to reconcile the actual results.

Reconciling your method of test development to this approach involves starting each test case with the validation statement.  In automated tests cases this line will look something like “expected.validate(actual);”, with the validation details encapsulated in the validate() method.  In manual tests, this line will be something like “Validate [web page]”, with a reference to the validation details.  Continue writing each preceding line of your test until the test data definition is in place.  For many, this will seem awkward at first, but I encourage you to follow this approach strictly for at least one software delivery cycle, then evaluate the overall results of this method.

Be reconciled!

Test Phases – the first step on a path to more resilient, efficient and informative test libraries

Having worked as an SDET on several great software development teams, I’ve had the opportunity to become familiar with various approaches to automated UI testing.  The truth is, there isn’t much variation at all.  The common approach is to develop a library of scripted automated test cases identical to a library of scripted manual test cases.

It may seem there is benefit to having the computer perform the same test steps that would be run manually, but there are heavy maintenance costs, and limitations to reporting and alignment with acceptance criteria.  This approach also fails to maximize the ability of the computer to perform a large number of validations in a minimum number of steps, providing accurate, detailed and meaningful results.

Every test case, whether manual or automated, includes three main phases:

  1. identify/create data to be used for the test
  2. navigate the necessary path through the UI using the identified data
  3. validate by comparing the actual results to the expected results.

These three phases are recognized in The Agile Alliance Guide which recommends their use in writing Acceptance Criteria for User Stories.  That guide uses the terms Given, When and Then, which are growing in use.  This format improves the readability of manual tests within a library and reveals commonalities, but there is little that can be done to take full advantage of identified commonlities.

Though some automated test libraries apply this format to group test steps and provide some code reuse, they tend to fall short in providing effective reuse at the level of the common test phases: initialization, execution, and validation.  Developing modules, if you will, at this higher “phase” level makes it possible to build a more resilient and efficient library of tests that provide more detailed and useful test results.

When tests reuse Initialization modules, reliable test data can be provided without the overhead of maintaining test-specific data.  When tests reuse Execution modules, resilient paths through the application can be maintained with minimal effort.  When tests reuse Validation modules, verification of specific requirements are readily available for review.

This modular approach results in much more thorough verification in fewer test executions, and a library of tests that invites change, making it much more responsive in an agile environment.

You can begin reconciling your method of test case development to this approach, whether working with an existing test library or starting from scratch.  The first step is to label clearly the three phases as they exist in tests you review or develop.  The comments “Initialization”, “Execution” and “Validation” (or if you prefer, Given, When and Then) can be used.  Doing so will provide the information needed to take this approach to the next level.

Be reconciled!