Uncategorized

Test Case Organization – interface alignment

Can you relate with this experience:

It’s the beginning of a release cycle, which includes a change to a feature which has existed in your application since before you were on the team.  From the first release of the application, your team has continued to develop and save test cases for regression testing, with the intent of keeping them updated.  The ever growing library of test cases are filed by release and feature.  After browsing through the library for some time without locating the tests for this feature, and with a deadline looming, you decide to develop your test cases from scratch, filing them as new feature tests for the current release.

The result of this approach is an unwieldy library of test cases, rarely referenced or reused.  I’ve come to believe the primary cause is the way in which test cases are organized and archived.  In most cases they are aligned by release.  This alignment favors and supports program and project management, over release and regression testing.  In comparison, the cost of testing is far higher than the cost of managing the overall release effort.

Beginning with the end in mind, the end-goal of any requirement and development effort is a feature utilized by an end-user.  That feature is most often part of a service or user interface.  The feature will be delivered as part of a release, but once delivered the feature is part of an interface.  So, the destination (the end) is the interface, NOT the release.

Since that is the case, what would the previous experience be like if the tests were developed and archived by interface rather than release:

It’s the beginning of a release cycle, which includes a change to a feature which has existed in your application for since before you were on the team.  From the first release of the application, your team has continued to develop and save test cases for regression testing, with the intent of keeping them updated.  The ever growing library of test cases are filed by interface.  After briefly browsing through the library to the interface being updated, with a deadline looming, you have plenty of time to review the existing test cases, make the necessary updates to align them with the new requirements, and annotate each with the current release and the requirement verified.

The result of this approach is an efficiently reusable library of tests suitable for both release and regression testing.  Each file will contain a history of each requirement verified, and release supported.  Accounting for the testing of all changes in a release will take locating and reviewing test files, but with this organization even that effort will be efficient.

Reconciling your current test library with this approach is best done over a period of time, starting with your current release.  The cost of transferring all existing tests to this new approach in one effort will be far higher than any benefit you may experience.  Instead, draw a line in the sand, leave the past behind, and develop just enough testing to support your current release effort.  For a time there may be need to reference previous tests during regression efforts, but this need will fade.

Be reconciled!

Validation Design – a more comprehensive approach

Habit #2 of Stephen Covey’s, “The 7 Habits of Highly Effective People” says,

Begin with the end in mind

The end of a well written test case is the validation.  And in my experience, beginning test development with the validation helps inform the development of the other phases of the test case — initialization and execution.

The most common practice in manual testing is to write a separate test case for verification of a specific requirement.  Each “passed” test case represents alignment of the application with a very specific requirement.  For manual testing, this approach makes sense, ensuring accurate and comprehensive accounting of each requirement.

However, this approach fails to leverage the computers ability to quickly and accurately verify any number of requirements, without the risk of overlooking a single one.  Though there is benefit to the speed at which the computer runs tests written in a manual style, automating tests using this approach will never unleash the full potential of the computer.

It is a widely held belief that a well written test verifies one and only one requirement.  To fully realize the potential of the computer, this definition must be expanded for automated testing: a well written test validates that the current state of a specific entity meets all related requirements.  Examples of entities are: web page, database record, or browser cookie.

Consider it this way: Given the related systems in an initial state, When a path is executed through the application to a specific destination, Then the state of a specific entity can be validated.  Combined with a modular test design, this approach supports well agile approaches like Business Driven Development (BDD).

Rather than numerous test cases using equivalent test data, navigating the same path through the application, with each verifying a very specific requirement, instead a single test can use one set of test data in an initial state, execute the necessary path, and validate any number of requirements for an entity.  The validation of an entity can be described using an individual verification for each requirement, making it easier to use the requirements to reconcile the actual results.

Reconciling your method of test development to this approach involves starting each test case with the validation statement.  In automated tests cases this line will look something like “expected.validate(actual);”, with the validation details encapsulated in the validate() method.  In manual tests, this line will be something like “Validate [web page]”, with a reference to the validation details.  Continue writing each preceding line of your test until the test data definition is in place.  For many, this will seem awkward at first, but I encourage you to follow this approach strictly for at least one software delivery cycle, then evaluate the overall results of this method.

Be reconciled!

Test Phases – the first step on a path to more resilient, efficient and informative test libraries

Having worked as an SDET on several great software development teams, I’ve had the opportunity to become familiar with various approaches to automated UI testing.  The truth is, there isn’t much variation at all.  The common approach is to develop a library of scripted automated test cases identical to a library of scripted manual test cases.

It may seem there is benefit to having the computer perform the same test steps that would be run manually, but there are heavy maintenance costs, and limitations to reporting and alignment with acceptance criteria.  This approach also fails to maximize the ability of the computer to perform a large number of validations in a minimum number of steps, providing accurate, detailed and meaningful results.

Every test case, whether manual or automated, includes three main phases:

  1. identify/create data to be used for the test
  2. navigate the necessary path through the UI using the identified data
  3. validate by comparing the actual results to the expected results.

These three phases are recognized in The Agile Alliance Guide which recommends their use in writing Acceptance Criteria for User Stories.  That guide uses the terms Given, When and Then, which are growing in use.  This format improves the readability of manual tests within a library and reveals commonalities, but there is little that can be done to take full advantage of identified commonlities.

Though some automated test libraries apply this format to group test steps and provide some code reuse, they tend to fall short in providing effective reuse at the level of the common test phases: initialization, execution, and validation.  Developing modules, if you will, at this higher “phase” level makes it possible to build a more resilient and efficient library of tests that provide more detailed and useful test results.

When tests reuse Initialization modules, reliable test data can be provided without the overhead of maintaining test-specific data.  When tests reuse Execution modules, resilient paths through the application can be maintained with minimal effort.  When tests reuse Validation modules, verification of specific requirements are readily available for review.

This modular approach results in much more thorough verification in fewer test executions, and a library of tests that invites change, making it much more responsive in an agile environment.

You can begin reconciling your method of test case development to this approach, whether working with an existing test library or starting from scratch.  The first step is to label clearly the three phases as they exist in tests you review or develop.  The comments “Initialization”, “Execution” and “Validation” (or if you prefer, Given, When and Then) can be used.  Doing so will provide the information needed to take this approach to the next level.

Be reconciled!

Welcome!

The variety of things I have wanted to publish online caused me to put off posting to my blog.  I was also concerned about starting a blog without a clear purpose.  Recently, however, my daughter started The Rusty Dog, a blog about her adventures with foster dogs (which I HIGHLY recommend, even for those not interested in dogs), which has inspired me to get started.

I struggled for quite a while, wanting to begin with a topic that establishes well the cornerstone of my beliefs, However, I found that virtually impossible.  Every belief is supported by many others, is only understood well in the context of the rest; they only hold together as a whole.  Reading my posts will be like watching an artist work on a huge canvas, moving from one seemingly random section to another, painting just enough in each to somewhat make sense on its own, leaving a feeling there is more yet to be displayed.  Only as section grow in size and detail, intersecting with others, does the observer begin to truly understand what the artist saw from the very start.

I now believe we are all like this: others only truly know and understand us if they have shared many varied life experiences with us, over time.  So my blog will contain many varied topics, from philosophy and theology, to family and relationships, to jobs and software development, and much more.  However, I believe there is one common theme that will run throughout my posts, regardless of the topic: reconciliation.  I believe all life is about reconciliation.  Hence, the title of my blog, “Life Reconciled”.

While I invite questions, comments and suggestions, and will be willing at times to “defend” my beliefs, this is primarily a venue for those interested in what I have to share, not for comparing or scrutinizing various beliefs.  My hope is my blog will be a resource for family, friends, acquaintances and future generations to learn who I truly am; for me to pass on the little understanding, wisdom and knowledge I have collected through my short life here on earth.

I hope you choose to visit often, as it will give me the opportunity to get to know you, as well.

Be reconciled!