Enhance Data-Driven Testing with TestConfigurations+ SpiraAp

Background: Existing Test Configuration Functionality

We released the test configurations functionality in SpiraTest back in v5.3 in July 2017. which allowed customers to create pairwise datasets and associate them with test sets. This lets customers create a data-set of possible testing configurations and run their test sets through all the different combinations with ease.

For example:

Scenarios: Testing combinations of 5 browsers, 4 operating systems, and 10 user logins (200 total configurations).
Capabilities: Test sets could execute all combinations automatically, streamlining manual and automated testing workflows.

However, the current functionality has some limitations, which we have just addressed with our new TestConfigurations+ SpiraApp add-on.

Overview of the Current Functionality

The data-driven testing functionality built in to Spira lets you create independent lists of values as custom lists. For example, you can have a list of web browsers and operating systems as custom lists:

(and similarly for web browsers)

Then you can create a test configuration set that maps these values to test case parameters ${browserName} and ${operatingSystem} that accept the different values in each test execution.

For manual tests, these parameters are displayed in lieu of static text allowing you to dynamically change the sample data that the tester sees. For automated tests these parameter values are passed directly to the automated testing tool as input test data.

Normally, you will associate this set of test data with a test set that contains the list of test cases to be executed:

However, there are some limitations in the current functionality:

When you execute the test set, it will execute each test case in turn (e.g., TC4, TC6, TC12, TC13 in turn) for a single test configuration (e.g., Windows 11 + Edge). Some customers would prefer the grouping to work the opposite way, where for a given test case (e.g., TC4), you execute it sequentially with all possible data combinations before moving on to the next test case in the set.
There is no way to change the ordering of test cases per data combination, i.e. it will execute them in the same order (TC4, TC6, TC12, TC13) for each browser/platform, whereas you might want to change the order for certain combinations.
Since all test runs are ultimately executions of the same test cases, the default test set reports and dashboard widgets will only show the results of the most recent data combination, not the complete set of all possible combinations. This can give misleading results when you want to make sure that a test set has passed for all possible combinations.

What the New SpiraApp Provides

To address these limitations and provide more flexibility, the new SpiraApp lets you generate a new static test set from an existing test set linked to a test configuration. This means you can easily reorder, edit, or remove test cases from the new test set to fine-tune your testing while maintaining the rich and dynamic parameters provided by test configurations. It takes a Test Set that is linked to a Test Configuration Set and will generate a new test set that contains all of the permutations converted into discrete test set test cases with individual parameters set. In addition, SpiraApp lets you choose whether to group by test case or test configuration, solving the grouping issue as well.

Once you have installed and activated the TestConfigurations+ SpiraApp, go to the SpiraApp product settings:

By default the population of test cases in the generated test set will be grouped by test case then by test data configuration. By checking this box you will switch the grouping to be by test data configuration then test case. We will show you how it looks in either direction.

When you click on an existing test set that is mapped to a test configuration set, you will see the new option to Populate Test Cases:

When you click this button, the SpiraApp will generate a new test set that is a clone of this one that contains all of the test cases expanded out by each unique parameter data combination:

Note how each test case is listed six times, once for each test configuration combination. The parameter values are now visible in the parameters column per test case.

If we changed the grouping setting to Group by Configuration = Yes, we would have seen the following instead:

In this case, you can now see the same four test cases repeated for each test configuration.

In either case, you can now fine-tune the test set to remove any invalid cases and/or adjust the execution order. You can also override the parameter values for specific combinations if necessary, giving the test manager maximum flexibility.

Key Benefits

Improved Flexibility: Tailor test sets to match your exact needs.
Better Organization: Group and reorder test cases based on specific testing scenarios.
Accurate Reporting: Ensure all combinations are accounted for in dashboards and reports.

How to Get Started?

Download the App:
- Visit the TestConfigurations+ SpiraApp in the Spira download center
Install on Spira:
- Once you have downloaded the SpiraApp, you can upload it into your instance of Spira (cloud and on-premise are both fully supported).
Activate and Configure:
- Follow the setup instructions to enable advanced test configuration management.