New Generic xUnit Automated Testing Framework Integration Re

Overview

When you run automated tests using a framework such as JUnit, the test results are often generated in a standard XML format known as the xUnit report format. This XML file contains a list of the test cases that were executed, the results that occurred, and potentially a list of associated screenshots or other attachments.

A sample report file looks like this:

<?xml version="1.0" encoding="UTF-8"?>
<!--
This is a basic JUnit-style XML example to highlight the basic structure.
-->
<testsuites time="15.682687">
    <testsuite name="LIS.Registration" time="6.605871">
        <testcase name="registration1" classname="LIS.Registration" time="2.113871" />
        <testcase name="registration2" classname="LIS.Registration" time="1.051" />
        <testcase name="registration3" classname="LIS.Registration" time="3.441" />
    </testsuite>
    <testsuite name="LIS.Authentication" time="9.076816">
        <testsuite name="LIS.Authentication.Login" time="4.356">
            <testcase name="login1" classname="LIS.Authentication.Login" time="2.244" />
            <testcase name="login2" classname="LIS.Authentication.Login" time="0.781" />
            <testcase name="login3" classname="LIS.Authentication.Login" time="1.331" />
        </testsuite>
        <testcase name="auth1" classname="LIS.Authentication" time="2.508" />
        <testcase name="auth2" classname="LIS.Authentication" time="1.230816" />
        <testcase name="auth3" classname="LIS.Authentication" time="0.982">
            <failure message="Assertion error message" type="AssertionError">
                Resource: Login Page
                File: /Login.aspx
                URL: https://www.libraryinformationsystem.org
                Login failed with error message: XYZ
            </failure>
        </testcase>
    </testsuite>
</testsuites>

Our new xUnit Framework Integration utility will read and parse the xUnit XML file and import the test results into Spira using our standard APIs. We have written this initial version using Python and published it on the Inflectra PyPI download site. Our plans also include creating a NodeJS version. Over time, we plan on adding samples that show using this new tool with specific frameworks such as Playwright and WebDriverIO.

Key Benefits:

Universal Framework Compatibility: Works seamlessly with any testing framework that generates xUnit report formats, ensuring wide-ranging applicability and giving freedom to users to deploy any preferred testing tool without worrying about compatibility with Spira.
Streamlined Quality Management: Automatically integrates test results from diverse frameworks into Spira's platform, thus saving time, reducing complexity in managing test results, and enhancing focus on development.
User-Friendly Design: Implemented as a Python module for ease of use, with plans for NodeJS version expansion. This will simplify the integration process, making it accessible to teams of all skill levels.
Advanced Functionality for Experts: Offers additional features for complex mapping and build generation in Spira, providing detailed insights and control for advanced users.
Open Source Community Support: The source code is available on GitHub, inviting collaboration and continuous improvement and supporting community involvement, leading to innovative approaches.
Aligned with Inflectra's Mission of helping customers deliver better software efficiently.

Using the Integration

The integration consists of a Python module called spira_xunit_reader.py that is executed after your xUnit tests are executed and the appropriate XML report file has been created. This module will then read the results in the xUnit report file and send the results to Spira, mapping each xUnit test case to a matching Spira test case.

For advanced users, you can also optionally map the xUnit test suites to corresponding Spira test sets and/or have the plugin generate a new build in Spira that contains all of the executed test cases and test suites.

First, run your tests using your existing framework code. That will generate an output file. We have two supplied example files you can use:

junit-basic.xml -- this is a minimalistic jUnit style example that contains several test cases nested in test suites with one test failure. It uses only the minimum set of attributes.
junit-complete.xml -- this is a more advanced, complete example that includes file attachments, URL attachments, errors, warnings, failures, and additional optional xUnit attributes such as assertions, properties, and standard error/output streams.

Next, you need to run the Spira results parser module spira_xunit_reader.py to upload the results to Spira. For example, with the sample tests, you can use either:

python spira_xunit_reader.py samples\junit-basic.xml spira.cfg

python spira_xunit_reader.py samples\junit-complete.xml spira.cfg

The second parameter is the location and name of the Spira configuration file spira.cfg. Once you run the XML parser, you should see a message similar to the following:

Sending test results to Spira at URL 'https://myserver/spiraservice.net'.
Successfully reported 4 test cases to Spira.

If there are any errors or warnings, they will be displayed instead.

Viewing the Results in Spira

Once the reporting of results has finished, you can view the results in Spira by going to the Test Runs tab. That will display the list of test runs:

If you click on one of the failed test runs, it will display the name of the test case from the xUnit XML file, together with the status (fail), the test set, release, and actual duration:

Scrolling down to the Console Output section displays the full name and path of the xUnit test case, the details of what caused the test to fail, and any test case properties or standard output/error streams.

This plugin will also import any screenshots or other file attachments if they are part of the report file: