SpiraTest, the Powerful Alternative to XRAY

Overview

XRAY is a popular test management plugin for Jira. It was developed as a plugin to Jira, available on the Atlassian marketplace for both Jira Server and Jira Cloud customers. Similar to other Jira native plugins Zephyr Squad and Zephyr Scale, XRAY uses the native Jira object model, data storage and user interface to provide the test management functionality. This means that it is a great initial choice for teams using Jira. However, as your testing needs progress and you need to perform more sophisticated testing, you will often run into its inherent limitations.

SpiraTest by contrast is a separate test management platform (available both in cloud and on-premise editions) that integrates seamlessly with Jira, but is not written completely as a Jira plugin. This means that it offers many advantages to a tool that is limited by being a Jira plugin. SpiraTest provides a unified platform to manage your manual, automated and exploratory testing, with real-time dashboards and reporting and end-to-end traceability.

Here are some of top advantages of SpiraTest over XRAY:

Unlimited Nested Test Case Parameters

Unlimited Storage of Test Data and Attachments

More Flexible Test Set Execution

Superior Reporting and Analytics

Significantly Improved User Experience

Improved Performance & User Experience

World-Class Technical Assistance

Full Compatibility between Cloud & Server Editions

Full, Robust API with No Limitations

License Flexibility, Including Concurrent Users

The Details

Here’s more information on why SpiraTest is your best choice for test management:

1. Unlimited Nested Test Case Parameters

Most types of software testing will involve testing with data, and consequently a key requirement is the need to be able to run the same test case with different combinations and permutations of data. In addition, you will often want to make your test cases modular, with different components (e.g. logging in, logging out) reused in multiple test cases. This requires the use of test case parameters and data-driven testing.

Due to the fact that it has to operate within Jira and use Jira issue types and associations to handle test cases, former customers tell us that XRAY has a limited set of parameters available, with only 20 discrete parameters available per-test case. Similarly, it will limit testers to having only 5-layes of nested test case. Some of these limitations are specific to the cloud version of XRAY.

By comparison, SpiraTest allows you to have an unlimited number of test case parameters in its test cases, with test cases being able to recursively call test cases an infinite number of levels.

We have designed SpiraTest to have no limits in general. Projects in SpiraTest can have unlimited test cases, unlimited test sets, unlimited test steps and unlimited test runs. Each of the test cases can have unlimited parameters, and the test cases can be nested to an infinite level.

2. Unlimited Storage of Test Data and Attachments

Software testing is as much an art as a science, so its often critical to have evidence of what was tested, and how it behaved. Similarly, they often say a picture is worth a thousand words; when it comes to testing, that is definitely the case. How many times have your testers been unable to reproduce an issue until they saw a screenshot or video of the problem in action.

Former customers of XRAY have told us that it is limited to having 50GB of storage for test case data and attachments. While this may seem like a reasonable amount of storage, once you start using it to manage large numbers of projects, you will quickly run up against that limit.

By comparison, SpiraTest comes with unlimited data storage, both of the core test case, test set and test run data itself, and any file attachments associated with the test artifacts. That way regardless of the number of projects and test cases, you will always have enough room for your critical data and testing evidence.

3. More Flexible Test Set Execution

Typically when you create a test case repository, you will want to compose them dynamically into different test sets (or test suites) that can be used for execution. This allows you to reuse the same test cases in different scenarios and use cases. To allow this to happen, the test management tool should let you pick and choose different test cases, combine into test sets / test suites and then execute these suites.

Former customers of XRAY have told us that the tool is limited to being effectively containers that organize the test cases (similar to test case folders in other tools), but they are not able to be used for test execution.

By comparison, SpiraTest lets you create a static organization of test cases based on folders, custom properties and freeform tags. These test cases can then be dynamically composed into different test sets from where they can be directly executed. The test sets are flexible enough to allow them to be used for both regression testing (where one person executes the entire set) or end-to-end scenario testing where different testers execute specific parts of the test set.

4. Superior Reporting and Analytics

Reporting and analysis is a critical part of the testing process. Test managers need to be able to see real-time data of their testing teams. For example, customers have told us that they want to be able to report across their organization using out of the box graphs and charts such as requirements test coverage status and test execution data such as time, steps, and number of tests executed.

Former customers of XRAY have told us that it has a limited selection of reports and graphs. If you want more than this, you need to create them yourself using the Jira reporting tools and the fact that XRAY is using standard Jira issue types.

By comparison, SpiraTest comes with an extensive set of reporting dashboards, both the initial “project home” and “program home” dashboards that give you a birds-eye view of the testing activities at both the project and program levels. This is supplemented by the fully-customizable reporting dashboard that includes both out of the box graphs/charts and user created ones. It also includes our powerful report writer that can generate documents, spreadsheets and other rich content. For additional reporting power, you can upgrade to SpiraPlan that includes an ODATA connector allowing you to use enterprise reporting tools such as PowerBI and Tableau.

5. Improved Traceability

Software testing is the art of making sure that the business risk associated with some piece of functionality has been adequately mitigated (to the point that any residual risk is acceptable for release). Therefore, one of the key ingredients of a test management tool is its ability to provide traceability to the source requirements to make sure there are no gaps. In addition, being able to see the links between a raised defect and the test that generated it will speed up reproducibility and verification that the issue has been fully fixed. Similarly in the age of supply-chain cyber-attacks, another aspect of traceability is to make sure all lines of code are appropriately traced to a business need.

Former customers tell us that XRAY only offers very basic requirements and test case traceability, with the plugin using a combination of Jira associations and issue ID tagging to handle the traceability needs.

By contrast, SpiraTest offers robust end-to-end traceability that lets you see the mapping between requirements and test cases, test cases and releases, test cases and defects, as well as associations between instances of the same artifact. The requirements traceability views make it easy to verify that all requirements have been adequately covered. The test case views let you easily see that all planned tests have been executed for a specific release, and the defects module lets you see the source requirements and test cases for each identified bug/defect.

For additional traceability, you can upgrade to the SpiraTeam or SpiraPlan edition to have such functionality as baselining, association versioning and source code-level traceability.

6. Improved Performance & User Experience

Testing is often the bottleneck for projects, so the time taken to manage and execute tests is a critical metric in most organizations. Therefore, the test management tool needs to be fast and responsive, with an intuitive user interface that makes it easy for business users and functional experts to record their test results.

Former customers have told us that the performance of XRAY (especially the cloud version) has not been optimal and that users are not particularly happy with the user interface. Some of this is inherent due to the fact that XRAY has to run as a plugin for Jira, and Jira was never designed to be a test management tool.

By comparison, SpiraTest has been designed for speed, performance and usability. With an emphasis on combining support for agile methodologies with excellence in quality engineering, SpiraTest was built from the group-up to do the job of test management. In addition, its seamless interface with Jira makes it easy for testers to work in SpiraTest and developers in Jira, sharing the data in real-time.

7. World-Class Technical Assistance

The features in a product only tell half the story. Software testing is a mission critical part of the software development lifecycle. When you have questions or need assistance, the company behind the product makes a big difference to whether you are successful or not.

Former customers have told us that they have been consistently dissatisfied with the timeliness and quality of support provided by XRAY. Part of the problem is the fact that Atlassian requires all technical support for marketplace plugins to be routed through their help desk and then triaged. This means that all support tickets for marketplace plugins will typically take several days to get answered, even for simple cases. For more complex cases, it could take over a week to get an answer.

By comparison, SpiraTest is backed by Inflectra, the world leader in customer service. We have a dedicated team of support professionals available 24 hours a day. Since we have our own direct support infrastructure, your tickets go directly from being logged to being assigned to a support specialist. We have a guaranteed SLA of one day turnaround, with most tickets being answered in an hour. If you read the testimonials for SpiraTest, you will see the that our exceptional support is one of the common themes.

8. Full Compatibility between Cloud & Server Editions

One of the trends in the past couple of years has been the adoption of cloud technologies, with companies looking to move from company-managed hardware in traditional data centers to some combination of Software as a Service (SaaS), Platform as a Service (PaaS) and Infrastructure as a Service (IaaS), which are the different types of cloud offering. With the evolving threat landscape as well as cost considerations, many companies are moving from a purely SaaS model to a combination of on-premise, private cloud, and public cloud. Even in the public cloud sphere, companies are adopting a mixture of SaaS, PaaS and IaaS depending on their needs.

Atlassian has made the determination that it’s moving to a completely SaaS cloud approach going forward, with its Server versions discontinued and all future development work happening on its Cloud version (vs. its legacy DataCenter product). This means that customers are being forced to move to the SaaS Cloud version regardless of their deployment preference. Complicating this choice is the fact that former customers have told us that XRAY’s cloud version is not the same as its Server/DataCenter version, with more restrictions, less functionality, worse performance and a different API. Former XRAY customers have also told us that moving from the on-premise version to the cloud was very difficult, with many data elements such as linked test cases, not migrating properly.

By comparison, Inflectra has made a public commitment to supporting its Cloud and On-Premise versions indefinitely, with all product updates released simultaneously. In addition, the user interface, plugin support, integrations and API are all 100% compatible across both versions. If a partner or customer creates a SpiraApp for the cloud version, it works on the on-premise version (and vice-versa). Finally, customers can migrate seamlessly from the cloud version to on-premise or vice-versa with no need for any professional services, it’s that easy.

9. Full, Robust API with No Limitations

One of the key non-functional requirements for any test management tool should be how easy is it to integrate with other products. That means it should have a wide range of connectors and integrations for other common development and project management tools as well as a wide variety of open source testing frameworks and commercial testing tools. Since many companies use in-house custom frameworks, or have very specific use cases, it should also have a well-documented, versioned API.

Former XRAY customers tell us that the API for accessing the data is complicated because it relies on using the Jira Cloud or Server API which sees everything as Jira issues rather than native test management objects. In addition, the Jira API is convoluted, with different versions for Cloud and Server, frequent breaking changes, and a v3 API that is in permanent beta and a v2 API that is legacy but still changing.

By comparison, SpiraTest comes with a robust, versioned API that has been designed with backwards compatibility and stability in mind. The API consists of 7 different versions with support for REST, SOAP and ODATA protocols. All modules in SpiraTest have their own dedicated API resources (test cases, requirements, test steps, test runs, test sets, etc.) that make it easy to integrate with. In addition, the API versioning means that code written today will continue to run in the future, even when there are major functionality changes.

10. License Flexibility, Including Concurrent Users

When all is said and done, one of the most important questions after looking at different tools for features, usability and flexibility is licensing and price. This is usually the final set of questions we get asked after every demonstration. Customers are looking for ways to manage their spend, keep costs under control, have predictability, and not spend money on licenses they are not using.

SpiraTest gives you the ultimate in flexibility in terms of licensing or pricing. Our cloud and on-premise subscriptions are displayed on our website, with current pricing available for all editions. You have the flexibility of monthly or annual options, and all licenses included unlimited named users, projects, test cases, storage and other artifacts. The license tiers are based solely on the number of concurrent users accessing the system. This lets you match the license level to your actual usage, and means you don’t need to have dedicated licenses for occasional users (e.g. an executive that just wants to view a dashboard or run a report).

XRAY on the other hand is limited to providing the same license options as Jira, since it is a marketplace plugin. That means you will need to purchase licenses for every named user in Jira, even if many of those users will not use XRAY at all. So if you have a Jira user base of 1,000 named accounts and you add XRAY, you will need to purchase 1,000 XRAY licenses. If your testing team is only 50 people, of which 20 would be working at peak usage, you’d only need a SpiraTest 20 concurrent user license.