Overview
A month ago, we received a request to extend our 30-day free trial of Rapise from a programmer, who was reviewing test automation tools as a proof-of-concept in the acquisition of automated UI testing software for Microsoft Dynamics AX in the context of a government procurement decision-making process. The review focused on three top tools: Visual Studio Coded UI, Rapise and AXeptance and compared them along the following five parameters:
- functionality
- ease of use & maintenance
- date-driven testing
- reliability
- speed.
We are presenting the review in its original form sans all identifiers of the government office for which it was developed.
Let’s take a closer look.
Functional. The UI tests have the desired results. This is the bare minimum of what a test automation software should be able to do.
Easy to create, to run and to maintain. The UI tests were somewhat easy to create, but more difficult to maintain. In particular, controls in DAX change properties constantly: a table cell becomes a text box when you click on it, the same window can have different titles, etc. This makes it difficult for Visual Studio’s smart match to find the controls. Maintenance of the tests poses an extra challenge: every time a step is re-recorded or a new step reuses a control, the controls used are added again to the UIMap, making it large and messy. Search properties for controls cannot be changed within the code, only through forms in VS. Tests can be run in sequence with acceptable feedback from Visual Studio; integration with Microsoft Test Manager is possible but was not tested.
Data-driven. When creating tests from the data from a file, a certain amount of hand-coding is required: for every method recorded, a method to add parameters is automatically generated and we can pass it a data column by adding a line of code. Adding data-driven code to the tests causes problems with smart match: rows in a table (such as Lines) are identified by their content and will not be recognized by VS if the tests are run in a different context or with different data.
Reliable. VS Coded UI Tests are not reliable because of many dependencies on the test environment. It is partly the fault of AX: it remembers which tabs are open or closed between transactions, for example, which can hide key controls from VS. On occasion, it is unclear why a test runs successfully one moment and fails the next: often, controls that were accessible before are no longer found by smart match Finally, user’s customizations in AX may cause certain tests to fail.
Quick. The execution of the tests is slow because of long pauses during difficult smart matches. Often, test time is comparable to the time it would take a human to execute the task: for example, the creation of a purchase requisition takes approximately two minutes.
Prerequisites:
- All tabs in the Purchase Requisition Line Details form must be closed before testing.
- The first test must be initiated from the Home page.
Likelihood to recommend Visual Studio Coded UI Testing to automate UI testing in Dynamics AX: 3/5
Functional. The UI tests created with Rapise do fulfill their goals.
Easy to create and to maintain. UI tests are more or less easy to create with Rapise. The record tool is occasionally buggy but is obvious when an action was not recorded (the problem probably lies with AX). Some controls in AX cause issues, especially tabs and text boxes, but there are efficient workarounds which are time-consuming to discover but then easy to use. Inflectra’s customer support is quick and very useful in this regard. Hand-coding in JavaScript is required, sometimes a few lines at a time. The tests can be difficult to maintain if there are major changes in UI, but this is not expected. Finally, controls are located using window names, which are dependent on the details of a PR or PO: the window name and window text must be changed by hand in the object properties using regular expressions to account for these variances.
Easy to run. The tests can be run in Rapise but the UI is not designed to do this efficiently: Inflectra sells another product, SpiraTest, to run, manage and report on tests, but this application was not tested here. Tests can be grouped together into “Test Suites” to run them sequentially.
Data-driven. It is easy to create an Excel spreadsheet to use for data-driven testing with Rapise. One or more lines of hand-coding are required, but Inflectra’s Knowledge Base provides all the information needed. Using different data does not affect a test’s reliability. However, one Excel file is required for each test.
Reliable. Rapise is reliable: tests executed in identical conditions yield identical results (unless AX fails to recognize a button click, as can happen with a human user). However, as can be seen above, the use of personalization within AX will cause the tests to fail. A controlled testing environment is required if the tests cannot be run on cleared usage data.
Quick. Rapise executes tests much more quickly than humans. It is three times faster than VS Coded UI.
Likelihood to recommend Rapise to automate UI testing in Dynamics AX: 4/5
Functional. The UI tests created with Axeptance do fulfill their goals when they can be recorded successfully.
Easy to create and to maintain. UI tests are easy to create in Axeptance when they are simple; more complex tests require extensive workarounds, some of which are detailed in Axeptance’s 120-page user guide. There are more workarounds required to create Axeptance tests than for the other tested programs.
Easy to run. Tests are easy to run but the program itself is finicky.
Data-driven. Axeptance allows the user to input data within the program in the form of a list or a range.
Reliable. Axeptance is not reliable.
Quick. Axeptance executes tests more slowly than Rapise.
Additionally, Axeptance requires the user to install it directly into Dynamics AX as a module where it will overwrite certain classes and add label files that do not load reliably. A proper testing tool should work “outside of the machine”, just like the user does.
Likelihood to recommend Axeptance to automate UI testing in Dynamics AX: 2/5
Do you know of other tools that should have been included in this review?
