November 6th, 2023 by Adam Sandman
As we described recently, we are planning on introducing many different forms of Artificial Intelligence (AI) into our products as part of the upcoming Spira and Rapise roadmaps. One of the first parts of executing on this AI strategy is the introduction of our OpenAI / ChatGPT SpiraApp plugin in our next release of Spira. In this article we illustrate some examples of this plugin in action, as well as highlight some possible areas for future enhancement.
Overview
In this examples, I have created a couple of different requirements in SpiraPlan. So that I could generate test cases, tasks, risks and BDD steps, I made sure that the requirements were of a type that supported BDD steps, and that the user I was logged in with had the permissions to create all four types of item. In each example, I used the new ChatGPT SpiraApp out of the box, with the default prompts and default parsing regular expressions.
Example 1: Reserving a Flight
Many of us have experience booking / reserving airline flights using websites such as Expedia, Travelocity, so I thought it would be an interesting example to write a simple one-line requirement "The ability to reserve a flight" and see what the AI model would generate:
Test Cases
The test cases that it generated seem plausible, covering different "happy path" and edge case scenarios:
| Test # | Name | Status | Est. Duration | Last Execution Date |
|---|---|---|---|---|
| TC391 | Verify that a user can reserve multiple flights in one reservation. | Not Run | -- | |
| TC390 | Verify that a user can successfully reserve an airline flight. | Not Run | -- | |
| TC393 | Verify that a user cannot reserve a flight if the flight is already fully booked. | Not Run | -- | |
| TC392 | Verify that a user cannot reserve a flight if the payment details are invalid. | Not Run | -- | |
| TC394 | Verify that a user cannot reserve a flight without providing all the required information. | Not Run | -- |
Now there are probably many other test cases that are missing, but imagine that the requirement was more detailed, for example "The ability to book a round-trip flight between two cities as a single reservation", then you would get more detailed test cases as well. One other idea would be to add functionality to our plugin that would let it generate more detailed requirements first. Then you could use those generated child requirements as the basis for the test cases.
If you click on one of the test cases inside Spira, there is just a single step right now with a basic description. However, it would be relatively easy to extend the plugin to subsequently call OpenAI for each generated test case and ask it to create the possible test steps for each one in turn.
Tasks
If you look at the tasks it generated, they cover the main development areas, from user interface to the back end processing of the reservations.
| Task # | Name | Status | Priority | Owned By | End Date | % Completed |
|---|---|---|---|---|---|---|
| TK60 | Design the user interface for the flight reservation system. | Not Started | -- | 0% | ||
| TK61 | Conduct user acceptance testing to gather feedback and make any necessary improvements to the system. | Not Started | -- | 0% | ||
| TK62 | Implement the functionality to search for available flights based on user input, such as departure and arrival cities and dates. | Not Started | -- | 0% | ||
| TK63 | Create a system to track and manage reserved flights, including updating seat availability and storing reservation information. | Not Started | -- | 0% | ||
| TK64 | Develop the functionality to confirm the flight reservation and generate a confirmation number for the user. | Not Started | -- | 0% | ||
| TK65 | Implement security measures to protect user information and payment details. | Not Started | -- | 0% | ||
| TK66 | Implement the functionality to allow the user to select a flight and reserve a seat. | Not Started | -- | 0% |
BDD Scenarios
The BDD option generated four basic scenarios: the success case, followed by three failure cases (invalid travel dates, invalid airports, and no available flights):
| Step | Description |
|---|---|
| 1 | Successful flight reservation Given I am on the flight reservation page When I enter my departure and destination airports And select my preferred travel dates And choose the number of passengers And click on the "Search" button Then I should see a list of available flights And I should be able to select a flight from the list And I should be redirected to the flight details page And I should see the flight details and fare And I should be able to enter passenger information And I should be able to review and confirm my reservation And I should receive a confirmation email with my reservation details |
| 2 | Invalid travel dates Given I am on the flight reservation page When I enter my departure and destination airports And select invalid travel dates (e.g. past dates) And choose the number of passengers And click on the "Search" button Then I should see an error message indicating that the travel dates are invalid |
| 3 | Invalid departure or destination airports Given I am on the flight reservation page When I enter an invalid departure or destination airport And select my preferred travel dates And choose the number of passengers And click on the "Search" button Then I should see an error message indicating that the airports are invalid |
| 4 | No available flights Given I am on the flight reservation page When I enter my departure and destination airports And select my preferred travel dates And choose the number of passengers And click on the "Search" button Then I should see a message indicating that there are no available flights |
Based on my experience using flight reservation software, that did feel like the three main failure cases, though of course there are more niche edge cases that we'd need to consider. It was impressive that was also able to generate the more detailed steps for the requirement.
Identifying Risks
Finally, I asked the plugin to generate a list of possible business and technical risks that could occur with this new feature:
The list that is generated is helpful, but does seem to follow the standard "-ilities" model, and some do feel like boilerplate (security, performance, etc.). That being said, several of the items were ones that I would have forgotten to include, and would be useful when developing a real flight reservation system.
Example 2: An E-Commerce Shopping Cart
One of the most common system features is the ability to have a "shopping cart". This UI paradigm is not only used for e-commerce, but often used in other applications as well. For example, when managing Reserved Instances (RIs) in Amazon Web Services (AWS) Elastic Cloud Compute (EC2) you put the items into a virtual shopping card. So I asked the plugin to generate the test cases and development tasks associated with implementing a shopping card:
Test Cases
The test cases it generated are more granular than the ones for the airline reservation example and cover a lot of good negative and positive tests, such as adding items with a zero to a shopping card and making sure you can add single and multiple items:
| Test # | Name | Status | Est. Duration | Last Execution Date |
|---|---|---|---|---|
| TC308 | Adding a single item to the shopping cart | Not Run | -- | |
| TC312 | Adding an item that is out of stock to the shopping cart | Not Run | -- | |
| TC311 | Adding an item to the shopping cart and removing it | Not Run | -- | |
| TC310 | Adding an item to the shopping cart and verifying the quantity | Not Run | -- | |
| TC309 | Adding an item to the shopping cart and verifying the total price | Not Run | -- | |
| TC313 | Adding an item to the shopping cart from different pages | Not Run | -- | |
| TC314 | Adding an item with a specified quantity to the shopping cart | Not Run | -- | |
| TC315 | Adding an item with a zero quantity to the shopping cart | Not Run | -- | |
| TC316 | Adding multiple items to the shopping cart | Not Run | -- |
Tasks
The suggested development tasks are a bit skewed to the back-end side of things, where there is a good level of definition. For the UI side, you'd need more detail on the screens and user interface in real life:
| Task # | Name | Status | Priority | Owned By | End Date | % Completed |
|---|---|---|---|---|---|---|
| TK74 | Create a user interface for displaying products and allowing users to add them to the shopping cart. | Not Started | -- | 0% | ||
| TK75 | Develop backend logic to handle updating the inventory of the products after a successful purchase. | Not Started | -- | 0% | ||
| TK76 | Develop backend logic to handle emptying the shopping cart and removing all items from the database. | Not Started | -- | 0% | ||
| TK77 | Develop backend logic to handle error scenarios and display appropriate error messages to the user. | Not Started | -- | 0% | ||
| TK78 | Develop backend logic to calculate the total price of the items in the shopping cart. | Not Started | -- | 0% | ||
| TK79 | Develop backend logic to handle removing items from the shopping cart when a user clicks on the "Remove" button. | Not Started | -- | 0% | ||
| TK80 | Develop backend logic to handle the checkout process, including validating user information and processing payment. | Not Started | -- | 0% | ||
| TK81 | Develop backend logic to handle updating the quantity of items in the shopping cart when a user changes the quantity input. | Not Started | -- | 0% | ||
| TK82 | Develop backend logic to handle adding items to the shopping cart when a user clicks on the "Add to Cart" button. | Not Started | -- | 0% |
One way to improve this would be for us to look through all of the task types and ask it to generate tasks of each type. Alternatively, we could enhance the plugin to ask the OpenAI model separately for all:
- Front-end development tasks
- Back-end development tasks
- Database development tasks
- API / Integration development tasks
Example 3: Booking a Hotel Room
You can tell I am a frequent traveler! As well as the example to reserve an airline flight, I also created a requirement for a feature that lets you book hotel rooms.
Test Cases
The plugin generated a very different style of test case for this example, everything was prefixed by "Verify" and it focused a lot more on the negative test cases as well as the one happy path successful case:
| Test # | Name | Status | Est. Duration | Last Execution Date |
|---|---|---|---|---|
| TC395 | Verify that a hotel room can be successfully booked when all required information is provided. | Not Run | -- | |
| TC400 | Verify that an error message is displayed when an invalid room type is selected. | Not Run | -- | |
| TC396 | Verify that an error message is displayed when the check-in date is in the past. | Not Run | -- | |
| TC398 | Verify that an error message is displayed when the check-out date is before the check-in date. | Not Run | -- | |
| TC397 | Verify that an error message is displayed when the hotel name is not provided. | Not Run | -- | |
| TC399 | Verify that an error message is displayed when the number of guests is not provided. | Not Run | -- |
It felt like a lot of the tests were limited to UI data validation tests, rather than testing the underlying business logic. Some of this could be due to the fact that the requirement was called Book a Hotel Room vs. Reserve a Hotel Room so the focus was on the literal booking part. If you compare the results of this example with the airline one, you can see how important it is to have a good requirement name when using this plugin.
Tasks
The tasks it generated seemed reasonable, and it had a better mix of UI and back-end tasks as compared to the shopping cart example:
| Task # | Name | Status | Priority | Owned By | End Date | % Completed |
|---|---|---|---|---|---|---|
| TK67 | Design the user interface for the hotel booking feature. | Not Started | -- | 0% | ||
| TK68 | Optimize the performance of the hotel booking feature to ensure fast and efficient processing. | Not Started | -- | 0% | ||
| TK69 | Implement the functionality to search for available hotel rooms based on specified dates and location. | Not Started | -- | 0% | ||
| TK70 | Implement the ability for the user to select a hotel room and proceed to the booking process. | Not Started | -- | 0% | ||
| TK71 | Test the hotel booking feature to ensure it is functioning correctly and meets the requirements. | Not Started | -- | 0% | ||
| TK72 | Implement the ability for the user to view and manage their hotel room bookings. | Not Started | -- | 0% | ||
| TK73 | Implement the functionality to validate and process the user's payment for the hotel room booking. | Not Started | -- | 0% |
BDD Scenarios
The suggested BDD scenarios were interesting this time. I was expecting it to simply give a single successful case and a few other failure cases. It actually gave us two successful cases, one where you simply booked a room as expected, and a second one where additional requests were entered. This is a common real-world use case (especially when I am traveling with my family), and it was impressive that it generated this.
| Step | Description |
|---|---|
| 1 | Unsuccessful hotel room reservation due to unavailability Given a user wants to reserve a hotel room When the user provides the necessary details such as check-in and check-out dates, number of guests, and preferred room type Then the system should display a list of available hotel rooms matching the user's preferences And all the rooms are already booked for the specified dates Then the system should display an error message indicating the unavailability of rooms |
| 2 | Successful hotel room reservation Given a user wants to reserve a hotel room When the user provides the necessary details such as check-in and check-out dates, number of guests, and preferred room type Then, the system should display a list of available hotel rooms matching the user's preferences And the user selects a specific room from the list And the system confirms the reservation by displaying a success message |
| 3 | Unsuccessful hotel room reservation due to invalid input Given a user wants to reserve a hotel room When the user provides invalid or incomplete details such as missing check-in or check-out dates Then the system should display an error message indicating the invalid input And the system should not proceed with the reservation process |
| 4 | Successful hotel room reservation with additional requests Given a user wants to reserve a hotel room When the user provides the necessary details such as check-in and check-out dates, number of guests, and preferred room type And the user also provides additional requests such as a specific floor or a view Then the system should display a list of available hotel rooms matching the user's preferences and additional requests And the user selects a specific room from the list And the system confirms the reservation by displaying a success message, including the additional requests. |
Identifying Risks
Finally, I asked the plugin to generate a list of possible business and technical risks that could occur with this new feature:
As we saw with the airline example, some of the risks felt a bit generic and boilerplate and could reasonably be applied to almost any requirement. However, some of the risks were very pertinent and relevant (double-booked rooms when using two systems that don't talk and integration with different hotel platform APIs).
Conclusion
As you can see from these examples, although the use of the ChatGPT / OpenAI SpiraApp is not a replacement for a good business analyst and test manager, it is a great time-saver. In the same way that Gmail Autocomplete is a great time-saver, using this new plugin will save countless hours of manual work and also uncover items that will have been missed by the team. Also, as you can see, we have many ideas for how to enhance the functionality once the first version is live next month.