Installing TestRunner

Right-click Modules/Pages and choose Import Public Module > TestRunner. You also need to install AiTester.

After installation find TestRunner and AiTester inside Modules/Pages. Note, that you need to install both TestRunner and AiTester for TestRunner to work properly.

Using TestRunner

Here are a few examples of what you can accomplish with TestRunner. Note that TestRunner queries test run data from Spira.

Rerun Failed Test Cases

One of  the popular methods of scheduling nightly test set runs is via a dedicated test that triggers multiple Spira.RunTestSet actions. With this method you may group test sets using RVL tabs. Let's assume that some of the test sets failed. How to automatically run them again after fixing all the issues or if you want to see if the failures were accidental? Simply use TestRunner.DoRerunFailedAndNotRun and pass the RVL file with Spira.RunTestSet calls as an argument.

DoRerunFailedAndNotRun will find all the test sets mentioned in all sheets of the passed RVL file, then determine which of them failed today and schedule them for immediate rerun.

Analyze Recent Failures and Generate PDF Report

With TestRunner.DoAnalyzeRunHistory you may query recent test runs (Test Run Name, Execution Date,  Status(Pass/Fail), Failure Reason)  and using TestRunner.DoAiReport send this info to AI to generate a report. The report is saved in PDF format.

In this example, we query all failures that occurred today in the project with ID 9. We then generate a PDF report and save it to the Documents module in Spira, within the folder with the specified ID.

Analyze Recent Failures and Draw a Plot with Python

By applying TestRunner.DoAiReportPy to the result of TestRunner.DoAnalyzeRunHistory you may draw a plot visualizing test execution data. To use DoAiReportPy make sure you have Python installed.

In this example, we query all test cases that failed at least once during the last week. We then generate a Python program using the hand-written prompt and AI, and run it.

Full prompt text:

Discover which test cases have at least one non-pass status (build an array). Go again through all data and capture available runs of these test cases. Draw all statuses (including Pass, etc.) for each such test case over time. Each test case - is a horizontal sequence of dots.  

Example plot:

Discover Flaky Test Cases and Generate HTML-based Plot

Using TestRunner.DoAnalyzeFlakyTestCases discover top-k flaky test cases and visualize the result with TestRunner.DoAiReportHtml.

In this example, we query the last 10,000 test runs from Spira. We query automated test runs and select the top 20 most flaky tests. The most flaky test is the one that has the lowest pass rate. Then, an HTML-based plot is created. The HTML file is uploaded to Spira.

Full prompt text: 

Discover which test cases have at least one non-pass status (build an array). Go again through all data and capture available runs of these test cases. Draw all statuses (including Pass, etc.) for each such test case over time. Each test case - is a horizontal sequence of dots.

Example plot:

Discover Root Cause of Recent Failures

Using TestRunner.DoAnalyzeFailures, send reports of recent failures to AI for analysis to generate a root cause analysis for each failure and a summary report.

In this example, we find test cases that failed today and send their test run reports to AI for analysis. As a result, for each failed test case, we have a PDF file describing the root cause of the failure, and we also have a summary PDF file that combines all individual reports.

After execution of TestRunner.DoAnalyzeFailures, in the Rapise report, you will find entries corresponding to discovered failures:

There will also be a link to the summary PDF at the end.

If you wish, you may upload the generated summary report into Spira:

Generating PDFs

TestRunner provides 3 ways of generating PDFs.

Convert Markdown to PDF

In many cases, LLMs return responses in Markdown format. It is also an easy-to-generate and widely used format. With TestRunner.DoMarkdownToPdf, you may convert an .md file to a .pdf file and, if you wish, automatically open the result.

In this example, we use AiTester.DoAnalyzeReport to send a report to the LLM and receive the root cause analysis.  We then write this analysis to an .md file and convert the .md file to PDF.

Convert TRP to PDF using Chrome/Edge

If you have Chrome or Edge installed, you can use one of them to generate a PDF version of a test execution report with TestRunner.DoReportToPdf. TestRunner will first generate HTML using the default template and after that will use the selected browser to convert the HTML to PDF.

This method is flexible, provides well formatted result, but may be a bit slow for really large reports (> 100MB).

Directly Convert TRP  to PDF

Use TestRunner.DoTrpToPdf to directly convert an execution report to PDF. This method is fast and memory efficient, but it does not support templates.

Sample AI Prompts

In this section find examples of prompts to use with TestRunner.DoAiReport* actions.

TestRunner.DoAiReport

  • Build a table describing failed test cases.
  • Create a report describing test case failures. Format as table. Group failures of the same test case and reduce the number of lines in the table for same errors.
  • Describe any patterns and regularities you can see in this data.
  • Find groups of test cases that usually fail together.
  • What are the top 10 longest test runs?
  • Do you observe any anomalies in this test run data?
  • Build a list of failed test cases and their fail reasons.
  • Create a report for this sequence of test runs. Include failures only into a summary table. Also provide analysis for all runs and list unique TestSet names.

TestRunner.DoAiReportPy

  • Draw status for each test case over time. Each test case - is a horizontal sequence of dots.
  • Discover which test cases have at least one non-pass status (build an array). Go again through all data and capture available runs of these test cases. Draw all statuses (including Pass, etc.) for each such test case over time. Each test case - is a horizontal sequence of dots. 
  • Draw fail status for each test case over time. Each test case - is a horizontal sequence of dots. Do not draw Pass points.

TestRunner.DoAiReportHtml

  • Discover which test cases have at least one non-pass status (build an array). Go again through all data and capture available runs of these test cases. Draw all statuses (including Pass, etc.) for each such test case over time. Each test case - is a horizontal sequence of dots.

Implementation Details

By default TestRunner puts intermediate files into AiReports subfolder of the main folder of your testing framework. You may change this folder using TestRunner.SetParameter with DataFolder key.

There are more keys to use with TestRunner.SetParameter.

  • ExcludeFromAnalysis - set comma separated list of test case names that you want to ignore in  DoAnalyzeFlakyTestCases and DoAnalyzeRunHistory.
  • MinNumberOfTestRuns - if a test case has less than this specified number of runs then it will be excluded from DoAnalyzeFlakyTestCases. Default value is 10.
  • MaxNumberOfTestRuns - maximum number of test runs that can be fetched from Spira. The default value is 100,000. The more data you fetch, the greater the load on the server and the longer it takes.
  • BaseName - set this property to avoid passing it explicitly to various methods of TestRunner. Some methods that generate data will prepend the base name to the file names they produce. Also, TestRunner.DoAiReport* methods will grab data from the files with this prefix.

DoAiReport passes whole report with the last screenshot to AI.

DoAiReportPy and DoAiReportHtml do not pass specific data except the user prompt.