Publishing test result files
Last updated
Was this helpful?
Last updated
Was this helpful?
In order to turn the Test Cases in Jira into a real living documentation it is essential to register the execution results of the Test Cases. Having the Test Case execution results registered in Jira might be also important as part of the traceability requirements.
You can publish scenario execution results (test results) to Jira with SpecSync using the publish-test-result command that works with many BDD tool and platform including SpecFlow, Reqnroll, Cucumber and PyTest (Check page for the full list).
This documentation page describes in detail the concept for publishing test results with SpecSync. The complete reference guide of the command line options can be found in the page. The configuration settings for test result publishing are in the .
The section below shows a few concrete examples. A simple usage of the command could be as simple as this:
The guide contains details about how to integrate the publish-test-results command to a build or release pipeline.
The automation source of the Test Cases are provided by the BDD scenarios and the automation solution implemented for them. The automation solution might use different tools (SpecFlow, Reqnroll, Cucumber, PyTest, etc.) and might run on different platforms (e.g. .NET, Java or Python).
SpecSync uses a concept that works with different tools and platforms: execute the scenarios as usual with their standard tooling and save the test results into a test result file of their native test result format. SpecSync understands many test result file formats and how they represent the scenario results and able to match them to the Test Cases, synchronized earlier with a push command.
The page contains the currently supported BDD tools and test result file formats. Please if your tool is not in the list and we are happy to provide support for that. You can also support custom test result file formats with a .
The exact way how you can get test result files depends on the test runner tool and the BDD framework you use. For example for .NET Reqnroll test projects the test results can be saved into a TRX file by providing the --logger trx;logfilename=<your-trx-file-name>.trx option for the dotnet test command.
For the SpecSync publish-test-results command, the path of the test result file can be specified using the --testResultFile command line option. If the test result file is not a TRX file, you also have to specify the file format using the --testResultFileFormat option (see for possible values).
The section below shows how some of the most commonly used BDD frameworks and platforms can be used.
The representation of the test results in Jira depends on the Test Case Management solution used.
Jira TCM Customization
Currently not supported.
Zephyr Scale
SpecSync creates a new Test Cycle during test result publishing with all test results that were included in the test result file(s). For scenario outlines a test result will be created with multiple test script result each of them representing a scenario outline example.
Additional fields of the Test Cycle and the individual Test Results can be specified using the --testRunSetting and the --testResultSetting options. Currently for the Test Cycle the name, description, iteration, folder, version fields and for the Test Result, the comment and the environment fields can be set.
Some test execution frameworks report skipped scenarios as Inconclusive. Publishing them as inconclusive result (that is a kind of failure) would make the overall test outcome and the detail statistics invalid.
My setting the publishTestResults/treatInconclusiveAs setting in the configuration file, you can map this result to another value, e.g. NotExecuted.
You can also specify multiple test result files. In this case SpecSync will merge the results and publish them as a single Test Run.
The --testResultFile command line option allows specifying multiple files, separated by a semicolon (;).
Multiple test results files can also be specified by specifying a folder name. In this case SpecSync will scan through the folder and uses all files that are supported by the specified format setting (e.g. all TRX files).
The following example shows how to run .NET Core SpecFlow tests using the dotnet test command and publish the test result file (bddtestresults.trx) to the default Test Configuration.
The following example shows how to run Cucumber Java tests using Maven and publish the test result file to the default Test Configuration.
The following example shows how to publish the results to a specific Environment using Zephyr Scale. In this case the tests are executed both with Chrome and Firefox, therefore we created two Environments (Chrome and Firefox).
We executed the tests for Chrome and saved the results to a file bddresults-chrome.trx. After that publishing the results to the Chrome configuration can be done with
The Test Runs and the individual Test Results within that can have additional settings (e.g. comment) that can be specified. These settings can be specified from the (e.g. --testRunSetting) or using the publishTestResults/testRunSettings and publishTestResults/testResultSettings .
In the specified values, you can also use different placeholders, e.g. the {br} placeholder to include a new line. For the complete list of placeholders that can be used, please check the .
When running mvn test the test result file is saved to a file name that matches to your package name under the target/surefire-reports folder, e.g. target/surefire-reports/TEST-eu.specsolutions.calculator.CucumberTest.xml. Check the for details.
The Test Suite in this case is the Test Suite that the scenarios are synchronized to (BDD Scenarios). In this example we use the to invoke the command, but the same would work with the other too.