Publishing test result files
In order to turn the Test Cases in Jira into a real living documentation it is essential to register the execution results of the Test Cases. Having the Test Case execution results registered in Jira might be also important as part of the traceability requirements.
You can publish scenario execution results (test results) to Jira with SpecSync using the publish-test-result command that works with many BDD tool and platform including SpecFlow, Cucumber and PyTest (Check Compatibility page for the full list).
This documentation page describes in detail the concept for publishing test results with SpecSync. The complete reference guide of the command line options can be found in the publish-test-results reference page. The configuration settings for test result publishing are in the publishTestResults configuration section.
The Examples section below shows a few concrete examples. A simple usage of the command could be as simple as this:
The Use SpecSync from build or release pipeline guide contains details about how to integrate the publish-test-results command to a build or release pipeline.
SpecSync operations, including the publish-test-results command supports "dry-run" mode using the --dryRun command line option. In dry-run mode, no change is made neither to Jira nor to the feature files, so you can test the impact of an operation without making an actual change.
Getting scenario execution result from your BDD tool
The automation source of the Test Cases are provided by the BDD scenarios and the automation solution implemented for them. The automation solution might use different tools (SpecFlow, Cucumber, PyTest, etc.) and might run on different platforms (e.g. .NET, Java or Python).
SpecSync uses a concept that works with different tools and platforms: execute the scenarios as usual with their standard tooling and save the test results into a test result file of their native test result format. SpecSync understands many test result file formats and how they represent the scenario results and able to match them to the Test Cases, synchronized earlier with a push command.
The Compatibility page contains the currently supported BDD tools and test result file formats. Please contact us if your tool is not in the list and we are happy to provide support for that. You can also support custom test result file formats with a SpecSync plugin.
The exact way how you can get test result files depends on the test runner tool and the BDD framework you use. For example for .NET Core SpecFlow test projects the test results can be saved into a TRX file by providing the --logger trx;logfilename=<your-trx-file-name>.trx option for the dotnet test command.
For the SpecSync publish-test-results command, the path of the test result file can be specified using the --testResultFile command line option. If the test result file is not a TRX file, you also have to specify the file format using the --testResultFileFormat option (see Compatibility for possible values).
The Examples section below shows how some of the most commonly used BDD frameworks and platforms can be used.
Publish test results to Jira
The representation of the test results in Jira depends on the Test Case Management solution used.
| TCM solution | Test result representation |
|---|---|
Jira TCM Customization | Currently not supported. |
Zephyr Scale | SpecSync creates a new Test Cycle during test result publishing with all test results that were included in the test result file(s). For scenario outlines a test result will be created with multiple test script result each of them representing a scenario outline example. Additional fields of the Test Cycle and the individual Test Results can be specified using the |
Publishing inconclusive test results
Some test execution frameworks report skipped scenarios as Inconclusive. Publishing them as inconclusive result (that is a kind of failure) would make the overall test outcome and the detail statistics invalid.
My setting the publishTestResults/treatInconclusiveAs setting in the configuration file, you can map this result to another value, e.g. NotExecuted.
Merging multiple test result files
You can also specify multiple test result files. In this case SpecSync will merge the results and publish them as a single Test Run.
The --testResultFile command line option allows specifying multiple files, separated by a semicolon (;).
Multiple test results files can also be specified by specifying a folder name. In this case SpecSync will scan through the folder and uses all files that are supported by the specified format setting (e.g. all TRX files).
Custom Test Run and Test Result settings
The Test Runs and the individual Test Results within that can have additional settings (e.g. comment) that can be specified. These settings can be specified from the command line (e.g. --testRunSetting) or using the publishTestResults/testRunSettings and publishTestResults/testResultSettings configuration settings.
In the specified values, you can also use different placeholders, e.g. the {br} placeholder to include a new line. For the complete list of placeholders that can be used, please check the reference.
Examples
SpecFlow (.NET Core)
The following example shows how to run .NET Core SpecFlow tests using the dotnet test command and publish the test result file (bddtestresults.trx) to the default Test Configuration.
Cucumber (Java, Maven)
The following example shows how to run Cucumber Java tests using Maven and publish the test result file to the default Test Configuration.
When running mvn test the test result file is saved to a file name that matches to your package name under the target/surefire-reports folder, e.g. target/surefire-reports/TEST-eu.specsolutions.calculator.CucumberTest.xml. Check the Maven Surefire Report Plugin documentation for details.
The Test Suite in this case is the Test Suite that the scenarios are synchronized to (BDD Scenarios). In this example we use the SpecSync native binaries to invoke the command, but the same would work with the other installation options too.
Publish using a specific environment with a custom description
The following example shows how to publish the results to a specific Environment using Zephyr Scale. In this case the tests are executed both with Chrome and Firefox, therefore we created two Environments (Chrome and Firefox).
We executed the tests for Chrome and saved the results to a file bddresults-chrome.trx. After that publishing the results to the Chrome configuration can be done with
Last updated