Publishing test result files
PreviousTest result publishing featuresNextSupport for Azure DevOps Test Plan / Test Suite based test execution
Last updated
Was this helpful?
Last updated
Was this helpful?
In order to turn the Test Cases in Azure DevOps into a real living documentation it is essential to register the execution results of the Test Cases. Having the Test Case execution results registered in Azure DevOps might be also important as part of the traceability requirements.
You can publish scenario execution results (test results) to Azure DevOps with SpecSync using the publish-test-result command that works with many BDD tool and platform including SpecFlow, Reqnroll, Cucumber and PyTest (Check page for the full list).
The automation source of the Test Cases are provided by the BDD scenarios and the automation solution implemented for them. The automation solution might use different tools (SpecFlow, Reqnroll, Cucumber, PyTest, etc.) and might run on different platforms (e.g. .NET, Java or Python).
SpecSync uses a concept that works with different tools and platforms: execute the scenarios as usual with their standard tooling and save the test results into a test result file of their native test result format. SpecSync understands many test result file formats and how they represent the scenario results and able to match them to the Test Cases, synchronized earlier with a push command.
The exact way how you can get test result files depends on the test runner tool and the BDD framework you use. For example for .NET Reqnroll test projects the test results can be saved into a TRX file by providing the --logger trx;logfilename=<your-trx-file-name>.trx option for the dotnet test command.
Azure DevOps records test results for Test Cases in a flexible way and to be able to understand how SpecSync needs to be configured it makes sense to review the most important aspects of the Azure DevOps test result behavior and their implication to the SpecSync synchronization process.
The test results in Azure DevOps are always connected to a Test Configuration, so you can choose a Test Configuration when publishing the test results with SpecSync.
Test results in Azure DevOps always belong to a Test Suite and you cannot record test results to Test Cases that are not included in any Test Suites.
As the results are connected to a single Test Suite, when a Test Case result is published to that Test Suite that result is not visible when you check the same Test Case in a different Test Suite.
Teams usually decide on the Test Suite they use as "living documentation", so the Test Suite that they check if they are interested in the test execution results.
Azure DevOps groups the test results together into a Test Run. The list of Test Runs can be found in the Test Plans / Runs section of the Azure DevOps project web portal.
SpecSync always attaches the test result file to the created Test Run, but you can attach additional files using the --attachedFiles option.
The test results can also be associated with a build (a concrete run of a build pipeline) to track the execution source.
SpecSync publishes the result of a scenario outline is cumulated from the results of the individual data variations (examples). In this cases the test result comment contains the individual results or the error messages. SpecSync also publishes these results as sub-results of the test result, but currently sub-results are not displayed in the Azure DevOps user interface (they can be retrieved though the API though). To check these results in detail, you need to open the test result file attached to the Test Run.
Some test execution frameworks report skipped scenarios as Inconclusive. Publishing them as inconclusive result (that is a kind of failure) would make the overall test outcome and the detail statistics invalid.
My setting the publishTestResults/treatInconclusiveAs setting in the configuration file, you can map this result to another value, e.g. NotExecuted.
You can also specify multiple test result files. In this case SpecSync will merge the results and publish them as a single Test Run.
The --testResultFile command line option allows specifying multiple files, separated by a semicolon (;).
Multiple test results files can also be specified by specifying a folder name. In this case SpecSync will scan through the folder and uses all files that are supported by the specified format setting (e.g. all TRX files).
The following example shows how to run .NET Core SpecFlow tests using the dotnet test command and publish the test result file (bddtestresults.trx) to the default Test Configuration. The Test Suite in this case is the Test Suite that the scenarios are synchronized to (BDD Scenarios).
The following example shows how to run Cucumber Java tests using Maven and publish the test result file to the default Test Configuration.
The following example shows how to publish the results to a specific Test Configuration. In this case the tests are executed both with Chrome and Firefox, therefore we created two Test Configurations (Chrome and Firefox) and assigned them to the Test Suite that the scenarios are synchronized to (BDD Scenarios).
We executed the tests for Chrome and saved the results to a file bddresults-chrome.trx. After that publishing the results to the Chrome configuration can be done with
The following example shows how to customize the settings for the publish-test-results command. (We use a SpecFlow .NET Core result file bddresults.trx as an example here.) The following customizations are applied:
The Test Configuration is specified in the config file (instead of command line)
The results are published to a Test Suite (BDD Results, in Test Plan ID 23) different from the one where the scenarios are synchronized to
The name of the Test Run has been specified
This documentation page describes in detail the concept for publishing test results with SpecSync. The complete reference guide of the command line options can be found in the page. The configuration settings for test result publishing are in the .
The section below shows a few concrete examples. A simple usage of the command could be as simple as this:
The guide contains details about how to integrate the publish-test-results command to a build or release pipeline.
You can find an overview of the Azure DevOps plans in the . The shows the different reports and charts you can create for the results.
The page contains the currently supported BDD tools and test result file formats. Please if your tool is not in the list and we are happy to provide support for that. You can also support custom test result file formats with a .
For the SpecSync publish-test-results command, the path of the test result file can be specified using the --testResultFile command line option. If the test result file is not a TRX file, you also have to specify the file format using the --testResultFileFormat option (see for possible values).
The section below shows how some of the most commonly used BDD frameworks and platforms can be used.
The Test Configuration represents the configuration your product was tested with. A configuration can be the target operating system, the target browser or the target mobile phone, but many teams do not need to represent these configuration differences in BDD scenario executions. Depending on your needs, you can configure multiple Test Configurations in Azure DevOps (see ) or just create a default configuration. Azure DevOps always creates a default Test Configuration called Windows 10 (can be renamed).
In case you you do not plan to publish results to multiple configurations, you can omit the Test Configuration setting or specify it in the config file by providing either the setting.
In case you publish results to different Test Configurations it is better to specify it using the .
For SpecSync this means you have to select a Test Suite to publish the results for. By default SpecSync uses the Test Suite configured for synchronizing the test cases (see ), but you can also choose different Test Suite as well using the setting or the . There is no restriction for the Test Suite type, you can specify static, query-based and requirement-based Test Suites as well.
Since you can specify the Test Suite for publishing, the publish-test-results command works even if you have not configured SpecSync to . But using that feature has other benefits as well so it is recommended to use it even if you publish the results to a different suite.
Alternatively you can use the feature of SpecSync, that allows you to publish the same test result to multiple Test Suites (even into all Test Suite where the Test Case is included). Because of Azure DevOps limitations, these Test Suites must belong the the same Test Plan.
The SpecSync publish-test-results command always creates a single Test Run. You can customize many details of the created Test Run using the and the options. For example you can set the name of the Test Run using the --runName option.
The Azure DevOps Test Runs have a Run Type setting that can either be Automated or Manual. SpecSync sets this flag by default based on whether it is configured to (synchronization/automation/enabled is set to true). The value can be overwritten by setting publishTestResults/runType.
Inconsistent setting of the Test Case automation flag and the Test Run run type might lead to "Mismatch in automation status of test case and test run" error in older Azure DevOps servers. See related for details.
When the SpecSync publish-test-results command is invoked from an Azure DevOps build pipeline, it will detect the ID of the currently executing build and automatically associates the results to that build. For other situations the build can also be associated using the --buildId or the --buildNumber command line options. To disable the automatic build association, you have to specify the --disablePipelineAssociation option (or set the --buildId option to an empty value before v3.3.3). For more information about using SpecSync from a pipeline, please check the guide.
In Azure DevOps the pipeline can only be associated to the created Test Run, if that pipeline is in the same Azure DevOps project as the synchronized Test Cases. If you try to publish test result from a different project, you will receive a pipeline not found error. See the for a workaround to this limitation.
Scenario outlines represent data-driven variation tests: the same test is executed with different input data. SpecSync .
The Test Runs and the individual Test Results within that can have additional settings (e.g. comment) that can be specified. These settings can be specified from the (e.g. --runComment) or using the publishTestResults/testRunSettings and publishTestResults/testResultSettings .
In the specified values, you can also use different placeholders, e.g. the {br} placeholder to include a new line. For the complete list of placeholders that can be used, please check the .
When running mvn test the test result file is saved to a file name that matches to your package name under the target/surefire-reports folder, e.g. target/surefire-reports/TEST-eu.specsolutions.calculator.CucumberTest.xml. Check the for details.
The Test Suite in this case is the Test Suite that the scenarios are synchronized to (BDD Scenarios). In this example we use the to invoke the command, but the same would work with the other too.
The following example shows how to run .NET Framework SpecFlow tests using the vstest.console command and publish the test result file (bddtestresults.trx) to the default Windows 10 Test Configuration. The runner saves the test result file to the TestResults folder by default. (.NET Framework tests can also be executed using the dotnet test command, see .)
The Test Suite in this case is the Test Suite that the scenarios are synchronized to (BDD Scenarios). In this example we use the to invoke the command, but the same would work with the other too.
