How to use SpecSync from build or release pipeline

Keeping the test cases in sync with the scenarios is important, therefore automating the synchronization process is recommended. This can be done for example from CI/CD build or release pipeline: you can configure an additional build steps that invoke the synchronization and test result publishing process.

The exact build steps and their order might be dependent on the project context, but a build pipeline that includes synchronization of the scenarios and publishes test results to Jira, usually follow the structure below (SpecSync specific steps are highlighted):

1. Initialize project (get sources, restore packages, etc.)

2. Build project (compile, build and publish your code)

3. Perform core tests (e.g. unit tests or other programmer tests)

4. SpecSync push — synchronize scenarios to Jira Test Cases

5. Run BDD tests and produce test result file (e.g. run SpecFlow tests)

6. SpecSync publish-test-results — publish the test results from the test result file to the synchronized Test Cases

In this documentation we first show how to add a build step to your pipeline that invokes SpecFlow and how to configure the authentication.

In the last sections (Performing synchronization (push) from build or release pipeline and Publishing test results from build or release pipeline) we show how to configure the SpecSync step for performing push and publish-test-results commands.

Adding SpecSync steps to your build or release pipeline

As SpecSync is a provided as a command line tool, the SpecSync commands can be added to the CI/CD pipeline as command line or script tasks. This provides high flexibility so that you can use it in any CI/CD pipeline systems, e.g. Jenkins, Bamboo or Azure DevOps. See details on how to configure these tasks below.

In order to diagnose synchronization issues, it is recommended to specify a log file for SpecSync execution. This can be done by specifying a --log <log-file-name>.log SpecSync command line option.

Adding a Command line task to invoke SpecSync

In the SpecSync command line task you need to invoke the SpecSync4Jira.exe or the dotnet executable with the necessary command line parameters. For that you need to ensure that SpecSync has been downloaded to the build agent.

If SpecSync has installed as a .NET Core tool, it can be invoked easily by invoking the dotnet command from your command line task.

In order to use any .NET tool, the tools have to be restored. If SpecSync is your first .NET tool in the project, than you need to add a step to do that to the beginning of the pipeline (usually before the normal dotnet restore command), as below.

dotnet tool restore

If you have installed SpecSync as a .NET Console App by adding a reference to the SpecSync.Jira.Console NuGet package in one of your projects, the build has probably restored this package already. You just need to figure out where the NuGet packages are downloaded. With the usual setup it is either the packages folder of your solution for older projects or the folder $(UserProfile)\.nuget\packages for newer, SDK-stype projects.

If you use the SpecSync native binaries, you have to make sure that the necessary binaries are downloaded and invoke the SpecSync command line tool from the downloaded location.

If you use the SpecSync official Docker image, you have to invoke the appropriate Docker commands from the command line task.

The following example shows how to invoke SpecSync push command from a Command line tool task from a restored NuGet package. See the following sections for details about the recommended authentication options and for the usual settings for push and publish-test-results commands.

cd src/Tests/MyProject.Specs
$HOME\.nuget\packages\SpecSync.Jira.Console\3.0.2\tools\SpecSync4Jira.exe push --disableLocalChanges --user "$SPECSYNC_PAT"

Authentication settings to perform SpecSync commands from build or release pipeline

For authentication, you can use a Jira user account with sufficient privileges (modify test cases).

Use a user account for the synchronization

In this section we show how the authentication can be configured using Personal Access Tokens (PAT), but the authentication can also be performed with user name and password in a similar way.

Step 1: Define a secret environment variable for the PAT in your pipeline

Add a new variable in the Variables section of your pipeline (e.g. SPECSYNC_PAT), specify the PAT of the account and make it secret.

Step 2: Use the environment variable in SpecSync commands

Once the variable is defined, you can pass its value to SpecSync using the --user command line option.

... push --disableLocalChanges --user "$(SPECSYNC_PAT)"

Performing synchronization (push) from build or release pipeline

The SpecSync push command is usually performed after the successful execution of core tests (e.g. unit tests), but before executing the BDD tests. This way you can ensure that the test cases are updated even if some of the BDD scenarios fail (especially when automated as integration or end-to-end test).

Adding the push step to the pipeline ensures that the Test Cases are updated to the exact same version that was used to execute the tests. If the changes have been synchronized locally then this step will just simply detect that the test cases are up-to-date and not change them.

The following table contains the settings that are important or usually configured for the push command.

Setting	Description
Working Directory	For the easiest configuration it is recommended to set the Working Directory property of the task to the folder where your feature-set is (e.g. to the SpecFlow project folder). This is typically the same folder where your SpecSync configuration file is located.
--user	The --user command line option can be used so specify the user credentials (e.g. Personal Access Token, PAT) for the user that needs to be used for the synchronization. See Synchronizing test cases from build for details.
--disableLocalChanges	Creating the initial link between scenarios and test cases requires a small change in the feature file (to include the test case tags). Such changes cannot be done in an automated process, because the changes cannot be committed to the source control. To handle this, SpecSync provides a --disableLocalChanges switch (see command line reference for more details). If this switch is provided, SpecSync will not synchronize new scenarios, but only updates the ones that are already linked to test cases.
Task conditions	The synchronization usually should be performed only for normal build executions and not for Pull Requests. Make sure your task is configured accordingly.

The following example shows a fully configured step that performs the SpecSync push command.

cd src/Tests/MyProject.Specs
dotnet specsync push --disableLocalChanges --user "$SPECSYNC_PAT"

Publishing test results from a pipeline

To be able to track the test execution results at the test cases, you can execute the tests like usual, save the test result to a test result file (in case of .NET this is a TRX file) and publish the test results using the SpecSync publish-test-results command.

Step 1: Prepare test execution task

In order to publish the test results, the test execution task has to be configured to save the results to a test result file (in case of .NET this is a TRX file). The exact way of doing that depends on the platform and the test execution tool you use.

cd src/Tests/MyProject.Specs
dotnet test src/Tests/MyProject.Specs/*.csproj --configuration $(BuildConfiguration) --logger trx;logfilename=bddtestresults.trx --results-directory $(Agent.TempDirectory)"

Step 2: Configure SpecSync task to publish test result file

In order to publish the test results, a SpecSync task has to be added right after the test execution task. See Adding SpecSync steps to your build or release pipeline for details about how to do this.

The following table contains the settings that are important or usually configured for the publish-test-results command.

Setting	Description
Working Directory	For the easiest configuration it is recommended to set the Working Directory property of the task to the folder where your feature-set is (e.g. to the SpecFlow project folder). This is typically the same folder where your SpecSync configuration file is located.
--user	The --user command line option can be used so specify the user credentials (e.g. Personal Access Token, PAT) for the user that needs to be used for the synchronization. See Synchronizing test cases from build for details.
--testResultFile	The test results file produced by the test execution step can be specified using the --testResultFile option. Make sure you specify the file from the right folder (usually $(Agent.TempDirectory)).
--testResultFileFormat	This setting is needed if the test result file is NOT a TRX file. For TRX files this setting can be omitted. The possible format values are listed in the Compatibility page.
--runName	There are many settings that can be used to customize the Test Run created by SpecSync. You can find these settings in the publish-test-results page. The --runName setting for example can be used to specify a name of your Test Run, so that it can be easily distinguished from the normal test execution results.
Task conditions	It is important that the publish-test-results command is performed even if the test execution failed. By default the tasks are only executed if all previous tasks succeeded. Also make sense to note that test result publishing usually should be performed only for normal build executions and not for Pull Requests. Make sure your task is configured accordingly.

The following example shows a fully configured step that performs the SpecSync publish-test-results command. The example assumes that the test results were saved to a file bddtestresults.trx in the folder $(Agent.TempDirectory) like it was configured in the example of the previous step.

cd src/Tests/MyProject.Specs
dotnet specsync publish-test-results --user "$SPECSYNC_PAT" --testResultFile $(Agent.TempDirectory)\bddtestresults.trx --runName "BDD Tests"