Customization: Publishing test results to multiple Test Suites

As described in the Publishing test result files page, the test results in Azure DevOps are always connected to a Test Suite. When a Test Case result is published to that Test Suite that result is not visible when you check the same Test Case in a different Test Suite.

The Multi-suite publish customization for SpecSync allows publishing the same test results to multiple Test Suites (even into all Test Suite where the Test Case is included). Because of Azure DevOps limitations, these Test Suites must belong the the same Test Plan.

The Multi-suite publish customization described here is an Enterprise feature.

The Multi-suite publish customization is not supported in TFS 2017.

In order to enable the customization, the customizations/multiSuitePublishTestResults section of the configuration has to be enabled. The list of all possible settings can be found in the customization configuration reference. Since the Test Suites have to be in the same Test Plan, specifying the testPlanId setting is mandatory.

The following example shows a basic configuration.

specsync.json

{
  ...
 "customizations": {
    "multiSuitePublishTestResults": {
      "enabled": true,
      "testPlanId": 567,
      "suites": [
        "SuiteA",
        "SuiteB",
      ]
    }
  }
  ...
}

The customization can be used in different ways:

Goal	Description
Publish test results to selected Test Suites within a Test Plan	For that the selected Test Suites have to be listed in the suites setting. The list can contain Test Suite names or Test Suite IDs in #1234 form. You can enable the includeSubSuites setting to include all suites of the listed ones.
Publish to all Test Suites within the Test Plan	For that the publishToAllSuites flag has to be set to true.
Publish to requirement-based Suites of the requirements linked to the scenarios via tags.	To be able to use this, the synchronization has to be configured to link Work Items using tags. To achieve this goal, the publishToRequirementBasedTestSuites flag has to be set to true. Optionally the work item links to be considered can be restricted using the linkTagPrefixes setting. The publishToRequirementBasedTestSuites setting can also be combined with the suites setting.

To be able to publish the same test results to multiple Test Suites, the created Test Run will contain duplicates of the test results. This makes the basic pass/fail statistics invalid.