SpecSync Documentation
Getting StartedConfigurationGuidesDownloads
Azure DevOps
Azure DevOps
  • Introduction to SpecSync for Azure DevOps
  • Getting started
    • Getting started using SpecFlow
    • Getting started using Cucumber or other Gherkin-based BDD tool
  • Installation & Setup
    • Install as .NET tool
    • Install as .NET Console App
    • Install as native binaries for Linux or macOS
    • Install as Docker image
    • Setup and Configure
  • Features
    • Push features
      • Pushing scenario changes to Test Cases
      • Configuring the format of the synchronized test cases
      • Synchronizing Scenario Outlines
      • Add new Test Cases to an Area or an Iteration
      • Mark Test Cases as Automated
      • Setting Test Case state on change
      • Update Test Case fields
      • Attach files to Test Cases using tags
      • Customization: Setting Test Case fields with default values
      • Customization: Update custom Test Case fields on push
      • Customization: Ignoring marked Test Case steps
      • Customization: Ignoring Test Case Tags
      • Customization: Ignore non-supported local tags
      • Customization: Mapping tags
      • Customization: Synchronizing scenarios from feature branches
      • Customization: Reset Test Case state after change
      • Customization: Automatically link changed Test Cases
      • Customization: Synchronize linked artifact titles
      • Customization: Add Test Cases to Suites
      • Customization: Do not synchronize title
    • Pull features
      • Pulling Test Case changes to local scenarios
    • Common synchronization features
      • Configuration key
      • Remote scope
      • Linking Work Items using tags
      • Synchronizing Test Case hierarchies using Test Suites
      • Include synchronized Test Cases to a Test Suite (deprecated)
      • Excluding scenarios from synchronization
      • Synchronization conflict resolution
      • Re-link scenarios to new Test Cases
    • Test result publishing features
      • Publishing test result files
      • Support for Azure DevOps Test Plan / Test Suite based test execution
      • Customization: Publishing test results to multiple Test Suites
    • General features
      • Azure DevOps authentication options
      • Configuration file
      • Hierarchical configuration files
      • Local test case conditions
      • Configuration wizards
      • SpecSync plugins
    • Customizations
    • Plugin list
  • Licensing
  • Guides
    • What is my Azure DevOps project URL?
    • How to define the local feature-set to be synchronized
    • Filters and scopes
    • How to synchronize automated test cases
    • How to use SpecSync from build or release pipeline
    • How to publish test results from pipelines using the VSTest task
    • How to use the SpecSync Azure DevOps pipeline tasks
    • How to link GitHub pull requests
    • How to upgrade to a newer version of SpecSync
    • How to attach files to test results
    • Using SpecSync with SpecFlow+
    • Using SpecSync with Cucumber
    • Using SpecSync with Cypress
    • Using SpecSync with Postman
    • Using SpecSync with TestNG
    • Using SpecSync on macOS or Linux
    • Using SpecSync inside a Docker container
    • How to handle Test Cases of multiple parallel application releases
    • Migrating from SpecSync v3 to v5
    • Migrating from SpecSync v2 to v3
    • Migrating from SpecSync v1 to v2
  • Changelog
  • Release Model and Roadmap
  • Downloads
  • Reference
    • Command line reference
      • init
      • upgrade
      • push
      • pull
      • publish-test-results
      • re-link
      • version
    • Configuration reference
      • toolSettings
      • local
      • remote
      • knownRemotes
      • synchronization
        • push
        • pull
        • automation
        • state
        • areaPath
        • iterationPath
        • links
        • attachments
        • format
        • fieldUpdates
      • hierarchies
      • publishTestResults
      • specFlow
      • reqnroll
      • customizations
    • Compatibility
    • Older versions
  • Contact
    • SpecSync Support
    • Troubleshooting
    • FAQ
  • Project Website
Powered by GitBook
On this page
  • Supported hierarchy types
  • Common hierarchy settings
  • The folders, foldersAndFiles and foldersAndDocumentNames hierarchy types
  • Skipping folder prefix
  • The levels hierarchy type
  • Customize node name
  • Handling non-matching levels
  • The tag hierarchy type
  • The single hierarchy type
  • The custom hierarchy type

Was this helpful?

  1. Features
  2. Common synchronization features

Synchronizing Test Case hierarchies using Test Suites

PreviousLinking Work Items using tagsNextInclude synchronized Test Cases to a Test Suite (deprecated)

Last updated 1 month ago

Was this helpful?

Besides and to Azure DevOps Test Cases, SpecSync can also help you to organize the remote Test Cases into hierarchical structures. In Azure DevOps that means that it can create and maintain Test Suite hierarchies and include the Test Cases to these Test Suites according to the configured rules. The test results can also be published to such a synchronized hierarchy.

The Test Case hierarchy synchronization is only available to SpecSync for Azure DevOps currently. The feature will be available for SpecSync for Jira soon.

In SpecSync a Test Case hierarchy is a configured set of rules that defines a hierarchical structure for the Test Cases. The hierarchy defines several nodes (Test Suites in Azure DevOps), that might contain synchronized Test Cases.

SpecSync will take care of

  • Creating the hierarchy nodes

  • Adding the Test Cases to the hierarchy nodes according to the configured rules

  • Moving the Test Cases between hierarchy nodes when the rules conditions change (e.g. scenario is moved to another feature file)

  • Removing the Test Case from the hierarchy node if it is detected to be removed (requires a configured ).

Within one hierarchy each synchronized Test Case is included only once, but you can define multiple hierarchies. The hierarchy nodes might contain Test Cases that are not synchronized by SpecSync. These Test Cases have to be added to the hierarchy node manually, but SpecSync will not remove them during synchronization.

Synchronizing to multiple hierarchies or hierarchies with more than 20 nodes requires an .

You can configure the hierarchies based on different commonly used rule-sets by choosing a hierarchy type. For example the following configuration defines a hierarchy that is based on the folder structure of the local test case documents. I.e. every folder and sub-folder becomes a hierarchy node.

The following configuration defines a hierarchy named folder-hierarchy based on the folders and sub-folders of the files and includes all local test cases to the defined nodes that does not have an @ignore tag. In this example, the root hierarchy node is mapped to the Scenarios by Folder Test Suite of the My plan Test Plan.

specsync.json
{
  ...
  "hierarchies": [
    {
      "name": "folder-hierarchy",
      "type": "folders",
      "condition": "not @ignore",
      "root": {
        "testPlan": "My plan",
        "name": "Scenarios by Folder"
      }
    }
  ],
  ...
}

Supported hierarchy types

Type
Description

folders

foldersAndFiles

foldersAndDocumentNames

levels

tag

single

custom

Common hierarchy settings

Regardless of the chosen hierarchy type there are some common settings that can be used.

  • name: The name of the hierarchy is an identifier that can be used to refer to the hierarchy for other features (e.g. for publishing test results to that hierarchy). If no name is specified the name default is used. The name has to be unique among the defined hierarchies and it is mandatory when multiple hierarchies are specified.

  • type: Specifies the type of the hierarchy. It is mandatory and has to be set to one of the available hierarchy types from the table above.

  • root: Specifies the root location in Azure DevOps, where the hierarchy should be mapped to. The root location specified here will be mapped to the root of the hierarchy. In Azure DevOps the root can be specified by specifying a Test Plan (testPlan setting) and either a Test Suite name, path or id. If only a Test Plan is specified, the root Test Suite of the plan will be used as the root. For most of the hierarchy types, specifying the root is mandatory.

  • ignoreAdditionalNodes: By default SpecSync generates a warning if the hierarchy in Azure DevOps contains additional nodes (nodes that are not defined by the hierarchy). If such additional nodes are required, it is recommended to set this setting to true to avoid unnecessary warnings.

  • disableUnderscoreTransformation: The _ character in the matched node names are automatically transformed to space by default. This behavior can be disabled by setting the disableUnderscoreTransformation hierarchy setting to true. This setting can be used for levels and tag hierarchy types.

The folders, foldersAndFiles and foldersAndDocumentNames hierarchy types

The folders, foldersAndFiles and foldersAndDocumentNames hierarchy types define a hierarchy based on the source documents of the local test cases.

The source documents are usually arranged into a logical folder and file structure that is also useful to maintain in Azure DevOps.

All three types uses the folders and sub-folders of the source documents (e.g. feature files) as a core structure, but foldersAndFiles and foldersAndDocumentNames also defines nodes based on the source documents (e.g. feature files) as well.

Let's assume the following local test case structure. The SpecSync configuration root folder is the folder that contains the specsync.json configuration file.

<SpecSync configuration root folder>
├ Authentication.feature
│  └ Scenario: User authenticates successfully with token (#1)
├ Payments/
│  ├ CardPayments.feature
│  │  ├ Scenario: Card payment is successful (#2)
│  │  └ Scenario: Card payment was cancelled (#3)
│  └ BankTransferPayments.feature
│     └ Scenario: Bank payment instructions are sent (#4)
└ Inventory/
   ├ MaintainInventory.feature
   │  ├ Scenario: A new product is added to the Inventory (#5)
   │  └ Scenario: A product is removed from the Inventory (#6)
   ├ DisplayInventory.feature
   │  └ Scenario: The details of a product is displayed (#7)
   └ Reporting/
      └ InventoryReports.feature
         └ Scenario: Inventory day close report is generated (#8)

Choosing the folders hierarchy type would generate the following hierarchy structure from the local test case structure above.

<root>: Contains #1
├ Payments: Contains #2, #3, #4
└ Inventory: Contains #5, #6, #7
   └ Reporting: Contains #8

Choosing the foldersAndFiles hierarchy type would define nodes from the files as well.

<root>: Empty
├ Authentication: Contains #1
├ Payments: Empty
│  ├ CardPayments: Contains #2, #3
│  └ BankTransferPayments: Contains #4
└ Inventory: Empty
   ├ MaintainInventory: Contains #5, #6
   ├ DisplayInventory: Contains #7
   └ Reporting: Empty
      └ InventoryReports: Contains #8

Choosing the foldersAndDocumentNames hierarchy type almost identical to foldersAndFiles except that it uses the name of the local test case document instead of the file name when available. For example for feature files it uses the feature name (the name you specify after the Feature: prefix in the feature file).

Assuming the feature names of the example files are the same of the file names except of spaces, the hierarchy would look like the following.

<root>: Empty
├ Authentication: Contains #1
├ Payments: Empty
│  ├ Card Payments: Contains #2, #3
│  └ Bank Transfer Payments: Contains #4
└ Inventory: Empty
   ├ Maintain Inventory: Contains #5, #6
   ├ Display Inventory: Contains #7
   └ Reporting: Empty
      └ Inventory Reports: Contains #8

Skipping folder prefix

In some cases the source documents are in a sub-folder of the SpecSync configuration root folder and you don't want to represent those folders in the hierarchy. Let's consider the following structure.

<SpecSync configuration root folder>
└ src/
   └ features/
      ├ Authentication.feature
      └ Inventory/
        ├ MaintainInventory.feature
        ├ DisplayInventory.feature
        └ Reporting/
            └ InventoryReports.feature

By default the src/features folders would become part of the hierarchy. For example with folders type, it would generate the following structure.

<root>
└ src
   └ features
      └ Inventory
        └ Reporting

To avoid generating hierarchy nodes from src/features, you can use the skipFolderPrefix setting:

specsync.json
{
  ...
  "hierarchies": [
    {
      "type": "folders",
      "skipFolderPrefix": "src/features"
    }
  ],
  ...
}

As a result, the following hierarchy is generated.

<root>
└ Inventory
  └ Reporting

The skipFolderPrefix setting can be used for folders, foldersAndFiles and foldersAndDocumentNames hierarchy types.

The levels hierarchy type

The levels hierarchy type defines a hierarchy where the different levels correspond to a particular property or categorization of the test cases. This is useful when the levels of the desired structure is defined according to some aspects.

Let's assume we would like to build a hierarchy with three levels:

  • level 1: area (e.g. "Payments")

  • level 2: component (e.g. "FrontEnd")

  • level 3: test category (e.g. "Regression")

With this specification the Test Cases would be included to hierarchy nodes, like "Payments / FrontEnd / Regression".

Consider the following feature file.

Features/Payments/CardPayments.feature
Feature: Card Payments

@component:FrontEnd
Rule: Card payment can be initiated from the UI

  @smoke
  Scenario: Card payment is started (#1)
  [...]
  
  @regression
  Scenario: Card payment is successful (#2)
  [...]

Rule: Card payment is logged

  @component:BackEnd
  @regression
  Scenario: Card payment is added to transaction log (#3)
  [...]

The folder of the feature file (Payments), the tags of the scenarios and the tags "inherited" from the Feature and Rule headers define the aspects of the test cases that we would like to use:

  • Scenario #1: Card payment is started

    • Area: Payments

    • Component: FrontEnd

    • Category: Smoke

  • Scenario #2: Card payment is successful

    • Area: Payments

    • Component: FrontEnd

    • Category: Regression

  • Scenario #3: Card payment is added to transaction log

    • Area: Payments

    • Component: BackEnd

    • Category: Regression

The following hierarchy configuration can be used to define the hierarchy with the "area", "component" and "test category" levels.

specsync.json
{
  ...
  "hierarchies": [
    {
      "type": "levels",
      "levels": [
        { // level 1: area
          "condition": "$sourceFile ~ Features/*/" // takes the folder below Features/
        },
        { // level 2: component
          "condition": "@component:*" // takes the component from tag
        },
        { // level 3: test category
          "conditionalName": {
            "@smoke": "Smoke",
            "@regression": "Regression"
          }
        }
      ]
    }
  ],
  ...
}

Synchronizing the feature file above would generate the following hierarchy.

<root>: Empty
└ Payments: Empty
   ├ FrontEnd: Empty
   │  ├ Smoke: Contains #1
   │  └ Regression: Contains #2
   └ BackEnd: Empty
      └ Regression: Contains #3

Customize node name

When a level is defined with a wildcard match the hierarchy node name will use the value that has matched to the wildcard. The default behavior can be changed by specifying the name level setting.

Let's say we would like to define the second level as FrontEnd Category and BackEnd Category instead of FrontEnd and BackEnd. This can be achieved by the following configuration.

specsync.json
{
  ...
  "hierarchies": [
    {
      "type": "levels",
      "levels": [
        { // level 1: area
          [...]
        },
        { // level 2: component
          "condition": "@component:*",
          "name": "{1} Category"
        },
        { // level 3: test category
          [...]
        }
      ]
    }
  ],
  ...
}

In the name level setting, the placeholder {1} refers to the first wildcard, {2} to the second wildcard (if present), and so on.

With this configuration, the generated hierarchy would become:

<root>
└ Payments
   ├ FrontEnd Category
   │  ├ Smoke
   │  └ Regression
   └ BackEnd Category
      └ Regression

The matched level name may contain multiple hierarchy path nodes separated by /. This can be used for special cases, for example a tag @frontEndSmoke could be mapped to the hierarchy node path FrontEnd/Smoke.

The _ character in the matched wildcards are automatically transformed to space by default. So for example the tag @category:FrontEnd_Category using the condition @category:* would define a node name FrontEnd Category. This behavior can be disabled by setting the disableUnderscoreTransformation hierarchy setting to true.

Handling non-matching levels

Ideally the value for each level can be detected based on the source path and the tags. In some cases some cases the scenarios are classified to a "default" category. For example the team may decide that if the scenario has no @component:xxx tag, it should belong to the "General" component. They might also decide to just include these Test Cases to the parent hierarchy node.

SpecSync provides multiple options to handle this situation by specifying the onNotMatching level setting. For example the following configuration uses a "General" component if the @component:xxx tag is missing.

specsync.json
{
  ...
  "hierarchies": [
    {
      "type": "levels",
      "levels": [
        { // level 1: area
          [...]
        },
        { // level 2: component
          "condition": "@component:*",
          "onNotMatching": "customName",
          "nameForNotMatching": "General"
        },
        { // level 3: test category
          [...]
        }
      ]
    }
  ],
  ...
}

The following table contains the possible values for the onNotMatching level setting. The default is ignore for top level and skipLevel for the other levels.

Value
Description

ignore

The Test Case is ignored and will not be included to this hierarchy. In the example above, if @component:xxx tag is missing the Test Case would not be included in the hierarchy.

skipLevel

The non-matching level is skipped, so all levels below would be promoted by one. In the example above, if @component:xxx tag is missing the Test Case would be included in a node like "Payments / Smoke".

includeToParent

The Test Case is included to the hierarchy node of the previous (parent) level node but the remaining levels are not processed. In the example above, if @component:xxx tag is missing the Test Case would be included in the "Payments" node, regardless of whether it has @smoke tag.

customName

A custom name is used instead that has to be specified using the nameForNotMatching level setting. In the example above, if @component:xxx tag is missing a "General" component level can be used instead.

The tag hierarchy type

The tag hierarchy type defines the hierarchy based on the hierarchy node paths specified using tags of a specified tag prefix.

Consider the following feature file.

CardPayments.feature
Feature: Card Payments

Rule: Card payment can be initiated from the UI

  @suite:Payments/FrontEnd/Smoke
  Scenario: Card payment is started (#1)
  [...]
  
  @suite:Payments/FrontEnd/Regression
  Scenario: Card payment is successful (#2)
  [...]

Rule: Card payment is logged

  @suite:Payments/BackEnd/Regression
  Scenario: Card payment is added to transaction log (#3)
  [...]

The scenarios in the example above are tagged with a @suite:xxx tag that specifies the hierarchy node path (Test Suite path in Azure DevOps).

The following hierarchy configuration can be used to define this hierarchy.

specsync.json
{
  ...
  "hierarchies": [
    {
      "type": "tag",
      "tagPrefix": "suite"
    }
  ],
  ...
}

Synchronizing the feature file above with this configuration would generate the following hierarchy.

<root>: Empty
└ Payments: Empty
   ├ FrontEnd: Empty
   │  ├ Smoke: Contains #1
   │  └ Regression: Contains #2
   └ BackEnd: Empty
      └ Regression: Contains #3

SpecSync uses colon : as a default separator for values specified with tag prefixes. The allowed separators can be configured using the synchronization/tagPrefixSeparators configuration setting. For example setting this to [':', '='] would allow defining the path with both @suite:Payments/FrontEnd/Smoke and @suite=Payments/FrontEnd/Smoke.

The / character in the tag name is used to define the hierarchy structure. If a node name needs to have the / character, it has to be wrapped with quotes (' or "). For example the tag @suite:"A/C"/Smoke defines a two-level hierarchy, with the first level named A/C and the second level as Smoke.

The _ character in the tag name is automatically transformed to space by default. So for example the tag @suite:Payments/FrontEnd_Category would define a node with the second level as FrontEnd Category. This behavior can be disabled by setting the disableUnderscoreTransformation hierarchy setting to true.

The single hierarchy type

The single hierarchy type defines a hierarchy with a single node and all Test Cases that match to the optionally defined condition are included to that node.

The following configuration defines a hierarchy with a single "BDD Scenarios" node(in Test Plan "My Plan") and includes all synchronized test cases that does not have an @ignore tag.

specsync.json
{
  ...
  "hierarchies": [
    {
      "type": "single",
      "condition": "not @ignore",
      "node": {
        "name": "BDD Scenarios",
        "testPlan": "My plan"
      }
    }
  ],
  ...
}

Specifying the root of the hierarchy using the root setting is not required, but if specified the settings in node overwrite the settings in root.

If condition is not specified, all Test Cases are included into the hierarchy node.

The custom hierarchy type

The custom hierarchy type defines a hierarchy by configuring each node separately. For each node you can specify the node name or path and a condition that specifies which Test Cases should be included in that node. The test case will be included to the first node with a matching condition.

Consider the following feature file.

Features/Payments/CardPayments.feature
Feature: Card Payments

Rule: Card payment can be initiated from the UI

  @smoke
  Scenario: Card payment is started (#1)
  [...]
  
  @regression
  Scenario: Card payment is successful (#2)
  [...]

Rule: Card payment is logged

  @regression
  Scenario: Card payment is added to transaction log (#3)
  [...]
specsync.json
{
  ...
  "hierarchies": [
    {
      "type": "custom",
      "nodes": [
        {
          "path": "Payments/Smoke",
          "condition": "$sourceFile ~ Features/Payments/ and @smoke"
        },
        {
          "path": "Payments/Regression",
          "condition": "$sourceFile ~ Features/Payments/ and @regression"
        },
        {
          "name: "General"
        }
      ]
    }
  ],
  ...
}

The node configurations within the nodes setting cannot be nested, but you can specify the nodes using path instead of name to build up a nested node hierarchy.

Synchronizing the feature file above with this configuration would generate the following hierarchy.

<root>: Empty
└ Payments: Empty
   ├ Smoke: Contains #1
   └ Regression: Contains #2, #3

The third configured node ("Generic") does not have a condition, so it will include all Test Cases that have not been included by the conditions of the prior nodes. This means that if there is an additional scenario that is not within the Features/Payments/ folder or does not have neither a @smoke nor a @regression tag, it will be included here.

This feature description describes the usage of this feature. You can also check the reference guide of the .

The following table contains an overview of the supported hierarchy types. The sections below contain further details for the different types. The can be used for each hierarchy type.

Defines the hierarchy based on the folders and sub-folders of the local test case documents (e.g. feature files). Each node will contain the local test cases (scenarios) of every document that is located directly in the related folder. See details .

Defines the hierarchy based on the folders, sub-folders and file names of the local test case documents (e.g. feature files). The file name extension is omitted. Each node will contain the test cases of the related local test case document. See details .

Similar to foldersAndFiles, but for the leaf nodes, it uses the name of the local test case document instead of the file name when available. For example for feature files it uses the feature name (the name you specify after the Feature: prefix in the feature file). See details .

Defines a hierarchy where the different levels correspond to a particular property of the test cases. E.g. the top level is defined by the area of the test: "Payment", "Inventory", etc; and the second level is defined by the test type: "UI", "API", etc. The Test Cases will be included in nodes, like "Payment / API". See details .

The hierarchy nodes are defined by tags with a prefix, e.g. @suite:Payment/API. See details .

Defines a hierarchy with a single node and all Test Cases that match to the optionally defined condition are included to that node. This hierarchy type can be seen as an direct alternative to the feature SpecSync provided earlier. See details .

Specifies a hierarchy by specifying each node with a condition that defines which test case should be included to that node. The test case will be included to the first node with a matching condition. See details .

condition: A to specify which test cases should be included to this hierarchy. By default all synchronized Test Cases is included.

If the files are not directly in the SpecSync configuration root folder, but in a sub-folder, you can consider using the skipFolderPrefix setting, see details .

The <root> node can be mapped to a root location in Azure DevOps, see for details. In Azure DevOps the root location can be a Test Suite in a Test Plan (or the root Test Suite of the Test Plan).

In some cases a particular aspect cannot be detected (e.g. the scenario has neither @regression nor @smoke tag). You can configure how SpecSync should handle this case with the onNotMatching and the nameForNotMatching level settings. See details .

The levels can be specified with conditions similar to the ones that are used for .

This hierarchy type can be seen as an direct alternative to the feature SpecSync provided earlier.

This hierarchy type can be seen as an direct alternative to the customization SpecSync provided earlier.

The following configuration defines a hierarchy similar to the example we defined using the .

synchronizing local test case details
test results
remote scope
Enterprise license
hierarchies configuration setting
local test case condition
updating Test Case fields
Test Suite synchronization
Add Test Cases to Suites
common hierarchy settings
below
common hierarchy settings
below
levels type
below
below
below
below
below
Test Suite synchronization
below
below