NUnit tests on Azure DevOps split across VMs

I have an Azure DevOps build pipeline with NUnit3Adaptor 4.5.0, running integration tests in parallel across 3 VMs. A number of our tests are in a single file, single setup, and many tests that run sequentially, each relying on the previous.

The point is to save the single expensive setup and use it across many tests in sequence, advancing the process and testing it, step-by-step:

[TestFixture]
public class LotsaSequentialTests
{
    [OneTimeSetUp]
    public void SetupForThisClass()
    {
        // Expensive data import and process setup; one-time here
        // ...
    }

    [Test, Order(1)]
    public void CheckO1()
    {
        // I change something in the data and make a check
        // ...
    }

    [Test, Order(2)]
    public void CheckO2()
    {
        // I change something else in the data and make a check
        // ...
    }

    [Test, Order(3)]
    public void CheckO3()
    {
        // I change something else in the data and make another check
        // ...
    }
}

The idea is that I can use a single expensive setup for 10 to 30 tests that run sequentially, each makes a tweak to the data reruns the process, does a check on this stage in the process, and the next test adds another change to the data set, reruns a process and does it’s check.

If one fails, all the subsequent tests fail and that works great for our scenario.

This Works with basedOnTestCases

The rationale is I don’t want to run the expensive data import and setup for each of the 10 to 30 tests, and then having to advance each test to some state before checking. This with NUnit3TestAdapter 4.5.0 and builds locally just fine, and did until recently on Azure DevOps.

The Issue

We recently switched the parallelism from distributionBatchType: basedOnTestCases to distributionBatchType: basedOnExecutionTime hoping to shorten overall time. We are getting some odd behavior. ADO is splitting the test class across multiple VMs, but now it is splitting them differently: rather than running all the tests in the class on a single VM, it is now splitting them across multiple VMs.

Therefore, not only is each VM is now doing the expensive setup (which is what I want to avoid), but worse it is running random sets of tests on each VM, which of course don’t have their predecessors, and so the entire scheme just fails.

Note we are not using NUnit parallelization here.

Desired Solution

With this setup, instruct Azure DevOps to run this class on a single VM. We have 1500 tests, and only 5 or 10 classes use this scheme, and those classes randomly fail depending on how ADO breaks them up.

Here is a snippet of the configuration:

stages:
  - stage: Build
    jobs: 
    - job: Build
      # perform the build and save the test artifacts 4to storage
      
  - stage: Test
    dependsOn: Build
    jobs:
    - job: Test
      strategy:
        parallel: 3
      steps:
      
      - task: DownloadPipelineArtifact@2
        # restore the compiled NUnit tests from storage
        
      - task: VSTest@2
        displayName: 'Run Tests'
        inputs:     
          testAssemblyVer2: |
           **\MyApp.DTO.UnitTests.dll
           **\MyApp.Migrations.IntegrationTests.dll
           **\MyApp.Services.UnitTests.dll
           **\MyApp.Services.IntegrationTests.dll
           **\MyApp.Web.IntegrationTests.dll
           **\MyApp.Services.Import.IntegrationTests.dll
           **\MyApp.Services.ApiTests.dll
           **\MyApp.Services.Thingy.IntegrationTests.dll
          runInParallel: true
          distributionBatchType: 'basedOnExecutionTime'   

There is at least One Recourse

One solution is to go to a single test, remove [Test, Order(x)] from each like so:

[TestFixture]
public class LotsaSequentialTests
{
    [OneTimeSetUp]
    public void SetupForThisClass()
    {
        // Expensive data import and process setup; one-time here
        // ...
    }

    [Test]
    public void CheckAllOfThem()
    {
        CheckO1();
        CheckO2();
        CheckO3();
        CheckO4();
        CheckO5();
        CheckO6();
        // ...
    }
}

But we lose the granularity of the display. Not a big deal, but I really want to learn more about Azure DevOps and taming the beast to my will.

Thank you.