Simple Fork-Join Framework With Matrix Builds

The recently released TeamCity 2023.11 comes with the long-awaited matrix build feature. Matrix build is a build that executes the same set of steps on different combinations of input parameters, producing a matrix with the result of every execution. This is a classic approach to testing code changes on different architectures and operating systems.

Matrix builds in TeamCity allow exactly this, while using the Fork-Join pattern under the hood. Let’s see how this works.

Applying the Fork-Join pattern to builds in TeamCity

The Fork-Join pattern is a programming technique designed for situations where certain tasks can be done in parallel. You spawn a set of tasks/threads (fork), wait for those tasks/threads to be executed (join), and then combine all the results.

Let’s assume that we’d like to use the same Fork-Join approach for our builds in TeamCity. A natural choice would be to create several build configurations for the parallel activities and add a composite build that would have snapshot dependencies on all these build configurations.

Here the Preparation build prepares something that can be used by the Task builds. Run All is a composite build that waits for all the tasks to finish.

If we’re using the Kotlin DSL, then we can create any number of such build configurations relatively easily. We might end up with something like:

import jetbrains.buildServer.configs.kotlin.*
import jetbrains.buildServer.configs.kotlin.buildSteps.script

version = "2023.11"

project {
   val tasks = mutableListOf<Task>()
   for (num in 1..3) {
       val task = Task(num)
       tasks.add(task)
       buildType(task)
   }

   buildType(RunAll(tasks))
   buildType(Preparation)
}

class RunAll(tasks: List<Task>) : BuildType({
   name = "Run All"

   type = Type.COMPOSITE

   dependencies {
       for (task in tasks) {
           snapshot(task) {}
       }
   }
})

class Task(taskNum: Int) : BuildType({
   id("Task$taskNum")
   name = "Task $taskNum"

   steps {
       script {
           scriptContent = """
               echo "Running Task $taskNum"
           """.trimIndent()
       }
   }

   dependencies {
       snapshot(Preparation) {}
   }
})

object Preparation : BuildType({...})

If we want to use a web interface, we’d probably first create a template and then base our Task build configurations on it. Obviously, using a web interface for setup will involve a lot more clicking, but in the end the result will be the same.

What does this have to do with matrix builds?

Let’s change our Kotlin DSL example by utilizing the matrix build feature:

import jetbrains.buildServer.configs.kotlin.*
import jetbrains.buildServer.configs.kotlin.buildSteps.script

version = "2023.11"

project {
   buildType(RunAll)
   buildType(Preparation)
}

object RunAll : BuildType({
   name = "Run All"

   features {
       matrix {
           param("taskNum", listOf(
               value("1"),
               value("2"),
               value("3")
           ))
       }
   }

   steps {
       script {
           scriptContent = """
               echo "Running Task %taskNum%"
           """.trimIndent()
       }
   }

   dependencies {
       snapshot(Preparation) {}
   }
})

object Preparation : BuildType({...})

You can see that our RunAll build configuration is now a normal / non-composite build that has a matrix build feature with a taskNum parameter, and this taskNum parameter is being used in the build step. There are no Task(s) and no snapshot dependencies on them; the only snapshot dependency left is Preparation.

When we trigger RunAll, TeamCity will analyze the matrix feature parameters and will create a new set of builds with the same settings as the triggered one, but each build gets its own value from the matrix parameter. The main build is then transformed into a composite one, and snapshot dependencies are added to the generated builds.

Essentially, the build becomes a build chain that resembles the same Fork-Join pattern:

All we had to do was to add a matrix build with a single parameter with multiple values and make our steps change behavior based on the value of the parameter (which we’d need to do in any case).

Another advantage is that the generated builds are being placed in the auto-generated build configurations, which are not being shown in the normal TeamCity build configuration views. This significantly reduces clutter. You can still navigate to the generated builds using direct links and investigate the build history there. 

Since the matrix build is a composite build, all the results are being accumulated and shown in a single place automatically. For instance, if individual builds run tests, then all of them will be visible on the matrix build’s Tests tab. Moreover, if individual builds publish artifacts, the artifacts will also be combined into a single artifact tree and shown in the matrix build.

As you can see, matrix builds can be used to parallelize activities in your builds. For instance, you might want to split a single test suite into several smaller ones. In this case, you can set up a matrix build with a “suite” parameter, whose value might be a list of all these separate test suites. Subsequently, you simply feed this suite parameter value into your build step without generating any extra build configurations and without observing them constantly in the UI. 

By the way, if you need to run different build steps depending on matrix build parameters, then you can use conditional steps.

Happy building!