Introducing the Unreal Engine Plugin for TeamCity

TeamCity is a popular choice in game development for several reasons, including its support for Perforce, its ability to be installed on premises, and its numerous configuration options. In addition to its main application as a general-purpose CI/CD solution, we’re also striving to bring dedicated support for various build tools, too. Having already provided a Unity plugin for several years, we’re now excited to officially introduce our Unreal Engine Support plugin!

We’ve been in close contact with several of our game development customers throughout the development of this new plugin to ensure it meets the needs of DevOps teams working on real-world Unreal Engine projects.

Setting up a proper build pipeline for an Unreal Engine game can be a daunting task, especially for those with limited experience of the platform’s unique peculiarities. In this blog post, we’ll walk you through a basic setup, while showcasing the plugin’s features and demonstrating advanced capabilities such as distributed BuildGraph support.

Key features

Here’s a quick overview of what you can expect from this version of the plugin:

• A dedicated runner tailored for most common use cases (BuildCookRun, BuildGraph, and automation tests).
• Build distribution based on your BuildGraph description.
• On-the-fly automation test reporting.
• Automatic discovery of Unreal Engine installations on the build machines.
• Compatibility with the latest 5.x versions of Unreal Engine, plus support for 4.x.

The demo

In this demo, we’ll be setting up a pipeline for two starter games shipped with Unreal: Cropout and Lyra. We’ll use AWS infrastructure (EC2 and FSx for OpenZFS) for build agents and TeamCity Cloud with the Unreal Engine plugin installed.

We’ll look at two scenarios: when the engine is already installed on an agent and when it’s built from source along with the game. At the moment, TeamCity Cloud doesn’t offer agents with the preinstalled Unreal Engine, but you can always add your own self-hosted agent with all the required SDKs. This is the approach we’ll stick to in this blog post.

Building Cropout with BuildCookRun

Building an Unreal project usually involves many steps, such as:

• Compilation for the target platforms in the specified configurations.
• Asset cooking (converting all of the assets into ones that can be read on the target platforms).
• Packaging of the project into a proper distribution format.
• And many more.

Then, of course, there’s the testing!

To begin with, let’s start with a simple scenario and run all of these stages sequentially on a single machine utilizing the standard BuildCookRun command.

For this scenario, we’ll use the latest vanilla Unreal Engine version available at the moment of writing, installed from EGL (Epic Games Launcher).

After successfully connecting the agent to our TeamCity Cloud server, we can see it in the Agents tab.

At this point, I’d like to take a moment to briefly explore some of the agent’s properties.

Here, looking at the picture above, we see that the agent discovered the engine and its version (we have more than one on our machine, as you can see). Keep a note of this information, as we’ll need it later for a build step configuration.

Nowadays, the common approach is to store all your configs as code under source control (i.e. configuration as code). In TeamCity, you can do this using Kotlin DSL. Of course, you can also go with the traditional UI configuration, but today, we’ll use the first approach (due to the popularity of YAML and the fact that it has become a de-facto standard, we’ve added it as a part of our recent release of TeamCity Pipelines – check it out if you haven’t already).

The Kotlin DSL configuration for building Cropout in TeamCity looks like this:

unrealEngine {
    engineDetectionMode = automatic {
        identifier = "5.4"
    }
    command = buildCookRun {
        project = "cropout/CropoutSampleProject.uproject"
        buildConfiguration = standaloneGame {
            configurations = "Development+Shipping"
            platforms = "Mac"
        }
        cook = cookConfiguration {
            maps = "Village+MainMenu"
            cultures = "en"
            unversionedContent = true
        }
        stage = stageConfiguration {
            directory = "./staged"
        }
        archive = archiveConfiguration {
            directory = "./archived"
        }
        pak = true
        compressed = true
        prerequisites = true
    }
    additionalArguments = "-utf8output -buildmachine -unattended -noP4 -nosplash -stdout -NoCodeSign"
}

All of the settings are pretty self-descriptive, but there are a couple worth paying closer attention to:

• engine detection mode
  
  Here, we can specify either “automatic” or “manual”. The former assumes you already have the engine installed on your agent and registered in the system.
  
  Let me take this opportunity to clarify what “registered” means: Whenever you install the Unreal Engine from EGL or build it from source, it writes to certain files on the target machine (or registry in the case of Windows). More information on that can be found here.
  
  This is essentially what the automatic detection mode is for. We read those files and publish identifiers as agent properties, so you can use them later to select the proper agent for your build.
  
  The “manual” mode allows you to set the exact path to the Unreal Engine root folder, which might be useful when you build from source. We’ll touch on that later.
  
• build configuration
  
  By adjusting this parameter, we can specify the type of build we’re creating, whether it’s a standalone game, client, server, or both client and server components.
  
  Depending on the selected value, the plugin will apply -client, -server, -noserver flags and manage -clientconfig, -serverconfig, -config, -targetplatform, and -servertargetplatform parameters accordingly.

So far, so good. Now, it’s time to add some automation tests and run the final pipeline.

unrealEngine {
    engineDetectionMode = automatic {
        identifier = "5.4"
    }
    command = runAutomation {
        project = "cropout/CropoutSampleProject.uproject"
        execCommand = runTests {
            tests = """
                StartsWith:JsonConfig
                Input.Triggers.Released
            """.trimIndent()
        }
        nullRHI = true
    }
    additionalArguments = "-utf8output -buildmachine -unattended -noP4 -nosplash -stdout -NoCodeSign"
}

As you may know, when working with the Unreal Engine Automation Framework, tests are usually executed using one of the following automation “subcommands”:

• RunAll – this is a pretty straightforward command that has the ability to run all required tests.
• RunFilter – with this command, tests tagged with one of the specified filters (which includes such values as Engine, Stress, Smoke, etc.) can be executed.
• RunTests – this is the most versatile command, as it allows you to specify the list of tests to run (including prefix filters with StartsWith, run group filters, and simple substring matches).

We tried to preserve the same wording in Kotlin DSL to keep it familiar for Unreal folks.

It’s also worth mentioning that the plugin parses the results of the tests on the fly and presents them in a nicely formatted manner native to TeamCity.

Here’s what the result of the build looks like:

Below, we can see the game binaries published to the Artifacts Storage. By default, artifacts are published to built-in storage; however, TeamCity also supports various external storage options. In the context of the cloud, it makes sense to configure publishing to S3.

The final configuration in the UI would then look like this:

And here’s the step with the test run:

All the settings are disabled since we’ve enabled Kotlin DSL configuration and disabled editing it in the UI.

Building Lyra with BuildGraph

We’re slowly moving towards a more complex example with Lyra. Since it’s a multiplayer game, let’s build a Linux server (as we know that everything high-performance should be run on Linux!), along with Windows and Linux game clients.

In the previous example, we ran all the steps sequentially on a single machine. Usually, this takes a significant amount of time. But there’s a better way, and it’s called BuildGraph. This is Unreal’s method for describing your build declaratively (or almost) as a set of tasks with dependencies among them. The different parts of the graph can then be executed together or split across different machines. The latter approach allows you to dramatically speed up the entire build process, especially for a big and complex project.

The nice thing about BuildGraph is it manages all of the intermediate artifacts between jobs automatically. All you need is shared storage. While building Lyra, we’ll be using this shared storage for two purposes: sharing data between nodes within a single build and maintaining a shared derived data cache (DDC).

As we mentioned earlier, we’ll also build the engine from the source. There are multiple reasons why you might want to do that, but in our specific case, it’s because there’s no way to build a Linux server from the engine installed from EGL (at least at the time this post was written). This is due to the lack of files and SDKs provided with this particular engine version.

Okay, that’s enough preamble – let’s get to grips with the code.

The structure of the project and corresponding streams in Perforce look like this (we assume that you already have the Unreal Engine source code from Epic’s Perforce):

Here’s the excerpt of the BuildGraph XML script that we’ll use for our build. This particular part of the script shows how we’ll build the editor along with the tools required for the compilation. It also contains a few tests, too. In our simple example, these are hardcoded. In a real-world scenario, you’d likely pass a list of tests to the script and employ more complicated logic.

...
<!-- Editors -->
  <ForEach Name="Platform" Values="$(EditorPlatforms)" Separator="+">
    <Property Name="Compiler" Value="$(AgentPrefixCompile)$(Platform)" />

    <Agent Name="Build Editor and tools $(Platform)" Type="$(Compiler)">
		    
		    ...

      <Property Name="ExtraToolCompileArguments" Value="$(ExtraToolCompileArguments) -architecture=arm64" If="'$(Platform)' == 'Mac'" />
      
      <Node Name="$(ToolsNodeName) $(Platform)" Requires="Setup Toolchain $(Platform)" Produces="#$(Platform)ToolBinaries">
        <Compile Target="CrashReportClient" Platform="$(Platform)" Configuration="Shipping" Arguments="$(ExtraToolCompileArguments)" Tag="#$(Platform)ToolBinaries"/>
        <Compile Target="CrashReportClientEditor" Platform="$(Platform)" Configuration="Shipping" Arguments="$(ExtraToolCompileArguments)" Tag="#$(Platform)ToolBinaries"/>
        <Compile Target="ShaderCompileWorker" Platform="$(Platform)" Configuration="Development" Arguments="$(ExtraToolCompileArguments)" Tag="#$(Platform)ToolBinaries"/>
        <Compile Target="UnrealLightmass" Platform="$(Platform)" Configuration="Development" Tag="#$(Platform)ToolBinaries"/>
        <Compile Target="InterchangeWorker" Platform="$(Platform)" Configuration="Development" Arguments="$(ExtraToolCompileArguments)" Tag="#$(Platform)ToolBinaries"/>
        <Compile Target="UnrealPak" Platform="$(Platform)" Configuration="Development" Arguments="$(ExtraToolCompileArguments)" Tag="#$(Platform)ToolBinaries"/>
        <Compile Target="BootstrapPackagedGame" Platform="$(Platform)" Configuration="Shipping" Arguments="$(ExtraToolCompileArguments)" Tag="#$(Platform)ToolBinaries" If="'$(Platform)' == 'Win64'"/>
      </Node>
      
      ...
      
      <Node Name="$(EditorNodeName) $(Platform)" Requires="$(ToolsNodeName) $(Platform)" Produces="#$(Platform)EditorBinaries">
        <Compile Target="$(ProjectName)Editor" Project="$(UProject)" Platform="$(Platform)" Configuration="Development" Arguments="$(ExtraEditorCompileArguments)" Tag="#$(Platform)EditorBinaries"/>
      </Node>

			<Property Name="AutomationTestsNodeName" Value="Run Tests $(Platform)" />
      <Node Name="$(AutomationTestsNodeName)" Requires="$(EditorNodeName) $(Platform)">
        <Property Name="TestArgs" Value="-Project=$(UProject) -NullRHI -Deterministic" />
        <Property Name="TestArgs" Value="$(TestArgs) -Test=UE.EditorAutomation -RunTest=&quot;StartsWith:Input&quot;" />
        <Property Name="TestArgs" Value="$(TestArgs) -Build=Editor -UseEditor" />
        <Command Name="RunUnreal" Arguments="$(TestArgs)" />
      </Node>
    </Agent>

    <Property Name="BuildNodes" Value="$(BuildNodes);$(EditorNodeName) $(Platform);$(AutomationTestsNodeName);" />
  </ForEach>
  
  ...

The full syntax is described in full on Epic’s website. Here, what we’re essentially doing is iterating over the list of platforms passed to the script via an option EditorPlatforms and building the Editor for each. One interesting feature of this part of the code is the Type property of the agent node.

<Agent ... Type="">

See this table from the documentation:

So, in order to run a set of tasks on a particular agent, you can manipulate this field. At the time of writing, to make everything work, this part requires a little configuration of your build agents. Namely, you should define two properties in the agent’s configuration file:

• unreal-engine.build-graph.agent.type: This could be any value (or a list of values separated by ;). The corresponding set of tasks will then be run only on an agent with at least one match.
• unreal-engine.build-graph.agent.shared-dir: As discussed earlier, in order to run distributed BuildGraph builds, we need a shared storage location. With this property, we set a path to this storage location on a specific agent.

This is what our setup for our EC2 build agents looks like:

• For Linux:

unreal-engine.build-graph.agent.type = CompileLinux;CookLinux;Linux
unreal-engine.build-graph.agent.shared-dir = /mnt/agent-shared-dir/intermediate

• And for Windows (for some reason, mounting the folder as a separate drive caused one of the Windows system calls to fail, so we opted for a network share in the configuration):

unreal-engine.build-graph.agent.type = CompileWin64;CookWin64;Win64
unreal-engine.build-graph.agent.shared-dir = \\\\fs-040b8d6dab476baf1.fsx.eu-west-1.amazonaws.com\\fsx\\

For real-world scenarios, it might make sense to differentiate between agents that perform cooking and those that perform compilation and ensure they have the appropriate specs.

Let’s quickly take a look at what the processes of compilation, cooking and packaging of the game look like. In our script, we’ve separated the logic for building a client and building a server (as they differ in terms of passed flags):

<ForEach Name="Platform" Values="$(ClientPlatforms)" Separator="+">
    <!-- COMPILATION -->
    <Property Name="Compiler" Value="$(AgentPrefixCompile)$(Platform)" />
    <Agent Name="Compile $(Platform) Client" Type="$(Compiler)">

      <Property Name="CompileNodeName" Value="Compile $(Platform) Client" />  
      <Node Name="$(CompileNodeName)" Requires="$(ToolsNodeName) $(Platform)" Produces="#$(Platform) Client Binaries">
        <ForEach Name="TargetConfiguration" Values="$(TargetConfigurations)" Separator="+">    
          <Compile Target="$(ProjectName)Client" Project="$(UProject)" Platform="$(Platform)" Configuration="$(TargetConfiguration)" Arguments="$(ExtraProjectCompileArguments)" />
        </ForEach>
      </Node>

      <Property Name="BuildNodes" Value="$(BuildNodes);$(CompileNodeName);" />
    </Agent>

    <!-- COOKING -->
    <Property Name="Cooker" Value="$(AgentPrefixCook)$(Platform)" />

    <Property Name="CookPlatformNodeName" Value="Cook $(Platform) Client" />
    <Agent Name="Cook $(Platform) Client" Type="$(Cooker)">
      <Property Name="CookPlatform" Value="$(Platform)" />
      <Property Name="CookPlatform" Value="Windows" If="'$(Platform)' == 'Win64'" />

      <Node Name="$(CookPlatformNodeName)" Requires="$(EditorNodeName) $(Platform)" Produces="#Cook $(Platform) Client Complete">
        <Cook Project="$(UProject)" Platform="$(CookPlatform)Client"/>
      </Node>
    </Agent>

    <Property Name="BuildNodes" Value="$(BuildNodes);$(CookPlatformNodeName);" />

    <!-- PACKAGING -->
    <Agent Name="Package $(Platform) Client" Type="$(Platform)">
      <Property Name="BCRArgs" Value="-Project='$(UProject)' -Platform=$(Platform) -NoCodeSign -Client" />

        <!-- Stage -->
      <Node Name="Stage $(Platform) Client" Requires="Compile $(Platform) Client;Cook $(Platform) Client">
        <ForEach Name="TargetConfiguration" Values="$(TargetConfigurations)" Separator="+">
          <Command Name="BuildCookRun" Arguments="$(BCRArgs) -Configuration=$(TargetConfiguration) -SkipBuild -SkipCook -Stage -Pak" />
        </ForEach>
      </Node>

      <!-- Package -->
      <Node Name="Package $(Platform) Client" Requires="Stage $(Platform) Client">
        <ForEach Name="TargetConfiguration" Values="$(TargetConfigurations)" Separator="+">    
          <Command Name="BuildCookRun" Arguments="$(BCRArgs) -Configuration=$(TargetConfiguration) -SkipBuild -SkipCook -SkipStage -Package" />
        </ForEach>
      </Node>

      <!-- Publish (Packages) -->
      <Node Name="Archive $(Platform) Client" Requires="Package $(Platform) Client">
        <ForEach Name="TargetConfiguration" Values="$(TargetConfigurations)" Separator="+">    
          <Command Name="BuildCookRun" Arguments="$(BCRArgs) -Configuration=$(TargetConfiguration) -SkipBuild -SkipCook -SkipStage -SkipPak -SkipPackage -Archive" />
        </ForEach>
      </Node>

      <Node Name="Publish $(Platform) Client" Requires="Archive $(Platform) Client">
        <Property Name="PublishPlatform" Value="$(Platform)" />
        <Property Name="PublishPlatform" Value="Windows" If="'$(Platform)' == 'Win64'" />
        <Log Message="##teamcity[publishArtifacts 'game/ArchivedBuilds/$(PublishPlatform)Client=>$(PublishPlatform)Client.zip']" />
      </Node>
    </Agent>

    <Property Name="BuildNodes" Value="$(BuildNodes);Archive $(Platform) Client;Publish $(Platform) Client" />
  </ForEach>

The script is pretty self-explanatory. We iterate over the list of platforms on which we want our client to run, which is again passed as an option. We compile for each of the platforms, cook the assets, and finally package everything. As the last step of the packaging process, we use a service message to publish the built client as a TeamCity build artifact.

As you might have noticed, in this part of the script, we have three agents representing three stages of the build process (compilation, cooking, and packaging). The compilation requires tools belonging to the editor, but not the editor itself:

<Node Name="$(CompileNodeName)" Requires="$(ToolsNodeName) $(Platform)" Produces="#$(Platform) Client Binaries">

Meanwhile, the cooking process requires the actual editor:

<Node Name="$(CookPlatformNodeName)" Requires="$(EditorNodeName) $(Platform)" Produces="#Cook $(Platform) Client Complete">

And since there is no dependency between these two processes, they can run in parallel on different machines (if you have enough machines available). This simple example demonstrates the power of using BuildGraph: You simply describe pieces of work that share dependencies, and then they run simultaneously.

In the interest of brevity, we won’t show the rest of the script that includes the server part, as it looks very similar to what we just described, with only the set of flags differing.

Finally, we’ve reached the part with the plugin configuration in TeamCity. Since most of the work is described in the BuildGraph XML script, the DSL configuration is fairly brief:

params {
    param("env.UE_SharedDataCachePath", "/mnt/agent-shared-dir/ddc")
    param("env.UE-SharedDataCachePath", "\\\\fs-040b8d6dab476baf1.fsx.eu-west-1.amazonaws.com\\fsx\\ddc")
}

steps {
    unrealEngine {
        id = "Unreal_Engine"
        name = "Build"
        engineDetectionMode = manual {
            rootDir = "engine"
        }
        command = buildGraph {
            script = "game/BuildProject.xml"
            targetNode = "BuildProject"
            options = """
                ProjectPath=%teamcity.build.checkoutDir%/game
                ProjectName=Lyra
                ClientPlatforms=Linux+Win64
                ServerPlatforms=Linux
                EditorPlatforms=Linux+Win64
                TargetConfigurations=Shipping
            """.trimIndent()
            mode = UnrealEngine.BuildGraphMode.Distributed
        }
        additionalArguments = "-utf8output -buildmachine -unattended -noP4 -nosplash -stdout -NoCodeSign"
    }
}

Here, we specify that we’d like to run the given BuildGraph script in distributed mode. Nevertheless, there’s always an option to run all the nodes sequentially on a single machine if we want to. We’ve also specified a couple of environment variables that are required for the engine to enable the usage of a shared DDC. Those folders should already be mounted to the agents connected to TeamCity.

Now, we can pass the following list of three platform options for our game: ClientPlatforms, ServerPlatforms, and EditorPlatforms.

In the UI, it looks like this:

When we launch the build, we get this build chain:

So, as we can see, the plugin has transformed the graph into a proper TeamCity build chain.

We can check the published artifacts on the corresponding tab of the specific build. Here’s what it looks like for the Linux server:

Before finishing up, we’d like to emphasize some important points to bear in mind:

• In order for the distribution process of BuildGraph to work, your build configuration should contain exactly one active build step.
• Currently, the plugin does not manage the retention period of the produced artifacts in the shared storage in any way. It’s your responsibility to set it up properly.
• Examples in this blog post are by no means production-ready. Your real-world scenarios will likely be more complex. However, they could serve as a good starting point.

Try it out!

You can get the demo code shown in this blog post from the GitHub repository.

If you’d like to try out the Unreal Engine plugin for TeamCity, you can download it from JetBrains Marketplace and install it on a TeamCity On-Premises server.

For TeamCity Cloud users, the Unreal Engine plugin has been pre-installed, allowing you to use it simply by adding an Unreal Engine build step. As a reminder, TeamCity Cloud agents don’t have Unreal Engine pre-installed, so you will need to use a self-hosted agent to run an Unreal Engine build.

An important note for those who have been using a preview version of the plugin (any version that starts with 0.x.x): Please take into account that your existing configurations will be broken, as we have made changes and redesigned several features in the 1.0.0 version. You may need to recreate those configurations using the new version of the plugin.

What’s next?

First off, we’re looking forward to receiving your feedback. Please feel free to get in touch by submitting a ticket via our issue tracker or by commenting on this blog post.

There are also some ideas we have in mind for further work, including:

• Providing a more in-depth build log analysis.
• Including integration with UnrealGameSync (UGS), with features like build status publication and the potential provision of an out-of-the-box metadata server.
• Adding detection of the SDKs available on build machines, possibly using Turnkey.
• Looking into Gauntlet Automation Framework and what we can do there.
• Looking into how we can leverage recent advancements in the build process introduced by Epic Games, namely, checking out Unreal Build Accelerator (UBA) and coordinating it on the agents from TeamCity.
• Open-sourcing the plugin. We believe that this will increase transparency and allow for the creation of a better plugin overall. Furthermore, going open source can be a lot of fun.
• Anything else you can think of! If you have an idea of something that would be nice to have, please feel free to reach out via any of the channels we mentioned above or simply by leaving a comment on this blog post.