Executing MSBuild Targets in Parallel – Part 1

[ In this series – Part 1, Part 2 ]

Coming in the next release of the MSBuild Extension Pack is a new task which allows you to easily run Targets in parallel. This will be limited to the .NET 4.0 releases, so next up that’s 4.0.5.0. (you can try the task now by getting changeset 74609).

Typically in MSBuild we execute a sequence of targets and MSBuild will do just that; execute sequentially. If you have simple scripts that run quickly then this may not be a problem. If however you have complex scripts that take considerable time to execute, the new parallel functionality may be useful to reduce execution times.

In the following samples I’ll work through how the tasks can be used and various things to be aware of.

Below I have a sample which executes three targets. In each target there is a Sleep task so we can provide some metrics as we extend the sample.

<Project ToolsVersion="4.0" DefaultTargets="Default" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
    <Import Project="$(MSBuildExtensionsPath)\ExtensionPack\4.0\MSBuild.ExtensionPack.tasks"/>
    <Target Name="Default" DependsOnTargets="Target1;Target2;Target3"/>
    <Target Name="Target1">
        <MSBuild.ExtensionPack.Framework.Thread TaskAction="Sleep" Timeout="1000"/>
    </Target>
    <Target Name="Target2">
        <MSBuild.ExtensionPack.Framework.Thread TaskAction="Sleep" Timeout="4000"/>
    </Target>
    <Target Name="Target3">
        <MSBuild.ExtensionPack.Framework.Thread TaskAction="Sleep" Timeout="2000"/>
    </Target>
</Project>

Executing the above sample we get the expected 7 second or so build time.

Assuming there are no dependencies between these targets, the fastest time we we can execute this is around 4 seconds, the time of our slowest target, plus the initialization cost.

Rather than having to run these targets sequentially, what I really want to do is run non-dependent targets in parallel and reduce the build time. In the next sample I’ve removed the DependsOnTargets and added the parallel task to execute BuildTargetsInParallel

<Project ToolsVersion="4.0" DefaultTargets="Default" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
    <Import Project="$(MSBuildExtensionsPath)\ExtensionPack\4.0\MSBuild.ExtensionPack.tasks"/>
    <Target Name="Default">
        <MSBuild.ExtensionPack.Framework.Parallel TaskAction="BuildTargetsInParallel" Targets="Target1;Target2;Target3"/>
    </Target>
    <Target Name="Target1">
        <MSBuild.ExtensionPack.Framework.Thread TaskAction="Sleep" Timeout="1000"/>
    </Target>
    <Target Name="Target2">
        <MSBuild.ExtensionPack.Framework.Thread TaskAction="Sleep" Timeout="4000"/>
    </Target>
    <Target Name="Target3">
        <MSBuild.ExtensionPack.Framework.Thread TaskAction="Sleep" Timeout="2000"/>
    </Target>
</Project>

Executing the above sample we see our build time drop to 4.60 seconds.

This is a great reduction in build time, but not as close to the four second mark as we may have expected. The reason that we have a slightly higher build time is that the parallel task needs to start MSBuild and initialize a new instance of your build file before it executes each target you have requested to execute in parallel. This is an important point to remember if you have a large build file and a high initialization cost. The aim is not to simply run every target in parallel, but rather to find an efficient way to run your build in parallel.

In this first parallel sample I ran all targets in parallel, initiating the build file four times. It may be more efficient to run sets of targets in parallel. The parallel task allows us to specify BuildTargetSetsInParallel.

<Project ToolsVersion="4.0" DefaultTargets="Default" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
    <Import Project="$(MSBuildExtensionsPath)\ExtensionPack\4.0\MSBuild.ExtensionPack.tasks"/>
    <ItemGroup>
        <MyTargetSets Include="1">
            <Targets>Target2</Targets>
        </MyTargetSets>
        <MyTargetSets Include="2">
            <Targets>Target1;Target3</Targets>
        </MyTargetSets>
    </ItemGroup>
    <Target Name="Default">
        <MSBuild.ExtensionPack.Framework.Parallel TaskAction="BuildTargetSetsInParallel" Targets="@(MyTargetSets)"/>
    </Target>
    <Target Name="Target1">
        <MSBuild.ExtensionPack.Framework.Thread TaskAction="Sleep" Timeout="1000"/>
    </Target>
    <Target Name="Target2">
        <MSBuild.ExtensionPack.Framework.Thread TaskAction="Sleep" Timeout="4000"/>
    </Target>
    <Target Name="Target3">
        <MSBuild.ExtensionPack.Framework.Thread TaskAction="Sleep" Timeout="2000"/>
    </Target>
</Project>

Executing this sample we get a build time of 4.18 seconds.

In this case we have initiated 3 rather than 4 instances of our build file and chosen to execute two parallel sets of targets. Note that I have the longest running target running by itself and have the remaining targets running sequentially on another instance. Given that the execution time of the remaining targets was less than Target2, there is no need to run more than two sets at once.

In this simple example we’ve managed to reduce our build time by over 42%.

In Part 2 I’ll cover how we maintain state when running in parallel as well as some limitations and tips.

Mike