Home Blog Running Different Tasks in Parallel Loops

Running Different Tasks in Parallel Loops

Listen to this blog post!

Table of contents:

In the previous parts, you learned two concepts for speeding up code:

Parallel loops: Execute the same script block multiple times. -ThrottleLimit sets the maximum number of threads that can run simultaneously.
Jobs: Execute a single script block in the background. By starting multiple jobs, you can shift several tasks to the background. Jobs do not have a -ThrottleLimit because they run independently of each other.

‍

To conclude this introductory section on parallelization, let’s combine the benefits of parallel loops and jobs: why not execute different script blocks within a parallel loop?

Executing different script blocks in a parallel loop combines the features of parallel loops and jobs. Unlike jobs, this approach has built-in throttling, allowing you to actively control how many threads are used to execute the list of script blocks.

PowerShell 7

Here is a script that can handle any number of tasks (script blocks). Note that a script block can span multiple lines and include hundreds of commands.

# list of tasks you need to do
$jobArray =
{ Get-Process },
{ Get-Service },
{ Get-Date }

# number of threads you want to invest
$ThrottleLimit = 3

# feed the script blocks in...
$result = $jobArray |
  # wrap each script block in an object with some metadata
  # so we later know where the result belongs
  ForEach-Object -Begin { $id = 0 } -Process {
    [PSCustomObject]@{
      Id     = $id++
      Code   = $_
      Result = $null
    }
  } |
  # execute script block in parallel
  ForEach-Object -Parallel {
    # execute code and save result, then return the wrapper object
    $_.Result = & $_.Code
    $_
  } -ThrottleLimit $ThrottleLimit |
  # group all results by their ID so they can be associated to the
  # original command
  Group-Object -Property Id -AsHashTable

# getting to the results of the first script block
$result[0].Result

These script blocks are executed by ForEach-Object -Parallel. However, there is one caveat: when you run script blocks in parallel, results may not arrive sequentially.

That’s why a separate ForEach-Object wraps each script block in an object with metadata: it receives a numeric ID and the original script block, so we can later easily associate results with the corresponding task.

At the end, Group-Object adds an index to the results using the id property. This makes it simple to access the results for a particular script block: the results for the first script block are always in $result[0], the results for the second script block in $result[1], and so on.

Windows PowerShell

The script for Windows PowerShell looks almost identical. It simply uses Invoke-Parallel instead of ForEach-Object -Parallel and requires that you have installed the PSParallel module from the PowerShell Gallery.

# list of tasks you need to do
$jobArray = 
{ Get-Process },
{ Get-Service },
{ Get-Date } 

# number of threads you want to invest
$ThrottleLimit = 3

# feed the script blocks in...
$result = $jobArray | 
    # wrap each script block in an object with some metadata
    # so we later know where the result belongs
    Foreach-Object -begin { $id=0 } -process {
        [PSCustomObject]@{
            Id = $id++
            Code = $_
            Result = $null
        }
    } |
    # execute script block in parallel
    Invoke-Parallel {
        # execute code and save result, then return the wrapper object
        $_.Result = & $_.Code
        $_ 
    } -ThrottleLimit $ThrottleLimit |
    # group all results by their ID so they can be associated to the
    # original command
    Group-Object -Property Id -AsHashTable

# getting to the results of the first script block
$result[0].Result

‍

Running Different Tasks in Parallel Loops

PowerShell 7

Windows PowerShell

Related links