Home Blog Parallelization: Speeding Up Tasks

Parallelization: Speeding Up Tasks

Listen to this blog post!

Table of contents:

PowerShell is single-threaded, so it can execute only one command at a time. However, you can speed up many tasks by running PowerShell code on separate threads.

In fact, this is quite simple when you use a parallel loop. Parallel loops were introduced in PowerShell 7 and can be added to Windows PowerShell through available modules.

Suitable Tasks

When you start with multithreading, make sure you pick tasks that are isolated and do not require communication with other tasks. In previous posts we created a number of functions to ping hosts and to port-scan networks — that’s an excellent use case for parallelization.

I’ll use the simple ping function from earlier posts and speed it up with parallel loops. Here is the function for reference:

function Ping-Computer
{
  param
  (
    [Parameter(ValueFromPipeline,Mandatory)]
    [string]
    $ComputerName,
    
    # timeout in milliseconds
    [int]
    $Timeout = 5000
  )
  
  begin
  {
    $obj = [System.Net.NetworkInformation.Ping]::new()
  }
  process
  {  
    $obj.Send($ComputerName, $timeout) |
      Select-Object -Property Status, Address, RoundTripTime, Success |
      ForEach-Object {
        # fill in the computername/IP address in case there was
        # no response
        $_.Address = $ComputerName
        # add a Boolean value
        $_.Success = $_.Status -eq [System.Net.NetworkInformation.IPStatus]::Success
        $_
      }
  }
  end
  {
    $obj.Dispose()
  }
}

‍

Serial Execution

In a traditional setup, if you wanted to ping your network — for example, an IPv4 range from 192.168.2.1 to 192.168.2.254 — you would use a loop:

1..254 | ForEach-Object {
    # MAKE SURE TO ADJUST THE IP ADDRESS TO MATCH YOUR NETWORK!
    $ip = "192.168.2.$_"

    Ping-Computer -ComputerName $ip -Timeout 1000
}

While this works, it takes about one second per computer — a total of roughly 254 seconds.

Parallel Execution…

With a parallel loop, you can drastically speed things up — provided you keep a few caveats in mind. Let’s start with PowerShell 7, where parallel loops are built in. To execute your task in parallel threads, modify the code above as follows:

1..254 | ForEach-Object -Parallel {
  # MAKE SURE TO ADJUST THE IP ADDRESS TO MATCH YOUR NETWORK!
  $ip = "192.168.2.$_"

  Ping-Computer -ComputerName $ip -Timeout 500
} -ThrottleLimit 128

IMPORTANT: Change the variable $ip if you want to ping a different IPv4 range. The provided code pings addresses from 192.168.2.1 to 192.168.2.254.

The code above instructs ForEach-Object to run the script block in parallel (-Parallel) and to use up to 128 threads simultaneously (-ThrottleLimit 128).

…and Failure!

Disappointingly, the code just throws a bunch of red error messages at you:

On Windows PowerShell, it doesn’t recognize the parameters -Parallel and -ThrottleLimit. That’s expected — Windows PowerShell has no built-in parallel loop. For Windows PowerShell, wait for the next part of this series, where we’ll add a parallel loop manually.
On PowerShell 7, it reports that the command Ping-Computer is missing. We’ll fix this in a moment — it’s a common issue when you start using additional threads to execute code.

Caveat: Understanding Threads

Threads are isolated PowerShell environments that do not share variables or functions with your main script. Because you defined Ping-Computer as a function in your main session, the parallel loop doesn’t recognize it — its code runs in multiple independent threads.

Fortunately, there’s an easy solution:

Modules: The best approach is to load functions from PowerShell modules. Since this happens automatically, any thread — including those used by the parallel loop — can access the required commands from modules. We’ll explore how to store your own functions in custom modules in part five.
Scope: If you don’t want to use modules, simply move your variable and function definitions inside the loop. That’s what we’ll do here:

1..254 | ForEach-Object -Parallel {


  # defining custom functions per thread
  function Ping-Computer {
    param
    (
      [Parameter(ValueFromPipeline, Mandatory)]
      [string]
      $ComputerName,

      # timeout in milliseconds
      [int]
      $Timeout = 5000
    )

    begin {
      $obj = [System.Net.NetworkInformation.Ping]::new()
    }
    process {
      $obj.Send($ComputerName, $timeout) |
        Select-Object -Property Status, Address, RoundTripTime, Success |
        ForEach-Object {
          # fill in the computername/IP address in case there was
          # no response
          $_.Address = $ComputerName
          # add a Boolean value
          $_.Success = $_.Status -eq [System.Net.NetworkInformation.IPStatus]::Success
          $_
        }
      }
      end {
        $obj.Dispose()
      }
    }

    # running  the task in parallel
    # MAKE SURE TO ADJUST THE IP ADDRESS TO MATCH YOUR NETWORK!
    $ip = "192.168.2.$_"
    Ping-Computer -ComputerName $ip -Timeout 500


} -ThrottleLimit 128

Success! What took four minutes before now finishes in just two to three seconds.

Throttling

It’s important to understand that parallel loops have built-in throttling (queuing). Otherwise, PowerShell could quickly run out of resources.

Imagine your loop processing a few thousand elements. Without throttling, it would spawn thousands of threads and quickly run your computer out of memory.

The -ThrottleLimit parameter sets the maximum number of threads the loop can run simultaneously. In the example above, even though there are 256 IP addresses to ping, the loop initially pings only 128 computers (-ThrottleLimit 128). Once a thread finishes, it’s reassigned to process another IP address.

Let’s summarize:

Parallel loops are ideal for executing the same task multiple times.
Throttling (queuing) is always built in.
Since parallel loops execute code in isolated threads and have no access to your main script, you must define everything inside the loop.
Only the information passed to the loop (for example, the numbers in the ping example above) is available inside the loop as $_.

‍