Recently, I received a task which required me to run a particular command on a several thousands of servers. Since execution of this command takes some time, it is just logical to run them in a parallel mode. And PowerShell Background Jobs are the first thing comes in mind.
But resources of my PC are limited — it cannot run more than a 100 jobs simultaneously. Unfortunately, PowerShell doesn’t have a built-in functionality for limiting background jobs yet.
Though, I’m not the first one who stuck with the same problem: official “Hey, Scripting Guy!” blog has introduced us a queue based on .NET Framework objects. But I couldn’t manage this solution to work and needed something simpler. After all, all we need are:
- a loop
- a counter, for how much jobs are active and running
- a variable, allowing next job in queue to start
Eventually, I came up with a piece of code like this:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 |
$maxConcurrentJobs = 100 #Max. number of simultaneously running jobs foreach($Object in $Objects) { #Where $Objects is a collection of objects to process. It may be a computers list, for example. $Check = $false #Variable to allow endless looping until the number of running jobs will be less than $maxConcurrentJobs. while ($Check -eq $false) { if ((Get-Job -State 'Running').Count -lt $maxConcurrentJobs) { $ScriptBlock = { #Insert the code of your workload here } Start-Job -ScriptBlock $ScriptBlock $Check = $true #To stop endless looping and proceed to the next object in the list. } } } |
Whereas my version is similar to a solution proposed at StackOverflow (which I only found AFTER completing my own version, of course), the SO version suffers from a bug where some items in queue may be skipped.
While PS Jobs are so easy to play with, Boe Prox claims that runspaces work much and much faster. Go, check it out, how you can use them to accelerate your job queue in his blog.
Thank you very much for the code snippet. Exactly what I was looking for.
Well, finally somebody who knows how to keep it simple. Nice and clean and works like a charm
Your version suffers from a while loop excessively checking/polling for the running status count to change
A better solution would be to use the built-in Wait-Job -Any command-let to check for threads finishing.
The problem with the stack overflow version is that it was doing an if else and it only waited in the else then moved on to the next iteration item without starting a job for it.
My suggestion is to check for the conditions to wait on first.
If the condition is true wait
then start the job – always happens.
$maxConcurrentJobs = 100
foreach ($object in $objects) {
$running = @(Get-Job -State Running)
$freeMemory = (Get-WmiObject win32_OperatingSystem).freephysicalmemory * 1KB
if ($freeMemory -lt 1.5GB -or $running.Count -ge $maxConcurrentJobs) {
$null = $running | Wait-Job -Any
}
$scriptBlock = {
#Insert the code of your workload here
Write-Output “Do your work here…”
}
Start-Job -ScriptBlock $ScriptBlock
Start-Sleep -Seconds 3 # not needed but gives some breathing room.
}
I added the memory check to avoid throwing System.OutOfMemoryException but that value is dependent on your system
Nice catch, man! Didn’t pay enough attention to Wait-Job at the time. Thank you!
@Kirill and @Mackenson
thanks both so much. It’s so simple but effective and exactly what I needed.
I had been asked to supply a script with a list of objects, could be 10, could be 200, whatever the DB spits out, but only do 10 at a time, and NOT break up the script into multiple runs of 10.
I used the “Wait-Job -Any” code and I just added in front of it … write-host ” Waiting … ”
Great stuff.
All great examples, thank you. Simplified it a bit by creating a reusable function for this:
“`powershell
Function Wait-MaxRunningJobsHC {
[CmdletBinding()]
Param (
[Parameter(Mandatory)]
[System.Management.Automation.Job[]]$Name,
[Parameter(Mandatory)]
[Int]$MaxThreads,
[Int]$FreeMemory = 1GB
)
Begin {
Function Get-FreeMemoryHC {
(Get-WmiObject win32_OperatingSystem).FreePhysicalMemory * 1KB
}
Function Get-RunningJobsHC {
@($Name).Where( { $_.State -eq ‘Running’ })
}
}
Process {
while (
((Get-FreeMemoryHC) -lt $FreeMemory) -or
((Get-RunningJobsHC).Count -ge $MaxThreads)
) {
$null = Wait-Job -Job $Name -Any
}
}
}
“`
Hope this might helpful for others