start-job and invoke-vmscript to build VMs in parallel

[squarebox]

I've got a script going to build VMs from a text delimited file. I started my scripting with a one row file (without start-job) but now have switched to using start-job so I can run VM builds in parallel.  

I'm finding the transcript no longer shows the output of my invoke-vmscript calls and some other stuff like since using start-job.  

$row doesn't populate in the transcript either, although the row above does ('Write-Verbose -Verbose "Variables supplied from $using:CSVpath") does. 

Any ideas? 🤔

cls


##################### Parallel VM Build #########################################
$maxConcurrentJobs = 2 # Set the maximum number of VMs to build in parallel

#################################################################################


############################## PREPARE TAB DELIMITED VARIABLES FROM FILE FOR LATER USAGE  #############################################################

# Import the CSV file
Write-Verbose -Verbose "Importing input file from $CSVpath"
Test-Path $CSVpath
$UpdatedCSV=Import-Csv -Delimiter "`t" -Path $CSVpath

############################# OPEN LOG FOLDER TO MONITOR TRANSCRIPTS #################################################

#Set log path (also needed inside the job)
$logpath="\\$IvantiServer\VMBuild\Logs"
invoke-item $logpath


###############################################################################################################################################################################################################
###############################################################################################################################################################################################################
Write-Verbose "Waiting for VMs to finish building..." -Verbose

################# Kick off Script timer #########################

$StopWatch = new-object system.diagnostics.stopwatch
$StopWatch.Start()
#################################################################

# Create array to hold job objects
$jobs = @()

# Loop through each row in the CSV to build VMs in parallel
foreach ($row in $UpdatedCSV) {
    # Start a background job for each VM
    $jobs += Start-Job -ScriptBlock {
        param($row)
        
        # Convert $CSV to their respective variables for the current row
        foreach ($header in $row.PSObject.Properties.Name) {
            Set-Variable -Name $header -Value $row.$header -Force
        }

#Convert $hostname to upper case
$hostname=$hostname.ToUpper()


########################################################################################################################

################# Bring variables in from outside the job ##################################

$LogServer=$using:IvantiServer
$VMLocalCredential=$using:VMLocalCredential
$cred=$using:cred

############################################################################################

#################### Setup logging #####################################################################################

$date=get-date -Format "yyyy-MM-dd HHmm"
$logpath="\\$LogServer\VMBuild\Logs"
$transcriptlog="$hostname-$date.log"
Write-Verbose -Message "Starting transcript" -Verbose

#Start transcript
if ($PSVersionTable.psversion.major -ge 2)  {Start-Transcript -Path "$logpath\transcripts\$transcriptlog"}

#################### Write info to top of transcript ###################################################################

Write-Verbose -Verbose "Script executed by $(whoami) on $(get-date)"
Write-Verbose -Verbose "Script version is $using:Version"
# Write input variables to transcript
Write-Verbose -Verbose "Variables supplied from $using:CSVpath"
$row #




#################### Get connected to vsphere vCenter server ##########################################################################

# Turn off deprecation warnings (won't work with start-job otherwise see https://www.lucd.info/knowledge-base/running-a-background-job/)
Set-PowerCLIConfiguration -DisplayDeprecationWarnings $false -Confirm:$false | Out-Null

# Connect to vCenter
if ($vCenterInstance) {$esx=$vCenterInstance}
Connect-VIServer $esx -Verbose -Credential $cred






##################################################### GET TEMPLATE AND RUN CHECKS ##############################################################################################

#Write-host "Deploying $Hostname" -foregroundcolor Green


#Get template
$SourceVMTemplate = Get-Template -Name $VMTemplate

########################################################################################################################


 
############################# Build the VM #########################################################


Write-Verbose -Message "Checking Virtual Machine with Name: [$Hostname] doesn't already exist." -Verbose
#Check VM doesn't already exist (break if it does)
if (Get-VM -Name $Hostname -erroraction silentlycontinue) {
    Write-Host -ForegroundColor Red "VM already exists.  Exiting"
    stop-transcript
    break
    }
else {#Build the VM if it doesn't already exist
    Write-Verbose -Message "Deploying Virtual Machine with Name: [$Hostname] using Template: [$SourceVMTemplate] and Customization Specification: [$CustomSpec] on Cluster: [$Cluster] and waiting for completion" -Verbose
    New-VM -Name $Hostname -Template $SourceVMTemplate -ResourcePool $cluster -Location $Location -Datastore $Datastore -DiskStorageFormat $DiskStorageFormat -OSCustomizationSpec $CustomSpec 
    }

 
Write-Verbose -Message "Virtual Machine $Hostname Deployed. Powering On" -Verbose
 
Start-VM -VM $Hostname | out-null
 

 
# ------This Section Targets and Executes the Scripts on the VM------
 
# We first verify that the guest customization has finished on on the new VM by using the below loops to look for the relevant events within vCenter. 
 
 

Write-Verbose -Message "Customization of VM $Hostname has started. Checking for Completed Status......." -Verbose
    while($True)
    {
        $DCvmEvents = Get-VIEvent -Entity $Hostname
        $DCSucceededEvent = $DCvmEvents | Where { $_.GetType().Name -eq "CustomizationSucceeded" }
        $DCFailureEvent = $DCvmEvents | Where { $_.GetType().Name -eq "CustomizationFailed" }
  
        if ($DCFailureEvent)
        {
            Write-Warning -Message "Customization of VM $Hostname failed" -Verbose
            return $False  
        }
 
        if ($DCSucceededEvent)  
        {
            break
        }
        Start-Sleep -Seconds 5
    }
Write-Verbose -Message "Customization of VM $Hostname Completed Successfully!" -Verbose



###################### Get network adapter info for later config ###############################################################################################################
# Script to gather network adapter information
$networkScript = @"
    Get-NetAdapter | Select-Object -Property InterfaceAlias
"@

# Execute the script to gather network adapter information on the target VM
$networkAdapters = Invoke-VMScript -ScriptText $networkScript -VM $Hostname -GuestCredential $VMLocalCredential

# Extracting everything after "InterfaceAlias" and excluding the line containing '--------------' and 'InterfaceAlias'
$interfaceInfo = $networkAdapters.ScriptOutput -split [System.Environment]::NewLine | Where-Object {$_ -and $_ -ne '--------------' -and $_ -notlike 'InterfaceAlias*'}

# Initialize an empty array to store custom objects
$interfaceAliases = @()

# Convert each line into a custom object
foreach ($line in $interfaceInfo) {
    $interfaceAliases += [PSCustomObject]@{
        InterfaceAlias = $line.Trim()  # Trim any leading or trailing whitespace
    }
}

# Output the array of custom objects
Write-Verbose "NIC Interfaces found:" -Verbose
$interfaceAliases

#######################################################################################################################################################################################



################# Create script text for IP address and DNS for NICs #################################################################################################################################

#IP to be assigned to NIC1
 if ($IP1) {
 if ($gw1) {#with Default Gateway
    Add-Script "New-NetIPAddress -IPAddress %1 -PrefixLength %2 -InterfaceAlias %3 -DefaultGateway %4" @($IP1, $SubnetLength1, $interfaceAliases.InterfaceAlias[0], $GW1)
    #Rename the NIC
    Add-Script "Get-NetAdapter | where InterfaceAlias -eq %1 | Rename-NetAdapter -NewName %2" @($interfaceAliases.InterfaceAlias[0], $NIC1Description) #set NIC description in Windows
    }
else #without default gateway
    {
    Add-Script "New-NetIPAddress -InterfaceAlias %3 -IPAddress %1 -PrefixLength %2 " @($IP1, $SubnetLength1, $interfaceAliases.InterfaceAlias[0])
    #Rename the NIC
    Add-Script "Get-NetAdapter | where InterfaceAlias -eq %1 | Rename-NetAdapter -NewName %2" @($interfaceAliases.InterfaceAlias[0], $NIC1Description) #set NIC description in Windows
    }
if ($DNSNIC1) {
    Add-Script "Set-DnsClientServerAddress -InterfaceAlias %2 -ServerAddresses %1" @($DNSNIC1,$NIC1Description) 
    }
}

########### Now Set the IP addresses ##################################################################################################################################

# After Customization Verification is done we change the IP of the VM to the value defined 
Write-Verbose -Message "Getting ready to change IP Settings on VM $Hostname with the following scripts:" -Verbose
$scripts

#Execute scripts to set NICs up
foreach ($script in $scripts)
{
    Invoke-VMScript -ScriptText $script[0] -VM $Hostname -GuestCredential $VMLocalCredential #| Out-Null
    if ($script[1]) {Restart-VM $Hostname}
}
 
# NOTE - The Below Sleep Command is due to it taking a few seconds for VMware Tools to read the IP Change so that we can return the below output. 
# This is strctly informational and can be commented out if needed, but it's helpful when you want to verify that the settings defined above have been 
# applied successfully within the VM. We use the Get-VM command to return the reported IP information from Tools at the Hypervisor Layer. 
Start-Sleep 30
$EffectiveAddress = (Get-VM $Hostname).guest.ipaddress[0]
Write-Verbose -Message "Assigned IP for VM [$Hostname] is [$EffectiveAddress]" -Verbose



################### DOMAIN JOIN ######################################################################################################
<#
#Works around issue with PSCreds not being passed properly.  
Thanks https://www.reddit.com/r/PowerShell/comments/v2iqp2/invokevmscript_using_local_admin/ 
and 
https://communities.vmware.com/t5/VMware-PowerCLI-Discussions/Problems-passing-Domain-credentials-to-script-block-on-a-remote/td-p/510612
#>

#Create Script text for domain join
if ($JoinDomainYN.ToUpper() -eq "Y") { 
$domainUn = $cred.UserName
$domainPw = $cred.GetNetworkCredential().Password
$DomJoin = @"
`$invokeDomainUn = '$domainUn'
`$invokeDomainPw = ConvertTo-SecureString -String '$domainPw' -AsPlainText -Force
`$invokeDomainCredObj = New-Object System.Management.Automation.PSCredential(`$invokeDomainUn,`$invokeDomainPw)
Add-Computer -DomainName $Domain -Credential `$invokeDomainCredObj
"@ 
}
Write-Verbose -Message "Joining $hostname to $domain domain" -Verbose
#Perform domain join
Invoke-VmScript -vm $Hostname -ScriptText $DomJoin -ScriptType powershell -GuestCredential $VMLocalCredential  
Write-Verbose -Message "Rebooting after domain join (waits for VMware Tools to respond before continuing script)" -Verbose
#Reboot
Restart-VM $Hostname


########################################## Install Tools ####################################################

#
########################### Prep (Copy and unzip) install files for Installs ###############################

#Script text for checking c:\temp exists
$CheckTempExists=@"
if (!(test-path c:\temp)) {md c:\temp}
"@

#Script text for expanding zip files
$UnpackZips=@"
Get-ChildItem c:\temp -Filter *.zip | Expand-Archive -DestinationPath C:\Temp -Force
"@

#Check c:\temp exists
Invoke-VmScript -vm $Hostname -ScriptText $CheckTempExists -ScriptType powershell -GuestCredential $VMLocalCredential -Verbose

#Copy Trellix Files to VM
Copy-VMGuestFile -Verbose -LocalToGuest -Force -Source "\\$LogServer\VMBuild\Software\Trellix\*.zip" -Destination "c:\temp" -VM $Hostname -GuestCredential $VMLocalCredential

#Unzip zips
Invoke-VmScript -vm $Hostname -ScriptText $UnpackZips -ScriptType powershell -GuestCredential $VMLocalCredential -Verbose


########################################## Install Trellix Agent ####################################################
Write-Verbose -Message "Installing Trellix Agent" -Verbose

$AgentInstall=@"
##Agent
# Specify the path to the Trellix Agent installer
`$installerPath = "C:\Temp\Agent\FramePkg.exe"

# Specify any installation parameters or options
`$installParameters = "/install=agent /s"

try {
    # Start the installation process
    #Write-Host -ForegroundColor green "Installing Trellix Agent"
    `$process = Start-Process -FilePath `$installerPath -ArgumentList `$installParameters -Wait -PassThru
    `$exitcode = `$process.ExitCode

    # Check the exit code of the installation process
    if (`$exitCode -eq 0) {
        Write-Host -ForegroundColor Green "Trellix Agent installation completed successfully."
    } else {
        Write-Host -ForegroundColor Red "Trellix Agent installation failed with exit code: `$exitCode"
    }
} catch {
   
    Write-Host "An error occurred: $($_.Exception.Message)"
    Write-Host -ForegroundColor Yellow "Install parameters were: `$installerPath `$installParameters"
     Get-WinEvent -LogName Microsoft-Windows-PowerShell/Operational -MaxEvents 10 | where LevelDisplayName -eq warning |  select timecreated,LevelDisplayName,message | ft -AutoSize -Wrap
}
"@

Invoke-VmScript -vm $Hostname -ScriptText $AgentInstall -ScriptType powershell -GuestCredential $VMLocalCredential -Verbose

################################### Install Trellix Endpoint Security ########################################
Write-Verbose -Message "Installing Trellix Endpoint Security" -Verbose

$ESPInstall=@"
##Agent
# Specify the path to the Trellix ESP installer
`$installerPath = "C:\Temp\Endpoint_Security_Platform\setupCC.exe"

try {
    # Start the installation process
    `$process = Start-Process -FilePath `$installerPath -Wait -PassThru
    `$exitcode = `$process.ExitCode

    # Check the exit code of the installation process
    if (`$exitCode -eq 0) {
        Write-Host -ForegroundColor Green "Trellix Agent installation completed successfully."
    } else {
        Write-Host -ForegroundColor Red "Trellix Agent installation failed with exit code: `$exitCode"
    }
} catch {
   
    Write-Host -ForegroundColor Red "An error occurred: $($_.Exception.Message)"
    Write-Host -ForegroundColor Yellow "Install parameters were: `$installerPath `$installParameters"
    Get-WinEvent -LogName Microsoft-Windows-PowerShell/Operational -MaxEvents 10 | where LevelDisplayName -eq warning |  select timecreated,LevelDisplayName,message | ft -AutoSize -Wrap
      # Get-WinEvent -LogName Microsoft-Windows-PowerShell/Operational -MaxEvents 10 |   select timecreated,LevelDisplayName,message | ft -AutoSize -Wrap
}
"@

Invoke-VmScript -vm $Hostname -ScriptText $ESPInstall -ScriptType powershell -GuestCredential $VMLocalCredential -Verbose


################################### Install Trellix Threat Prevention ########################################
Write-Verbose -Message "Installing Trellix Threat Prevention" -Verbose


$TPInstall=@"
##Agent
# Specify the path to the Trellix Threat Prevention installer
`$installerPath = "C:\Temp\Threat_Prevention\setupTP.exe"

try {
    # Start the installation process
    `$process = Start-Process -FilePath `$installerPath -Wait -PassThru
    `$exitcode = `$process.ExitCode

    # Check the exit code of the installation process
    if (`$exitCode -eq 0) {
        Write-Host -ForegroundColor Green "Trellix Agent installation completed successfully."
    } else {
        Write-Host -ForegroundColor Red "Trellix Agent installation failed with exit code: `$exitCode"
    }
} catch {
   
    Write-Host -ForegroundColor Red "An error occurred: $($_.Exception.Message)"
    Write-Host -ForegroundColor Yellow "Install parameters were: `$installerPath `$installParameters"
    Get-WinEvent -LogName Microsoft-Windows-PowerShell/Operational -MaxEvents 10 | where LevelDisplayName -eq warning |  select timecreated,LevelDisplayName,message | ft -AutoSize -Wrap
    # Get-WinEvent -LogName Microsoft-Windows-PowerShell/Operational -MaxEvents 10 |   select timecreated,LevelDisplayName,message | ft -AutoSize -Wrap
}
"@

Invoke-VmScript -vm $Hostname -ScriptText $TPInstall -ScriptType powershell -GuestCredential $VMLocalCredential -Verbose


#############################################################################################################





######################  FINAL REBOOT #######################################################################################

# Reboots the machine after provisioning one final time
Write-Verbose -Message "Rebooting $Hostname to Complete Provisioning" -Verbose

#Reboot VM for final time
Restart-VM $Hostname

Write-Verbose -Message "Waiting for VM Tools" -Verbose
Wait-Tools -VM $Hostname -TimeoutSeconds 300 | out-null

Write-Verbose -Message "Provisioning on $Hostname Complete.  Log file located @ $logpath\transcripts\$transcriptlog" -Verbose
 
 

################# End Script timer #########################

$StopWatch.Stop()
$Minutes = $StopWatch.Elapsed.TotalMinutes
if ($Minutes -lt 1) {
    $Minutes = 1
} else {
    $Minutes = [Math]::Round($Minutes)
}

Write-Verbose -Verbose "Script runtime in minutes was $Minutes."


#################################################################

#STOP transcribing
Stop-Transcript

############### Disconnect vCenter ##############################

Disconnect-VIServer -Server $esx -Force:$true -Confirm:$false


###############################################################################################################################################################################################################
###############################################################################################################################################################################################################
    } -ArgumentList $row
    
  
    # Throttle the number of concurrent jobs
    if ($jobs.Count -ge $maxConcurrentJobs) {
        $jobs | Wait-Job -Any
        $jobs = $jobs | Where-Object { $_.State -eq 'Running' }
    }
    
# Display the number of jobs left and their Hostnames
$jobsLeft = $jobs.Count
Write-Verbose "Waiting for $jobsLeft jobs to complete..." -Verbose

}

###############################################################################################################################################################################################################
###############################################################################################################################################################################################################


# Wait for all jobs to complete
$jobs | Wait-Job

# Retrieve and display job results
$results = $jobs | ForEach-Object {
    #$jobHostname = $_.Arguments[0].Hostname
    $result = Receive-Job -Job $_
    #foreach ($item in $result) {
    #    Write-Output "[$jobHostname] $item"
    #}
    Remove-Job -Job $_
}

# Display or use the results as needed
$results

[LucD]

Redirecting/capturing output, except for the success and error streams, in remote PS jobs, mini-shells and background PS jobs does not work.
See many issues (#5848, #7814, #9585, #3354) reported under the PS rep.
The (only) solution, write your own logging function and use that inside your background job.

Not sure what you mean by "$row doesn't populate in the transcript either ...".
Don't you get the correct content in $row inside the background job?

[squarebox]

Ugh I had a feeling that might be the case LucD.  😞 

Re $row - script works re $row but for whatever reason when the script runs $row (to return what's in $row variable) it doesn't return anything.  I was planning to try 'write-verbose -verbose $row' to see if that works to return the data through receive-job (although it will mess up the simple output I was hoping for).  

[LucD]

I would suggest writing the content of the $row variable in that log function I mentioned earlier.

[squarebox]

So, by logging function, would one way to do this be to take all my code that previously outputted fine to screen/transcript (like the results of invoke-vmscript) and store them in a variable (e.g. $result1=invoke-vmscript blah blah blah) and then use write-verbose -verbose $result1 to get them into the transcript so that receive-job can get hold of it after?  

[LucD]

I would sooner create a function that uses Out-File to a file.
And pass the name of that file in the ArgumentList to the Start-Job cmdlet.
Watch out with writing to the same file from multiple background jobs, there might be locks.

[squarebox]

Can you share an example of what you mean?  I'm not clear what I'd need to do to most of the code for that to work (given receive-job doesn't pull what would be there if I wasn't running it in side start-job I'm not sure how out-file helps)

[LucD]

Sure, the following snippet shows how I would tackle capturing all output streams in background jobs.
I hope that helps.

$code = {
  param($var1, $var2, $log)

  function Write-Log {
    param(
      [String]$Path,
      [Parameter(ValueFromPipeline = $true)]
      [PSoBject]$Object
    )

    "$(Get-Date -Format 'dd-MM-yy HH:mm:ss.ffff') - $PID - $($Object.ToString())" | Out-File -FilePath $Path -Append
  }

  Write-Output "Success" | Write-Log -Path $log
  Write-Error "Error" 2>&1 | Write-Log -Path $log
  Write-Warning "Warning" 3>&1 | Write-Log -Path $log
  $VerbosePreference = 'Continue'
  Write-Verbose "Verbose" 4>&1 | Write-Log -Path $log
  $DebugPreference = 'Continue'
  Write-Debug "Debug" 5>&1 | Write-Log -Path $log
  Write-Information "Information" 6>&1 | Write-Log -Path $log

  Write-Log -Path $log -Object $var1
  Write-Log -Path $log -Object "Variable `$Var2 = $var2"

}

$logPath = '.\MyLog'
Get-Item -Path "$($logPath)*.log" | Remove-Item -Confirm:$false -ErrorAction SilentlyContinue
$anyVar1 = 'abc'
$anyVar2 = 'xyz'
1..2 | ForEach-Object -Process {
  $jobLog = "$($logPath)_$($_).Log"
  Start-Job -ScriptBlock $code -ArgumentList $anyVar1, $anyVar2, $jobLog -Name "MyJob$($_)"
}
while ((Get-Job -Name "MyJob*").State -contains 'Running') { Start-Sleep -Seconds 2 }

Get-Content -Path "$($logPath)*.log" |
Sort-Object |
Out-File -FilePath .\full-log.log

Get-Content -Path .\full-log.log

[squarebox]

Thanks that's really helpful.