Bulk Testing PowerShell Scripts with the Tokenizer

In Part 1, we explored the internal PowerShell parser to see how it analyzes code and breaks it down into individual tokens.

Today, let's use this knowledge to build tools. Here’s how the tokenizer can analyze a file-based script:

# path to the script to check (make sure it exists!)
$Path = 'C:\test\some_script.ps1'

# content of script file must first be read (tokenizer cannot open file automatically)
$code = Get-Content -Path $Path -Raw -Encoding Default

# remaining code is unchanged
$syntaxErrors = $null
$tokens = [System.Management.Automation.PSParser]::Tokenize($code, [ref]$syntaxErrors)

$tokens

Essentially, this builds on the code we explored in Part 1. The tokenizer doesn’t care where the code comes from—it simply processes whatever you provide. By using Get-Content, a file-based script is read as a string and then passed to the tokenizer.

Error Checker Tool

For tools, it's essential to handle any number of script files. Simply integrate the code into a pipeline-aware function. Let’s create a tool that scans your PowerShell scripts and identifies those with syntax errors:

function Test-SyntaxError
{
  param
  (
    [String]
    [Parameter(Mandatory,ValueFromPipeline)]
    [Alias('FullName')]
    $Path
  )
  
  begin
  {
    $fileCount = 0
  }
  process
  {
    $fileCount++
    Write-Progress -Activity 'Testing PowerShell Scripts' -Status $fileCount
    
    $syntaxErrors = $null
    $code = Get-Content -Path $Path -Raw -Encoding Default
    $null = [Management.Automation.PSParser]::Tokenize($code, [ref]$syntaxErrors)
    
    if ($syntaxErrors)
    {
      [PSCustomObject]@{
        Name = Split-Path -Path $Path -Leaf
        ErrorMessage = $syntaxErrors[0].Message
        Line = $syntaxErrors[0].Token.StartLine
        Column = $syntaxErrors[0].Token.StartColumn
        Content = $syntaxErrors[0].Token.Content
        Path = $Path
      } 
    }  
  }
}

$myDocuments = [Environment]::GetFolderPath('myDocuments')

Get-ChildItem -Path $myDocuments -File -Filter *.ps1 -Include *.ps1 -Recurse |
  Test-SyntaxError |
  Format-Table

The result is a list of all PowerShell scripts in your Documents folder that contain syntax errors (ensure there’s at least one script with a missing quote or another syntax issue). The list includes the syntax error message and the tokens that caused the error, along with their exact location in the script.

Often, one syntax error leads to subsequent errors. That’s why the list shows only the first syntax error. Typically, fixing this error resolves all others as well.

Here’s is a sample result:

Name          ErrorMessage                                               Line Column Content                             Path                                   
----          ------------                                               ---- ------ -------                             ----                                   
logic.ps1     Unable to find type [System.Data.SQLite.SQLiteConnection].  131      4 System.Data.SQLite.SQLiteConnection ...
beispiele.ps1 Missing ')' in method call.                                   9     39                                     ...
12 ad kont... Could not find the module 'xActiveDirectory'.                 3      2 Import-DscResource -ModuleName '......
12 ad kont... Could not find the module 'xActiveDirectory'.                 3      2 Import-DscResource -ModuleName '... 
parallelis... Unexpected token '}' in expression or statement.             26      3 }                                   ...
Vorlage Me... Missing ')' in method call.                                  13     35                                     ...
logic.ps1     Unable to find type [System.Data.SQLite.SQLiteConnection].  131      4 System.Data.SQLite.SQLiteConnection...
using.ps1     Could not find the module 'C:\Users\tobia\OneDrive\Doku...    1      1 using module 'C:\Users\tobia\One... 
break und ... Missing statement block after 'else' keyword.                32      9                               

The tokenizer returns all information that would prevent a script from running, including missing modules or dependencies (as long as the script uses the #requires statement to specify the required modules, so the tokenizer can recognize them).

Creating Variable and Command Lists

Similarly, a tool could generate a list of variable names and/or commands used in a particular script, then report them for analysis or use them as the basis for documentation:

function Get-ScriptDetail
{
  param
  (
    [String]
    [Parameter(Mandatory,ValueFromPipeline)]
    [Alias('FullName')]
    $Path
  )
  
  begin
  {
    $fileCount = 0
    
    # blacklist of variable names we are not interested in (add more if needed)
    $ignoreVariable = '_', 'true', 'false', 'null', 'psscriptroot', 'psitem'
  }
  process
  {
    $fileCount++
    Write-Progress -Activity 'Analyzing PowerShell Scripts' -Status $fileCount
    
    $syntaxErrors = $null
    $code = Get-Content -Path $Path -Raw -Encoding Default
    $tokens = [Management.Automation.PSParser]::Tokenize($code, [ref]$syntaxErrors)
    
    if (!$syntaxErrors)
    {
      # get variable names
      $variableNames = $Tokens | 
          Where-Object Type -eq Variable | 
          Where-Object Content -notin $ignoreVariable |
          Sort-Object -Property Content -Unique |
          Select-Object -ExpandProperty Content
      
      # get command names
      $commandNames = $Tokens | 
          Where-Object Type -eq Command | 
          Sort-Object -Property Content -Unique |
          Select-Object -ExpandProperty Content
          
      [PSCustomObject]@{
        Name = Split-Path -Path $Path -Leaf
        VariableNames = $variableNames -join ','
        Commands = $commandNames -join ','
        Path = $Path
      } 
    }  
  }
}

$myDocuments = [Environment]::GetFolderPath('myDocuments')

Get-ChildItem -Path $myDocuments -File -Filter *.ps1 -Include *.ps1 -Recurse |
  Get-ScriptDetail |
  Out-GridView

The result is a list of all valid PowerShell scripts, including all variables and commands used.

Name                           VariableNames                                     Commands                                                                       
----                           -------------                                     --------                                                                       
skript.ps1                     path                                              ConvertTo-Csv,ForEach-Object,Get-Date,Import-Csv,Out-String,Select-Object,Se... 
print-certificate.ps1          datum,female,info,liste,nachname,ort,vorname,word Import-Csv,New-Object,Out-Null,Print-Cert,Select-Object,Start-Sleep,Where-Ob...
Test-Ping.ps1                  ComputerName,obj,TimeoutMillisec                  Select-Object,Write-Warning                                                    
Compact-Path.ps1               Length,Path,Shlwapi,StringBuilder                 Add-Type                                                                       
Get-IpV4Segment.ps1            end,From,ip,ipFromBytes,ipToBytes,start,To,x                                                                                     
Get-NetworkPrinterInfo.ps1     ComputerName,hash,NoEmpty,oid,SNMP                ForEach-Object,New-Object,Sort-Object                                          
Get-ResourceFileName.ps1       executable,result,url                             Assert-NotNull,Get-BinaryDownloadCommand,Test-Executable                       
Get-SystemFolderPathForDown... GetType,GuidDownloads,ptr,result,signature        Add-Type,Join-Path

Related links 

• ScriptRunners Script collection provides ready-to-use PowerShell scripts.
• Try out ScriptRunner here
• ScriptRunner: Book a demo with our product experts