HomePowershell TipsBulk Testing PowerShell Scripts with the Tokenizer

Bulk Testing PowerShell Scripts with the Tokenizer

May 22, 2025

Aleksandar Nikolić and Dr. Tobias Weltner

In Part 1, we explored the internal PowerShell parser to see how it analyzes code and breaks it down into individual tokens.

Today, let's use this knowledge to build tools. Heres how the tokenizer can analyze a file-based script:

# path to the script to check (make sure it exists!)
$Path = 'C:\test\some_script.ps1'

# content of script file must first be read (tokenizer cannot open file automatically)
$code = Get-Content -Path $Path -Raw -Encoding Default

# remaining code is unchanged
$syntaxErrors = $null
$tokens = [System.Management.Automation.PSParser]::Tokenize($code, [ref]$syntaxErrors)

$tokens

Essentially, this builds on the code we explored in Part 1. The tokenizer doesnt care where the code comes fromit simply processes whatever you provide. By using Get-Content, a file-based script is read as a string and then passed to the tokenizer.

Error Checker Tool

For tools, it's essential to handle any number of script files. Simply integrate the code into a pipeline-aware function. Lets create a tool that scans your PowerShell scripts and identifies those with syntax errors:

function Test-SyntaxError
{
  param
  (
    [String]
    [Parameter(Mandatory,ValueFromPipeline)]
    [Alias('FullName')]
    $Path
  )
  
  begin
  {
    $fileCount = 0
  }
  process
  {
    $fileCount++
    Write-Progress -Activity 'Testing PowerShell Scripts' -Status $fileCount
    
    $syntaxErrors = $null
    $code = Get-Content -Path $Path -Raw -Encoding Default
    $null = [Management.Automation.PSParser]::Tokenize($code, [ref]$syntaxErrors)
    
    if ($syntaxErrors)
    {
      [PSCustomObject]@{
        Name = Split-Path -Path $Path -Leaf
        ErrorMessage = $syntaxErrors[0].Message
        Line = $syntaxErrors[0].Token.StartLine
        Column = $syntaxErrors[0].Token.StartColumn
        Content = $syntaxErrors[0].Token.Content
        Path = $Path
      } 
    }  
  }
}

$myDocuments = [Environment]::GetFolderPath('myDocuments')

Get-ChildItem -Path $myDocuments -File -Filter *.ps1 -Include *.ps1 -Recurse |
  Test-SyntaxError |
  Format-Table

The result is a list of all PowerShell scripts in your Documents folder that contain syntax errors (ensure theres at least one script with a missing quote or another syntax issue). The list includes the syntax error message and the tokens that caused the error, along with their exact location in the script.

Often, one syntax error leads to subsequent errors. Thats why the list shows only the first syntax error. Typically, fixing this error resolves all others as well.

Heres is a sample result:

 Name          ErrorMessage                                               Line Column Content                             Path                                   
----          ------------                                               ---- ------ -------                             ----                                   
logic.ps1     Unable to find type [System.Data.SQLite.SQLiteConnection].  131      4 System.Data.SQLite.SQLiteConnection ...
beispiele.ps1 Missing ')' in method call.                                   9     39                                     ...
12 ad kont... Could not find the module 'xActiveDirectory'.                 3      2 Import-DscResource -ModuleName '......
12 ad kont... Could not find the module 'xActiveDirectory'.                 3      2 Import-DscResource -ModuleName '... 
parallelis... Unexpected token '}' in expression or statement.             26      3 }                                   ...
Vorlage Me... Missing ')' in method call.                                  13     35                                     ...
logic.ps1     Unable to find type [System.Data.SQLite.SQLiteConnection].  131      4 System.Data.SQLite.SQLiteConnection...
using.ps1     Could not find the module 'C:\Users\tobia\OneDrive\Doku...    1      1 using module 'C:\Users\tobia\One... 
break und ... Missing statement block after 'else' keyword.                32      9

The tokenizer returns all information that would prevent a script from running, including missing modules or dependencies (as long as the script uses the #requires statement to specify the required modules, so the tokenizer can recognize them).

Creating Variable and Command Lists

Similarly, a tool could generate a list of variable names and/or commands used in a particular script, then report them for analysis or use them as the basis for documentation:

function Get-ScriptDetail
{
  param
  (
    [String]
    [Parameter(Mandatory,ValueFromPipeline)]
    [Alias('FullName')]
    $Path
  )
  
  begin
  {
    $fileCount = 0
    
    # blacklist of variable names we are not interested in (add more if needed)
    $ignoreVariable = '_', 'true', 'false', 'null', 'psscriptroot', 'psitem'
  }
  process
  {
    $fileCount++
    Write-Progress -Activity 'Analyzing PowerShell Scripts' -Status $fileCount
    
    $syntaxErrors = $null
    $code = Get-Content -Path $Path -Raw -Encoding Default
    $tokens = [Management.Automation.PSParser]::Tokenize($code, [ref]$syntaxErrors)
    
    if (!$syntaxErrors)
    {
      # get variable names
      $variableNames = $Tokens | 
          Where-Object Type -eq Variable | 
          Where-Object Content -notin $ignoreVariable |
          Sort-Object -Property Content -Unique |
          Select-Object -ExpandProperty Content
      
      # get command names
      $commandNames = $Tokens | 
          Where-Object Type -eq Command | 
          Sort-Object -Property Content -Unique |
          Select-Object -ExpandProperty Content
          
      [PSCustomObject]@{
        Name = Split-Path -Path $Path -Leaf
        VariableNames = $variableNames -join ','
        Commands = $commandNames -join ','
        Path = $Path
      } 
    }  
  }
}

$myDocuments = [Environment]::GetFolderPath('myDocuments')

Get-ChildItem -Path $myDocuments -File -Filter *.ps1 -Include *.ps1 -Recurse |
  Get-ScriptDetail |
  Out-GridView

The result is a list of all valid PowerShell scripts, including all variables and commands used.

Name                           VariableNames                                     Commands                                                                       
----                           -------------                                     --------                                                                       
skript.ps1                     path                                              ConvertTo-Csv,ForEach-Object,Get-Date,Import-Csv,Out-String,Select-Object,Se... 
print-certificate.ps1          datum,female,info,liste,nachname,ort,vorname,word Import-Csv,New-Object,Out-Null,Print-Cert,Select-Object,Start-Sleep,Where-Ob...
Test-Ping.ps1                  ComputerName,obj,TimeoutMillisec                  Select-Object,Write-Warning                                                    
Compact-Path.ps1               Length,Path,Shlwapi,StringBuilder                 Add-Type                                                                       
Get-IpV4Segment.ps1            end,From,ip,ipFromBytes,ipToBytes,start,To,x                                                                                     
Get-NetworkPrinterInfo.ps1     ComputerName,hash,NoEmpty,oid,SNMP                ForEach-Object,New-Object,Sort-Object                                          
Get-ResourceFileName.ps1       executable,result,url                             Assert-NotNull,Get-BinaryDownloadCommand,Test-Executable                       
Get-SystemFolderPathForDown... GetType,GuidDownloads,ptr,result,signature        Add-Type,Join-Path

Your ultimate PowerShell Cheat Sheet

Unleash the full potential of PowerShell with our handy poster. Whether you're a beginner or a seasoned pro, this cheat sheet is designed to be your go-to resource for the most important and commonly used cmdlets.The poster is available for download and in paper form.

Get your poster here!

Related links