Home Powershell Tips Advanced Script Analysis in PowerShell: Tokenizing File-Based Scripts for Syntax Validation

Advanced Script Analysis in PowerShell: Tokenizing File-Based Scripts for Syntax Validation

In the previous parts, we explored the PowerShell parser and how it can be used to analyze script content.

Today, lets look at more advanced insights that could even help address security issues. PowerShell scripts can use native .NET methods, and it can be highly useful to identify the methods used by scripts:

Curiosity: Which .NET methods are in use, why, and are they potentially interesting or unsafe?
Best practices: Using .NET methods should be a secondary choice. For readability, a script shouldn't use raw .NET methods when equivalent PowerShell cmdlets are available.
Security risks: Some methods may be considered a security risk and could be prohibited in your enterprise.

.NET methods are represented by the token type 'Member'. Heres a script to list all .NET methods used by a PowerShell script:

function Get-ScriptNetMethod
{
  param
  (
    [String]
    [Parameter(Mandatory,ValueFromPipeline)]
    [Alias('FullName')]
    $Path
  )
  
  begin
  {
    $fileCount = 0
  }
  process
  {
    $fileCount++
    Write-Progress -Activity 'Analyzing PowerShell Scripts' -Status $fileCount
    
    $syntaxErrors = $null
    $code = Get-Content -Path $Path -Raw -Encoding Default
    $tokens = [Management.Automation.PSParser]::Tokenize($code, [ref]$syntaxErrors)
    
    if (!$syntaxErrors)
    {
      # get .NET methods
      $methods = $Tokens | 
          Where-Object Type -eq Member | 
          Sort-Object -Property Content -Unique |
          Select-Object -ExpandProperty Content
          
      if ($methods)
      {
        [PSCustomObject]@{
          Name = Split-Path -Path $Path -Leaf
          NetMethods = $methods -join ','
          Path = $Path
        }
      } 
    }  
  }
}

# check all PowerShell scripts located anywhere in your Documents folder
$myDocuments = [Environment]::GetFolderPath('myDocuments')

Get-ChildItem -Path $myDocuments -File -Filter *.ps1 -Include *.ps1 -Recurse |
  Get-ScriptNetMethod

The result may look like this:

Compact-Path.ps1               Mandatory,new,PathCompactPathEx,ToString,ValueFromPipeline,ValueF...
Get-SpecialFolderPath.ps1      GetFolderPath,Mandatory,ValueFromPipeline                           
Test-Online.ps1                Add,AddArgument,AddScript,AsyncWaitHandle,BeginConnect,BeginInvok...
Test-Ping.ps1                  Address,Dispose,E,Mandatory,N,new,Send,Status,ValueFromPipeline     
Test-Port.ps1                  Address,Client,Close,ConnectAsync,Connected,Dispose,HostName,IP,I...
Compact-Path.ps1               Mandatory,new,PathCompactPathEx,ToString,ValueFromPipeline,ValueF...
Get-SpecialFolderPath.ps1      GetFolderPath,Mandatory,ValueFromPipeline

This result may help roughly identify the focus of a script, but upon reflection, listing .NET methods without knowing their type and origin is of limited use.

This illustrates the limitation of the tokenizer: it can tell you the meaning of a given word in code, but it cannot provide context.

Accessing the AST (Abstract Syntax Tree)

For more advanced analysis, you can access the Abstract Syntax Tree (AST). Before diving into what an AST is, heres how to access it:

# path to a PowerShell script to analyze (make sure it exists!)
$path = "C:\test\some_file.ps1"

# empty variables, these must exist. The method returns syntax errors and
# tokens in these variables (by reference) after the call
$syntaxErrors = $null
$tokens = $null

# parse the file
$ast = [System.Management.Automation.Language.Parser]::ParseFile($Path, [ref] $tokens, [ref]$syntaxErrors)

When you run this code, you get three pieces of information about your script:

Syntax Errors: As before, $syntaxErrors contains any syntactical errors.
Tokens: Now more refined, including nested tokens.
AST: A structured representation of relationships in the code.

At first glance, though, the information returned in $AST may seem disappointing. Where are all the contextual insights?

Understanding the AST

The AST describes relationships in your script using various types. Once you understand these, you can better appreciate its tremendous value.

Add this line to the code above to reveal all AST node types:

$ast.FindAll({$true}, $true) | ForEach-Object { $_.GetType().Name }

It returns the type names of all AST components found in the script specified by $path. To gain meaningful insights, make sure to specify a PowerShell script with a substantial amount of code

This is what the output could look like:

ScriptBlockAst
NamedBlockAst
FunctionDefinitionAst
ScriptBlockAst
ParamBlockAst
AttributeAst
ParameterAst
AttributeAst
NamedAttributeArgumentAst
ConstantExpressionAst
NamedAttributeArgumentAst
ConstantExpressionAst
NamedAttributeArgumentAst
ConstantExpressionAst
TypeConstraintAst
VariableExpressionAst
ParameterAst
AttributeAst
NamedAttributeArgumentAst
ConstantExpressionAst
NamedAttributeArgumentAst
ConstantExpressionAst
TypeConstraintAst
VariableExpressionAst
ParameterAst
AttributeAst
NamedAttributeArgumentAst
ConstantExpressionAst
TypeConstraintAst
VariableExpressionAst
ConstantExpressionAst
NamedBlockAst
TryStatementAst
StatementBlockAst
AssignmentStatementAst
VariableExpressionAst
CommandExpressionAst
InvokeMemberExpressionAst

The AST begins at the top of your script with a ScriptBlockAst, followed by more specific types that describe the code in detail. For example, a ParameterAst describes a param() block, and a VariableExpressionAst describes a variable assignment.

Upon closer inspection, you'll see an InvokeMemberExpressionAst() because the script in the example calls a .NET method. While the simple tokenizer can only return the names of methods used in your script, the AST provides much more detail, which we will explore in Part 4.

Your ultimate PowerShell Cheat Sheet

Unleash the full potential of PowerShell with our handy poster. Whether you're a beginner or a seasoned pro, this cheat sheet is designed to be your go-to resource for the most important and commonly used cmdlets.The poster is available for download and in paper form.

Get your poster here!

Advanced Script Analysis in PowerShell: Tokenizing File-Based Scripts for Syntax Validation

Accessing the AST (Abstract Syntax Tree)

Understanding the AST

Your ultimate PowerShell Cheat Sheet

Related links