How To Use Web Services in PowerShell - Invoke-WebRequest Deep Dive

Let’s focus on a few things that the more basic Invoke-WebRequest can do for you. 

Binary Downloads

Since it does not modify, convert, or interpret the content, Invoke-WebRequest is ideal and efficient for downloading binaries. Below is a simple example of downloading a Star Wars image—you get the idea.

$url = 'https://cdn.jumpradio.de/thema/star-wars-148-resimage_v-variantBig16x9_w-1280.jpg?version=3723'
$ImagePath = "$env:temp\starwars.png"

Invoke-WebRequest -UseBasicParsing -Uri $url -OutFile $ImagePath
Invoke-Item -Path $ImagePath 

When downloading resources via HTTPS, most web servers require at least TLS 1.2 or higher for security reasons. If you encounter errors, you may want to enable TLS 1.2 manually:

$AllProtocols = [System.Net.SecurityProtocolType]'Ssl3,Tls,Tls11,Tls12'
[System.Net.ServicePointManager]::SecurityProtocol = $AllProtocols 

Analyzing WebSite Structures

Another thing Invoke-WebRequest is good at is analyzing the underlying connection and website structure.

For example, here is a GitHub repository with a collection of useful PowerShell cheat sheets: https://github.com/PrateekKumarSingh/CheatSheets/tree/master/Powershell

You can download these manually or clone the repository. However, thanks to Invoke-WebRequest, you can also “scrape” data from websites with arbitrary structures. This is how Invoke-WebRequest can find links on a website:

$url = "https://github.com/PrateekKumarSingh/CheatSheets/tree/master/Powershell"
$page = Invoke-WebRequest -UseBasicParsing -Uri $url
# get only links with a title
$page.Links | Where-Object title | Select-Object -Property title, href 

And this is what the result looks like:

title                                                                     href                                                              
-----                                                                     ----                                                              
AD_PowerShell_QuickReference_By_JonathanMedd.pdf                          /PrateekKumarSingh/CheatSheets/blob/master/Powershell/AD_PowerS...
AD_PowerShell_QuickReference_By_JonathanMedd.pdf                          /PrateekKumarSingh/CheatSheets/blob/master/Powershell/AD_PowerS...
LyncServer_PowerShell_Cheatsheet.pdf                                      /PrateekKumarSingh/CheatSheets/blob/master/Powershell/LyncServe...
LyncServer_PowerShell_Cheatsheet.pdf                                      /PrateekKumarSingh/CheatSheets/blob/master/Powershell/LyncServe...
Office365_PowerShell_Cheatsheet_By_Patrick_Lambert.pdf                    /PrateekKumarSingh/CheatSheets/blob/master/Powershell/Office365...
Office365_PowerShell_Cheatsheet_By_Patrick_Lambert.pdf                    /PrateekKumarSingh/CheatSheets/blob/master/Powershell/Office365...
PowerShell DSC Quick Reference for WMF 4 0 Preview by Stefan Stranger.pdf /PrateekKumarSingh/CheatSheets/blob/master/Powershell/PowerShel...
PowerShell DSC Quick Reference for WMF 4 0 Preview by Stefan Stranger.pdf /PrateekKumarSingh/CheatSheets/blob/master/Powershell/PowerShel...
PowerShell_CmdLine_Conversion_Guide_AD.pdf                                /PrateekKumarSingh/CheatSheets/blob/master/Powershell/PowerShel...
PowerShell_CmdLine_Conversion_Guide_AD.pdf                                /PrateekKumarSingh/CheatSheets/blob/master/Powershell/PowerShel...
PowerShell_Security_Logging_CheatSheet.pdf                                /PrateekKumarSingh/CheatSheets/blob/master/Powershell/PowerShel...
PowerShell_Security_Logging_CheatSheet.pdf                                /PrateekKumarSingh/CheatSheets/blob/master/Powershell/PowerShel...
PowerShell_v3_LanguageReference.pdf                                       /PrateekKumarSingh/CheatSheets/blob/master/Powershell/PowerShel...
PowerShell_v3_LanguageReference.pdf                                       /PrateekKumarSingh/CheatSheets/blob/master/Powershell/PowerShel...
Powershell_Cheat_Sheet_ByRamblingCookieMonster.JPG                        /PrateekKumarSingh/CheatSheets/blob/master/Powershell/Powershel...
Powershell_Cheat_Sheet_ByRamblingCookieMonster.JPG                        /PrateekKumarSingh/CheatSheets/blob/master/Powershell/Powershel...
Powershell_Exchange_2013_By_PacketPublishing.pdf                          /PrateekKumarSingh/CheatSheets/blob/master/Powershell/Powershel...
Powershell_Exchange_2013_By_PacketPublishing.pdf                          /PrateekKumarSingh/CheatSheets/blob/master/Powershell/Powershel...
Powershell_Functions_PipeLines_Parameters_Templates_CheatSheet.pdf        /PrateekKumarSingh/CheatSheets/blob/master/Powershell/Powershel...
Powershell_Functions_PipeLines_Parameters_Templates_CheatSheet.pdf        /PrateekKumarSingh/CheatSheets/blob/master/Powershell/Powershel...
Powershell_OS_Support_Matrix.jpg                                          /PrateekKumarSingh/CheatSheets/blob/master/Powershell/Powershel...
Powershell_OS_Support_Matrix.jpg                                          /PrateekKumarSingh/CheatSheets/blob/master/Powershell/Powershel...
Powershell_Support_Matrix.JPG                                             /PrateekKumarSingh/CheatSheets/blob/master/Powershell/Powershel...
Powershell_Support_Matrix.JPG                                             /PrateekKumarSingh/CheatSheets/blob/master/Powershell/Powershel...
Powershell_v4_by_PowershellMagazine.jpg                                   /PrateekKumarSingh/CheatSheets/blob/master/Powershell/Powershel...
Powershell_v4_by_PowershellMagazine.jpg                                   /PrateekKumarSingh/CheatSheets/blob/master/Powershell/Powershel...
SharePoint_PowerShell_Cheat_Sheet.pdf                                     /PrateekKumarSingh/CheatSheets/blob/master/Powershell/SharePoin...
SharePoint_PowerShell_Cheat_Sheet.pdf                                     /PrateekKumarSingh/CheatSheets

Website Scrapers

Now you are in the “PowerShell business” and can use it to automatically download all PDF documents referenced on the website:

# examine website structure
$url = "https://github.com/PrateekKumarSingh/CheatSheets/tree/master/Powershell"
$page = Invoke-WebRequest -UseBasicParsing -Uri $url

# take all links on the website...
$page.Links | 
    # that have a title...
    Where-Object title | 
    # that point to a PDF...
    Where-Object href -like *.pdf |
    # take title and URL...
    Select-Object -Property title, href |
    # do any translation that may be necessary
    # for GitHub, you need to change the URL to "raw" to access the underlying
    # binary PDF objects
    ForEach-Object {
        $relativeUrl = $_.href -replace '/blob/', '/raw/'
        $absoluteUrl = "https://github.com/$relativeUrl"
        
        # local file name is the folder defined above + the title
        $destinationFile = Join-Path -Path $destinationFolder -ChildPath $_.title

        # use the same cmdlet for downloading each file
        Invoke-WebRequest -UseBasicParsing -Uri $absoluteUrl -OutFile $destinationFile
    }

# Once downloaded, open in File Explorer
explorer $destinationFolder 

Running this script “scrapes” all the PDF cheat sheets from that GitHub repository. The files are placed into the “PSCheats” folder on your desktop, but you can, of course, easily change the destination folder.

(Simple) Web Services

Invoke-WebRequest does not process the returned data, so you can also use it to access simple web services that return plain text. Here’s an example:

[Byte]$number = Read-Host -Prompt 'Enter a positive number between 1-100'

$result = Invoke-WebRequest -UseBasicParsing -Uri "http://numbersapi.com/$number" | 
    Select-Object -ExpandProperty Content

"Fun fact for ${number}: $result" 

When you enter a number, the web service returns a fun fact. The next time you’re invited to an anniversary, you can add a fun fact to your greeting card:

Enter a positive number between 1-100: 12
Fun fact for 12: 12 is the number of function keys on most PC keyboards (F1 through F12). 

Conclusions

In a nutshell, Invoke-WebRequest is perfect for whenever you want to examine website connections, website structure, or process plain information, such as downloading binary files or retrieving plain text.

Related links 

• ScriptRunner ActionPacks provide ready-to-use PowerShell scripts.
• Try out ScriptRunner here
• ScriptRunner: Book a demo with our product experts