Home Powershell Tips How To Use Web Services in PowerShell - Invoke-WebRequest Deep Dive

How To Use Web Services in PowerShell - Invoke-WebRequest Deep Dive

Lets focus on a few things that the more basic Invoke-WebRequest can do for you.

Binary Downloads

Since it does not modify, convert, or interpret the content, Invoke-WebRequest is ideal and efficient for downloading binaries. Below is a simple example of downloading a Star Wars imageyou get the idea.

$url = 'https://cdn.jumpradio.de/thema/star-wars-148-resimage_v-variantBig16x9_w-1280.jpg?version=3723'
$ImagePath = "$env:temp\starwars.png"

Invoke-WebRequest -UseBasicParsing -Uri $url -OutFile $ImagePath
Invoke-Item -Path $ImagePath

When downloading resources via HTTPS, most web servers require at least TLS 1.2 or higher for security reasons. If you encounter errors, you may want to enable TLS 1.2 manually:

$AllProtocols = [System.Net.SecurityProtocolType]'Ssl3,Tls,Tls11,Tls12'
[System.Net.ServicePointManager]::SecurityProtocol = $AllProtocols

Analyzing WebSite Structures

Another thing Invoke-WebRequest is good at is analyzing the underlying connection and website structure.

For example, here is a GitHub repository with a collection of useful PowerShell cheat sheets: https://github.com/PrateekKumarSingh/CheatSheets/tree/master/Powershell

You can download these manually or clone the repository. However, thanks to Invoke-WebRequest, you can also scrape data from websites with arbitrary structures. This is how Invoke-WebRequest can find links on a website:

$url = "https://github.com/PrateekKumarSingh/CheatSheets/tree/master/Powershell"
$page = Invoke-WebRequest -UseBasicParsing -Uri $url
# get only links with a title
$page.Links | Where-Object title | Select-Object -Property title, href

And this is what the result looks like:

title                                                                     href                                                              
-----                                                                     ----                                                              
AD_PowerShell_QuickReference_By_JonathanMedd.pdf                          /PrateekKumarSingh/CheatSheets/blob/master/Powershell/AD_PowerS...
AD_PowerShell_QuickReference_By_JonathanMedd.pdf                          /PrateekKumarSingh/CheatSheets/blob/master/Powershell/AD_PowerS...
LyncServer_PowerShell_Cheatsheet.pdf                                      /PrateekKumarSingh/CheatSheets/blob/master/Powershell/LyncServe...
LyncServer_PowerShell_Cheatsheet.pdf                                      /PrateekKumarSingh/CheatSheets/blob/master/Powershell/LyncServe...
Office365_PowerShell_Cheatsheet_By_Patrick_Lambert.pdf                    /PrateekKumarSingh/CheatSheets/blob/master/Powershell/Office365...
Office365_PowerShell_Cheatsheet_By_Patrick_Lambert.pdf                    /PrateekKumarSingh/CheatSheets/blob/master/Powershell/Office365...
PowerShell DSC Quick Reference for WMF 4 0 Preview by Stefan Stranger.pdf /PrateekKumarSingh/CheatSheets/blob/master/Powershell/PowerShel...
PowerShell DSC Quick Reference for WMF 4 0 Preview by Stefan Stranger.pdf /PrateekKumarSingh/CheatSheets/blob/master/Powershell/PowerShel...
PowerShell_CmdLine_Conversion_Guide_AD.pdf                                /PrateekKumarSingh/CheatSheets/blob/master/Powershell/PowerShel...
PowerShell_CmdLine_Conversion_Guide_AD.pdf                                /PrateekKumarSingh/CheatSheets/blob/master/Powershell/PowerShel...
PowerShell_Security_Logging_CheatSheet.pdf                                /PrateekKumarSingh/CheatSheets/blob/master/Powershell/PowerShel...
PowerShell_Security_Logging_CheatSheet.pdf                                /PrateekKumarSingh/CheatSheets/blob/master/Powershell/PowerShel...
PowerShell_v3_LanguageReference.pdf                                       /PrateekKumarSingh/CheatSheets/blob/master/Powershell/PowerShel...
PowerShell_v3_LanguageReference.pdf                                       /PrateekKumarSingh/CheatSheets/blob/master/Powershell/PowerShel...
Powershell_Cheat_Sheet_ByRamblingCookieMonster.JPG                        /PrateekKumarSingh/CheatSheets/blob/master/Powershell/Powershel...
Powershell_Cheat_Sheet_ByRamblingCookieMonster.JPG                        /PrateekKumarSingh/CheatSheets/blob/master/Powershell/Powershel...
Powershell_Exchange_2013_By_PacketPublishing.pdf                          /PrateekKumarSingh/CheatSheets/blob/master/Powershell/Powershel...
Powershell_Exchange_2013_By_PacketPublishing.pdf                          /PrateekKumarSingh/CheatSheets/blob/master/Powershell/Powershel...
Powershell_Functions_PipeLines_Parameters_Templates_CheatSheet.pdf        /PrateekKumarSingh/CheatSheets/blob/master/Powershell/Powershel...
Powershell_Functions_PipeLines_Parameters_Templates_CheatSheet.pdf        /PrateekKumarSingh/CheatSheets/blob/master/Powershell/Powershel...
Powershell_OS_Support_Matrix.jpg                                          /PrateekKumarSingh/CheatSheets/blob/master/Powershell/Powershel...
Powershell_OS_Support_Matrix.jpg                                          /PrateekKumarSingh/CheatSheets/blob/master/Powershell/Powershel...
Powershell_Support_Matrix.JPG                                             /PrateekKumarSingh/CheatSheets/blob/master/Powershell/Powershel...
Powershell_Support_Matrix.JPG                                             /PrateekKumarSingh/CheatSheets/blob/master/Powershell/Powershel...
Powershell_v4_by_PowershellMagazine.jpg                                   /PrateekKumarSingh/CheatSheets/blob/master/Powershell/Powershel...
Powershell_v4_by_PowershellMagazine.jpg                                   /PrateekKumarSingh/CheatSheets/blob/master/Powershell/Powershel...
SharePoint_PowerShell_Cheat_Sheet.pdf                                     /PrateekKumarSingh/CheatSheets/blob/master/Powershell/SharePoin...
SharePoint_PowerShell_Cheat_Sheet.pdf                                     /PrateekKumarSingh/CheatSheets

Website Scrapers

Now you are in the PowerShell business and can use it to automatically download all PDF documents referenced on the website:

# examine website structure
$url = "https://github.com/PrateekKumarSingh/CheatSheets/tree/master/Powershell"
$page = Invoke-WebRequest -UseBasicParsing -Uri $url

# take all links on the website...
$page.Links | 
    # that have a title...
    Where-Object title | 
    # that point to a PDF...
    Where-Object href -like *.pdf |
    # take title and URL...
    Select-Object -Property title, href |
    # do any translation that may be necessary
    # for GitHub, you need to change the URL to "raw" to access the underlying
    # binary PDF objects
    ForEach-Object {
        $relativeUrl = $_.href -replace '/blob/', '/raw/'
        $absoluteUrl = "https://github.com/$relativeUrl"
        
        # local file name is the folder defined above + the title
        $destinationFile = Join-Path -Path $destinationFolder -ChildPath $_.title

        # use the same cmdlet for downloading each file
        Invoke-WebRequest -UseBasicParsing -Uri $absoluteUrl -OutFile $destinationFile
    }

# Once downloaded, open in File Explorer
explorer $destinationFolder

Running this script scrapes all the PDF cheat sheets from that GitHub repository. The files are placed into the PSCheats folder on your desktop, but you can, of course, easily change the destination folder.

(Simple) Web Services

Invoke-WebRequest does not process the returned data, so you can also use it to access simple web services that return plain text. Heres an example:

[Byte]$number = Read-Host -Prompt 'Enter a positive number between 1-100'

$result = Invoke-WebRequest -UseBasicParsing -Uri "http://numbersapi.com/$number" | 
    Select-Object -ExpandProperty Content

"Fun fact for ${number}: $result"

When you enter a number, the web service returns a fun fact. The next time youre invited to an anniversary, you can add a fun fact to your greeting card:

Enter a positive number between 1-100: 12
Fun fact for 12: 12 is the number of function keys on most PC keyboards (F1 through F12).

Conclusions

In a nutshell, Invoke-WebRequest is perfect for whenever you want to examine website connections, website structure, or process plain information, such as downloading binary files or retrieving plain text.

Your ultimate PowerShell Cheat Sheet

Unleash the full potential of PowerShell with our handy poster. Whether you're a beginner or a seasoned pro, this cheat sheet is designed to be your go-to resource for the most important and commonly used cmdlets.The poster is available for download and in paper form.

Get your poster here!