How to Download a File Using an Alternative PowerShell wget Command

You need to download files from the internet, but you don't like clicking links over and over again? You will probably want to automate your job if you regularly download files from the web. Isn't PowerShell the perfect alternative to wget in PowerShell?

PowerShell comes with the ability to download files. To download files with PowerShell, you must know which cmdlets and .NET classes to use and how to use them.

This article explains how to download files from the web using PowerShell.

Prerequisites

There are some prerequisites to follow the examples in this article since it is a learn-by-doing article. Here are the requirements.

• Computers running Windows 10 or higher. These scripts/commands will be run on this computer.
• (Recommended) Windows PowerShell 5.1 or PowerShell 7.1.
• Windows PowerShell 5.1 is already included in Windows 10.
• A website that hosts the files to be downloaded.
• Try Tele2 Speedtest's free service for non-authenticated file downloads.

To test file downloads with authorization, you may need to build your HTTP file server. HFS by Rejetto is an example of a free HTTP file server.

You can use PowerShell to download files from URLs in four different ways

PowerShell can download files without requiring third-party tools using four methods. These include:

• Request-WebRequest
• Call-RestMethod
• Transfer-Bits-Start
• WebClient class for .NET.

Whatever method you choose, the logic and components to make them work are the same. To download files, there must be a source URL pointing to the file's location, as well as a destination path. The webserver may also require you to enter credentials.

Each of these four methods is described in the following sections. The choice of how you would use PowerShell to download files ultimately lies with you.

Invoke-WebRequest as an alternative to Wget in PowerShell

Using the Invoke-WebRequest cmdlet is the first way to download files in PowerShell. The most frequently used cmdlet in this article, Invoke-WebRequest, can download HTTP, HTTPS, and FTP links.

Invoke-WebRequest can be used with credentials when the source location requires logging in.

Using the syntax below, you can download a file with the minimum parameters necessary.

"Invoke-WebRequest -Uri <source> -OutFile <destination>"

For example, the code below downloads a file with the name 10MB.zip from a website. Then it saves the downloaded file to C:\dload\10MB.zip. You may copy the code below and paste it into your PowerShell session to test.

"# Source file location

$source = 'http://speedtest.tele2.net/10MB.zip'

# Destination to save the file

$destination = 'c:\dload\10MB.zip'

#Download the file

Invoke-WebRequest -Uri $source -OutFile $destination"

Below is a demonstration of what happens after running the code above in PowerShell. You can see that the file was successfully downloaded.

Would it be possible if the source required authentication before access was granted? The following code, for example, downloads a file from a private website.

"$source = 'https://mirror.lzex.ml/100MB.zip'

$destination = 'c:\dload\100MB.zip'

Invoke-WebRequest -Uri $source -OutFile $destination"

Unfortunately, the download failed due to unauthorized access.

You should add a credential to the request using the -Credential parameter if authentication is required. It prompts you for the credentials (username and password) on line 1 and stores them in the variable $credential.

"$credential = Get-Credential

$source = 'https://mirror.lzex.ml/100MB.zip'

$destination = 'c:\dload\100MB.zip'

Invoke-WebRequest -Uri $source -OutFile $destination -Credential $credential"

Here is a demonstration of what you'd expect to see when you run the above code in PowerShell. The Get-Credential cmdlet required PowerShell credentials, as you can see. I was able to download the file using the credential with Invoke-WebRequest this time.

How to use the PowerShell Get-Credential cmdlet and all things credentials

When using Invoke-WebRequest, watch out for parsing errors

The most important thing to remember when using Invoke-WebRequest in Windows PowerShell is that it uses the Internet Explorer engine by default. It is possible to get the error below when using Invoke-WebRequest on computers without Internet Explorer.

You'll have to re-issue your command this time with the -UseBasicParsing option.

"Invoke-WebRequest -Uri <source> -OutFile <destination> -UseBasicParsing"

Windows PowerShell may report the following error message: The response content cannot be parsed since Internet Explorer's engine is not installed, or the first-launch configuration is incomplete. Select the UseBasicParsing parameter and try again.

PowerShell Core 6.0 uses only basic parsing for the Invoke-WebRequest cmdlet. Therefore, the -UseBasicParsing parameter is no longer required.

Using the Invoke-RestMethod

Invoke-RestMethod is more about sending an HTTP or HTTPS request to a RESTful web service. The cmdlet is better suited to requests interacting with REST APIs, such as the Microsoft Graph API.

Invoke-RestMethod is an excellent method for downloading files directly from the Internet. Do not be fooled into believing otherwise. For downloading files from a direct web link, there is not much difference between using Invoke-RestMethod and Invoke-WebRequest.

Use the following syntax to download a file using Invoke-RestMethod. The command uses the same parameters as Invoke-WebRequest.

"Invoke-RestMethod -Uri <source> -OutFile <destination>"

In the example code below, the file is downloaded from the URL value in the $source variable. Then, saved to the path defined in the $destination variable.

"$source = 'http://speedtest.tele2.net/10MB.zip'

$destination = 'c:\dload\10MB.zip'

Invoke-RestMethod -Uri $source -OutFile $destination"

Using the -Credential parameter, you can provide credentials if the source requires authentication. The example below prompts for credentials and stores them in the $credential variable. -Credential is then passed the value of the $credential variable.

Since the file link is HTTP, instead of HTTPS, it means that you are sending an unencrypted authentication. In general, HTTP sources should not be used for security reasons. The -AllowUnencryptedAuthentication switch must be added to your command if you must use an HTTP source.

"$credential = Get-Credential

$source = 'http://speedtest.tele2.net/10MB.zip'

$destination = 'c:\dload\10MB.zip

Invoke-RestMethod -Uri $source -OutFile $destination -Credential $credential -AllowUnencryptedAuthentication"

Using Start-BitsTransfer

Start-BitsTransfer was designed specifically for the transfer of files between client and server computers. PowerShell cmdlet is dependent on the Windows operating system's Background Intelligent Transfer Service (BITS).

Due to the requirement for BITS, the Start-BitsTransfer cmdlet cannot be run on non-Windows computers. As a result, Start-BitsTransfer enjoys the benefits of BITS itself. Here are a few:

• Awareness of network bandwidth and usage.
• Interruptions (resume, auto-resume, pause, etc.)
• Downloading multiple files in the background.
• The ability to prioritize download jobs.

Downloading a File

To download a file with Start-BitsTransfer in PowerShell, you must specify a source and destination. Using the script below, you only need to change the $source and $destination values according to your requirements.

"$source = 'http://speedtest.tele2.net/100MB.zip

$destination = 'c:\dload\100MB.zip'

Start-BitsTransfer -Source $source -Destination $destination"

As you can see from the demo below, the file is downloaded to the path c:\dload\100MB.zip.

In the event that the destination is not specified, Start-BitsTransfer downloads and saves the files in the current working directory. The file is downloaded to the same directory if you run Start-BitsTransfer from C:/dload.

When a download requires authentication, Start-BitsTransfer has a -Credential parameter that accepts a PSCredential object.

Downloading Multiple Files

Creating a CSV file with two columns will demonstrate downloading multiple files. The file should be named filelist.txt. In the first column, you should include a link to the source, and in the second column, you should include the destination path. Here is an example.

"# source,destination

http://speedtest.tele2.net/1MB.zip,c:\dload\1MB.zip

http://speedtest.tele2.net/10MB.zip,c:\dload\10MB.zip

http://speedtest.tele2.net/100MB.zip,c:\dload\100MB.zip

Related: Managing CSV Files in PowerShell with Import-Csv"

Once the CSV file is ready, use the command below to begin the file download. The command uses Import-Csv to import the CSV file and passes its contents to Start-BitsTransfer.

"Import-Csv .\filelist.csv | Start-BitsTransfer"

See how the code above works in the demo below. You can see the download progress as it starts. There is no PowerShell prompt during the download.

Imagine you want to start the download process as a background job. You only need to add the -Asynchronous switch to the end of the Start-BitsTransfer command to accomplish this.

"Import-Csv .\filelist.csv | Start-BitsTransfer -Asynchronous"

The initial state of each job would be connected. Below is a screenshot showing the job id for each file download.

Having started the download process, you should check whether the download has been completed. Use the Get-BitsTransfer cmdlet to determine the status of the download job. The download jobs' status has changed to Transferred, as you can see below.

Using WebClient and HttpClient Classes (.NET Framework)

Due to its .NET-based nature, PowerShell can leverage the power of .NET itself. PowerShell supports two .NET classes for downloading files; WebClient and HttpClient.

* When to use WebClient vs. HttpClient vs. HttpWebRequest if you want to know more about these two .NET classes in more development and technical way. The next section will show you how to use WebClient and HttpClient in PowerShell to download files from the web.

Using System.Net.WebClient to download a file

A WebClient object must be created as a System.Net.WebClient type. The $webClient object in the following example represents the new System.Net.WebClient object. Using the DownloadFile() method, we download the file from the source.

Using PowerShell Data Type Accelerators to Speed Up Coding

To test the code, copy and paste it into your PowerShell session. There will be no progress or output on the screen unless there is an error. PowerShell, however, will remain locked until the download is complete.

"# Define the source link and destination path

$source = 'http://speedtest.tele2.net/10MB.zip'

$destination = 'c:\dload\10MB.zip'

# Create the new WebClient

$webClient = [System.Net.WebClient]::new()

# Download the file

$webClient.DownloadFile($source, $destination)"

Use the following code if the source requires authentication to allow the file download. In the first line, the credential is prompted and stored in the $credentials variable. The value of $credential is then included in the file download request.

"# Prompt for username and password

$credentials = Get-Credential

$source = 'http://speedtest.tele2.net/10MB.zip'

$destination = 'c:\dload\10MB.zip'

# Create the new WebClient

$webClient = [System.Net.WebClient]::new()

# Add the credential

$webClient.Credentials = $credentials

# Download the file

$webClient.DownloadFile($source, $destination)"

Microsoft recommends against using the WebClient class for new development. Use the System.Net.Http.HttpClient class instead."

Microsoft recommends the HTTPClient class in place of the WebClient class, which is obsolete. No worries. The next section describes how to use PowerShell's HttpClient class to download files from the web.

Downloading a File using System.Net.Http.HttpClient

To create a System.Net.Http.HttpClient, you need to create the WebClient first. This code downloads a file from the $source to the $destination. To understand what each line of code does, please refer to the comments above each line.

Run the code below in your PowerShell session to test it.

"# Set the source and destination

$source = 'http://speedtest.tele2.net/10MB.zip'

$destination = 'c:\dload\10MB.zip'

# Create the HTTP client download request

$httpClient = New-Object System.Net.Http.HttpClient

$response = $httpClient.GetAsync($source)

$response.Wait()

# Create a file stream to pointed to the output file destination

$outputFileStream = [System.IO.FileStream]::new($destination, [System.IO.FileMode]::Create, [System.IO.FileAccess]::Write)

# Stream the download to the destination file stream

$downloadTask = $response.Result.Content.CopyToAsync($outputFileStream)

$downloadTask.Wait()

# Close the file stream

$outputFileStream.Close()"

When downloading a file requires authentication, you need to add the credential to the HttpClient object. To include credentials with the file download request, create a new System.Net.Http.HttpClientHandler object.

You can run the code below in PowerShell to test it. PowerShell scripts can also be run. The code in this example is saved as download-file.ps1.

"# Set the source and destination

$source = 'http://speedtest.tele2.net/10MB.zip'

$destination = 'c:\dload\10MB.zip'

# Prompt for credentials

$credentials = Get-Credential

# Create the HTTP client download request with credentials

$handler = New-Object System.Net.Http.HttpClientHandler

$handler.Credentials = $credentials

$httpClient = New-Object System.Net.Http.HttpClient($handler)

$response = $httpClient.GetAsync($source)

$response.Wait()

# Create a file stream to pointed to the output file destination

$outputFileStream = [System.IO.FileStream]::new($destination, [System.IO.FileMode]::Create, [System.IO.FileAccess]::Write)

# Stream the download to the destination file stream

$downloadTask = $response.Result.Content.CopyToAsync($outputFileStream)

$downloadTask.Wait()

# Close the file stream

$outputFileStream.Close()"

The following demo shows the results when the PowerShell script is run to download the file.

Initially, the directory contained only the script file. A prompt asks for the user name and password. The script then downloads the file. You can see that the new file is now inside the destination directory after downloading it.

Conclusion

Windows PowerShell and PowerShell Core come with built-in capabilities to download files, acting as an alternative to PowerShell wget. You can use PowerShell to download password-protected sources, single files, or multiple files.

This article covers both Windows PowerShell and PowerShell Core ways to download files. With the exception of Start-BitsTransfer, these methods can be used on both Windows and non-Windows systems.

You can also translate what you learn into scripts since PowerShell is more than a command prompt. You can then automate what you learn. Now you don't have to copy URLs, click links, or wait for downloads manually.