SFTP Automation for Windows PowerShell with WinSCP .NET assembly

The MFT Gateway allows you the ability to easily send and receive files over the secure AS2 protocol for Internet based file transfer. While the MFT Gateway offers many integration options such as a REST API with webhooks and direct AWS S3 based file integration, many organizations might still choose to integrate existing systems using SFTP.

While SFTP is natively available on Linux and similar operating systems, automation of SFTP on Windows usually requires purchased software. This article explains how you can use the popular free and open-source software WinSCP as a library and use Windows PowerShell scripting to automate a file exchange.

The MFT Gateway SFTP File Structure

The default folder structure exposed for SFTP integration is of the following simplified form, and since a single AS2 message can contain multiple files, or the same file name can be repeated by a partner, the MFT Gateway creates a unique directory to contain files of each AS2 message. At a high level, this results in an inbox folder structure as follows.

../inbox/<unique-directory>/<file-name>

The above mentioned inbox folder structure is the “default” file structure used by MFT Gateway. But if the Remove Subdirectory With Random Message ID and/or Add Custom Subdirectory options were enabled under the File Structure settings of this partner, the SFTP file structure will be different from the above. In such a case, the sample scripts mentioned below should be adjusted accordingly to match the actual file structure.

Now an SFTP script must list contents of each unique directory within the inbox and download one or more files that might exist under it.

WinSCP Scripting

WinSCP has very good support for scripting, along with detailed documentation on how to use those capabilities. While using the scripting interface directly is recommended for simple tasks not requiring any control structures, WinSCP recommends the use of the WinSCP .NET assembly and the COM library, for more complex automation needs.

Using WinSCP with Windows PowerShell

To get started, download the WinSCP .NET assembly from the download page, and refer to the installation instructions. It is best to install the distribution to a location without spaces in the path. Note that by default, executing PowerShell scripts will be disabled. Trying to run a PowerShell script without an execution policy will usually result in an error like below.

PS C:\opt\WinSCP-5.21.5-Automation\Demo> .\mftg-sftp-download.ps1
.\mftg-sftp-download.ps1 : File C:\opt\WinSCP-5.21.5-Automation\Demo\mftg-sftp-download.ps1 cannot be loaded because running scripts is disabled on this system. For more information, see about_Execution_Policies at
https:/go.microsoft.com/fwlink/?LinkID=135170.

You can choose to lift this restriction by executing the following cmdlet from a PowerShell administrative console.

Set-ExecutionPolicy Unrestricted

Or you may bypass this by executing scripts as follows.

PS C:\opt\WinSCP-5.21.5-Automation\Demo> powershell -ExecutionPolicy Bypass -File .\mftg-sftp-download.ps1

Loading the Assembly

To load the assembly, use the ‘Add-Type’ cmdlet, along with the full path to the WinSCP DLL.

Load WinSCP .NET assembly

Add-Type -Path "C:\opt\WinSCP-5.21.5-Automation\WinSCPnet.dll"

Setting up a connection

To setup a connection, create a sessionOptions variable and set the hostname, username and the SSH keypath and fingerprint etc. You would also want to define a variable transferOptions, to select he ‘Binary’ mode of transfer for files.

# Setup session options
$sessionOptions = New-Object WinSCP.SessionOptions -Property @{
    Protocol = [WinSCP.Protocol]::Sftp
    HostName = "sftp.mftgateway.com"
    UserName = "asankha"
    SshPrivateKeyPath = "C:\opt\WinSCP-5.21.5-Automation\Demo\asankha.ppk"
    SshHostKeyFingerprint = "ssh-rsa 2048 12:b7:cf:7a:55:9a:77:43:f2:87:d6:43:09:92:88:15"
}

$transferOptions = New-Object WinSCP.TransferOptions
$transferOptions.TransferMode = [WinSCP.TransferMode]::Binary

Iterating directories and getting a file count

Remember our main requirement to be able to handle files within unique subdirectories containing one or more files? So, we will list the root directory and get a count of the total number of files, and also the individual path names.

try {
    # Connect to obtain file enumeration
    Write-Host "Connecting..."
    $session = New-Object WinSCP.Session
    $session.Open($sessionOptions)

    $remotePath = "/mftg-asankha.com/AS2/files/ACP_PROD/P1/inbox/"
    $localPath  = "C:\opt\WinSCP-5.21.5-Automation\Demo\inbox\"

    Write-Host "Starting files enumeration..."
    $directory = $session.ListDirectory($remotePath)
    $subdirs   = $directory.Files.FullName
}    
finally
{
    # Disconnect, clean up
    $session.Dispose()
}

[System.Collections.ArrayList]$msgs = @()
foreach ($sub in $subdirs)
{
    [void]$msgs.Add($sub)
}

$count = $msgs.Count
Write-Host "Total messages found for download: $count"   

Downloading individual files

Finally, we are ready to start downloading individual files, and we will create a new session for this task, and we can optionally delete the files after we successfully download them — by uncommenting the ‘$downloadSession.RemoveFiles($file)’ line below.

try {
  Write-Host "Starting download ..."
  $downloadSession = New-Object WinSCP.Session
  $downloadSession.Open($sessionOptions)

  while ($True)
  {
   $count = ($msgs).Count 
   if ($count -eq 0) {
    break
   }
     
   $file = ($msgs)[0]
   ($msgs).RemoveAt(0)

   if ($file -match '/..$') {
    Write-Host "Skipping parent directory"
    break
   }

   $count = ($msgs).Count
  
   $remote = -join($file, "/");

   Write-Host "Downloading $remote to $localPath ..."
   $transferResult = $downloadSession.GetFiles($remote, $localPath, $False, $transferOptions)

   # Did the download succeeded?
   if (!$transferResult.IsSuccess)
   {
    # Print error (but continue with other files)
    Write-Host ("Error downloading file $file " + "$($transferResult.Failures[0].Message)")
   } else {
    ($stats).count++
    # Delete remote file and directory after successful download
    # $downloadSession.RemoveFiles($file)
   }
  }

  Write-Host "Download completed"
 }
 finally
 {
  $downloadSession.Dispose()
 }

Sample Execution Output

Execution of this script will yield an output as below. Notice that we have bypassed the PowerShell execution policy for simplicity. Once you have tested your script, you may choose to schedule it for production use.

PS C:\opt\WinSCP-5.21.5-Automation\Demo> powershell -ExecutionPolicy Bypass -File .\mftg-sftp-download.ps1
Connecting...
Starting files enumeration...
Total messages found for download: 5
Starting download ...
Downloading /mftg-asankha.com/AS2/files/ACP_PROD/P1/inbox/956246844612951/ to C:\opt\WinSCP-5.21.5-Automation\Demo\inbox\ ...
Downloading /mftg-asankha.com/AS2/files/ACP_PROD/P1/inbox/956251031647737/ to C:\opt\WinSCP-5.21.5-Automation\Demo\inbox\ ...
Downloading /mftg-asankha.com/AS2/files/ACP_PROD/P1/inbox/956251096571533/ to C:\opt\WinSCP-5.21.5-Automation\Demo\inbox\ ...
Downloading /mftg-asankha.com/AS2/files/ACP_PROD/P1/inbox/956251102345887/ to C:\opt\WinSCP-5.21.5-Automation\Demo\inbox\ ...
Skipping parent directory
Download completed
Took 00:00:19.7675095
Downloaded 4 files

Complete working script

Here is the complete working script for my Windows 11 laptop where this was developed.

# Sample WinSCP script to download MFT Gateway received files using SFTP
#
# @author Asankha - Aayu Technologies LLC - 11 Jan 2022
#
# Download WinSCP .Net assembly from https://winscp.net/eng/downloads.php#additional and extract to "C:\opt\WinSCP-5.21.5-Automation" as an example
#
# Execution example
#    PS C:\opt\WinSCP-5.21.5-Automation\Demo> powershell -ExecutionPolicy Bypass -File .\mftg-sftp-download.ps1
#    Connecting...
#    Starting files enumeration...
#    Total messages found for download: 5
#    Starting download ...
#    Downloading /mftg-asankha.com/AS2/files/ACP_PROD/AdvanceAutoParts/inbox/956246844612951/ to C:\opt\WinSCP-5.21.5-Automation\Demo\inbox\ ...
#    Downloading /mftg-asankha.com/AS2/files/ACP_PROD/AdvanceAutoParts/inbox/956251031647737/ to C:\opt\WinSCP-5.21.5-Automation\Demo\inbox\ ...
#    Downloading /mftg-asankha.com/AS2/files/ACP_PROD/AdvanceAutoParts/inbox/956251096571533/ to C:\opt\WinSCP-5.21.5-Automation\Demo\inbox\ ...
#    Downloading /mftg-asankha.com/AS2/files/ACP_PROD/AdvanceAutoParts/inbox/956251102345887/ to C:\opt\WinSCP-5.21.5-Automation\Demo\inbox\ ...
#    Skipping parent directory
#    Download completed
#    Took 00:00:19.7675095
#    Downloaded 4 files

try {
    [Console]::TreatControlCAsInput = $True
    Start-Sleep -Seconds 1
    $Host.UI.RawUI.FlushInputBuffer()

    # Load WinSCP .NET assembly
    Add-Type -Path "C:\opt\WinSCP-5.21.5-Automation\WinSCPnet.dll"
 
    # Setup session options
    $sessionOptions = New-Object WinSCP.SessionOptions -Property @{
        Protocol = [WinSCP.Protocol]::Sftp
        HostName = "sftp.mftgateway.com"
        UserName = "asankha"
        SshPrivateKeyPath = "C:\opt\WinSCP-5.21.5-Automation\Demo\asankha.ppk"
        SshHostKeyFingerprint = "ssh-rsa 2048 12:b7:cf:7a:55:9a:77:43:f2:87:d6:43:09:92:88:15"
    }
 
    $transferOptions = New-Object WinSCP.TransferOptions
    $transferOptions.TransferMode = [WinSCP.TransferMode]::Binary

    try {
        # Connect to obtain file enumeration
        Write-Host "Connecting..."
        $session = New-Object WinSCP.Session
        $session.Open($sessionOptions)

        $remotePath = "/mftg-asankha.com/AS2/files/ACP_PROD/AdvanceAutoParts/inbox/"
        $localPath  = "C:\opt\WinSCP-5.21.5-Automation\Demo\inbox\"
 
        Write-Host "Starting files enumeration..."
        $directory = $session.ListDirectory($remotePath)
        $subdirs   = $directory.Files.FullName
    }    
    finally
    {
        # Disconnect, clean up
        $session.Dispose()
    }

    [System.Collections.ArrayList]$msgs = @()
    foreach ($sub in $subdirs)
    {
        [void]$msgs.Add($sub)
    }

    $count = $msgs.Count

    Write-Host "Total messages found for download: $count"   

    $started = Get-Date
    $stats = @{
        count = 0
        bytes = [long]0
    }

    try {
  Write-Host "Starting download ..."
  $downloadSession = New-Object WinSCP.Session
  $downloadSession.Open($sessionOptions)

  while ($True)
  {
   $count = ($msgs).Count 
   if ($count -eq 0) {
    break
   }
     
   $file = ($msgs)[0]
   ($msgs).RemoveAt(0)

   if ($file -match '/..$') {
    Write-Host "Skipping parent directory"
    break
   }

   $count = ($msgs).Count
  
   $remote = -join($file, "/");

   Write-Host "Downloading $remote to $localPath ..."
   $transferResult = $downloadSession.GetFiles($remote, $localPath, $False, $transferOptions)

   # Did the download succeeded?
   if (!$transferResult.IsSuccess)
   {
    # Print error (but continue with other files)
    Write-Host ("Error downloading file $file " + "$($transferResult.Failures[0].Message)")
   } else {
    ($stats).count++
    # Delete remote file and directory after successful download
    # $downloadSession.RemoveFiles($file)
   }
  }

  Write-Host "Download completed"
 }
 finally
 {
  $downloadSession.Dispose()
 }

    $ended = Get-Date
    Write-Host "Took $(New-TimeSpan -Start $started -End $ended)"
    Write-Host ("Downloaded $($stats.count) files")
    exit 0
}
catch
{
    Write-Host "Error: $($_.Exception.Message)"
    exit 1
}