Free cookie consent management tool by TermsFeed AWS S3 Integration | Aayu Technologies Cookies preferences
EDI and AS2 Gateways by Aayu Technologies
mft gateway logo as2 gateway logo edi generator logo
Link Search Menu Expand Document
logo
Contact

AWS S3 Integration

1. Introduction

The MFT Gateway utilizes Amazon Simple Storage Service, Amazon S3 as the storage for all files exchanged through the system. Each tenant or account within the MFT Gateway is assigned its own unique S3 Bucket. If you are deploying your own MFT Gateway, with dedicated hosting, or on-cloud hosting within your own AWS account, the S3 region can be selected from the available AWS regions.

This section guides you on using direct S3 bucket access, for integration of AS2 communications with external systems.

2. AWS S3 Integration for AS2 & SFTP

2.1 Access Keys

MFT Gateway allows access to the S3 bucket via AWS access keys. To obtain a new access key pair, expiring any previously issued credentials, navigate to the S3 Integration Page. Before the first credential creation, you will need to tick the checkbox to confirm that you have read and understood the terms. Then click ‘Enable S3’, and your key pair will be downloaded as a comma separated value (CSV) format file. Note the S3 bucket name assigned to your account, as it will be required for all subsequent access.

The keys can be downloaded only once. You can create a new key pair replacing the previous keys.

S3 Integration

Once an access key pair is created, you can either disable S3 integration, or generate a new credential pair, which will invalidate previously issued key pairs. S3 access disable or modify

Once the keys are ready, you can use them to access the bucket through the AWS SDK (available for several programming languages) or ready-made tools like s3cmd, or the AWS CLI.

e.g.

export AWS_ACCESS_KEY_ID=yourAccessKey
export AWS_SECRET_ACCESS_KEY=yourAccessSecret
export AWS_DEFAULT_REGION=us-east-1

aws s3 ls s3://<mftg-yourtenantname>

2.2 Bucket Content Layout

The following hierarchy exists for each bucket, created for each MFT Gateway account that has at least one station or one AS2 / SFTP partner. All files related to AS2 communication is stored under the AS2/ prefix and files used for SFTP communication is stored under SFTP/ prefix. Refer to the section 3.1 for more details on AS2 content storage.

  • AS2
    • files
    • headers
    • keystore
    • raw-mdn
    • raw-message
    • send
    • tmp
  • SFTP
    • send
    • partners
    • external

2.3 Default S3 Lifecycle rules

MFT Gateway enables S3 lifecycle rules by default to manage transactional objects stored in buckets. Objects stored under the following directories of the S3 bucket will be moved to non-current status after 35 days, and will be permanently deleted after another 35 days. i.e. Objects will be deleted 70 days since creation.

  • AS2/files
  • AS2/raw-message
  • AS2/raw-mdn
  • AS2/headers
  • SFTP/partners/
  • SFTP/external/

Objects stored in AS2/tmp directory will be moved into non-current status after 1 day, and will be permanently deleted after 2 days from the creation.

3. Automation via S3 API/Tools for AS2 messages

3.1 Bucket Content Layout for AS2 messages

<Station-AS2-ID> refers to the AS2 ID of the local Station defined in the MFT Gateway, while <Partner-AS2-ID> refers to the AS2 ID of the remote partner. The AS2/raw-message contains the encrypted AS2 messages as-sent, or as-received, while the AS2/files contains the actual payload files without, or after decryption. It should be noted that AS2/keystore and AS2/tmpare special locations and end-users should not access them, and are listed here only for completeness.

  • AS2/files [AS2 message Attachments]
    • <Station-AS2-ID>/<Partner-AS2-ID>/inbox
    • <Station-AS2-ID>/<Partner-AS2-ID>/outbox
  • AS2/headers [Raw HTTP headers of messages]
    • <Station-AS2-ID>/<Partner-AS2-ID>/inbox
    • <Station-AS2-ID>/<Partner-AS2-ID>/outbox
  • AS2/keystore [Key stores]
    • https.jks [SSL/TLS certificates and key pairs]
    • identity.jks [AS2 Station encryption/signature certificates]
    • trust.jks [AS2 Partner certificates]
  • AS2/raw-mdn [Raw MDNs]
    • incoming
    • outgoing
  • AS2/raw-message [Raw AS2 messages]
    • <Station-AS2-ID>/<Partner-AS2-ID>/inbox
    • <Station-AS2-ID>/<Partner-AS2-ID>/outbox
  • AS2/send [Send location]
    • <Station-AS2-ID>/<Partner-AS2-ID>/
  • AS2/tmp [Temporary file location]

3.2 Sending AS2 Files

To submit a file to be sent over AS2, copy it (i.e. PutObject) to the S3 location AS2/send/<station-AS2-id>/<partner-AS2-id>/. If you want to send a single file, you can copy it directly to this location (or S3 prefix). If you have more than one file to be submitted as a single AS2 message, zip the files into a single archive, and then upload the archive.

e.g.

aws s3 cp <local-file> s3://<mftg-bucketname>/AS2/send/<station-AS2-id>/<partner-AS2-id>/

After a file has been submitted for send, the system will be automatically triggered, and will perform the AS2 communications process to send the file to the designated partner. The processed file will be placed into the AS2/files/<Station-AS2-ID>/<Partner-AS2-ID>/outbox location.

Files uploaded with same name within a 6-hour window will be considered as duplicates and will not be processed. If you need to submit files with the same name, add a postfix to the filename, such as an extension. (Duplicated files will be moved to AS2/duplicate/<Station-AS2-ID>/<Partner-AS2-ID>/)

S3 triggers only apply for the content directly uploaded to the above mentioned directory / S3 prefix. Any content copied or moved within the S3 bucket itself, will be ignored.

3.3 Receiving AS2 Files

Received files will be persisted to the AS2/files/<station-AS2-id>/<partner-AS2-id>/inbox. You can fetch/download these payload files through the (GetObject) call.

e.g.

aws s3 cp s3://<mftg-bucketname>/AS2/files/<station-AS2-id>/<partner-AS2-id>/inbox/<timestamp-random-number>/<attachment-name> <local-path>

3.4 Accessing MDNs, sent files, HTTP headers and raw messages

If required, the MDN’s sent and received, files successfully sent, raw (encrypted) messages or message headers can be downloaded from the S3 locations as listed in the bucket layout.

3.5 Customizing S3 prefix of AS2 message attachments

By default, each message will have a unique path / S3 prefix, with the timestamp followed by a random number, within which the actual attachments will be found. e.g.

/AS2/files/<station-AS2-id>/<partner-AS2-id>/inbox/<timestamp-random-number>/<attachment-name>
/AS2/files/<station-AS2-id>/<partner-AS2-id>/outbox/<timestamp-random-number>/<attachment-name>

You can customize default S3 prefix by updating the AS2 Partner’s File Structure settings.

3.5.1 Save directly into Inbox or Outbox

Enable only Remove Subdirectory With Random Message ID toggle and update the partner. The attachments of incoming AS2 messages will be persisted directly in to inbox, and attachments of outgoing AS2 messages will be directly persisted in to outbox. e.g.

/AS2/files/<station-AS2-id>/<partner-AS2-id>/inbox/<attachment-name>
/AS2/files/<station-AS2-id>/<partner-AS2-id>/outbox/<attachment-name>

Note : Removing the subdirectory with a randomly generated message ID carries the risk of potential file overwrites if files with the same name exist.

3.5.2 Add a Custom Subdirectory

Enable only Add Custom Subdirectory toggle and update the partner. The attachments of AS2 messages will be persisted in to inbox or outbox under the subdirectory name and then a random generated ID. e.g.

/AS2/files/<station-identifier>/<partner-AS2-id>/inbox/<custom-subdirectory>/<random-message-id>/<attachment-name>
/AS2/files/<station-identifier>/<partner-AS2-id>/outbox/<custom-subdirectory>/<random-message-id>/<attachment-name>

3.5.3 Save in to a custom subdirectory without the random generated ID

To save directly in to the custom subdirectory, enable both Remove Subdirectory With Random Message ID and Add Custom Subdirectory toggles. Message persisting paths will be as follows:

/AS2/files/<station-identifier>/<partner-AS2-id>/inbox/<custom-subdirectory>/<attachment-name>
/AS2/files/<station-identifier>/<partner-AS2-id>/outbox/<custom-subdirectory>/<attachment-name>

4. Automation via S3 API/Tools for SFTP messages

4.1 Bucket Content Layout for SFTP messages

<Partner-SFTP-ID> refers to the ID for the partner given when creating SFTP partner. SFTP/external is the location where SFTP partners receive their messages, and it’s also used by partners to send SFTP messages. Therefore, SFTP/external location can only be accessed using the SFTP keys generated for the SFTP partners. You are not allowed to upload files or download files from here.

  • SFTP/send [Send location]
    • <Partner-SFTP-ID>/
  • SFTP/partner[SFTP message Attachments]
    • <Partner-SFTP-ID>/inbox
    • <Partner-SFTP-ID>/inbox
  • SFTP/external [Partner’s Location. You do not have access to upload or download files from here]
    • <Partner-SFTP-ID>/send
    • <Partner-SFTP-ID>/inbox
    • <Partner-SFTP-ID>/outbox

4.2 Sending Files through SFTP

To submit a file to be sent through SFTP, copy it (i.e. PutObject) to the S3 location SFTP/send/<partner-SFTP-id>/. If you want to send a single file, you can copy it directly to this location (or S3 prefix). If you have more than one file to be submitted over SFTP, zip the files into a single archive, and then upload the archive.

e.g.

aws s3 cp <local-file> s3://<mftg-bucketname>/SFTP/send/<partner-SFTP-id>/

After a file has been submitted for send, the system will be automatically triggered, and will perform the SFTP communications process to send the file to the designated partner. The processed file will be placed into the SFTP/partners/<Partner-SFTP-ID>/outbox location

Files uploaded with same name within a 6-hour window will be considered as duplicates and will not be processed. If you need to submit files with the same name, add a postfix to the filename, such as an extension. (Duplicated files will be moved to SFTP/partners/<Partner-SFTP-ID>/duplicate)

S3 triggers only apply for the content directly uploaded to the above mentioned directory / S3 prefix. Any content copied or moved within the S3 bucket itself, will be ignored.

4.3 Receiving SFTP Files

Received files will be persisted to the SFTP/partners/<partner-SFTP-id>/inbox. You can fetch/download these payload files through the (GetObject) call.

e.g.

aws s3 cp s3://<mftg-bucketname>/SFTP/partners/<partner-SFTP-id>/inbox/<timestamp-random-number>/<attachment-name> <local-path>

4.4 Customizing S3 prefix of SFTP message attachments

By default, each message will have a unique path / S3 prefix, with the timestamp followed by a random number, within which the actual attachments will be found. e.g.

/SFTP/partners/<partner-SFTP-id>/inbox/<timestamp-random-number>/<attachment-name>
/SFTP/partners/<partner-SFTP-id>/outbox/<timestamp-random-number>/<attachment-name>

You can customize default S3 prefix by updating the SFTP Partner. Go to Manage SFTP Partner View –> Advanced Options –> File Structure.

Update SFTP Partner View

4.4.1 Save directly into Inbox or Outbox

Enable only Remove Subdirectory With Random Message ID toggle and update the partner. The attachments of incoming SFTP messages will be persisted directly in to inbox, and attachments of outgoing SFTP messages will be directly persisted in to outbox. e.g.

/SFTP/partners/<partner-SFTP-id>/inbox/<attachment-name>
/SFTP/partners/<partner-SFTP-id>/outbox/<attachment-name>

Note : Removing the subdirectory with a randomly generated message ID carries the risk of potential file overwrites if files with the same name exist.

4.4.2 Add a Custom Subdirectory

Enable only Add Custom Subdirectory toggle and update the partner. The attachments of SFTP messages will be persisted in to inbox or outbox under the given subdirectory name and then a random generated ID. e.g.

/SFTP/partners/<partner-SFTP-id>/inbox/<custom-subdirectory>/<random-message-id>/<attachment-name>
/SFTP/partners/<partner-SFTP-id>/outbox/<custom-subdirectory>/<random-message-id>/<attachment-name>

4.4.3 Save in to a custom subdirectory without the random generated ID

To save directly in to the custom subdirectory, enable both Remove Subdirectory With Random Message ID and Add Custom Subdirectory toggles. Message persisting paths will be as follows:

/SFTP/partners/<partner-SFTP-id>/inbox/<custom-subdirectory>/<attachment-name>
/SFTP/partners/<partner-SFTP-id>/outbox/<custom-subdirectory>/<attachment-name>