With modern technology embracing microservices or API’s, it’s a valid question to ask about the large scale, continued use and importance of file transfers in business, government or healthcare industry. The Applicability Statement 2 (AS2) protocol defined by the IETF, remains the most widely accepted and adopted standard for file transfers across the world.
All businesses, government authorities and other organizations require the efficient exchange of information with other parties. Modern computer systems and mobile applications need to exchange information with other systems in a very fast and transactional manner, to provide the best performance and user experience. This applies well for a purchase from an ecommerce website, or for an online fund transfer. When large amounts of data, or business critical, personal or sensitive information needs to be exchanged, they are still placed into files, encrypted / secured, and exchanged in a standards based manner. This helps secure the content independent of the transmission technology or path, provide at-rest encryption, or retry failed transmissions, or failed processing reliably. Files also work naturally well with batch processing systems or mainframe systems, which still drives many of the largest organizations.
APIs and microservices work well where transactional messaging and high performance are important. However, they do not usually issue industry standard acknowledgements with digital signatures over the original requests made, allowing for legally valid proof of receipt. While SOAP messages with WS-Security allows XML signatures and encryption, it’s not widely used in business, nor define widely used standards. Most web based API calls depend on the underlying transport level SSL/TLS encryption for in-transit encryption of content.
The replay of messages to recover transmission errors, or partner system unavailability due to maintenance or downtime, or to recover from batch processing errors, exposes a whole new set of problems. Another significant reason would be the still wide use of mainframe or mini computer systems by large organizations, that perform batch processing, and generate large amounts of data, as files. The use of Electronic Data Interchange (EDI) standards such as the US X12 or the UN EDIFACT standard, also fits naturally with these systems and files.
We can summarize the advantages of file transfers as follows
- Ability to transmit large amounts of data (several hundred megabytes or more)
- Ability to easily retransmit a failed request again, and to easily recognize duplicate transmissions
- Ability to encrypt the contents to ensure only the authorized recipient can read the contents
- Ability to digitally sign the contents so that the recipient can confirm the content as sent by the sender
- Ability to guarantee that data remained unchanged during transmission
- Ability to request for and receive a digitally signed receipt for a file, which is legally recognized for non-repudiation
- Ability to process data through batch processing systems, and retry failed attempts and error recovery
- Natural fit for batch processing, EDI and mainframe and mini computer systems used by large organizations
File Transfer Protocol (FTP) and Secure FTP has been used to simply exchange files in a fast and efficient manner. Value Added Networks (VANs) also connected several circles of businesses together over private networks in the early days, but have since been deprecated due to high costs, with the advent of the Internet.
Depending on the business and industry, the files exchanged can vary from Electronic Data Interchange (EDI) US X12 or UN EDIFACT files, XML, JSON, CSV, Flat files or binary files. For example, a single file sent over AS2 by a buyer such as Walmart, could include several EDI 850 Purchase Orders as shown below.
ISA*00* *00* *08*925485US00*ZZ*VENDOR*210729*1538*:*00501*850100003*0*P*>~ GS*PO*925485US00*VENDORXXXXXX*20210729*1538*850100003*X*005010~ ST*850*100004~ BEG*00*SA*5712201804**20210729~ ... first Purchase Order detail (omitted for clarity).. AMT*1*3.75~ CTT*5~ AMT*GV*18.75~ SE*47*100004~ ST*850*100005~ BEG*00*SA*5712201805**20210729~ ... second Purchase Order detail (omitted for clarity).. AMT*1*3.75~ CTT*1~ AMT*GV*3.75~ SE*47*100010~ GE*7*850100003~ IEA*1*850100003~