This document describes performance testing carried out on a dedicated deployment of the MFT Gateway. The objective of the test was to test the ability of the MFT Gateway to process a steady stream of 1MB files at 100 transactions per second. The tests included the following:
- Receipt of 1MB messages - target 100 TPS using the Large payload EC2 endpoints
- Receipt of 1MB messages - target 100 TPS using the API Gateway endpoints
- Sending of outgoing messages of 1MB in size - target 100 TPS
- Long-running test of 12 hours - target over 1 million messages across 12 or more hours
The complete test was conducted within the Amazon AWS environment. The load generator was a single EC2 instance, with the MFT Gateway deployment receiving the messages over the API Gateway and Large payload (EC2) endpoints.
The Lambda functions performing the core AS2 operations of the MFT Gateway was configured as follows. Note that the key Lambda functions were allocated 2048M of memory, and allowed the maximum execution delay of 15 minutes.
- MFT-Gateway-AS2-Servlet-Receiver / 2048M / 15 min : Process uploads via Large payload EC2
- MFT-Gateway-AS2-Sender / 2048M / 15 min : AS2 sender
- MFT-Gateway-AS2-Receiver / 2048M / 15 min : API Gateway exposed AS2 receiver
- MFT-Gateway-AS2-Async-Processor / 2048M / 15 min : Async MDN processor
- as2Servicesapiss3UploadTrigger / 512M / 5 min
EC2: c3.2xlarge / 8 vcpu / 15G RAM / NW: high / 64-bit
To generate a steady stream of large encrypted and signed messages, we utilized JavaBench, a variant of the popular ApacheBench tool, as described and used in the ESB Performance Testing framework. We saved a signed and encrypted AS2 message in binary form into a file, and this file was submitted by the load generator across multiple iterations, and at the specified concurrency. During testing, initially we hit the network bandwidth limit allowed on the EC2 instance size selected. This was overcome by selecting a ‘c3.2xlarge’ instance with ‘high’ network performance. The average NW utilization at the load generator was ~18 MB/s (megabytes, per second)
- Binary file of size 1055953 (ie. signed and encrypted size)
- AS2: signed SHA-1 and encrypted 3DES 168-bit, requesting sync MDN
AS2-From: hiruload User-Agent: MFT Gateway Disposition-Notification-To: firstname.lastname@example.org AS2-Version: 1.1 From: email@example.com Message-Id: <firstname.lastname@example.org> Content-Transfer-Encoding: binary AS2-To: loadtest Disposition-Notification-Options: signed-receipt-protocol=required,pkcs7-signature; signed-receipt-micalg=required,sha1 Subject: 360 200 "" 5 1MB LB Test01 MIME-Version: 1.0 Content-Type: application/pkcs7-mime; name="smime.p7m"; smime-type=enveloped-data
The AS2 message load simulated 200 concurrent users, with each such user performing 360 requests immediately, one after the other. Each request sends the saved AS2 (signed and encrypted) message of ~1MB to the designated URL. This is repeated across 5 iterations. So the total requests made are 200 x 360 x 5 = 360,000.
nohup ./run.sh 360 200 "" 5 > run.log-$(date +%T) &
EC2: 6 x t3a.medium / 2 vcpu / 4G / 5Gbps burst The large payload endpoints were configured with an auto-scaling EC2 server cluster, fronted by an Application Load Balancer with stickiness set to off. The auto-scale group had 6 EC2 instances of the above mentioned ‘t3a.medium’ configuration, with each running Apache Tomcat 8.5.51 with Java 1.8.0_302-b08, and configured with 2G of heap memory (i.e. -Xms2G / -Xmx2G). Note that the task of these EC2 instances is simply the acceptance of AS2 messages over HTTP/S and uploading of the request to S3. The expectation was to configure a single EC2 instance / Tomcat node, to process a 100 concurrent connections at a time. Thus, the following Tomcat configuration was used.
<Connector port="8080" protocol="org.apache.coyote.http11.Http11NioProtocol" address="0.0.0.0" asyncTimeout="600000" connectionTimeout="600000" maxKeepAliveRequests="10000" processorCache="300" acceptCount="0" maxThreads="100" maxConnections="100" disableUploadTimeout="false" redirectPort="8443" />
<Connector port="8090" protocol="HTTP/1.1" address="0.0.0.0" asyncTimeout="30000" connectionTimeout="30000" maxKeepAliveRequests="5" processorCache="300" acceptCount="0" maxThreads="5" maxConnections="5" disableUploadTimeout="false" />
# S3 transfer related properties s3.core.threads=20 s3.max.threads=200 s3.idle.keepalive.sec=60 s3.multipart.upload.threshold=5248000 # AWS clients related properties aws.request.timeout=-1 aws.socket.timeout=120000 aws.client.execution.timeout=0 aws.max.error.retry=0 aws.max.connections=150 # Thread pool related properties executor.core.threads=20 executor.max.threads=200 executor.idle.keepalive.sec=60
To test the sending out of AS2 messages, we created a shell script copying a file of size 1MB into the S3 path to be picked up by the MFT Gateway AS2 send flow. Files were copied at a rate of 100 per second. These files were signed and encrypted, and then sent over to the API Gateway based endpoints as a loopback test.
The following observations were recorded during the execution of the first load test, sending requests to the large payload endpoints, on the EC2 load balancer.
We can see a steady CPU utilization at around 17% during the testing. The network utilization also reflects this, and everything seems exactly as expected.
We can see that the load balacer was responding back to each request with a response time of approximately 2,000 ms - or 2 seconds. We also see a few HTTP 4xx errors and ELB 4xx errors.
The successful executions, HTTP 2xx success responses, active connections and network utilization appear normal and as expected.
We can see that each instance was utilizing around 90% of the CPU resources during the testing, and all nodes exhibited similar behaviour.
On the memory usage, we notice one instance which utilized more memory than the others. Its memory use gradually increased upto around 60% while the other five nodes, maintained utilization around 30%.
This is the main Lambda function that processes received AS2 messages. It decrypts te raw message, validates the signature, and stores the attachments into the S3 bucket, and updates the metadata in the DynamoDB database. During the testing, we can observe invocations across the five iterations, with an average execution duration of 588ms.
The memory utilization for the processing seems to be between 253 MB to 273 MB for most executions, and the concurrent executions was around 170 on average.
The raw output captured via JavaBench across the five iterations are as follows.
Running iteration 0 Document Path: http://large-payload.service.loadtest.mftgateway.com/receiver Average Document Length: 3840 bytes Concurrency Level: 200 Time taken for tests: 661.030229 seconds Complete requests: 71994 Failed requests: 3 Write errors: 3 Total transferred: 76298805070 bytes Requests per second: 108.91 [#/sec] (mean) Time per request: 1,836.348 [ms] (mean) Time per request: 9.182 [ms] (mean, across all concurrent requests) Transfer rate: 418.32 [Kbytes/sec] received 115,005.75 kb/s sent 115,424.08 kb/s total Running iteration 1 Document Path: http://large-payload.service.loadtest.mftgateway.com/receiver Average Document Length: 3847 bytes Concurrency Level: 200 Time taken for tests: 646.062181 seconds Complete requests: 71998 Failed requests: 1 Write errors: 1 Total transferred: 76303495614 bytes Requests per second: 111.44 [#/sec] (mean) Time per request: 1,794.667 [ms] (mean) Time per request: 8.973 [ms] (mean, across all concurrent requests) Transfer rate: 428.74 [Kbytes/sec] received 117,676.76 kb/s sent 118,105.50 kb/s total Running iteration 2 Document Path: http://large-payload.service.loadtest.mftgateway.com/receiver Average Document Length: 3849 bytes Concurrency Level: 200 Time taken for tests: 648.861845 seconds Complete requests: 71998 Failed requests: 1 Write errors: 1 Total transferred: 76303642486 bytes Requests per second: 110.96 [#/sec] (mean) Time per request: 1,802.444 [ms] (mean) Time per request: 9.012 [ms] (mean, across all concurrent requests) Transfer rate: 427.11 [Kbytes/sec] received 117,169.02 kb/s sent 117,596.13 kb/s total Running iteration 3 Document Path: http://large-payload.service.loadtest.mftgateway.com/receiver Average Document Length: 3849 bytes Concurrency Level: 200 Time taken for tests: 646.041599 seconds Complete requests: 72000 Failed requests: 0 Write errors: 0 Total transferred: 76305752140 bytes Requests per second: 111.45 [#/sec] (mean) Time per request: 1,794.560 [ms] (mean) Time per request: 8.973 [ms] (mean, across all concurrent requests) Transfer rate: 428.98 [Kbytes/sec] received 117,683.78 kb/s sent 118,112.75 kb/s total Running iteration 4 Document Path: http://large-payload.service.loadtest.mftgateway.com/receiver Average Document Length: 3849 bytes Concurrency Level: 200 Time taken for tests: 651.216166 seconds Complete requests: 71998 Failed requests: 1 Write errors: 1 Total transferred: 76303628078 bytes Requests per second: 110.56 [#/sec] (mean) Time per request: 1,808.984 [ms] (mean) Time per request: 9.045 [ms] (mean, across all concurrent requests) Transfer rate: 425.55 [Kbytes/sec] received 116,745.42 kb/s sent 117,170.97 kb/s total
The load test issued a total of 359,988 completed AS2 requests at the large payload endpoint. A total of 12 requests failed. The MFT Gateway was able to consistently receive and process the messages of 1MB, at a rate above 110 TPS. The MFT Gateway received a total of 359,992 messages successfully, which were saved to the DynamoDB metadata store and S3 storage. The difference between this number and the success count at the load generator is due to 4 requests which completed processing at the MFT Gateway, but not receiving the successful HTTP response at the load generator before the connection timedout.
This load test was a loopback test. It signed, encrypted and sent a text file of 1,048,576 bytes (~1MB) - out of the MFT Gateway, and back into the MFT Gateway. The files were received to the API Gateway based endpoints. A total of 360,000 files were copied to be sent out and received.
The ‘MFT-Gateway-AS2-Sender’ lambda function was responsible for the creation and sending out of AS2 messages, to the API Gateway endpoint. This was allocated 2,048 MB (~2G) of RAM, and provided with the maximum 15 minute timeout.
The processing took around ~1,725ms on average, and had close to ~300 concurrently executing instances of the Lambda function.
The ‘MFT-Gateway-AS2-Receiver’ lambda function was responsible for the processing of incoming AS2 messages, received via the API Gateway endpoint. This was allocated 2,048 MB (~2G) of RAM, and provided with the maximum 15 minute timeout.
The processing took around ~840ms on average, and had close to ~190 concurrently executing instances of the Lambda function.
The API Gateway endpoint which fronted the AS2 receiver Lambda showed the following metrics. The average latency was around ~885ms.
The load test triggered the sending out of 360,000 text files of ~1MB in size. All of the files were received successfully, and matched both the received S3 bucket statistics and DynamoDB meta data. The test executed for 51 minutes, yielding a TPS of ~118.
This load test was executed for a period over 12 hours, to measure the capability of the system under continuous load. Due to the costs and overhead involved, this test was executed against one Large payload EC2 instance at a concurrency of 30 users (i.e. concurrent connections), with each executing 2,000 requests across 24 iterations. The total messages transferred was 1 x 30 x 2000 x 24 = 1,440,000 messages. The test was executed similarly to the test described in Section 2.1.
The EC2 instance was using around ~80% CPU and around 63% of the memory allocated to the system.
The Servlet running on Tomcat utilized around ~280MB of heap memory during the test.
A total of 1,440,000 messages of ~1MB was handled across a period over 12 hours, on a file of size ~1MB. The test executed for around 13 hours, giving an TPS of 31. Extrapolating this across 24 hours, and 6 node EC2 auto-scale group yields around 16 million messages of ~1MB per day.
This load test was performed in October 2021, as a preparation for a customer proof-of-concept performance test, where a 100 TPS load, and processing of a million messages per day was the target. During the testing, it was observed that the MFT Gateway can surpass these metrics many times over. With the scalability of the underlying AWS cloud platform, any increase of load will actually translate into a test of the underlying cloud platform.
The Aayu Technologies team will be happy to open up this deployment for any customer proof-of-concept or load testing, or allow this test to be carried out on a customer owned AWS account for full transparency and observability.
We welcome any suggestions and feedback on this performance test, and expect to run an updated test again in a few months.