STAC Report: Amazon io2 Block Express and kdb+ 4.0 (tick analytics)

First Amazon-engineered STAC-M3 solution.

10 May 2022

STAC recently performed the first STAC-M3 Benchmarks on a stack engineered by Amazon. The solution involved kdb+ 4.0 on 8 Amazon EC2 instances that accessed data stored on Amazon EBS io2 Block Express. The results are available here.

STAC-M3 is the set of industry standard enterprise tick-analytics benchmarks for database software/ hardware stacks that manage large time series of market data ("tick data"). In this project, we ran the baseline benchmark suite (code named Antuco).

The stack under test (SUT) was KX’s kdb+ 4.0 DBMS configured to run in distributed mode on 8 Amazon EC2 x2iedn.32xlarge instances, each with 128 vCPUs on Intel® Ice Lake CPUs, 4096 GiB of DRAM, and a single 100 Gbps Amazon Elastic Network Interface. The EC2 instances were a mix of shared and dedicated. The STAC-M3 Antuco database was distributed over four 4000GiB Amazon EBS io2 Block Express multi-attach volumes, with a separate EBS volume for the write tests. Amazon chose to highlight that this solution:

• Outperformed all publicly disclosed results in the 10-user volume curve mean response time benchmark (STAC-M3.β1.10T.VOLCURV.TIME):
• 2.2x speed-up versus the previous best result (SUT ID KDB210428)
• When compared to a solution on AWS distributed across 15 EC2 instances and data stored in a parallel filesystem across 40 AWS instances (SUT ID KDB210507), was faster in 15 of 17 mean response-time benchmarks, including:
• 21x speed-up in 10-user volume curve (STAC-M3.β1.10T.VOLCURV.TIME)
• When compared to an on-prem solution involving network-attached flash storage and distributed across 9 database servers (SUT ID KDB210929), was faster in all 17 mean response-time benchmarks, including:
• 11x speed-up in single-user stats with unpredictable intervals (STAC-M3.β1.1T.STATS-UI.TIME)
• When compared to an on-prem solution using 15 database servers accessing a single all-flash storage appliance via a parallel file system (SUT ID KDB211014), was faster in 13 of 17 mean response-time benchmarks, including:
• 4.8x speed-up in 10-user volume curve (STAC-M3.β1.10T.VOLCURV.TIME)

For details, please see the report at the link above. Premium subscribers have access to the code used in this project as well as the micro-detailed configuration information for the solution. To learn about subscription options, please contact us.