STAC Report: kdb+ 4.0 on WekaFS and AWS under STAC-M3 (tick analytics)

WekaFS on AWS set 6 Kanaga records

3 June 2021

STAC recently performed STAC-M3 Benchmarks on a stack involving kdb+, the WekaFS parallel filesystem, and an AWS EC2 cluster. The results are available here.

STAC-M3 is the set of industry standard enterprise tick-analytics benchmarks for database software/ hardware stacks that manage large time series of market data ("tick data"). In this project, we ran both the baseline benchmark suite (code named Antuco) and the optional scaling suite (code named Kanaga).

The stack under test (SUT) was Kx’s kdb+ 4.0 database system distributed across 15 Amazon EC2 C5n.9xlarge instances, using the WekaIO (Weka) WekaFS parallel filesystem to access data stored in a cluster of 40 Amazon EC2 i3en.6xlarge instances that provided a total of 546 TiB of SSD capacity. Weka chose to highlight the following from the results:

In terms of mean-response time benchmarks, this kdb+ 4.0 solution in Amazon Web Services with 15 database server nodes and 40 storage nodes:

• Outperformed all publicly disclosed results in 3 of 24 Kanaga benchmarks
  
  
• Versus a kdb+ 4.0 solution running on a 10-node cluster with 60TB of persistent memory (KDB200603):
• was faster in 16 of 24 Kanaga and 9 of 17 Antuco benchmarks

• Versus a kdb+ 3.6 solution on a parallel file system with 15 database servers accessing all-flash storage appliances (KDB200915):
• was faster in 20 of 24 Kanaga benchmarks and 4 of 17 Antuco benchmarks
• Versus a kdb+ 3.6 solution involving a previous version of Weka with 14 on-prem database nodes and 18 storage nodes (KDB200401):
• was faster in 3 of 24 Kanaga and 4 of 17 Antuco benchmarks
• Versus a kdb+ 3.6 solution involving 9 database servers accessing networked flash storage (KDB200914):
• was faster in 15 of 17 Antuco benchmarks

In terms of throughput benchmarks (megabytes per second), this solution:

• Outperformed all publicly disclosed results in 3 of 5 Kanaga benchmarks
  (STAC-M3.β1.1T.{3,4,5}YRHIBID.MBPS)

For details, please see the report at the link above. Premium subscribers have access to the code used in this project as well as the micro-detailed configuration information for the solution. To learn about subscription options, please contact us.