STAC Report: kdb+ 3.6 on WekaIO Matrix and a Penguin cluster under STAC-M3 (tick analytics)

Solution sets 8 records in scale tests

3 June 2019

STAC recently performed STAC-M3 Benchmarks on a stack involving kdb+, the WekaIO Matrix parallel filesystem, and a server cluster from Penguin Computing. The results are available here.

STAC-M3 is the set of industry standard enterprise tick-analytics benchmarks for database software/hardware stacks that manage large time series of market data ("tick data"). In this project, we ran both the baseline benchmark suite (code named Antuco) and the optional scaling suite (code named Kanaga). 

The stack under test was Kx's kdb+ 3.6 database system distributed across 7 Penguin Computing Relion XO1132g servers, with data stored in a cluster of 8 Penguin Computing Relion XE1112 servers with a total of 105 TiB of SSD capacity, all accessed via WekaIO Matrix 3.2.2 software. WekaIO chose to highlight that this solution:

• Outperformed all publicly disclosed results in 8 of 24 Kanaga mean-response time benchmarks (STAC-M3.β1.100T.YR1VWAB-12D-HO.TIME to STAC-M3.β1.100T.YR4VWAB-12D-HO.TIME, plus STAC-M3.β1.1T.2YRHIBID.TIME to STAC-M3.β1.1T.5YRHIBID.TIME)
• Outperformed all publicly disclosed results in 4 of 5 Kanaga throughput benchmarks (STAC-M3.β1.1T.2YRHIBID.MBPS through STAC-M3.β1.1T.5YRHIBID.MBPS)
• Was faster in 15 of 17 mean-response time benchmarks in the Antuco suite versus a kdb+ solution involving a Lustre-based on-premise cluster (SUT ID KDB150528), including:
  • 11x speedup in year-high bid (STAC-M3.β1.1T.YRHIBID.TIME)
  • 2.1x speedup in market snapshots (STAC-M3.β1.10T.MKTSNAP.TIME)
• Was faster in 13 of 17 mean-response time benchmarks in the Antuco suite versus a solution in which kdb+ ran in distributed mode on a public cloud (SUT ID KDB181001), including:
  • 2.15x the speed for STAC-M3.β1.100T.STATS-UI.TIME
  • 2.36x the speed for STAC-M3.β1.1T.VWAB-D.TIME
• Achieved 168 milliseconds for year-high bid (STAC-M3.β1.1T.YRHIBID.TIME), which is within 3 milliseconds of the best publicly disclosed result for this benchmark across all architectures.

Details are in the STAC Report at the link above. Premium subscribers have access to the code used in this project and the micro-detailed configuration information for the solution. (To learn about subscription options, please contact us.)