STAC Report: kdb+ 3.6 with DDN EXAScaler & AI400X Flash appliances under STAC-M3 (tick analytics)

Solution sets multiple throughput records

13 October 2020

STAC recently performed STAC-M3 Benchmarks on a stack involving kdb+, the DDN EXAScaler parallel file system, and two DDN AI400X All-Flash storage appliances. The results are available here.

STAC-M3 is the set of industry standard enterprise tick-analytics benchmarks for database software/ hardware stacks that manage large time series of market data ("tick data"). In this project, we ran both the baseline benchmark suite (code named Antuco) and the optional scaling suite (code named Kanaga).

The stack under test was Kx's kdb+ 3.6 database system distributed across 15 Intel S2600BPB servers accessing two DDN AI400X All-Flash appliances via the DDN EXAScaler 5.1.1 parallel filesystem. DDN chose to highlight that this solution:

• Outperformed all publicly disclosed results in all year-high bid throughput benchmarks (STAC-M3.β1.1T.*.BPS)
• Versus a kdb+ solution based on a parallel file system with 14 database servers and 18 storage servers (SUT ID KDB200401):
  • was faster in 4 of 17 Antuco mean response time benchmarks, including:
    • 9.4x speedup in NBBO (STAC-M3.β1.1T.NBBO.TIME)
    • 46% faster in 10-user market snapshot (STAC-M3.β1.10T.MKTSNAP.TIME)
    • 15% faster in 10-user aggregate stats (STAC-M3.β1.10T.STATS-AGG.TIME)
    • 3% faster in 100-user interval stats (STAC-M3.β1.100T.STATS-UI.TIME)
  • was within 0.2% of the mean response time for 10-user volume curve (STAC-M3.β1.10T.VOLCURV.TIME)

• Versus a kdb+ solution with 4 database servers and networked flash storage (SUT ID KDB190430):
  • was faster in 14 of 17 Antuco mean response time benchmarks, including:
    • 12x speedup in 10-user market snapshot (STAC-M3.β1.10T.MKTSNAP.TIME)
    • 6x speedup or greater in all four user counts doing intervalized stats (STAC-M3.β1.*.STATS-UI.TIME)
    • 5x speedup in 10-user aggregate stats (STAC-M3.β1.10T.STATS-AGG.TIME)
  • was faster in 20 of 24 Kanaga mean response time benchmarks, including:
    • 19x speedup or greater in all four 10-user market snapshots (STAC-M3.β1.10T.YR[n]-MKTSNAP.TIME)
    • 3x speedup or greater in year-high bid for all four years (STAC-M3.β1.1T.[n]YRHIBID.TIME)

  Details are in the STAC Report at the link above. Premium subscribers have access to the code used in this project and the micro-detailed configuration information for the solution. (To learn about subscription options, please contact us.)