STAC Report: kdb+ 4.0 with an Intel persistent memory pool under STAC-M3 (tick analytics)

First solution with distributed Optane Persistent Memory sets 8 records across scale and baseline benchmarks

20 August 2020

STAC recently performed STAC-M3™ Benchmarks on a stack involving kdb+ 4.0 and a server cluster supporting a pool of Intel® Optane DC Persistent Memory. The results are available here.

STAC-M3 is the set of industry standard enterprise tick-analytics benchmarks for database software/hardware stacks that manage large timeseries of market data ("tick data"). In this project, we ran both the baseline benchmark suite (code named Antuco) and the optional scaling suite (code named Kanaga).

The stack included Kx's kdb+ 4.0 running on a cluster of ten 2-socket Supermicro SYS-6019U-TN4RT servers. Each server had two Intel® Xeon® Platinum 8280L CPUs @ 2.70GHz, 192GB of DRAM, and 6TB of Intel® Optane DC Persistent Memory. The database storage was distributed over the cluster's 60TB pool of persistent memory. The kdb+ processes ran in Docker v1.13 containers orchestrated by Red Hat OpenShift v3.11. The SUT was embedded in an implementation of the Intel® Data Management Platform, a reference solution targeted at data scientists and data engineers within large organizations. Additional high-level configuration information is available in the report linked above (and micro-detailed configuration information is also available at the same link).

Intel® chose to highlight that this solution:

• Outperformed all publicly disclosed results in 5 of 24 Kanaga mean-response time benchmarks:
  • 50-user 12-day VWAB benchmark year 1 (STAC-M3.β1.50T.YR1VWAB-12D-HO.TIME).
  • 10-user market snapshot: benchmark years 2 through 4 (STAC-M3.β1.10T.YR[n]-MKTSNAP.TIME), including:
  • 23.9x speedup in benchmark year 2 (STAC-M3.β1.10T.YR2-MKTSNAP.TIME)
  • 20.9x speedup in benchmark year 4 (STAC-M3.β1.10T.YR4-MKTSNAP.TIME)

• Outperformed all publicly disclosed results in 3 of 17 Antuco mean-response time benchmarks, including:
  • 1.2x speedup in 100-user interval stats (STAC-M3.β1.100T.STATS-UI.TIME)
  • 2.7x speedup in 100-user 12-day VWAB – no overlap (STAC-M3.β1.100T.VWAB-12D.NO.TIME)
  • 4.3x speedup in 10-user market snapshot (STAC-M3.β1.10T.MKTSNAP.TIME)

Detailed results are in the STAC Report. Premium subscribers have access to the code used in this project as well as the configuration details. (To learn about subscription options, please contact us.)