STAC Report: kdb+ 4.0 (sharded) with Lenovo and Intel Cooper Lake under STAC-M3 (tick analytics)

Lenovo & Intel set records using new kdb+ STAC Pack

20 July 2021

STAC recently performed the baseline and scaling suites of STAC-M3™ Benchmarks on the first stack where kdb+ utilized data distributed across the nodes and available locally from Optane™ Persistent Memory (PMem) via the memory bus. This is in contrast to previous kdb+ stacks that either utilized data replication or a shared filesystem.  The results are available here.

STAC-M3 is the set of industry standard enterprise tick-analytics benchmarks for database software/ hardware stacks that manage large time series of market data ("tick data"). In this project, we ran both the baseline benchmark suite (code named Antuco) and the optional scaling suite (code named Kanaga).

The stack under test (SUT) was KX’s kdb+ 4.0 database system configured to run in sharded mode on a 2-node cluster of 4-socket Lenovo ThinkSystem SR860 V2 servers, each with 4 x Intel^® Xeon Platinum 8380HL (Cooper Lake) 28-core CPUs @ 2.9GHz and 12TiB Intel^® Optane™ Persistent Memory 200 Series. Lenovo chose to highlight that this solution:

• Outperformed all publicly disclosed results in 10 of 17 Antuco benchmarks:
  • 4 of the benchmarks had speed-ups exceeding 2.4x compared to the previous best result
    (STAC-M3.v1.100T.VWAB-12D-NO.TIME, STAC-M3.ß1.10T.STATS-AGG.TIME, STAC-M3.ß1.1T.YRHIBID.TIME, STAC-M3.ß1.1T.QTRHIBID.TIME)
• Outperformed all publicly disclosed results in 9 of 10 Kanaga benchmarks on solutions configured for 2 or 3 years of data
• When compared to a single-node SUT with PMem configured with Sector mode namespaces, an earlier version of the kdb+ STAC Pack, and just 2 years of data (KDB201109):
  • had speed-ups greater than 3x in 3 of 10 two-year Kanaga benchmarks
    (STAC-M3.ß1.1T.TRUNC.2YRHIBID.TIME, STAC-M3.ß1.50T.TRUNC.YR{1,2}VWAB-12D-HO.TIME)
  • was faster in 13 of 17 Antuco benchmarks, including:
    • 6.1x speed-up in 100-user 12-day VWAB (STAC-M3.v1.100T.VWAB-12D-NO.TIME)
    • 4.5x or more speed-up in year high bid and 10-user aggregate stats
      (STAC-M3.ß1.1T.YRHIBID.TIME, STAC-M3.ß1.10T.STATS-AGG.TIME)

For details, please see the report at the link above. Premium subscribers have access to the code used in this project as well as the micro-detailed configuration information for the solution. To learn about subscription options, please contact us.