STAC Report: Sharded kdb+ on Google Cloud NVMe (tick analytics)

Google Cloud boosts its STAC-M3 results.

9 May 2022

Google Cloud was the first cloud provider to authorize public release of STAC-M3 results a few years ago. Recently, they asked STAC to audit STAC-M3 benchmarks on an alternative architecture (with upgraded components). The results are available here.

STAC-M3 is the set of industry standard enterprise tick-analytics benchmarks for database software/ hardware stacks that manage large time series of market data ("tick data"). In this project, we ran both the baseline benchmark suite (code named Antuco) and the optional scaling suite (code named Kanaga).

The stack under test (SUT) was KX’s kdb+ 4.0 DBMS configured to run in sharded mode on a cluster of 12 custom N2 Google Compute Engine instances, each with 32 vCPUs on Intel® Cascade Lake CPUs, 160GiB of RAM, and a single 32Gbps network interface. The sharded data were stored in 24 x 375GiB local NVMe SSDs per instance, for a system-wide total of 105TiB. Google Cloud chose to highlight that this solution:

• Outperformed all publicly disclosed results in the version of Year-High Bid that allows caching (STAC-M3.ß1.1T.YRHIBID-2.TIME)
• When compared to a previous stack in Google Cloud that used an earlier compatibility rev of the STAC Pack, Google Persistent SSD, and 13 Skylake instances (KDB181001):
• Was faster in 13 of 17 Antuco mean response-time benchmarks, including 18x the speed in the version of Year-High Bid that allows caching (STAC-M3.ß1.1T.YRHIBID-2.TIME)
• Was faster in 22 of 24 Kanaga mean response-time benchmarks, including over 10x the speed in market snapshot (STAC-M3.ß1.10T.YR[2,3,4,5]-MKTSNAP.TIME)
• Had 9x higher throughput in Year-High Bid (STAC-M3.ß1.1T.YRHIBID.MBPS)
• When compared to a stack in another public cloud with compute distributed across 15 instances and data stored in a parallel filesystem across 40 instances (KDB210507):
• Was faster in 11 of 17 Antuco mean response-time benchmarks, including 8x the speed in the version of Year-High Bid that allows caching (STAC-M3.ß1.1T.YRHIBID-2.TIME)
• Was faster in 13 of 24 Kanaga mean response-time benchmarks, including over 14x the speed in market snapshot benchmarks (STAC-M3.ß1.10T.YR[2,3,4,5]-MKTSNAP.TIME)

For details, please see the report at the link above. Premium subscribers have access to the code used in this project as well as the micro-detailed configuration information for the solution. To learn about subscription options, please contact us.