STAC Report: kdb+ tick analytics at scale on GCP

First public cloud solution with publicly disclosed STAC-M3 results outperformed previously tested Lustre-based on-prem solution

25 October 2018

STAC recently performed STAC-M3 benchmarks on a stack consisting of Kx Systems’ kdb+ 3.6 database system distributed across 13 x Google Cloud Platform (GCP) custom instances (32vCPU, 160GB DRAM, Skylake requested), accessing Google Persistent SSD. The solution was subjected to both the baseline (Antuco) and scaling (Kanaga) suites of STAC-M3.

The report is available here.

STAC-M3 is the set of industry standard enterprise tick-analytics benchmarks for database solutions that manage large time series of market data (tick data). STAC-M3 delivers dozens of test results, which are presented through a variety of tables and visualizations in this report. Google chose to highlight the following:

• These are the first public STAC-M3 results based on a public cloud solution.
• This solution, composed of "off the shelf" GCP offerings, outperformed a kdb+ solution involving a Lustre-based on-premise cluster (SUT ID KDB150528) in 14 of the 17 required baseline (Antuco) benchmarks (from 1.3x to 7.8x speedup)
• It also outperformed the Lustre-based solution in 16 of 16 of the scale (Kanaga) benchmarks that were reported for that solution (from 1.6x to 12.6x speedup)*

* KDB150528 operated on only 4 years of data. For that dataset size, the Kanaga suite has 16 benchmarks. The GCP solution operated on 5 years of data, which results in 24 benchmarks.
This GCP solution outperformed a bare metal solution based on Broadwell EX and 6TB DRAM (SUT ID KDB160425) in 8 of the 15 required benchmarks.

Details are in the STAC Reports at the links above. Premium subscribers also have access to the code used in this project and the micro-detailed configuration information for the solution. (To learn about subscription options, please contact us.)

This project follows on the heels of an earlier project in which we tested kdb+ running in memory in a GCP "Ultramem" instance. Google has now disclosed results for all of the dozens of STAC-M3 use cases, on datasets ranging from 3TB in memory to 55TB on Persistent SSD.