STAC Report: kdb+ with Dell EMC Isilon F800 Flash Array under STAC-M3 (tick analytics)

NFS-based solution beats previously tested Lustre-based solution in most benchmarks

16 June 2019

STAC recently performed STAC-M3 Benchmarks on a stack involving the kdb+3.6 database system with the Dell EMC Isilon F800 All-Flash Scale-Out NAS. The STAC Report is available here.

STAC-M3 is the set of industry standard enterprise tick-analytics benchmarks for database software/hardware stacks that manage large time series of market data ("tick data"). In this project, we ran both the baseline benchmark suite (code named Antuco) and the optional scaling suite (code named Kanaga). 

The stack under test was Kx's kdb+ database distributed across 4 Dell EMC PowerEdge R940 servers using NFS over 40GbE to access a single 4U Isilon All-Flash Scale-Out NAS chassis with 192 TiB physical capacity. Dell EMC chose to highlight that this solution based on networked flash storage was:

• 2.2x to 5.5x the speed of a solution using a single 4-socket server with direct-attached Intel Optane and NAND flash SSDs (SUT ID KDB181009) in the 100-user, 12-day VWAB operation on each year of the STAC-M3 Kanaga dataset (STAC-M3.β1.100T.YR1VWAB-12D-HO.TIME, … STAC-M3.β1.100T.YR5VWAB-12D-HO.TIME)
• Faster than a solution using a competitive flash array (SUT ID KDB140415) in 9 of 17 mean-response time benchmarks in the STAC-M3 Antuco suite, including:
  • Over 3.1x the speed in 12-day VWAB queries (STAC-M3.β1.100T.VWAB-12D-NO.TIME)
  • Over 2x the speed in year-high bid and quarter-high bid queries (STAC-M3.β1.1T.YRHIBID.TIME and STAC-M3.β1.1T.QTRHIBID.TIME, respectively.
• Faster than a Lustre-based solution (SUT ID KDB150528) in 11 of 17 mean-response time STAC-M3 Antuco benchmarks:
  • Over 7.8x the speed in 12-day VWAB queries (STAC-M3.β1.100T.VWAB-12D-NO.TIME)
  • About 7.9x the speed  in week-high bid (STAC-M3.β1.1T.WKHIBID.TIME)
• Faster than the same Lustre-based solution in 15 of 16 STAC-M3 Kanaga benchmarks*, including:
  • Over 24.2x speedup in STAC-M3.β1.1T.YR3VWAB-12D-HO.TIME
  • Over 16.4x speedup in STAC-M3.β1.50T.YR3VWAB-12D-HO.TIME
  • Over 9.4x speedup in STAC-M3.β1.100T.YR3VWAB-12D-HO.TIME

* Of the 16 measured for KDB150528, which operated on only 4 years of data. The Dell EMC solution operated on 5 years of data, which results in 24 benchmarks.

Details are in the STAC Report at the link above. Premium subscribers have access to the code used in this project and the micro-detailed configuration information for the solution. (To learn about subscription options, please contact us.)