STAC Report: STAC-ML Tacana Suite on Myrtle.ai VOLLO

Sliding window inference using FPGA as an accelerator.

4 May 2023

Last fall, STAC performed a STAC-ML™ Markets (Inference) audit of Myrtle.ai VOLLO on the fixed-window suite (code named Sumaco). As a follow-on, Myrtle.ai recently asked STAC to perform an additional STAC-ML audit using the sliding-window suite (code named Tacana). For this audit, the hardware remained the same, but the solution used updated versions of the STAC Pack, VOLLO SDK, and VOLLO Accelerator. The STAC Report is now available here.

STAC-ML Markets (Inference) is the technology benchmark standard for solutions that can be used to run inference on realtime market data. Designed by quants and technologists from some of the world's leading financial firms, the benchmarks test the latency, throughput, realized precision, energy efficiency, and space efficiency of a technology stack across three model sizes and different numbers of model instances (NMI).

The stack consisted of the STAC-ML™ Pack for Myrtle.ai VOLLO™ (Rev B) using the Myrtle.ai VOLLO SDK v0.2.0 to control the VOLLO Accelerator v0.2.0 application that was loaded on 4 x BittWare IA-840f (Intel® Agilex™ AGF027 FPGA) Cards in a BittWare TeraBox™ 1402B Server.

Myrtle.ai wished to highlight several results from this report:

• For LSTM_A (the smallest model) the 99p latency was:¹
• From 5.07 µs to 5.08 µs across 1, 2 & 4 model instances tested (NMI)
• 5.97 µs with 8 NMI
• 6.96 µs with 24 NMI
• For LSTM_B the 99p latency was:²
• 6.89 µs with 1 NMI
• 6.77 µs with 2 NMI
• 7.75 µs with 8 NMI
• For LSTM_C (the largest model) the 99p latency was:³
• 31.0 µs with 1 NMI
• LSTM_A with 24 NMI achieved the following throughput and efficiency:⁴
• 1.4M inferences / second
• 1.4M inferences / second / cubic foot
• 2.3M inferences / second / kW

Premium subscribers have access to extensive visualizations of all test results, the micro-detailed configuration information for the solutions tested, the code used in this project, and the ability to run these same benchmarks in the privacy of their own labs. To learn about subscription options, please contact us.

---------------
¹STAC-ML.Markets.Inf.T.LSTM_A.[1,2,4,8,24].LAT.v1
²STAC-ML.Markets.Inf.T.LSTM_B.[1,8].LAT.v1
³STAC-ML.Markets.Inf.T.LSTM_C.1.LAT.v1
⁴ STAC-ML.Markets.Inf.T.LSTM_A.24.[TPUT,SPACE_EFF,ENERG_EFF].v1