Introducing STAC-ML Markets (Inference)

A new benchmark for machine learning inference

10 May 2022

The STAC Benchmark Council has finalized STAC-ML™ Markets (Inference), a new technology benchmark standard for solutions that can run machine-learning inference on realtime financial data for uses such as low-latency trading or rapid backtesting of models.

The specifications, as well as the accompanying test harness, reference implementation, analytics toolkit, and documentation, are in the hands of several tech vendors who are preparing to release STAC-ML results soon. The materials are also available now to financial firms so that they can obtain the same kinds of performance insights on their inhouse systems and “mark those systems to market” by comparing with the results of vendor-optimized solutions.

Like other STAC Benchmarks in areas like time-series analytics, risk simulation, and trading latency, STAC-ML was developed by quants and technologists at some of the world's leading financial firms. Technology vendors also provide important input, but STAC Benchmarks are ultimately in the hands of those who must deliver business value from technology in the real world.

As an example of STAC-ML, STAC has released results for two different stacks under test (SUTs). Both used the same hardware and software (including the ONNX runtime), but one was designed for latency while the other was designed for throughput.

The results are not the last word on the performance that could be achieved from the SUT components (the implementation code was not hardware-optimized, and the SUTs were not tuned below the application layer). Rather, these STAC Reports illustrate the kinds of insights STAC-ML benchmarks can provide, while underscoring the sensitivity of performance results to the objectives of the solution architect.

Detailed performance and quality analyses and configuration information are available to premium subscribers, along with a research note that walks through the lessons learned from optimizing this technology stack for the two performance objectives.

To learn more about how you can access the benchmark specs, tools, detailed information, and research note, please contact us.