STAC-ML Working Group

The STAC-ML Working Group develops and promotes benchmark standards for key machine learning (ML) workloads in finance (in which we include "AI"). This enables customers, vendors, and STAC to make apples-to-apples comparisons of techniques and technologies.

Much of the group's current attention is on large language models (LLMs), which have potentially large top- and bottom-line impacts for financial firms. Whether providing fast insights on financial documents, making it easier for business people to analyze the vast stores of structured data already in financial institutions, or simply improving a firm's understanding of its internal operations, LLMs have opened up entirely new possibilities.

But LLMs aren't the only type of ML or AI important to institutions like banks, insurers, asset managers, and hedge funds. Deep learning and statistical ML are used in a variety of workflows, such as detecting fraud, asssessing customer risk, and interacting with clients. In the hyper-competitive world of trading and investment, where the execution of models is already largely automated, firms are now using ML to reduce time to market for new algorithms, improve model quality, or reduce costs.

Standardized, customer-driven benchmarks help with technology discovery and assessment. This is especially important for the AI stack, which is both blessed and cursed by an enormous range of options, including ML algorithms, frameworks, libraries, and processor architectures--not to mention the "lower stack" technologies such as server, storage, and networking hardware and the software to make them work well for AI. The options are further increased by machine-learning-as-a-service offerings from all the major cloud providers, as well as countless software and software-as-a-service providers promising to simplify, accelerate, or otherwise enhance ML workflows.

The first ML workload for which the working group has developed benchmarks is real-time inference of signals from event-driven financial data: STAC-ML (Markets) Inference. To learn about these benchmarks, select one of the reports below.

To influence the group's ongoing work to create business-relevant ML benchmark standards, please click on the "Enable me!" button to the right. Or if you have other thoughts on ML or AI, please contact us.