STAC Big Data SIG

Big data has become a big topic at STAC. All it takes to confirm this is a quick glance at discussions and presentations at STAC Summits over the past few years (particularly NY and London).

The STAC Big Data Special Interest Group (a group of end-user organizations and providers of big data solutions) defines "big data" as a workload that is difficult or expensive to handle using traditional technologies, largely due to data scale or complexity. Banks, exchanges, hedge funds, and other financial institutions are turning to non-traditional technologies such as Hadoop, Spark, NoSQL, NewSQL, graph databases, and in-memory databases to handle these workloads, as well as new compute, storage, and network technologies.

STAC published a white paper in May 2014 on some of the big data use cases in banking and securities. That paper was a byproduct of an initiative in the STAC Big Data SIG to develop technology-agnostic benchmarks based on big data workloads. As we say in the white paper:

Few areas cry out for good technology benchmarks more than the Wild West of big data. Solution designers face dozens of new software and hardware products. They must understand which products and design patterns are suited to which use cases. And they must determine whether these products deliver not only the transformative capabilities that they promise, but also the boring‐yet‐critical functionality taken for granted in traditional architectures. The need for rigorous big data benchmark standards is especially strong in finance, where the opportunity to turn information into money is huge, but the cost, quality, and security constraints grow stronger by the day.

The good news is that a set of big data benchmark proposals is now in development (accessible to SIG members). Those should be runnable before long, at which point the SIG will establish a research agenda to address shared interests. Along the way, the group will continue to discuss new technologies with potential to solve big data challenges.

See this FAQ for eligibility requirements and other details.