• Contact
  • Register
    (If you're a human, don't change the following field)
    Your first name.
    (If you're a human, don't change the following field)
    Your first name.

    Enter your email above, then click "Sign Up" to join the STAC mail list and (optionally) register to access materials on the site. Click for terms.

STAC-AI™ Test Harness

This repository is used in STAC-AI™ LANG6 (Inference-Isolated) benchmark tests and audits. It contains everything needed to run LLM Inference Benchmarks using 2 popular open-source model-serving frameworks including:

  • The STAC-AI™ LANG6 (Inference-Isolated) Benchmark Specifications Workbook (Excel)
  • Benchmark Documentation (PDF)
  • Test Harness automation scripts (Python and make)
  • Results analysis and visualization (iPython notebooks)
  • Reference implementations for Hugging Face Text Generation Inference (TGI) and vLLM (Python, docker)

The Test Harness is accompanied by an additional repository containing sets of test data. Custom testing will also require a STAC Pack: An implementation developed and optimized for a given hardware and/or software architecture.

The reference implementations are designed to download open-source models on the first use of the model, or to use local models. Official tests must use model versions curated by STAC; Contact STAC for details.

If you have been previously permissioned to this software, please log in using your corporate email at
https://stacresearch.beanstalkapp.com/. Otherwise, please click the link below to request access.

Please log in to see file attachments. If you are not registered, you may register for no charge.