Back

Tenstorrent’s Galaxy Blackhole reaches industry-leading 5,300 tokens/sec/user decode speed on gpt-oss-120b

OpenAI's gpt-oss-120b running at up to 5,300 tokens/sec/user on eight Tenstorrent Galaxy systems.

Nov 16, 2026

•
Share

The model and the hardware it runs on

gpt-oss-120b is an open-weight language model from OpenAI. It uses a Mixture-of-Experts (MoE) architecture, routing each token through just 4 of its 128 experts rather than activating the entire model. This allows it to use approximately 5.1 billion of its 117 billion parameters per token, improving computational efficiency.

Tenstorrent Galaxy packs 32 Blackhole chips into a single 6U server with 6.2 GB of on-chip SRAM and 1 TB of GDDR6 DRAM. Galaxy connects over standard Ethernet, allowing systems to scale from a single Galaxy to tens of machines.

Tenstorrent's architecture is built around efficient data placement and data movement. High-bandwidth on-chip SRAM keeps data close to compute, while GDDR6 provides the capacity for larger models. During token generation, moving model weights between memory and compute can become a major performance bottleneck. This project focuses on minimizing that bottleneck to accelerate inference and deliver tokens to users faster.

How gpt-oss-120b runs on an eight-Galaxy cluster

Every layer is split across a 4×4 mesh of 16 Blackhole chips. Expert weights on the critical path are stored in on-chip SRAM in BF4, our block floating-point format, while the remaining weights are streamed from DRAM.

Next, the pipeline is recycled. Unlike wafer-scale designs that dedicate a pipeline stage to each layer, we reuse stages: the full weights live in DRAM, and each stage loads the next layer's weights into SRAM while it computes the current one. That gives us the per-user speed of a 24-Galaxy deployment on just 8.

Between stages, chips pass activations over direct, low-latency Ethernet links. The pipeline is laid out to follow the system's physical cabling. Together, these pieces make a system that scales: the same design runs on anything from a single Galaxy to tens of machines, so you can choose when to trade cost against performance.

5,300 tokens/sec/user at peak, 3,000 on average

GSM8K (Grade School Math 8K) is a set of about 8,500 grade-school math word problems from OpenAI. Each takes several steps of reasoning to reach a single numeric answer. It's one of the most common public benchmarks for language models, and labs routinely report it in model cards.

On the full GSM8K dataset, our deployment of eight Galaxy systems (256 Blackhole chips) averaged 3,000 tokens/sec/user and scored 95.8%, with speed peaking at up to 5,300 tokens/sec/user. These figures cover decode only, using GSM8K's short prompts, and don't include time to first token

Tenstorrent Galaxy runs gpt-oss-120b fast for each user, keeps the model accurate, and scales from one system to tens of machines. If you're planning inference capacity for low-latency workloads, talk to our team about how Galaxy fits your deployment, or explore Galaxy.