AI inference benchmarks:real rigs, real numbers.

Featured leaderboards

Frequently asked questions

How is tokens-per-second measured in these benchmarks?

Every result is community-submitted. The convention is decode throughput: tokens generated divided by (completion time minus time-to-first-token), averaged over the full generation. That isolates decode speed from prefill — using total wall-clock time instead inflates the number on long prompts, because it folds the one-time prefill cost into the per-token rate.

What is TTFT (time to first token)?

TTFT is the latency from request sent to the first output token — the prefill phase. It is reported separately from decode tokens-per-second because they measure different things. Below roughly 500ms feels interactive; TTFT grows with prompt length and with batch size under load.

Which inference runtimes and hardware can I benchmark?

The submission form covers the common local-inference runtimes — vLLM, Ollama, llama.cpp, LM Studio, SGLang, TensorRT-LLM, and MLX — across NVIDIA, AMD, Apple Silicon, and Intel. Quantizations include GGUF, AWQ, and GPTQ, and multi-GPU plus Mixture-of-Experts configurations are supported with their own fields (GPU count, tensor-parallel split, active vs total parameters).

How do I post my own benchmark result?

Paste your llama-bench or nvidia-smi output and the form auto-fills the speed and VRAM fields; then add your model, quantization, and hardware and publish. Posting and commenting need a free account — reading every result is fully public, no sign-up required.

What makes two benchmark results directly comparable?

Four things must match: the model (family, parameter count, and instruct-vs-base variant), the GPU SKU (an RTX 3090 and a 3090 Ti are different rows), the quantization bit-width and family (GGUF Q4_K_M is not interchangeable with AWQ-int4 or GPTQ-int4), and the inference runtime. Differ on any one of those and the numbers describe different configurations, not a head-to-head.