About TokenDyno

A dynamometer for LLMs — continuous, independent measurement of real inference speed, set against vendor claims. No manual runs, no cherry-picked results — just a worker that fires every ~10 minutes and writes what it finds.

Why it exists

Vendor TPS claims are unreliable. They are measured under conditions the vendor controls — ideal prompts, empty queues, sometimes first-token time folded into the rate — and they almost never reflect what a real user sees on a real request. TokenDyno exists to measure the thing that actually matters to a consumer of these APIs: how fast does this model generate text for me, right now, on the plan I would buy?

Independent, continuous measurement is the only way to get that number. One-off benchmarks go stale; vendor dashboards advertise peaks. A worker that hits the same endpoint every ~10 minutes produces a living record that tracks outages, throttling, and quiet regressions vendors never announce.

How it works

A long-running worker benchmarks every model on a ~10-minute cycle across three providers, using the same prompt and the same hybrid measurement method for every run. Results are stored in Postgres and served via a small Hono API. The site is a static Astro build that hydrates chart islands client-side. See the methodology page for the full measurement spec.

Because every provider is benchmarked on the same engine with the same method, the numbers are directly comparable — you can put an Ollama model next to an OpenCode Zen model next to an OpenCode Go model and the comparison is fair.

Multi-provider, same yardstick

TokenDyno benchmarks three providers side by side:

  • Ollama — hosted API
  • OpenCode Zen — pay-per-use API (Zen endpoint)
  • OpenCode Go — monthly subscription plan (Go endpoint)

Same hardware, same prompt, same measurement method, same ~10-minute cadence. The only thing that varies is the provider.

Who runs this

This site is built and maintained by Anton Gulin. The engine (worker, API, and all sites) is a personal project. Infrastructure costs come out of pocket, which is why donations and referrals via the affiliate links on model pages are appreciated.

Support this project

If you find it useful, the best thing you can do is share it. Financial support via GitHub Sponsors helps keep the API subscriptions running (and therefore keeps the data fresh).

Sponsor

Contact

Bugs, methodology questions, or sponsorship enquiries: i@anton.qa