How the LLM Live Probe works

A tiny, identical test sent to every model, over and over, so you can see which are up, correct and fast right now.

The test

Each model gets one message: Reply with exactly the word pong. Settings: temperature 0, at most 64 output tokens, streaming on, 30-second timeout, no retries.

We allow 64 output tokens rather than ~5 because many current models "think" briefly before answering, and those hidden reasoning tokens count against the limit. Models normally run with their provider's default reasoning setting. A few get a per-model setting because their default either thinks past the limit or bills a lot of hidden tokens for a one-word reply: google/gemini-3.8-flash ({"effort": "low"}), qwen/qwen3.7-plus ({"enabled": false}), x-ai/grok-4.3 ({"enabled": false}). If a model still spends its whole budget thinking and never says pong, that counts as a wrong answer. That reflects what an agent making a quick, cheap call would actually get.

What counts as correct

We take the reply, trim whitespace, drop one trailing period, and lowercase it. It passes only if what's left is exactly pong. So "Pong." passes; "pong!" or "Sure: pong" fails. Any error (rate limit, timeout, server error, content filter) also counts as a fail.

Schedule

Every 15 minutes, triggered by a Cloudflare cron with a GitHub Actions schedule as backup. All models are probed in the same run, from a GitHub-hosted runner in the US.

Measurements

Score

Score = 60% correctness + 40% speed. Correctness = % of probes in the last 24 hours that passed. Speed = the average of two percentile ranks among the ranked models: median first-token time and median total time (100 = fastest, 0 = slowest; medians use passing probes only). Both parts are shown on the leaderboard so you can sort by either.

Models

Probed through OpenRouter, which routes to the providers:

Direct control checks (same prompt, straight to the provider):

Limits

Excluded runs

Runs we dropped from the stats because our own probe was misconfigured (not the models' fault):

Data