How we test celeris-1: speed and quality on the same footing

celeris-1 is built to be fast and accurate at the same time, not one at the cost of the other. To show that honestly we ran it against GPT-5, GPT-5 mini, Google’s Gemini 2.5 Flash, and Amazon Nova Micro on two public benchmarks — MMLU-Pro for reasoning, and a suite of short JSON tasks that mirror most production traffic — through one harness with identical prompts, scoring, and datasets. Every model’s quality and latency come from the same run.

What we compared

Four models:

• celeris-1
• GPT-5 (reasoning off)
• GPT-5 mini (reasoning off)
• Gemini 2.5 Flash (thinking off, MMLU-Pro only)
• Amazon Nova Micro

GPT-5, GPT-5 mini, and Gemini 2.5 Flash are reasoning models. Reasoning mode intentionally trades latency for accuracy, and applications that require interactive latency typically disable it — so we compare against the deployed low-latency configuration, with reasoning / thinking at minimal. That same setting produces both the accuracy and the latency numbers; every number for a model comes from one run.

MMLU-Pro: reasoning under load

420 questions · 5-shot chain-of-thought · strict scoring

MMLU-Pro is a harder MMLU: ten options per question instead of four, across fourteen subjects, weighted toward reasoning over recall. Guessing scores ~10%.

We ran a fixed 420-question set — 30 per category — five-shot with chain-of-thought exemplars, scored strictly: the model has to state its answer as the answer is (X) and match the gold letter. No partial credit.

Up and to the left wins — more accurate, faster. celeris-1 is alone there at 81.7% and 116 ms. GPT-5 (reasoning off) is close on accuracy at 80.2% but takes 2.3 seconds, about 20× longer. Gemini 2.5 Flash reaches 75.2% and GPT-5 mini 73.8%, both around 2.5–2.8 seconds; Nova Micro trails at 54.5%.

celeris-1 was not trained or finetuned on MMLU-Pro. None of the benchmark’s questions or answers were in its training data — the 81.7% is general reasoning, not a memorized test.

Simple workflows: the jobs LLMs actually do

10 JSON tasks · valid-JSON-only · 3 runs each

Most production calls aren’t reasoning gauntlets. They’re short structured jobs — classify a ticket, extract fields from an invoice, route a request — with one right answer.

We built ten such tasks: four classification, three extraction, and one each of normalization, ranking, and routing. Each requires valid JSON only, scored on exact match / field accuracy, run three times per model (median reported).

All four models in the workflow suite score 100% — perfect JSON and exact-match across the suite — so this is purely a speed test. (Gemini 2.5 Flash was benchmarked on MMLU-Pro only.)

celeris-1 answers in 23.5 ms; Nova Micro 412 ms, GPT-5 mini 840 ms, GPT-5 1,065 ms — 18× to 45× slower for the same output. Across a pipeline firing thousands of calls, that is the difference between instant and stalled.

We didn’t count the network

How much of each latency is the model, and how much is the network round trip? We removed the question: every latency we report for GPT-5, GPT-5 mini, and Nova Micro is the provider’s own server-side response time (openai-processing-ms, Bedrock latencyMs). No network is counted against them — the figures are model compute only, the same thing celeris-1’s number measures.

We also logged the full end-to-end time. The network added only 60–150 ms per call and is excluded from every chart above. Moving the client into the same datacenter would save milliseconds, not seconds — the gap is the model.

What it adds up to

Optimizing for speed and intelligence together is the whole point. celeris-1 matches or beats the fast configurations of GPT-5, GPT-5 mini, Gemini 2.5 Flash, and Nova Micro on reasoning, ties them at 100% on real-world tasks, and runs 18–45× faster on model time. That puts a frontier-level answer inside the hot path — every keystroke, every turn of a conversation, every step of an agent loop.

Methodology notes

• MMLU-Pro: TIGER-Lab/MMLU-Pro, a fixed 420-question block (30 per category × 14 categories), five-shot chain-of-thought, strict the answer is (X) scoring against the gold label.
• Workflow suite: ten JSON tasks (classification, extraction, normalization, ranking, routing), scored for schema validity plus exact match / field accuracy; three measured runs per task, median reported.
• Configurations: GPT-5 and GPT-5 mini run with reasoning set to minimal. Nova Micro via Amazon Bedrock. Latency is single-request (no batching), reported as the median (p50).
• Latency attribution: every reported latency is the provider’s own server-side response time (openai-processing-ms, Bedrock latencyMs), so the network round trip is excluded entirely. The full end-to-end time is shown alongside only for transparency.
• No contamination: celeris-1 was not trained or finetuned on MMLU-Pro. Its questions and answers were never in the training data; the score reflects general reasoning, not memorization of the benchmark.
• Scoring note: Nova Micro states its answers in LaTeX (\boxed{X}) rather than the exact template, so its answers were extracted with a lenient parser before scoring; without it, the same answers would be undercounted as format errors.