What are evals? Claude Code, Codex, Cursor and Grok
By Flavio
An eval is a fixed prompt, a real agent run, and a check you can repeat. How Claude Code, Codex, Cursor and the Grok API score that run.
An eval is a test for a model or an agent. You keep the prompt fixed, you run the agent, and you score the result against a check you can repeat. The score is what lets you see a regression when the prompt, the plugin, or the model changes.
A benchmark leaderboard is one kind of eval, run by someone else on a public suite. How AI models are measured is that topic. This entry is the smaller kind you run on your own task, inside or beside the coding tools.
How evals work
One run of an agent is a noisy sample. The useful pattern is a case you can run again: a prompt a real user would type, a fresh session, and one or more graders. A grader can be a regex on the reply, a check that a tool was called, a file the agent was supposed to write, or a second model judging the reply against a rubric.
You save the score. The next change is a comparison, not a feeling.
Evals in each tool
Claude Code ships a plugin eval command. Codex and Cursor give you a way to run the agent from a script and score it yourself. Grok Build does not document an eval command. xAI publishes a cookbook that grades the API.
| Claude Code | Codex | Cursor | Grok Build | |
|---|---|---|---|---|
| Built-in eval command | claude plugin eval |
Not documented | Not a CLI slash command | Not documented |
| What it runs | Your plugin, against cases in evals/ |
codex exec, which you can score from JSON events |
The Cursor SDK, Agent.prompt |
Not the TUI. An API cookbook grades grok-4.7 |
| A baseline | The same cases with the plugin unloaded | You write the comparison | You score the working tree after the run | The cookbook compares reasoning efforts |
Claude Code
claude plugin eval runs a plugin against test cases and scores them. Each case is a prompt plus one or more graders. Graders can be a regex on the reply, a check that a tool was called, or a rubric a second model judges. claude plugin eval init asks what a good result looks like, proposes cases, tries them, and writes the files. You can also write the files yourself.
A suite lives in evals/ inside the plugin. Each case is a directory with a prompt.md, a case.yaml, or both, and at least one grader. From the plugin root:
claude plugin eval .Each case runs three times by default with the plugin loaded, and three times without it. A case’s score is the mean of its runs. It passes when the score meets --threshold, which defaults to 1.0. The summary table shows WITH, W/OUT, and the difference. A high score on both sides means the plugin was not what made the case pass.
Every run, and every judge grader, is a real model call on your account. It counts against the plan or the API bill. The cost column is a list-price estimate. The plugin evals page lists a minimum Claude Code version. Check claude --version against that page before you rely on the command.
claude plugin validate is a different command. It checks the plugin’s files for syntax and schema errors. It does not run the agent.
OpenAI Codex
Codex does not document a plugin-eval command. The way to score it from a script is codex exec. By default, progress goes to stderr and the final message goes to stdout. Add --json and stdout becomes a JSON Lines stream. Event types include thread.started, turn.started, turn.completed, turn.failed, item.*, and error. Item types include agent messages, command executions, and file changes. turn.completed carries a usage object.
--output-schema points at a JSON Schema file for the final response shape. Codex validates tool output against it. That is a structured-output check, not a full grader suite.
codex exec starts in a read-only sandbox unless you set one. The non-interactive page says --full-auto is no longer supported. For edits inside the workspace, the documented replacement is --sandbox workspace-write.
See non-interactive mode and codex exec.
Cursor
Cursor’s evals page is about running the agent inside your own harness, through the Cursor SDK. Agent.prompt creates an agent, sends one prompt, waits until the run finishes, and disposes. The page calls that the primitive for a stateless eval task. After the run, you score the working tree with a test runner, a judge, or an exact match. The sample model id is grok-4.7.
SDK runs follow the same pricing, request pools, and Privacy Mode rules as the IDE and Cloud Agents. Eval traffic is tagged so it shows up under the SDK label in the team usage dashboard. The page says default API rate limits are for development, not a full sweep, and it gives an email address for higher limits on a public leaderboard or a large internal eval.
There is no /eval in the CLI slash-command reference.
Grok Build
The Grok Build command list has no eval command. xAI’s cookbook has a separate example, Reasoning Effort Evals, that grades the API. You give it a prompt and test cases. It runs each case through grok-4.7 at low, medium, high, and xhigh, grades the answers, and compares accuracy with cost. Code checks grade the fields that have a right answer. An LLM judge grades the free text and does not see which effort wrote the answer.
That cookbook is not a grok eval command. It is a small app you run against the API, with XAI_API_KEY. The reasoning effort entry is the setting those four levels refer to.
Details checked against each vendor’s documentation in November 2026.
A real example
You wrote a skill that should fire when someone asks how a newsletter signup validates an email. Claude Code’s eval command is the one that compares the plugin with a bare session. From the plugin root, after the cases exist:
claude plugin eval . --case email-field --runs 1 --ablation none--runs 1 and --ablation none are the documented way to try one arm once while you edit the skill description. The docs say a single run is noisy. Confirm the change at the default three runs, with the no-plugin arm, before you trust the delta.
On Codex, the matching building block is a scripted run whose events you can grade:
codex exec --json "Add a test that rejects an empty email in the newsletter form" --output-last-message /tmp/newsletter-eval.txt--json is what makes the tool calls visible to the script. The final message file is the reply. Your grader still has to decide whether the test file is the one you asked for. Cursor’s SDK page uses the same split: the SDK returns the run, and your harness scores the tree.
When to use evals
Use them when you maintain a skill or a plugin and you want to know if a wording change stopped the agent from picking it. Claude Code’s with-and-without table is built for that. Use them when you are choosing a reasoning effort and you have cases with answers you can check. The xAI cookbook is that comparison.
Use a handful of cases, not a leaderboard, when the behavior is specific to your repo. The free lesson Build a small evaluation starts at about 20 examples with the expected properties written down.
Skip a model judge when a command already answers the question. A test suite that must pass is a grader. Goal mode can keep an agent working until that command succeeds. An eval is the offline version: you run the case on purpose, and you keep the score. Give coding agents checks they can learn from is about putting those checks in the repo so the agent hits them during the task.
Pitfalls
A perfect score with the plugin unloaded means the plugin didn’t do the work. Claude Code prints that as WITH and W/OUT. Reading only WITH hides it.
One run is a weak sample. Claude Code’s default is three runs each arm, and the docs tell you not to trust a one-run delta. The same caution applies to a single codex exec or a single Agent.prompt.
Evals spend the same allowance as a normal session. Claude Code says so directly. Cursor says SDK eval traffic shows up in the team usage dashboard under the SDK label. A suite with a judge model spends a second round of tokens on top of the agent run.
codex exec will not edit files until you set a sandbox that allows it. A grader that expects a new test file needs workspace-write, in a checkout you can throw away. --full-auto is the old flag. The current page says it is no longer supported.
The cookbook’s judge is only as fair as the setup it describes: the judge doesn’t see which effort wrote the answer, and a pairwise comparison is shown in both orders. A judge that sees the label, or that you wrote to prefer one model, is a different test.
Related on flaviocopes.com
- How AI models are measured and compared is the public-benchmark side, including what a leaderboard does and doesn’t tell you.
- Which AI model is best for coding? is a hands-on bake-off on one small project, which is an eval with the cases visible.
- Give coding agents checks they can learn from puts the check in the repo so a later task hits it without a separate harness.
- Build a small evaluation is the short lesson on stable inputs and recorded failures.
Related glossary entries
- Goal mode: Goal mode keeps a coding agent working until a condition holds. How /goal works in Claude Code, Codex, the Cursor CLI and Grok Build.
- Reasoning effort: Reasoning effort sets how long a model thinks before it answers. The levels in Claude Code and Codex, what Cursor's plans fix, and how Grok sets effort.
- Spec-driven development: Spec-driven development writes the behavior and the check before an agent edits. How plan mode and goals support that in these four tools.
Want me to talk about your product? You can sponsor this site.