How to speed up tests for AI coding agents

A coding agent can edit a function quickly and then spend much longer waiting to learn whether the change works.

If that feedback is slow, every correction is expensive. The agent makes a change, starts the checks, waits, reads an error, and starts again, and a small misunderstanding can travel through several rounds of this. I wrote about putting a retry limit in a state machine around that loop.

Before running more agents or changing models, I would look at that loop. I want a check that answers the question currently blocking the work. Sometimes that’s a single test, sometimes it needs the application and a real database.

Useful feedback starts with a precise question

Suppose an agent is fixing a bug in a task manager web app: archived tasks still show up in search.

We can describe the expected behavior without talking about the implementation. An archived task shouldn’t appear in ordinary search. It should still be in the archive, and restoring it should bring it back to search. I wrote about that habit in What teaching can tell us about prompting.

Those expectations give us a small group of checks, and none of them involve the settings screens.

A test that only checks whether a filtering function was called tells us much less. The function can be called with the wrong value, or another layer can put the archived task back into the response.

So I would start at the narrowest point where the wrong behavior is visible, like the search query or the API response. The test creates the task in its archived state and checks the actual result.

Keep cheap decisions close to the code

Some behavior can be checked without starting the application. If the rule for which tasks match a search lives in one function, we can test it with a few explicit inputs.

Here’s a search.js module:

export function searchTasks(tasks, query) {
  return tasks.filter((task) => !task.archived && task.title.includes(query))
}

And a test for it, search.test.js, using the test runner built into Node.js:

import { test } from 'node:test'
import assert from 'node:assert/strict'
import { searchTasks } from './search.js'

const tasks = [
  { title: 'Write the launch email', archived: true },
  { title: 'Update the home page' },
  { title: 'Write the release notes' },
]

test('search skips archived tasks', () => {
  const titles = searchTasks(tasks, 'Write').map((task) => task.title)
  assert.deepEqual(titles, ['Write the release notes'])
})

test('search matches part of the title', () => {
  const titles = searchTasks(tasks, 'home').map((task) => task.title)
  assert.deepEqual(titles, ['Update the home page'])
})

Notice that the test imports the real function. Copying the filter expression into the test would give us two copies of the same idea, and the application could change while the copied test keeps passing.

If reaching the behavior means constructing the whole application, I would look at the dependencies before adding another abstraction. Sometimes extracting a small decision helps, and sometimes a focused integration test is already the simpler answer. Either way, when the archive rule changes, the failing check should point at the rule.

Test the boundary that could be wrong

A unit test can’t tell us whether a database query filters correctly if the database is replaced with a fake that already returns the expected records.

This is an easy mistake when you ask an agent to add tests. It builds an elaborate mock, calls the handler, and proves that the handler returns the mock’s answer.

For the archived task bug, I would want at least one check that runs the real query against a test database: create visible and archived tasks, run the query, and look at the returned IDs. That check doesn’t need a browser or a real user account if neither is relevant to the query.

Other bugs need other boundaries. A disabled button needs a browser. A permission check needs the server’s actual authorization path. An export bug may need you to open the generated file.

Give the agent a focused command

A repository should make its test entry points easy to find. If the only documented command starts every service and runs every check, that’s what an agent will use. It may also guess a command that does less than its name suggests.

I would document how to run one relevant file, how to check a larger area, and which checks are required before finishing. Those commands can live in AGENTS.md or the README, next to the development instructions.

With the Node.js test runner, node --test finds and runs every test file in the project:

node --test --test-reporter=spec

✔ search skips archived tasks (1.027208ms)
✔ search matches part of the title (0.057959ms)
ℹ tests 2
ℹ suites 0
ℹ pass 2
ℹ fail 0
ℹ cancelled 0
ℹ skipped 0
ℹ todo 0
ℹ duration_ms 63.446333

Pass a file and --test-name-pattern to run only the tests whose name matches:

node --test --test-reporter=spec --test-name-pattern="archived" search.test.js

✔ search skips archived tasks (1.009542ms)
ℹ tests 1
ℹ suites 0
ℹ pass 1
ℹ fail 0
ℹ cancelled 0
ℹ skipped 0
ℹ todo 0
ℹ duration_ms 63.082666

With two tests in one file the difference is nothing, but the same command shape keeps a run small in a project with hundreds of test files.

In a Vitest project, where tests import from vitest, Vitest’s filtering guide shows the same filters by file name, by line, and by test name:

npx vitest run search
npx vitest run search.test.js:11
npx vitest run -t "archived"

The guide also points out that with a name filter alone, Vitest still has to run each test file to discover which tests match. Including the file avoids that work.

I would ask the agent to report what it selected. “Tests passed” is ambiguous when it means one file from a much larger suite.

Focused checks and release checks serve different moments

During the repair, I want fast feedback on the behavior being changed. Before accepting it, I want to check the surrounding assumptions. The archive fix can affect archive counts, restore behavior and pagination, and a test that proves the original bug is gone doesn’t cover those.

So the checks widen gradually. Reproduce the bug, fix it, run the related area, then run the project’s required checks.

How far to go depends on the change. Updating a label needs different evidence from replacing the search query, and a small text edit doesn’t justify a new test suite.

I would have the agent explain its selection in terms of risk. “Search and archive share the query, so I checked both” helps me assess the work. A list of commands without their scope doesn’t.

If another change lands before integration, the checks have to run on the combined code. An earlier passing run doesn’t cover later edits.

Work through one repair from failure to acceptance

Let’s make the example more specific. Say search shows one result per page, and the test database has three tasks matching “Write”, newest first:

The first focused check asks the real search query for its first page. We expect the release notes. The broken query returns the archived launch email, which reproduces the bug.

An agent could “fix” that by removing archived tasks from the response after fetching the first page. Now the archived task disappears, but the page is empty. The visible tasks are still in the database, beyond the slice the query fetched.

The acceptance case has to say what the user should get. With a page size of one, search should return the release notes first, then the pricing page copy. If the interface shows a total, it should count two matches. Those expectations catch the incomplete repair without a browser.

While the agent edits, it keeps rerunning this focused group against the query. Once it passes, I’d widen the checks to the archive query and restoring. The launch email should still be in the archive, and restoring it should put it back in search where the app’s existing sort order says it belongs.

Then the browser flow that connects these operations: archive a result, search again, open the archive, restore it. This checks whether the interface sends the right request and shows the changed results, which a query test can’t tell you.

Finally, run the project’s required checks on the finished change. The report should keep the query tests, the browser flow and the broader checks apart. If the browser flow failed, the passing query test is still useful evidence, but the repair isn’t done.

The dataset is small on purpose, so every expected result is easy to explain. The time saved comes from catching the wrong repair early, before running the rest of the application around it again and again.

Measure the waiting before optimizing it

A slow test run can spend its time in unexpected places. The tests themselves can be quick while every run rebuilds the application. Setup may download dependencies, or a failed network request may wait for a timeout before the useful assertion starts.

The two runs above both report about 63 milliseconds in total, while each test takes one millisecond or less. For a tiny file, almost all the time is starting the runner.

I would look at the runner’s timing output and separate setup from execution. Then I’d fix the slowest unnecessary step and measure again under the same conditions.

Look at cold and warm runs separately. A cache can make a repeated local run feel fast while every clean CI job pays the full setup cost. So when you report a timing, say which kind of run it was.

Faster tests help a feedback loop that repeats many times. The whole task can still be held up by review or unclear requirements.

Parallel tests need separate state

Running checks concurrently helps when they’re independent and the machine has capacity. It can also expose hidden dependencies. Two tests use the same account, one deletes a task while another expects it to exist. Each passes alone, and together they fail now and then.

For browser tests, Playwright provides isolated browser contexts, so each test gets separate cookies and browser storage. The backend still needs attention, because separate browser sessions can edit the same database row.

I would use unique test records, or isolated databases where it makes sense. Each test creates the state it needs instead of relying on a previous test to leave it behind.

This matters for agents in particular. A test that only passes in a certain order is hard to reproduce from a single failure, and the agent ends up retrying when the fix is to remove the shared state.

Sharding helps after the suite can run independently

A larger suite can be split across machines. This is called sharding, and Playwright’s configuration supports running a selected shard of the suite.

How much it helps depends on how the work is divided and how much setup each job repeats. If one group has a long end-to-end flow and the others finish quickly, the whole run still waits for that group. If every job spends most of its time booting the same expensive environment, more jobs multiply that cost.

I would look at the distribution before increasing the shard count. For a small application, a single machine with a focused test command is fine. I’d add distributed runs when a measured bottleneck justifies maintaining them.

Browser tests should act on what the user sees

A browser test gets fragile when it depends on the accidental shape of the HTML. If it finds the archive button through a long chain of CSS classes, a harmless layout change breaks it, and the agent spends time fixing a test while the user flow still works.

Playwright’s best-practice guide recommends testing user-visible behavior, with locators based on roles and accessible names.

So I would open the task list, find the archive control by its accessible name, click it, and check the resulting list. An assertion that a component exists skips the action we care about.

Headless runs are convenient for repeatable checks. Changes to visual layout, keyboard interaction, or the feel of an editing workflow still need someone to use the app.

Make failures explain what happened

An agent works better with a failure it can investigate. “Expected true, received false” forces it to reconstruct the scenario.

Here’s the start of what the focused run prints when the !task.archived check is missing from searchTasks():

✖ search skips archived tasks (1.922167ms)
ℹ tests 1
ℹ suites 0
ℹ pass 0
ℹ fail 1
ℹ cancelled 0
ℹ skipped 0
ℹ todo 0
ℹ duration_ms 64.803709

✖ failing tests:

test at search.test.js:11:1
✖ search skips archived tasks (1.922167ms)
  AssertionError [ERR_ASSERTION]: Expected values to be strictly deep-equal:
  + actual - expected

    [
  +   'Write the launch email',
      'Write the release notes'
    ]

The test name says which rule broke, and the diff names the task that shouldn’t be there. That’s because the test compares the list of titles with deepEqual, instead of checking a boolean.

For browser failures, keep the evidence. A trace or screenshot shows whether the control was missing, a request failed, or the app reached a different page. Logs should describe the failure without exposing test credentials.

I would also tell a test failure apart from an environment failure. A database that never started tells us nothing about archive behavior, and the runner should make that difference visible.

If a check is flaky, passing a few times in a row doesn’t explain the earlier failure. Investigate it or say clearly that it’s uncertain. Quietly adding retries makes a green result harder to trust.

Keep the expected behavior independent

An agent can write an implementation and a test that agree on the wrong behavior.

Say it decides archived tasks should be deleted, then writes a test proving they disappear from search. Both changes agree with each other, and both break the requirement that you can restore a task later.

I want the expected behavior settled before I look at the implementation. A short description of the user’s action and result gives the review something independent to compare against.

For a bug fix, I’d check that the regression test fails on the old code and passes after the fix, like the failing run above. A test added after the change that never caught the bug is weaker evidence.

For a low-risk change, I’d also ask whether a new test is worth maintaining. Existing checks and a direct look can be enough. Generating tests is cheap, but carrying irrelevant ones through every future change isn’t.

Find the slowest useful check

I would take the last change where an agent spent a lot of time waiting and look at what happened. Which check was it waiting for? What did that check tell it? Could a smaller check have answered the immediate question? Did the final verification still cover the surrounding behavior?

Then I’d improve that one path: document the focused command, remove unnecessary setup, make the failure easier to read. The free Testing JavaScript Applications course covers the underlying choices across functions, databases, browsers, and CI.

Once the path works, the agent uses it on the next change, and I’d measure that before deciding whether the project needs more test machinery.

Want me to talk about your product? You can sponsor this site.