# Build a tiny site-audit crawler with Workers and Queues

> Build a bounded Cloudflare crawler with Workers, Queues, and D1 to find broken pages, missing titles, and redirect chains in a local test site.

Author: [Flavio Copes](https://flaviocopes.com/about/) | Published: 2026-10-30 | Topics: [Cloudflare](https://flaviocopes.com/tags/cloudflare/) | Canonical: https://flaviocopes.com/workers-site-audit-crawler/

A site audit starts with ordinary requests. Fetch a page, inspect its HTML, find links, and repeat. The difficult part is deciding what to fetch next without losing work, following an endless URL pattern, or crawling somewhere you didn't intend.

[OpenSEO](https://flaviocopes.com/openseo/) has a much larger audit engine with durable crawl state. Let's build a small teaching version using Cloudflare Workers, Queues, and D1.

It will report missing titles, HTTP failures, and redirect chains. We'll run everything locally against a deliberately broken fixture site. There are no paid provider calls or remote resource-creation steps.

## Decide what the crawler can reach

The target origin comes from configuration. The HTTP API accepts no target URL. Discovered links and redirect targets must stay on that exact origin, including the port.

Our limits are 20 admitted URLs, depth 2, three followed redirects per URL, eight seconds per fetch sequence, and 256 KiB of HTML. We remove queries and fragments during normalization. That deliberately excludes query-based pagination and treats query variants as the same URL.

The two-minute run deadline prevents a delayed job from beginning another page indefinitely. An audit result can contain errors and still finish its bounded work.

This is a local lab with unauthenticated endpoints. Keep it bound to loopback. A shared service also needs authentication, per-user run limits, an operator-controlled target allowlist, and a crawl policy for the sites you own. Don't expose this local API as a public URL checker.

## Create the project

Make a folder named `tiny-site-audit`, with a `src` directory. Save the following complete files at the indicated paths. You'll need a recent Node.js and npm.

First, `package.json` gives us commands for the fixture, local database, and Worker:

```json
{
  "name": "tiny-site-audit",
  "private": true,
  "type": "module",
  "scripts": {
    "dev": "wrangler dev --ip 127.0.0.1 --port 8787 --test-scheduled",
    "db": "wrangler d1 execute audit --local --file schema.sql",
    "fixture": "node fixture.mjs"
  },
  "devDependencies": { "wrangler": "latest" }
}
```

## Connect the local resources

Save this as `wrangler.jsonc`. The all-zero database ID is a local-development value, not a provisioned cloud database. Wrangler simulates the bindings locally. `workers_dev: false` also avoids accidentally assigning a public workers.dev endpoint if you later change the setup.

The deployed queue is configured for one active consumer batch. Local simulation does not support that concurrency setting, so duplicate deliveries can fetch a URL concurrently. SQL still deduplicates the frontier. Failures get three retries before moving to a dead-letter queue, whose handler records a terminal failure. The one-second retry delay keeps this lab quick.

```jsonc
{
  "name": "tiny-site-audit",
  "main": "src/index.js",
  "compatibility_date": "2026-10-17",
  "compatibility_flags": ["nodejs_compat"],
  "workers_dev": false,
  "vars": { "TARGET_ORIGIN": "http://127.0.0.1:9123" },
  "d1_databases": [{
    "binding": "DB",
    "database_name": "audit",
    "database_id": "00000000-0000-0000-0000-000000000000"
  }],
  "queues": {
    "producers": [{ "binding": "PAGES", "queue": "audit-pages" }],
    "consumers": [
      {
        "queue": "audit-pages",
        "max_batch_size": 1,
        "max_batch_timeout": 1,
        "max_concurrency": 1,
        "max_retries": 3,
        "retry_delay": 1,
        "dead_letter_queue": "audit-dead"
      },
      { "queue": "audit-dead", "max_batch_size": 1, "max_retries": 3 }
    ]
  },
  "triggers": { "crons": ["* * * * *"] },
  "observability": { "enabled": true, "head_sampling_rate": 1 }
}
```

## Keep the frontier in D1

Save this as `schema.sql`. A primary key deduplicates each URL within a run. A row starts as `ready`, becomes `sent` after dispatch, and eventually becomes `done` or `failed`.

`ready` rows also form a durable outbox. Recording a discovered URL and sending its queue message are separate operations. Keeping the unsent work in SQL lets a later dispatch recover from an interruption between them.

```sql
CREATE TABLE IF NOT EXISTS runs (
  id TEXT PRIMARY KEY,
  started INTEGER NOT NULL
);
CREATE TABLE IF NOT EXISTS pages (
  run TEXT NOT NULL REFERENCES runs(id),
  url TEXT NOT NULL,
  depth INTEGER NOT NULL,
  state TEXT NOT NULL DEFAULT 'ready',
  result TEXT,
  PRIMARY KEY (run, url)
);
CREATE INDEX IF NOT EXISTS frontier ON pages(state);
```

## Fetch, inspect, and save a page

Save this complete file as `src/index.js`. `inspect()` follows redirects manually so it can check each destination. It reads the body in chunks and rejects oversized HTML before parsing it with HTMLRewriter.

The consumer inserts discovered children and completes the current page in one [D1 batch](https://developers.cloudflare.com/d1/worker-api/d1-database/#batch). These writes succeed together or roll back together.

```javascript
const MAX_PAGES = 20
const MAX_DEPTH = 2
const MAX_BYTES = 262144

export function normalize(value, base, origin) {
  try {
    const url = new URL(value, base)
    if (url.origin !== origin || url.username || url.password) return null
    if (!['http:', 'https:'].includes(url.protocol)) return null
    url.search = ''
    url.hash = ''
    return url.href
  } catch {
    return null
  }
}

async function inspect(url, origin) {
  const controller = new AbortController()
  const timer = setTimeout(() => controller.abort(), 8000)
  const chain = []
  let reader
  try {
    for (let redirects = 0; ; redirects++) {
      const response = await fetch(url, {
        redirect: 'manual', signal: controller.signal,
        headers: { 'User-Agent': 'TinySiteAudit/1' }
      })
      chain.push({ url, status: response.status })
      if ([301, 302, 303, 307, 308].includes(response.status)) {
        await response.body?.cancel()
        const next = normalize(response.headers.get('location') || '', url, origin)
        if (!next) throw new Error('Redirect leaves configured origin')
        if (redirects === 3) throw new Error('Redirect limit reached')
        url = next
        continue
      }
      const html = (response.headers.get('content-type') || '').includes('text/html')
      if (!html) {
        await response.body?.cancel()
        return { status: response.status, chain, html, links: [] }
      }
      const chunks = []
      let size = 0
      reader = response.body?.getReader()
      if (reader) {
        for (;;) {
          const { done, value } = await reader.read()
          if (done) break
          size += value.byteLength
          if (size > MAX_BYTES) throw new Error('HTML byte limit reached')
          chunks.push(value)
        }
      }
      let title = ''
      const links = new Set()
      const parsed = new HTMLRewriter()
        .on('title', { text(chunk) { title += chunk.text } })
        .on('a[href]', { element(element) {
          const link = normalize(element.getAttribute('href'), url, origin)
          if (link && links.size < 100) links.add(link)
        } })
        .transform(new Response(new Blob(chunks), {
          headers: { 'content-type': 'text/html; charset=utf-8' }
        }))
      await parsed.arrayBuffer()
      return {
        status: response.status, chain, html,
        missingTitle: !title.trim(), links: [...links]
      }
    }
  } catch (error) {
    await reader?.cancel().catch(() => {})
    return { error: error.name === 'AbortError' ? 'Fetch timeout' : error.message, chain, links: [] }
  } finally {
    clearTimeout(timer)
  }
}

const statement = (env, sql, ...args) => env.DB.prepare(sql).bind(...args)

async function dispatch(env) {
  const { results } = await statement(env,
    "SELECT run, url FROM pages WHERE state = 'ready' LIMIT 20").all()
  for (const page of results) {
    await env.PAGES.send(page)
    await statement(env,
      "UPDATE pages SET state = 'sent' WHERE run = ? AND url = ? AND state = 'ready'",
      page.run, page.url).run()
  }
}

async function consume(env, message, dead) {
  const { run, url } = message.body
  const page = await statement(env,
    'SELECT p.*, r.started FROM pages p JOIN runs r ON r.id = p.run WHERE p.run = ? AND p.url = ?',
    run, url).first()
  if (!page || ['done', 'failed'].includes(page.state)) return
  if (dead || Date.now() - page.started > 120000) {
    await statement(env,
      "UPDATE pages SET state = 'failed', result = ? WHERE run = ? AND url = ?",
      JSON.stringify({ error: dead ? 'Queue retries exhausted' : 'Run deadline reached' }),
      run, url).run()
    return
  }
  const origin = new URL(env.TARGET_ORIGIN).origin
  if (normalize(url, origin, origin) !== url) throw new Error('Invalid stored URL')
  const result = await inspect(url, origin)
  const writes = []
  if (page.depth < MAX_DEPTH) {
    for (const link of result.links) {
      writes.push(statement(env,
        `INSERT OR IGNORE INTO pages(run, url, depth)
         SELECT ?, ?, ? WHERE (SELECT COUNT(*) FROM pages WHERE run = ?) < ?`,
        run, link, page.depth + 1, run, MAX_PAGES))
    }
  }
  writes.push(statement(env,
    "UPDATE pages SET state = 'done', result = ? WHERE run = ? AND url = ?",
    JSON.stringify(result), run, url))
  await env.DB.batch(writes)
}

export default {
  async fetch(request, env) {
    const path = new URL(request.url).pathname
    if (request.method === 'POST' && path === '/start') {
      const run = crypto.randomUUID()
      const origin = new URL(env.TARGET_ORIGIN).origin
      await env.DB.batch([
        statement(env, 'INSERT INTO runs VALUES (?, ?)', run, Date.now()),
        statement(env, 'INSERT INTO pages(run, url, depth) VALUES (?, ?, 0)', run, `${origin}/`)
      ])
      await dispatch(env)
      return Response.json({ run }, { status: 202 })
    }
    if (request.method === 'POST' && path === '/dispatch') {
      await dispatch(env)
      return new Response('Dispatched ready rows')
    }
    const match = path.match(/^\/runs\/([a-f0-9-]+)$/)
    if (request.method === 'GET' && match) {
      const { results } = await statement(env,
        'SELECT url, depth, state, result FROM pages WHERE run = ? ORDER BY url', match[1]).all()
      const pending = results.some(p => ['ready', 'sent'].includes(p.state))
      return Response.json({
        state: !results.length ? 'unknown' : pending ? 'running' :
          results.some(p => p.state === 'failed') ? 'failed' : 'finished-within-limits',
        pages: results.map(p => ({ ...p, result: p.result ? JSON.parse(p.result) : null }))
      })
    }
    return new Response('Not found', { status: 404 })
  },
  async queue(batch, env) {
    for (const message of batch.messages) {
      await consume(env, message, batch.queue === 'audit-dead')
      await dispatch(env)
      message.ack()
    }
  },
  async scheduled(event, env) {
    await dispatch(env)
  }
}
```

## Why duplicate delivery doesn't lose the frontier

[Queues can deliver a message more than once](https://developers.cloudflare.com/queues/reference/delivery-guarantees/). A completed row makes a duplicate a no-op. The consumer still tries dispatching any outstanding outbox entries before acknowledging the message.

The dispatcher sends first, then conditionally changes `ready` to `sent`. An interruption after the send can produce a duplicate. Changing the state before sending would create a worse gap: a row could look sent even though no queue message existed.

A fast consumer can complete a row before the dispatcher updates it. Because the conditional update only touches `ready`, it cannot overwrite `done`. Admission is checked inside each SQL insert, so concurrent discoveries still respect the page cap. A unique key prevents duplicate rows; it does not promise exactly-once HTTP fetching. In particular, the local simulator can process duplicate deliveries concurrently.

HTTP errors and fetch limits are audit findings, stored in the result. Infrastructure errors, such as a failed database write, throw and use queue retries. The [dead-letter handler](https://developers.cloudflare.com/queues/configuration/dead-letter-queues/) marks exhausted work as `failed`. If storage itself remains unavailable, the run stays visibly incomplete rather than claiming success.

The scheduled handler flushes the durable outbox. Local cron events don't automatically fire every minute in `wrangler dev`; `/dispatch` provides an explicit local recovery trigger.

## Make a site with known problems

Save this as `fixture.mjs`. It runs only on `127.0.0.1:9123`. The outside redirect points to another loopback port, which the crawler must reject before fetching. The fixture also includes a slow response, a redirect loop, and a link beyond our depth limit.

```javascript
import { createServer } from 'node:http'

createServer((request, response) => {
  const path = new URL(request.url, 'http://127.0.0.1:9123').pathname
  const redirects = { '/old': '/middle', '/middle': '/final', '/loop': '/loop', '/outside': 'http://127.0.0.1:9999/private' }
  if (redirects[path]) {
    response.writeHead(302, { location: redirects[path] }).end()
    return
  }
  response.setHeader('content-type', 'text/html')
  if (path === '/') {
    response.end(`<title>Reading club</title>
      <a href="/empty?sort=new#top">No title</a><a href="/empty">Duplicate</a>
      <a href="/missing">Broken</a><a href="/old">Redirect chain</a>
      <a href="/loop">Loop</a><a href="/outside">Outside origin</a>
      <a href="/big">Too big</a><a href="/final">Final</a>
      <a href="/level1">Depth check</a><a href="/slow">Timeout</a>`)
  } else if (path === '/level1') response.end('<title>Level 1</title><a href="/level2">Next</a>')
  else if (path === '/level2') response.end('<title>Level 2</title><a href="/level3">Too deep</a>')
  else if (path === '/slow') setTimeout(() => response.end('<title>Late</title>'), 10000)
  else if (path === '/empty') response.end('<h1>No title</h1>')
  else if (path === '/final') response.end('<title>Book list</title>')
  else if (path === '/big') response.end('x'.repeat(300000))
  else response.writeHead(404).end('<title>Missing page</title>')
}).listen(9123, '127.0.0.1', () => console.log('Fixture on port 9123'))
```

## Test the queue-driven result

Save this as `test.mjs`. It checks normalization, starts a run through the Worker, then waits for actual queue processing. Assertions inspect the findings instead of relying on a reassuring log message.

```javascript
import assert from 'node:assert/strict'
import { setTimeout } from 'node:timers/promises'
import { normalize } from './src/index.js'

const origin = 'http://127.0.0.1:9123'
assert.equal(normalize('/empty?q=1#part', origin, origin), `${origin}/empty`)
assert.equal(normalize('http://127.0.0.1:9999/private', origin, origin), null)
assert.equal(normalize('http://user@127.0.0.1:9123/', origin, origin), null)
const { run } = await (await fetch('http://127.0.0.1:8787/start', { method: 'POST' })).json()
let report
for (let attempt = 0; attempt < 60; attempt++) {
  report = await (await fetch(`http://127.0.0.1:8787/runs/${run}`)).json()
  if (report.state !== 'running') break
  await setTimeout(1000)
}
assert.equal(report.state, 'finished-within-limits')
assert.equal(report.pages.length, 11)
const page = path => report.pages.find(p => new URL(p.url).pathname === path)?.result
assert.equal(page('/empty').missingTitle, true)
assert.equal(page('/missing').status, 404)
assert.equal(page('/old').chain.length, 3)
assert.equal(page('/loop').error, 'Redirect limit reached')
assert.equal(page('/outside').error, 'Redirect leaves configured origin')
assert.equal(page('/big').error, 'HTML byte limit reached')
assert.equal(page('/slow').error, 'Fetch timeout')
assert.equal(page('/level3'), undefined)
assert.ok(report.pages.every(p => p.depth <= 2))
console.log(`Checked ${report.pages.length} URLs; report: /runs/${run}`)
```

## Run the whole lab

Install the development dependency and initialize only the local database:

```bash
npm install
npm run db
```

Start `npm run fixture` in one terminal. In another terminal, start `npm run dev`. Leave both running. The [Queues local-development guide](https://developers.cloudflare.com/queues/configuration/local-development/) describes the simulated producer and consumer environment.

In a third terminal, run:

```bash
node test.mjs
```

The test prints a report path after checking 11 admitted URLs. Open `http://127.0.0.1:8787` followed by that path, or request it with curl, to inspect the JSON.

`/missing` has status 404. `/empty` has `missingTitle: true`. `/old` records two redirects followed by a successful page. `/loop`, `/outside`, `/big`, and `/slow` produce bounded errors. `/level3` is absent because its discovery would exceed depth 2.

The duplicate links to `/empty` produce one row. The report's `finished-within-limits` state means every admitted URL reached an outcome. It doesn't mean the site has no problems, or that the crawler explored every page on it.

To flush recorded but unsent work locally:

```bash
curl -X POST http://127.0.0.1:8787/dispatch
```

Run it twice; completed pages remain completed. The queue can contain redundant deliveries, while the SQL frontier keeps one result per URL.

## What this small crawler leaves out

This fetches HTML without running page JavaScript. It doesn't execute a browser, measure Lighthouse performance, or check links hidden behind interaction. It also doesn't implement robots.txt policy. Use it on the fixture while learning, and add the site's permitted crawl policy before adapting it to an owned public site.

The page cap counts unique admitted URLs, including the root and failed pages. Redirect requests have their own separate bound. Depth-first versus breadth-first discovery and shortest-path depth are larger frontier decisions; this version preserves the first admitted depth.

For an actual deployed audit, add request pacing and per-origin controls, authenticate run creation, monitor stale runs and the dead-letter queue, and define retention for saved results. A failure should leave something inspectable: which URL was attempted, its response chain, and whether it reached an audit outcome or exhausted execution retries.