Build a tiny site-audit crawler with Workers and Queues

A site audit starts with ordinary requests. Fetch a page, inspect its HTML, find links, and repeat. The difficult part is deciding what to fetch next without losing work, following an endless URL pattern, or crawling somewhere you didn’t intend.

OpenSEO has a much larger audit engine with durable crawl state. Let’s build a small teaching version using Cloudflare Workers, Queues, and D1.

It will report missing titles, HTTP failures, and redirect chains. We’ll run everything locally against a deliberately broken fixture site. There are no paid provider calls or remote resource-creation steps.

Decide what the crawler can reach

The target origin comes from configuration. The HTTP API accepts no target URL. Discovered links and redirect targets must stay on that exact origin, including the port.

Our limits are 20 admitted URLs, depth 2, three followed redirects per URL, eight seconds per fetch sequence, and 256 KiB of HTML. We remove queries and fragments during normalization. That deliberately excludes query-based pagination and treats query variants as the same URL.

The two-minute run deadline prevents a delayed job from beginning another page indefinitely. An audit result can contain errors and still finish its bounded work.

This is a local lab with unauthenticated endpoints. Keep it bound to loopback. A shared service also needs authentication, per-user run limits, an operator-controlled target allowlist, and a crawl policy for the sites you own. Don’t expose this local API as a public URL checker.

Create the project

Make a folder named tiny-site-audit, with a src directory. Save the following complete files at the indicated paths. You’ll need a recent Node.js and npm.

First, package.json gives us commands for the fixture, local database, and Worker:

{
  "name": "tiny-site-audit",
  "private": true,
  "type": "module",
  "scripts": {
    "dev": "wrangler dev --ip 127.0.0.1 --port 8787 --test-scheduled",
    "db": "wrangler d1 execute audit --local --file schema.sql",
    "fixture": "node fixture.mjs"
  },
  "devDependencies": { "wrangler": "latest" }
}

Connect the local resources

Save this as wrangler.jsonc. The all-zero database ID is a local-development value, not a provisioned cloud database. Wrangler simulates the bindings locally. workers_dev: false also avoids accidentally assigning a public workers.dev endpoint if you later change the setup.

The deployed queue is configured for one active consumer batch. Local simulation does not support that concurrency setting, so duplicate deliveries can fetch a URL concurrently. SQL still deduplicates the frontier. Failures get three retries before moving to a dead-letter queue, whose handler records a terminal failure. The one-second retry delay keeps this lab quick.

{
  "name": "tiny-site-audit",
  "main": "src/index.js",
  "compatibility_date": "2026-10-17",
  "compatibility_flags": ["nodejs_compat"],
  "workers_dev": false,
  "vars": { "TARGET_ORIGIN": "http://127.0.0.1:9123" },
  "d1_databases": [{
    "binding": "DB",
    "database_name": "audit",
    "database_id": "00000000-0000-0000-0000-000000000000"
  }],
  "queues": {
    "producers": [{ "binding": "PAGES", "queue": "audit-pages" }],
    "consumers": [
      {
        "queue": "audit-pages",
        "max_batch_size": 1,
        "max_batch_timeout": 1,
        "max_concurrency": 1,
        "max_retries": 3,
        "retry_delay": 1,
        "dead_letter_queue": "audit-dead"
      },
      { "queue": "audit-dead", "max_batch_size": 1, "max_retries": 3 }
    ]
  },
  "triggers": { "crons": ["* * * * *"] },
  "observability": { "enabled": true, "head_sampling_rate": 1 }
}

Keep the frontier in D1

Save this as schema.sql. A primary key deduplicates each URL within a run. A row starts as ready, becomes sent after dispatch, and eventually becomes done or failed.

ready rows also form a durable outbox. Recording a discovered URL and sending its queue message are separate operations. Keeping the unsent work in SQL lets a later dispatch recover from an interruption between them.

CREATE TABLE IF NOT EXISTS runs (
  id TEXT PRIMARY KEY,
  started INTEGER NOT NULL
);
CREATE TABLE IF NOT EXISTS pages (
  run TEXT NOT NULL REFERENCES runs(id),
  url TEXT NOT NULL,
  depth INTEGER NOT NULL,
  state TEXT NOT NULL DEFAULT 'ready',
  result TEXT,
  PRIMARY KEY (run, url)
);
CREATE INDEX IF NOT EXISTS frontier ON pages(state);

Fetch, inspect, and save a page

Save this complete file as src/index.js. inspect() follows redirects manually so it can check each destination. It reads the body in chunks and rejects oversized HTML before parsing it with HTMLRewriter.

The consumer inserts discovered children and completes the current page in one D1 batch. These writes succeed together or roll back together.

const MAX_PAGES = 20
const MAX_DEPTH = 2
const MAX_BYTES = 262144

export function normalize(value, base, origin) {
  try {
    const url = new URL(value, base)
    if (url.origin !== origin || url.username || url.password) return null
    if (!['http:', 'https:'].includes(url.protocol)) return null
    url.search = ''
    url.hash = ''
    return url.href
  } catch {
    return null
  }
}

async function inspect(url, origin) {
  const controller = new AbortController()
  const timer = setTimeout(() => controller.abort(), 8000)
  const chain = []
  let reader
  try {
    for (let redirects = 0; ; redirects++) {
      const response = await fetch(url, {
        redirect: 'manual', signal: controller.signal,
        headers: { 'User-Agent': 'TinySiteAudit/1' }
      })
      chain.push({ url, status: response.status })
      if ([301, 302, 303, 307, 308].includes(response.status)) {
        await response.body?.cancel()
        const next = normalize(response.headers.get('location') || '', url, origin)
        if (!next) throw new Error('Redirect leaves configured origin')
        if (redirects === 3) throw new Error('Redirect limit reached')
        url = next
        continue
      }
      const html = (response.headers.get('content-type') || '').includes('text/html')
      if (!html) {
        await response.body?.cancel()
        return { status: response.status, chain, html, links: [] }
      }
      const chunks = []
      let size = 0
      reader = response.body?.getReader()
      if (reader) {
        for (;;) {
          const { done, value } = await reader.read()
          if (done) break
          size += value.byteLength
          if (size > MAX_BYTES) throw new Error('HTML byte limit reached')
          chunks.push(value)
        }
      }
      let title = ''
      const links = new Set()
      const parsed = new HTMLRewriter()
        .on('title', { text(chunk) { title += chunk.text } })
        .on('a[href]', { element(element) {
          const link = normalize(element.getAttribute('href'), url, origin)
          if (link && links.size < 100) links.add(link)
        } })
        .transform(new Response(new Blob(chunks), {
          headers: { 'content-type': 'text/html; charset=utf-8' }
        }))
      await parsed.arrayBuffer()
      return {
        status: response.status, chain, html,
        missingTitle: !title.trim(), links: [...links]
      }
    }
  } catch (error) {
    await reader?.cancel().catch(() => {})
    return { error: error.name === 'AbortError' ? 'Fetch timeout' : error.message, chain, links: [] }
  } finally {
    clearTimeout(timer)
  }
}

const statement = (env, sql, ...args) => env.DB.prepare(sql).bind(...args)

async function dispatch(env) {
  const { results } = await statement(env,
    "SELECT run, url FROM pages WHERE state = 'ready' LIMIT 20").all()
  for (const page of results) {
    await env.PAGES.send(page)
    await statement(env,
      "UPDATE pages SET state = 'sent' WHERE run = ? AND url = ? AND state = 'ready'",
      page.run, page.url).run()
  }
}

async function consume(env, message, dead) {
  const { run, url } = message.body
  const page = await statement(env,
    'SELECT p.*, r.started FROM pages p JOIN runs r ON r.id = p.run WHERE p.run = ? AND p.url = ?',
    run, url).first()
  if (!page || ['done', 'failed'].includes(page.state)) return
  if (dead || Date.now() - page.started > 120000) {
    await statement(env,
      "UPDATE pages SET state = 'failed', result = ? WHERE run = ? AND url = ?",
      JSON.stringify({ error: dead ? 'Queue retries exhausted' : 'Run deadline reached' }),
      run, url).run()
    return
  }
  const origin = new URL(env.TARGET_ORIGIN).origin
  if (normalize(url, origin, origin) !== url) throw new Error('Invalid stored URL')
  const result = await inspect(url, origin)
  const writes = []
  if (page.depth < MAX_DEPTH) {
    for (const link of result.links) {
      writes.push(statement(env,
        `INSERT OR IGNORE INTO pages(run, url, depth)
         SELECT ?, ?, ? WHERE (SELECT COUNT(*) FROM pages WHERE run = ?) < ?`,
        run, link, page.depth + 1, run, MAX_PAGES))
    }
  }
  writes.push(statement(env,
    "UPDATE pages SET state = 'done', result = ? WHERE run = ? AND url = ?",
    JSON.stringify(result), run, url))
  await env.DB.batch(writes)
}

export default {
  async fetch(request, env) {
    const path = new URL(request.url).pathname
    if (request.method === 'POST' && path === '/start') {
      const run = crypto.randomUUID()
      const origin = new URL(env.TARGET_ORIGIN).origin
      await env.DB.batch([
        statement(env, 'INSERT INTO runs VALUES (?, ?)', run, Date.now()),
        statement(env, 'INSERT INTO pages(run, url, depth) VALUES (?, ?, 0)', run, `${origin}/`)
      ])
      await dispatch(env)
      return Response.json({ run }, { status: 202 })
    }
    if (request.method === 'POST' && path === '/dispatch') {
      await dispatch(env)
      return new Response('Dispatched ready rows')
    }
    const match = path.match(/^\/runs\/([a-f0-9-]+)$/)
    if (request.method === 'GET' && match) {
      const { results } = await statement(env,
        'SELECT url, depth, state, result FROM pages WHERE run = ? ORDER BY url', match[1]).all()
      const pending = results.some(p => ['ready', 'sent'].includes(p.state))
      return Response.json({
        state: !results.length ? 'unknown' : pending ? 'running' :
          results.some(p => p.state === 'failed') ? 'failed' : 'finished-within-limits',
        pages: results.map(p => ({ ...p, result: p.result ? JSON.parse(p.result) : null }))
      })
    }
    return new Response('Not found', { status: 404 })
  },
  async queue(batch, env) {
    for (const message of batch.messages) {
      await consume(env, message, batch.queue === 'audit-dead')
      await dispatch(env)
      message.ack()
    }
  },
  async scheduled(event, env) {
    await dispatch(env)
  }
}

Why duplicate delivery doesn’t lose the frontier

Queues can deliver a message more than once. A completed row makes a duplicate a no-op. The consumer still tries dispatching any outstanding outbox entries before acknowledging the message.

The dispatcher sends first, then conditionally changes ready to sent. An interruption after the send can produce a duplicate. Changing the state before sending would create a worse gap: a row could look sent even though no queue message existed.

A fast consumer can complete a row before the dispatcher updates it. Because the conditional update only touches ready, it cannot overwrite done. Admission is checked inside each SQL insert, so concurrent discoveries still respect the page cap. A unique key prevents duplicate rows; it does not promise exactly-once HTTP fetching. In particular, the local simulator can process duplicate deliveries concurrently.

HTTP errors and fetch limits are audit findings, stored in the result. Infrastructure errors, such as a failed database write, throw and use queue retries. The dead-letter handler marks exhausted work as failed. If storage itself remains unavailable, the run stays visibly incomplete rather than claiming success.

The scheduled handler flushes the durable outbox. Local cron events don’t automatically fire every minute in wrangler dev; /dispatch provides an explicit local recovery trigger.

Make a site with known problems

Save this as fixture.mjs. It runs only on 127.0.0.1:9123. The outside redirect points to another loopback port, which the crawler must reject before fetching. The fixture also includes a slow response, a redirect loop, and a link beyond our depth limit.

import { createServer } from 'node:http'

createServer((request, response) => {
  const path = new URL(request.url, 'http://127.0.0.1:9123').pathname
  const redirects = { '/old': '/middle', '/middle': '/final', '/loop': '/loop', '/outside': 'http://127.0.0.1:9999/private' }
  if (redirects[path]) {
    response.writeHead(302, { location: redirects[path] }).end()
    return
  }
  response.setHeader('content-type', 'text/html')
  if (path === '/') {
    response.end(`<title>Reading club</title>
      <a href="/empty?sort=new#top">No title</a><a href="/empty">Duplicate</a>
      <a href="/missing">Broken</a><a href="/old">Redirect chain</a>
      <a href="/loop">Loop</a><a href="/outside">Outside origin</a>
      <a href="/big">Too big</a><a href="/final">Final</a>
      <a href="/level1">Depth check</a><a href="/slow">Timeout</a>`)
  } else if (path === '/level1') response.end('<title>Level 1</title><a href="/level2">Next</a>')
  else if (path === '/level2') response.end('<title>Level 2</title><a href="/level3">Too deep</a>')
  else if (path === '/slow') setTimeout(() => response.end('<title>Late</title>'), 10000)
  else if (path === '/empty') response.end('<h1>No title</h1>')
  else if (path === '/final') response.end('<title>Book list</title>')
  else if (path === '/big') response.end('x'.repeat(300000))
  else response.writeHead(404).end('<title>Missing page</title>')
}).listen(9123, '127.0.0.1', () => console.log('Fixture on port 9123'))

Test the queue-driven result

Save this as test.mjs. It checks normalization, starts a run through the Worker, then waits for actual queue processing. Assertions inspect the findings instead of relying on a reassuring log message.

import assert from 'node:assert/strict'
import { setTimeout } from 'node:timers/promises'
import { normalize } from './src/index.js'

const origin = 'http://127.0.0.1:9123'
assert.equal(normalize('/empty?q=1#part', origin, origin), `${origin}/empty`)
assert.equal(normalize('http://127.0.0.1:9999/private', origin, origin), null)
assert.equal(normalize('http://user@127.0.0.1:9123/', origin, origin), null)
const { run } = await (await fetch('http://127.0.0.1:8787/start', { method: 'POST' })).json()
let report
for (let attempt = 0; attempt < 60; attempt++) {
  report = await (await fetch(`http://127.0.0.1:8787/runs/${run}`)).json()
  if (report.state !== 'running') break
  await setTimeout(1000)
}
assert.equal(report.state, 'finished-within-limits')
assert.equal(report.pages.length, 11)
const page = path => report.pages.find(p => new URL(p.url).pathname === path)?.result
assert.equal(page('/empty').missingTitle, true)
assert.equal(page('/missing').status, 404)
assert.equal(page('/old').chain.length, 3)
assert.equal(page('/loop').error, 'Redirect limit reached')
assert.equal(page('/outside').error, 'Redirect leaves configured origin')
assert.equal(page('/big').error, 'HTML byte limit reached')
assert.equal(page('/slow').error, 'Fetch timeout')
assert.equal(page('/level3'), undefined)
assert.ok(report.pages.every(p => p.depth <= 2))
console.log(`Checked ${report.pages.length} URLs; report: /runs/${run}`)

Run the whole lab

Install the development dependency and initialize only the local database:

npm install
npm run db

Start npm run fixture in one terminal. In another terminal, start npm run dev. Leave both running. The Queues local-development guide describes the simulated producer and consumer environment.

In a third terminal, run:

node test.mjs

The test prints a report path after checking 11 admitted URLs. Open http://127.0.0.1:8787 followed by that path, or request it with curl, to inspect the JSON.

/missing has status 404. /empty has missingTitle: true. /old records two redirects followed by a successful page. /loop, /outside, /big, and /slow produce bounded errors. /level3 is absent because its discovery would exceed depth 2.

The duplicate links to /empty produce one row. The report’s finished-within-limits state means every admitted URL reached an outcome. It doesn’t mean the site has no problems, or that the crawler explored every page on it.

To flush recorded but unsent work locally:

curl -X POST http://127.0.0.1:8787/dispatch

Run it twice; completed pages remain completed. The queue can contain redundant deliveries, while the SQL frontier keeps one result per URL.

What this small crawler leaves out

This fetches HTML without running page JavaScript. It doesn’t execute a browser, measure Lighthouse performance, or check links hidden behind interaction. It also doesn’t implement robots.txt policy. Use it on the fixture while learning, and add the site’s permitted crawl policy before adapting it to an owned public site.

The page cap counts unique admitted URLs, including the root and failed pages. Redirect requests have their own separate bound. Depth-first versus breadth-first discovery and shortest-path depth are larger frontier decisions; this version preserves the first admitted depth.

For an actual deployed audit, add request pacing and per-origin controls, authenticate run creation, monitor stale runs and the dead-letter queue, and define retention for saved results. A failure should leave something inspectable: which URL was attempted, its response chain, and whether it reached an audit outcome or exhausted execution retries.

Want me to talk about your product? You can sponsor this site.