How to build a scraping tool

By

Build a scraping tool with Node.js or Cloudflare Workers: fetch, Cheerio, caching, alerts, browser rendering, and what changes when Google fights back.

~~~

After reading this tweet I got interested. Pieter Levels replaced a $249/month ScrapingBee plan with a scraper running on his own server. He uses Playwright with a real Google Chrome, waits about 30 seconds between requests, pays about $1/month in proxy bandwidth, and gets around 90% of his Google searches through.

I wanted to understand how it works. So I built it.

I ran the Node.js and browser tests in this post from my laptop on September 9, 2026, and I’ll tell you what worked and what got blocked as we go.

The tweet was about Google, but I didn’t want to build the whole thing around Google, because most scraping has nothing to do with search results. So we’ll build the tool around something I actually need, which is checking hosting pricing pages. Then in the second half we’ll point the same tool at Google and see what has to change when the site doesn’t want to be scraped.

What a scraping tool is

A scraper downloads a page and pulls some data out of it.

A scraping tool is everything around that. It accepts requests, runs them one at a time, caches results, keeps a log, and tells you when something breaks. ScrapingBee sells you the tool. Here we build our own.

There are two kinds of scrapers, and it helps to name them now because we’ll use both.

An HTTP scraper sends a request and parses the HTML that comes back. In Node.js this is fetch() plus a parser like Cheerio. It’s fast and it uses almost no memory, and it works as long as the site returns the full page and doesn’t try to stop you.

A browser scraper drives a real browser. Playwright or Puppeteer open the page, the JavaScript runs, and you read the DOM after rendering. It’s slower and heavier, but it sees what a person would see.

We start with the first kind.

What people scrape, and how hard each one is

I want to give you an idea of what people point scrapers at, because the difficulty varies a lot. I checked each of these with a single request and a Chrome user agent, unless I say otherwise.

Pricing pages. Hosting providers, SaaS competitors, payment processors. Most are plain HTML with the prices in a table. This is what we’ll build against.

Changelogs, docs and career pages. Also plain HTML. Useful if you want to keep an eye on what a competitor is shipping or hiring for.

News sites and blogs. Almost every one of them has an RSS feed, so use that. A feed is made for programs and it doesn’t break when the layout changes.

Job boards, real estate, used cars. Company career pages are easy. The big aggregators usually sit behind Cloudflare or Akamai, and a plain fetch() gets a challenge page instead of the content.

Hacker News, GitHub, Reddit. They have APIs. Hacker News has a free public one, GitHub’s is generous, Reddit’s costs money past a small free tier. When there’s an API, scraping the HTML is the wrong tool.

Airbnb and Booking. I fetched the Airbnb search page for Lisbon. I got a 200 and 1.1 MB of HTML, but the listings weren’t in the markup. They were in a JSON blob inside a script tag, next to references to an internal StaysSearch GraphQL operation the page calls when you scroll. So on a site like this you parse the JSON, or you call the internal API yourself. Their terms forbid both. Prices also depend on the dates, the currency and where your IP is, so what a server in Frankfurt sees isn’t what a user in Milan sees.

Amazon. One product page request returned the full page, title and price included. Amazon is known for switching to a “Robot Check” page after a handful of requests from the same datacenter IP, and prices again change with location. There’s a Product Advertising API for affiliates, which is the route Amazon wants you to take.

Google. Requires JavaScript since 2025, shows a CAPTCHA to headless browsers. This is the hardest one I know of, and it gets the second half of the post.

Before scraping anything, check if there’s an API or a feed, and read the terms. If there’s an API, use it. If the terms say no, you’re taking on some risk, and we’ll talk about how much at the end.

The escalation ladder

Every site fits somewhere on this list. Start from the top and stop at the first step that works.

  1. Look for an API. Open the network tab, click around, look for JSON responses.
  2. Check for a sitemap or an RSS feed.
  3. Use fetch() and Cheerio. This covers most content sites. It’s where we start.
  4. Use fetch() with TLS impersonation, with curl_cffi in Python or primp. Cloudflare and Akamai fingerprint the TLS handshake, and Node’s handshake is recognizable as Node. Pretending to be Chrome at that level gets you past the check without a browser.
  5. Use a headless browser, chromium.launch({ channel: 'chromium' }) in Playwright. You need this for pages that render their data with JavaScript.
  6. Use a headful patched browser with a persistent profile. Google needs this.
  7. Add proxies.
  8. Pay for an API.

Going down the list, each step costs more to run and more to keep working. In this post we do step 3 for the pricing pages and step 6 for Google.

Step 1: the simplest scraper

Let’s start with the DigitalOcean Droplet pricing page.

I keep hand-checked pricing data for hostingpicker.dev, and going through dozens of pages like this one by hand takes me an afternoon every month. I’d like a script to tell me when a number changes.

You need Node.js 24 LTS (22 or newer also works). Create a project and install Cheerio:

mkdir scraper && cd scraper
npm init -y
npm install cheerio@1

@1 gives you the latest 1.x release.

Create a file called digitalocean.mjs:

import * as cheerio from 'cheerio'

const UA = 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/154.0.0.0 Safari/537.36'

const res = await fetch('https://www.digitalocean.com/pricing/droplets', {
  headers: { 'user-agent': UA },
})

const $ = cheerio.load(await res.text())

const plans = $('table').first().find('tbody tr').map((_, tr) => {
  const c = $(tr).find('td').map((_, td) => $(td).text().trim()).get()
  return { memory: c[0], vcpu: c[1], transfer: c[2], ssd: c[3], hourly: c[4], monthly: c[5] }
}).get()

console.log(plans)

Run it:

node digitalocean.mjs

This is what I got:

[
  { memory: '512 MiB', vcpu: '1 vCPU', transfer: '500 GiB', ssd: '10 GiB', hourly: '$0.00595', monthly: '$4.00' },
  { memory: '1 GiB', vcpu: '1 vCPU', transfer: '1,000 GiB', ssd: '25 GiB', hourly: '$0.00893', monthly: '$6.00' },
  { memory: '2 GiB', vcpu: '1 vCPU', transfer: '2,000 GiB', ssd: '50 GiB', hourly: '$0.01786', monthly: '$12.00' },
  { memory: '2 GiB', vcpu: '2 vCPUs', transfer: '3,000 GiB', ssd: '60 GiB', hourly: '$0.02679', monthly: '$18.00' },
  { memory: '4 GiB', vcpu: '2 vCPUs', transfer: '4,000 GiB', ssd: '80 GiB', hourly: '$0.03571', monthly: '$24.00' },
  { memory: '8 GiB', vcpu: '4 vCPUs', transfer: '5,000 GiB', ssd: '160 GiB', hourly: '$0.07143', monthly: '$48.00' },
  { memory: '16 GiB', vcpu: '8 vCPUs', transfer: '6,000 GiB', ssd: '320 GiB', hourly: '$0.14286', monthly: '$96.00' }
]

Seven plans, in about 200 milliseconds, and we didn’t need a browser.

Let me explain the two choices in that code.

The user-agent header makes the request look like it comes from Chrome. Many sites give you a reduced page, or an error, when the user agent is missing or is Node’s default one. For sites that don’t actively fight scrapers, this header is usually all you need.

Then the selector, $('table').first(). The page has several tables, one per Droplet type, and the first one is the Basic plans. I could have used a class name instead. The classes on this page look like SimpleTablestyles__StyledSimpleTable-sc-1fkj6yy-9, though. That’s a hash from the CSS tooling and it changes on every deploy. When you don’t control the site, select on structure, like the first table and the rows in its body, and stay away from generated class names.

Cheerio’s .map() returns a Cheerio collection, so we call .get() at the end to turn it into a plain array.

When fetch is not enough

I tried the same thing on Hetzner’s cloud page. I got a 200 and 163 KB of HTML, and none of the prices were in it. The page builds the pricing table with JavaScript after it loads.

That’s the case for step 5 of the ladder, a browser. Hetzner doesn’t fight back, so plain Playwright in headless mode is fine there. Later in the post we build a browser scraper for Google, and the same shape works for Hetzner if you take out the stealth parts.

Step 2: one module per target

A tool that only knows DigitalOcean isn’t a tool. Each site we scrape will have its own quirks, so let’s give each one its own module, all with the same interface.

Create a targets folder and move the code into targets/digitalocean.mjs:

import * as cheerio from 'cheerio'

const UA = 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/154.0.0.0 Safari/537.36'

export async function scrape() {
  const res = await fetch('https://www.digitalocean.com/pricing/droplets', {
    headers: { 'user-agent': UA },
  })
  if (!res.ok) throw new Error('status ' + res.status)

  const $ = cheerio.load(await res.text())

  const plans = $('table').first().find('tbody tr').map((_, tr) => {
    const c = $(tr).find('td').map((_, td) => $(td).text().trim()).get()
    return { memory: c[0], vcpu: c[1], transfer: c[2], ssd: c[3], hourly: c[4], monthly: c[5] }
  }).get()

  if (plans.length === 0) throw new Error('no plans found')
  return plans
}

Every target exports a scrape(params) function that returns the data or throws. The server we write next only knows about that function. Whether a target uses fetch() or a browser is the target’s business.

I added two throw lines.

The first one, on !res.ok, is the obvious failure. The second one is the one I care about. When DigitalOcean redesigns the page, the request still returns 200, the first table is something else or isn’t there, and the code returns an empty array. If we let that through, the tool saves an empty result and I find out a month later that my pricing data is stale. So an empty result is an error.

Also, the first time a target works, save a copy of the HTML next to the code. When the page changes, having the old version to compare with makes the fix a lot faster.

Step 3: making it a service

We have a module. Now let’s make it something other projects can call.

We’ll write a small HTTP API with a queue and a cache. The queue makes sure two scrapes never run at the same time, however many callers there are. The cache makes sure we don’t fetch the same page twice for the same data.

We’ll only use what Node ships with: node:http for the server and node:sqlite for the cache. Node 22.13 and newer include SQLite. I wrote about it in the built-in Node.js SQLite module.

Create server.mjs:

import http from 'node:http'
import { DatabaseSync } from 'node:sqlite'
import * as digitalocean from './targets/digitalocean.mjs'

const targets = { digitalocean }

const db = new DatabaseSync('./tool.db')
db.exec(`CREATE TABLE IF NOT EXISTS results (
  key TEXT PRIMARY KEY,
  json TEXT,
  created_at INTEGER
)`)

const getCached = db.prepare('SELECT json, created_at FROM results WHERE key = ?')
const setCached = db.prepare('INSERT OR REPLACE INTO results VALUES (?, ?, ?)')

const DAY = 24 * 60 * 60 * 1000
const wait = (ms) => new Promise((r) => setTimeout(r, ms))

let queue = Promise.resolve()

function enqueue(target, params) {
  const job = queue.then(() => target.scrape(params))
  queue = job.catch(() => {}).then(() => wait(target.pause?.() ?? 5000))
  return job
}

async function handle(name, params) {
  const target = targets[name]
  if (!target) throw new Error('unknown target')

  const key = name + ':' + JSON.stringify(params)
  const cached = getCached.get(key)
  if (cached && Date.now() - cached.created_at < DAY) {
    return JSON.parse(cached.json)
  }

  const results = await enqueue(target, params)
  setCached.run(key, JSON.stringify(results), Date.now())
  return results
}

http
  .createServer(async (req, res) => {
    const url = new URL(req.url, 'http://localhost')

    if (req.headers.authorization !== `Bearer ${process.env.API_TOKEN}`) {
      res.writeHead(401)
      return res.end()
    }

    if (url.pathname !== '/scrape') {
      res.writeHead(404)
      return res.end()
    }

    const { target, ...params } = Object.fromEntries(url.searchParams)

    try {
      const results = await handle(target, params)
      res.writeHead(200, { 'content-type': 'application/json' })
      res.end(JSON.stringify(results))
    } catch (err) {
      res.writeHead(503, { 'content-type': 'application/json' })
      res.end(JSON.stringify({ error: err.message }))
    }
  })
  .listen(3100)

Let’s go through it.

targets is the registry. To add a site you add a file in targets/ and one line here. The target query parameter picks the module, and every other query parameter goes to scrape() as params. DigitalOcean ignores them. The Google target we write later reads q from there.

The queue is a promise chain. Each request appends its scrape to the end of the chain, then a pause. The caller gets its results as soon as its own scrape is done, and the next scrape waits for the pause. If ten callers show up at once, they’re served one after the other and none of them has to know about the pacing. The .catch(() => {}) is there so a failed scrape doesn’t break the chain for the ones behind it.

The pause is 5 seconds unless the target exports a pause() function. Five seconds between requests to a pricing page is fine. Google will need much more, and we’ll put that number inside the Google target, next to the site it belongs to.

The cache keeps results for 24 hours per key, where the key is the target name plus its parameters. Pricing pages don’t change more than once a day, so after the first call everything is served from SQLite. For a target with parameters, like a search query, the same query twice is one scrape.

The Authorization check is there so nobody else can use your tool. Set API_TOKEN to something long and random.

Run the server:

API_TOKEN=change-me node server.mjs

And call it from another terminal:

curl -H "Authorization: Bearer change-me" \
  "http://localhost:3100/scrape?target=digitalocean"

On my laptop the first call took 228 ms and the second one, from the cache, took 7 ms. Without the header you get a 401. With ?target=hetzner you get {"error":"unknown target"}, because we haven’t written that one.

Step 4: monitoring

A scraper breaks quietly. The site changes, the tool keeps running, and you notice weeks later because your own site shows old numbers. I want the tool to tell me instead.

Let’s record every run in the same SQLite database. Add this next to the results table:

db.exec(`CREATE TABLE IF NOT EXISTS runs (
  at INTEGER,
  key TEXT,
  ok INTEGER,
  error TEXT
)`)

const logRun = db.prepare('INSERT INTO runs VALUES (?, ?, ?, ?)')

Then a function that runs a scrape, logs how it went, and counts failures in a row:

let failures = 0

async function run(key, fn) {
  try {
    const results = await fn()
    logRun.run(Date.now(), key, 1, null)
    failures = 0
    return results
  } catch (err) {
    logRun.run(Date.now(), key, 0, err.message)
    failures++
    if (failures === 3) await alert(`Scraper: 3 failures in a row (${err.message})`)
    throw err
  }
}

The alert is one request to Telegram’s Bot API:

async function alert(text) {
  if (!process.env.TELEGRAM_TOKEN) return console.error(text)
  await fetch(`https://api.telegram.org/bot${process.env.TELEGRAM_TOKEN}/sendMessage`, {
    method: 'POST',
    headers: { 'content-type': 'application/json' },
    body: JSON.stringify({ chat_id: process.env.TELEGRAM_CHAT_ID, text }),
  })
}

To set it up, create a bot with @BotFather in Telegram, send it a message, and read your chat ID from https://api.telegram.org/bot<token>/getUpdates. Ten minutes of work. While developing, without a token, the alert prints to the terminal.

Now change enqueue() to go through run(). It needs the key, so we pass it in:

function enqueue(key, target, params) {
  const job = queue.then(() => run(key, () => target.scrape(params)))
  queue = job.catch(() => {}).then(() => wait(target.pause?.() ?? 5000))
  return job
}

And in handle(), call enqueue(key, target, params).

Two numbers are worth looking at in the runs table. How many runs succeeded per day, and how many items came back per successful run. If a page changes in a way that leaves one row in the table, the first number stays fine while the second one drops, so check both. Here is a query for the first one:

SELECT date(at / 1000, 'unixepoch') AS day,
       round(100.0 * sum(ok) / count(*)) AS success_pct,
       count(*) AS runs
FROM runs GROUP BY day ORDER BY day DESC LIMIT 14;

Step 5: where to run it

The tool runs on your laptop. Now we need a machine that runs it all the time.

For an HTTP scraper on sites that don’t fight back, almost anything works. The choice gets harder once you add a browser and start caring about the reputation of your IP address, which is what happens with Google. I’ll go through the options with both in mind.

Your own computer at home

A Mac mini, a Raspberry Pi or an old laptop. You run node server.mjs with launchd or in a tmux session, and your other apps reach it through a tunnel like Tailscale or Cloudflare Tunnel.

I ran all the tests for this post from a laptop on a normal home connection. A home IP is a residential IP, the kind sites trust the most, and you’re already paying for it. This doesn’t matter for a pricing page, and it matters a lot for Google.

The problem is that it’s your own IP. If the scraper gets flagged, you get CAPTCHAs in your own browser too. Home connections also go down, change address, and sit behind a router you have to configure. For a few hundred requests a day I think this is the cheapest good option. For something people pay for, I’d use a server.

A VPS

A small virtual server. The HTTP tool needs almost nothing. With a browser, headful Chrome with one tab needs between 500 MB and 1 GB of RAM, so 4 GB is comfortable. Prices checked on September 9, 2026:

  • Hetzner CX23, 2 vCPUs and 4 GB, €5.49 a month in Germany or Finland, after the June 2026 price increase.
  • DigitalOcean Basic Droplet, 2 vCPUs and 4 GB, $24 a month. More expensive, but you can pick datacenters in the US, Europe and Asia, so the scraper can sit in the same region as your users. The $12 Droplet with 2 GB works too if you keep one tab open. These are the numbers our scraper pulled in step 1.

You get a public IP that is only yours, full control of the machine, and a bill that doesn’t move with usage. The IP is a datacenter IP, which Google trusts less than a home connection, but one clean datacenter IP with good pacing handles a few thousand requests a day. If it gets flagged, you destroy the server and create a new one, and you have a new IP.

Here is the setup. I’ll use Ubuntu 24.04. If you’ve never set up a server, my free Ubuntu VPS course covers the first hour, and the SSH course covers connecting to it safely.

SSH in and create a user for the tool. Don’t run it as root:

adduser --disabled-password --gecos "" scraper

Install Node.js:

curl -fsSL https://deb.nodesource.com/setup_24.x | bash -
apt install -y nodejs

Copy your project to /home/scraper/scraper and install the dependencies as the scraper user:

su - scraper
cd scraper && npm install

Now let’s make it a service. My free systemd course has all the details, but this unit is enough. Create /etc/systemd/system/scraper.service:

[Unit]
Description=Scraping tool
After=network-online.target

[Service]
User=scraper
WorkingDirectory=/home/scraper/scraper
EnvironmentFile=/home/scraper/scraper/.env
ExecStart=/usr/bin/node server.mjs
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target

Put API_TOKEN=... and the Telegram variables in /home/scraper/scraper/.env, then enable the service:

systemctl daemon-reload
systemctl enable --now scraper
systemctl status scraper

Restart=always brings the service back 10 seconds after a crash. The database is on disk, so the cache and the log survive restarts.

Port 3100 is only for your other applications, so don’t expose it to the internet. Either bind the server to 127.0.0.1 and run your apps on the same machine, or add a firewall rule that only allows your other servers in. The Linux Server Security course shows how.

Also read your provider’s acceptable use policy. Hetzner’s doesn’t forbid scraping, but it forbids scanning other networks and spoofing source IPs, and DigitalOcean’s is similar. One slow process fetching pricing pages is not something they care about. The legal question that matters is the target site’s terms, and we get to that at the end.

exe.dev

exe.dev gives you Linux VMs over SSH for $20 a month. You get a pool of 2 vCPUs and 8 GB of RAM to split across up to 50 VMs, so the scraper can have its own VM next to your other projects. I wrote a deep dive into exe.dev if you want the details.

Creating a VM takes about a second, so you can try things freely. Every VM also gets a private HTTPS URL with exe.dev login in front of it, so you don’t need to think about exposing port 3100. Your other projects call https://<vm>.exe.xyz/scrape, and nobody else can.

What makes it less suitable for the Google part is that your VM has no public IP of its own. Outbound traffic leaves through addresses exe.dev owns and shares between customers. If someone else on the same address scrapes Google carelessly, you pay for it too, and you can’t get a fresh IP by recreating the VM. For pricing pages none of this matters. For Google I’d pick a VPS, or add a proxy.

The setup steps above work the same on exe.dev. The default image has Docker, Python and Go but not Node.js, so install Node first, and skip the firewall part.

Cloudflare Workers

A Cloudflare Worker is a good home for the HTTP version of this tool. It can fetch a page on a schedule, save the result in KV, and expose a small API without a server running all day.

The Node.js service we built won’t move there unchanged. Workers don’t listen with node:http, and node:sqlite can’t write a local database that survives between requests. We replace the HTTP server with a fetch() handler, the cron job with a scheduled() handler, and SQLite with KV or D1.

Create a Worker project, install Cheerio, and create a KV namespace:

npm create cloudflare@latest scraper-worker
cd scraper-worker
npm install cheerio@1
npx wrangler kv namespace create RESULTS

Put the namespace ID printed by the last command in wrangler.jsonc:

{
  "$schema": "node_modules/wrangler/config-schema.json",
  "name": "scraper-worker",
  "main": "src/index.js",
  "compatibility_date": "2026-09-09",
  "kv_namespaces": [
    {
      "binding": "RESULTS",
      "id": "paste-the-namespace-id-here"
    }
  ],
  "triggers": {
    "crons": ["0 6 * * *"]
  }
}

Cloudflare cron expressions use UTC. This one runs every day at 06:00 UTC.

Now put this in src/index.js:

import * as cheerio from 'cheerio'

const UA = 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/154.0.0.0 Safari/537.36'

async function scrapeDigitalOcean() {
  const res = await fetch('https://www.digitalocean.com/pricing/droplets', {
    headers: { 'user-agent': UA },
  })
  if (!res.ok) throw new Error('status ' + res.status)

  const $ = cheerio.load(await res.text())
  const plans = $('table').first().find('tbody tr').map((_, tr) => {
    const c = $(tr).find('td').map((_, td) => $(td).text().trim()).get()
    return { memory: c[0], vcpu: c[1], transfer: c[2], ssd: c[3], hourly: c[4], monthly: c[5] }
  }).get()

  if (plans.length === 0) throw new Error('no plans found')
  return plans
}

async function update(env) {
  const plans = await scrapeDigitalOcean()
  await env.RESULTS.put('digitalocean', JSON.stringify({
    updatedAt: new Date().toISOString(),
    plans,
  }))
  return plans
}

export default {
  async scheduled(_controller, env) {
    await update(env)
  },

  async fetch(request, env) {
    if (request.headers.get('authorization') !== `Bearer ${env.API_TOKEN}`) {
      return new Response(null, { status: 401 })
    }

    const path = new URL(request.url).pathname

    if (path === '/scrape') {
      return Response.json(await update(env))
    }

    if (path === '/result') {
      const result = await env.RESULTS.get('digitalocean')
      return new Response(result, {
        status: result ? 200 : 404,
        headers: { 'content-type': 'application/json' },
      })
    }

    return new Response(null, { status: 404 })
  },
}

Add the API token as a secret, then deploy:

npx wrangler secret put API_TOKEN
npx wrangler deploy

The scheduled handler refreshes the data once a day. /result returns the saved copy, while /scrape forces a fresh run. KV is enough for the latest result. Use D1 if you also want the complete run history and SQL monitoring queries from step 4.

If a page needs JavaScript, Workers can call Cloudflare Browser Run through a browser binding. Add this to wrangler.jsonc:

{
  "browser": {
    "binding": "BROWSER"
  }
}

Then a Worker can ask a headless browser to render a page and extract matching elements:

const response = await env.BROWSER.quickAction('scrape', {
  url: 'https://www.hetzner.com/cloud/',
  elements: [{ selector: 'table tbody tr' }],
})

const data = await response.json()

This is a good fit for a JavaScript-rendered pricing page like Hetzner. Browser Run handles the browser, while the Worker handles the schedule, storage and API. The compatibility date in the example is new enough for quickAction().

For several targets, put jobs into a Cloudflare Queue so scrapes don’t all start together. For one daily pricing page, the cron handler is enough.

GitHub Actions can run the same HTTP scrape on a schedule and commit the result. That works too, but a Worker plus KV avoids creating a commit every day and gives your other applications an API to call.

When the target fights back: Google

Up to here we assumed the site gives us the page when we ask for it. Google doesn’t. Let’s point the same tool at Google search results and see what has to change. The queue, the cache, the log and the alert stay as they are. The part that changes is how the target gets the page.

What gets blocked, and why

Before writing the target I wanted to know what Google blocks today. I tried four approaches from my laptop, same IP, same afternoon.

1. Plain curl. The same kind of request that worked on DigitalOcean:

curl -A "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/154.0.0.0 Safari/537.36" \
  "https://www.google.com/search?q=node+sqlite+tutorial&hl=en"

I got a 200 with a 92 KB HTML page and no results in it. The page links to /httpservice/retry/enablejs and asks you to enable JavaScript. Google stopped serving results to clients without JavaScript at the start of 2025, so an HTTP scraper can’t work on Google anymore, whatever headers you send.

2. Headless Chromium. The default Playwright launch:

const browser = await chromium.launch({ headless: true })

Google sent me to google.com/sorry/, the CAPTCHA page, on the first request. Playwright’s default headless mode uses chrome-headless-shell, the old headless implementation. It’s a separate build that differs from desktop Chrome in many measurable ways, and it even puts HeadlessChrome/ in the user agent. Most open source SERP projects use this mode, which is why they stopped working.

3. Real Chrome, headful, stock Playwright. This is the setup from the tweet: real Google Chrome, a visible window, a persistent profile.

const context = await chromium.launchPersistentContext('./profile', {
  channel: 'chrome',
  headless: false,
})

I expected this to work and it didn’t. /sorry/ again. I also tried a more human sequence, open google.com, accept the cookie dialog, type the query one letter at a time, press Enter. Same CAPTCHA. So the browser was real, the window was visible, the IP was fine, and the problem had to be in how Playwright talks to the browser.

4. Real Chrome, headful, patched Playwright. One reply under the tweet suggested replacing playwright with patchright. Patchright is a patched build of Playwright with the same API. You install it and change one import:

import { chromium } from 'patchright'

Same code, same IP, a minute later. Ten results on the first try. I ran four more queries over the next half hour and they all worked.

So in 2026 it’s not enough to hide that the browser is headless. Google also checks whether a debugger is attached to it.

Why stock Playwright leaks

Playwright, Puppeteer and Selenium all control Chrome through the Chrome DevTools Protocol (CDP). To run your page.evaluate() calls, they enable the Runtime domain of that protocol, and when that domain is on the browser behaves a little differently. A page can notice with a few lines of JavaScript.

The classic check goes like this. The page defines an object with a getter, logs it with console.debug, and checks whether the getter ran. In a normal browser it never runs, because nobody is looking at the console. When CDP’s Runtime domain is enabled, the debugger serializes the object for the DevTools console, and that calls the getter.

Chrome fixed that particular check in 2025, but there are others. Playwright creates named isolated worlds, it injects bindings like __playwright__binding__, it launches Chrome with the --enable-automation flag (which sets navigator.webdriver to true), and error stack traces can reveal local file paths.

Patchright never calls Runtime.enable. It runs your scripts in isolated contexts, removes the automation flags, and adds --disable-blink-features=AutomationControlled. The price is that you lose the browser console in your scripts, and you can only use Chromium-based browsers.

I don’t expect this to keep working forever. Patchright usually publishes a new version a few days after each Playwright release, and what I describe here worked on the day I tested it. When it breaks, the alert from step 4 will tell me.

The Google target

You need Google Chrome installed. Install Patchright:

npm install patchright@1

Patchright follows Playwright’s version numbers, so @1 gives you the latest 1.x release.

Create targets/google.mjs:

import { chromium } from 'patchright'

let context

async function getContext() {
  if (context) return context

  context = await chromium.launchPersistentContext('./profile', {
    channel: 'chrome',
    headless: false,
    viewport: { width: 1366, height: 850 },
    locale: 'en-US',
    timezoneId: 'Europe/Rome',
  })

  await context.route('**/*', (route) => {
    const type = route.request().resourceType()
    if (['image', 'media', 'font'].includes(type)) return route.abort()
    return route.continue()
  })

  return context
}

export const pause = () => 20000 + Math.random() * 20000

export async function scrape({ q }) {
  const context = await getContext()
  const page = await context.newPage()

  try {
    await page.goto(
      'https://www.google.com/search?q=' + encodeURIComponent(q) + '&hl=en',
      { waitUntil: 'domcontentloaded' }
    )

    if (page.url().includes('/sorry/')) throw new Error('blocked')

    const accept = page.getByRole('button', { name: /accept all/i })
    if (await accept.count()) await accept.first().click()

    await page.waitForSelector('#search', { timeout: 15000 })

    const results = await page.locator('#search a:has(h3)').evaluateAll((links) =>
      links.map((a) => {
        const block = a.closest('[data-hveid]')
        return {
          title: a.querySelector('h3').innerText,
          url: a.href,
          cite: block?.querySelector('cite')?.innerText,
          snippet: block?.querySelector('.VwiC3b')?.innerText,
        }
      })
    )

    for (const r of results) {
      if (r.url.includes('/goto?')) {
        const res = await context.request.get(r.url, { maxRedirects: 0 })
        r.url = res.headers()['location'] || r.url
      }
    }

    if (results.length === 0) throw new Error('no results')
    return results
  } finally {
    await page.close()
  }
}

It has the same interface as the DigitalOcean target, a scrape() that returns data or throws. There’s a lot more going on inside, so let me go through it from the top.

The browser is launched once and kept open in the context variable. Launching Chrome takes a few seconds and a lot of CPU, and a browser that stays open with a warm profile is both faster and closer to how a real person’s browser behaves. Each search opens a new tab and closes it at the end.

Then the launch options. launchPersistentContext('./profile') gives Chrome a user data directory on disk, so cookies, local storage and the cookie consent choice survive between runs. A browser that starts from an empty profile on every request looks like a bot, and a profile that has been used for weeks looks like someone’s browser. channel: 'chrome' uses the Google Chrome installed on the machine instead of the Chromium build Playwright downloads, so the codecs, fonts and version string are the ones Google expects. headless: false opens a real window. New headless mode uses the same engine, but a window still avoids a few small differences, and on a server we’ll give Chrome a virtual display so it costs nothing. viewport and locale are plausible values. Pick a common desktop size and don’t change it between requests. timezoneId should match where your IP is. A German IP with a Tokyo clock is one of the inconsistencies fingerprinting scripts look for.

context.route() blocks images, media and fonts. The results page still renders, since we only want text. In my test this removed 19 of the 30 requests a Google results page makes. If you ever put residential proxies in front of this, and they charge per gigabyte, this is what keeps the bill at a dollar instead of ten. Leave scripts and stylesheets alone though. The page needs them, and a browser that never loads them stands out.

In Europe, Google shows a cookie consent dialog on the first visit. The accept lines click “Accept all” if it’s there. The choice is saved in the profile and the dialog doesn’t come back.

Then the block check. When Google blocks you nothing throws, you get a 200 with a CAPTCHA page in it. So we look at the URL for /sorry/ and throw. After extraction we also throw on zero results, because zero results for a normal query means a block or a layout change, and either way I want an error in the log. The queue’s pause then keeps us from retrying right away, which matters, because hammering the CAPTCHA page makes the block last longer. When I got /sorry/ with stock Playwright, I waited about 20 seconds before trying Patchright, and the IP was still fine. Blocks at this level are short unless you retry in a loop.

The extraction selector is #search a:has(h3), every result link that contains a title. Google changes its class names all the time, but a link wrapping an h3 has been the structure for years. Each result lives inside an element with a data-hveid attribute, so we go up to it and read the cite (the breadcrumb, like https://nodejs.org › api › sqlite) and the snippet. The snippet class VwiC3b will stop working one day. When it does, this file is the only one to fix.

The for loop over the results is something I found while testing. Google wraps the result links now, so the href looks like https://www.google.com/goto?url=CAESZQHrOzAV... and the url parameter is encrypted. You can’t decode it, but you can follow it. context.request.get() with redirects disabled uses the browser’s cookies, so Google answers with a 302 and the real destination in the Location header. In my test all ten links resolved to the actual pages, like nodejs.org/api/sqlite.html.

And pause(). The tweet mentioned waiting about 30 seconds between searches, and I think this matters as much as the browser setup. Google rate limits each IP based on how it behaves. A person searches a few times a minute at most, with irregular gaps. A script that sends a query every 2 seconds isn’t a person, even with a perfect browser. So the Google target exports a pause() of 20 to 40 seconds, and the queue uses it instead of the 5 second default. That’s 90 to 180 searches an hour, or 2,000 to 4,000 a day, from one IP. If your product needs 500 searches a day, you’re done, no proxies. If you need 20,000 a day, one IP won’t get you there however you pace it. Do this math before buying anything.

Now register the target in server.mjs:

import * as digitalocean from './targets/digitalocean.mjs'
import * as google from './targets/google.mjs'

const targets = { digitalocean, google }

Restart the server and run a search:

curl -H "Authorization: Bearer change-me" \
  "http://localhost:3100/scrape?target=google&q=node+sqlite+tutorial"

A Chrome window opens, the search runs, and you get JSON back. On my laptop the first search took about 20 seconds, most of it Chrome starting up. The same query again came from the cache. And the runs table now has both targets in it:

digitalocean:{}                   | 1 |
google:{"q":"node sqlite tutorial"} | 1 |

The Google-specific code is all in one file, and nothing else in the tool changed.

Running a browser on a server

The VPS from step 5 has no screen, and Chrome wants one. Xvfb is a virtual X server that gives Chrome a display nobody looks at.

On the server, install Google Chrome from Google’s own package, and Xvfb:

wget -q -O /tmp/chrome.deb https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb
apt install -y /tmp/chrome.deb xvfb

Installing the .deb also adds Google’s apt repository, so apt upgrade keeps Chrome current. You want that. A Chrome three versions behind is rare among real users, and rare is what detection looks for.

Chrome refuses to run as root unless you pass --no-sandbox, and we don’t want that flag. One more reason the service runs as the scraper user.

Test it once:

API_TOKEN=change-me xvfb-run -a node server.mjs

xvfb-run -a starts a virtual display, sets the DISPLAY variable for the Node process, and shuts the display down when Node exits. Chrome thinks it has a screen.

Then change one line in the systemd unit:

ExecStart=/usr/bin/xvfb-run -a -s "-screen 0 1366x850x24" /usr/bin/node server.mjs

The screen size matches the viewport in the target. Run systemctl daemon-reload and systemctl restart scraper, and the browser runs on the server the same way it ran on your laptop. The profile is on disk, so it survives restarts too.

A hosted browser

If you’d rather not run Chrome yourself, Browserbase and Browserless run it for you. You keep the target code and connect to their browser over CDP:

const browser = await chromium.connectOverCDP(`wss://connect.browserbase.com?apiKey=${process.env.BB_KEY}`)

Browserbase’s Developer plan is $20 a month for 100 browser hours, then $0.12 per hour, and includes 1 GB of residential proxy traffic, then $12 per GB. Prices checked on September 9, 2026. A Google search takes about 10 seconds, so 2,000 searches a day is about 170 browser hours a month, around $30.

You still own the extraction code and the queue, and they own the browser, the stealth patches and the IPs. I haven’t tested either against Google, so measure the success rate before depending on one. Also keep in mind the profile lives on their side, sessions are metered, and if their patches stop working there’s nothing you can update yourself.

Why Browser Run doesn’t replace the Google setup

Cloudflare Browser Run can reuse a browser session. You can disconnect, reconnect on the next Worker request, and use a Durable Object when several requests need the same session. A browser closes after 60 seconds of inactivity by default, and you can extend that to 10 minutes.

That is useful for many scraping jobs, but it isn’t the setup that worked for Google in my test. Browser Run uses headless Chromium. It doesn’t give us a headful Patchright browser with an on-disk profile that ages for weeks on one IP.

I would use Browser Run for Hetzner and other JavaScript-rendered pages. I would not assume it gets through Google without testing it first. GitHub Actions has the same cold-profile problem, plus an Azure IP shared with many other automated jobs.

Fingerprint hygiene

If your success rate is good, stop here. Every stealth tweak you add is one more thing that can be inconsistent with the rest.

If you get blocked more than you’d expect, look at what a detection script sees before guessing. Point the browser at CreepJS or browserscan.net and take a screenshot. They show whether navigator.webdriver is set, whether the timezone matches the IP, whether the canvas fingerprint looks like a headless machine. Patchright passes both with the configuration above.

Some things that help:

  • Don’t override the user agent in the browser target. Chrome sends its version in many places, like the sec-ch-ua headers and navigator.userAgentData, and a spoofed user agent that disagrees with them is a stronger signal than the real one. This is different from the fetch() target, where there’s no browser and the header is all the site gets.
  • Let the profile age. Visit a few normal pages when you first create it.
  • Keep Chrome updated.
  • Use one profile per IP. Don’t share a profile across different proxy exits.

One reply suggested Brave instead of Chrome. Brave has fingerprint randomization and an ad blocker built in, and uses less memory because it doesn’t load ads. It’s Chromium-based, so executablePath: '/usr/bin/brave-browser' is the only change. I haven’t tested it against Google. Brave randomizes canvas and audio values per site, which is good for privacy but I’m not sure it’s good for scraping, where you want to look like an ordinary Chrome. Try it if Chrome stops working for you, and measure.

Proxies, only if you need them

In the tweet thread, the first attempt with residential proxies failed. Those IPs had already been used for scraping, and Google blocked more than half of them right away. A second provider worked, and it stays unnamed in the thread, because once a provider becomes popular with scrapers its IPs get blocked too.

My advice is to run without proxies first. Measure your success rate from your server’s IP for a week. One clean datacenter IP with good pacing is often enough for a few thousand searches a day, and proxies add cost, complexity, and another thing that can fail.

If you do need them, here is what’s on the market.

Datacenter proxies are cheap and fast, and Google has usually flagged them already. Your VPS IP is one of these. Buying more of them rarely helps with Google. They work fine with most other sites.

ISP proxies, also called static residential proxies, are IPs hosted in datacenters but registered under a consumer ISP. They look residential, they don’t rotate, and they cost a few dollars per IP per month. Good for a small pool you keep warm.

Rotating residential proxies send your traffic through real home connections. You pay per gigabyte, usually between $1 and $8 per GB depending on the provider and the volume. This is where the $1/month in the tweet comes from: text-only pages through a metered pool. You get a different IP on every request, or a “sticky” session that keeps the same IP for a few minutes. The next section is about these.

Mobile proxies exit through mobile carrier networks. Thousands of real users share each IP, so blocking one is expensive for Google. They’re also the most expensive to buy.

IPv6 proxies are a cheap niche. Google serves IPv6, and a /64 block gives you an enormous number of addresses for a few dollars. One reply in the thread recommended them for search scraping. They stop working the moment the target rate limits by prefix instead of by address, so I’d treat them as a bonus, not a plan.

Playwright accepts a proxy in the launch options:

const context = await chromium.launchPersistentContext('./profile', {
  channel: 'chrome',
  headless: false,
  proxy: {
    server: 'http://gate.provider.example:7777',
    username: process.env.PROXY_USER,
    password: process.env.PROXY_PASS,
  },
})

Most providers put the rotation and geography settings in the username, something like user-country-de-session-abc123. Check their docs. Use a sticky session for the duration of one search, so the page load and the redirect resolution come from the same IP.

And match your fingerprint to the exit IP. If the proxy exits in Germany, set locale: 'de-DE' and timezoneId: 'Europe/Berlin'. A browser that says it’s in Rome while its IP is in Frankfurt is what detection scripts look for.

Several replies under the tweet brought this up, and I think it’s the least understood part of the whole topic, so let’s spend some time on it.

What they are

When you send a request through a residential proxy, the request leaves the internet from someone’s home connection. Google sees the IP of a Comcast or Vodafone customer instead of your server’s. Blocking that IP would block a real customer, so Google is careful with it, and that caution is what you pay the proxy provider for.

The provider gives you access to a pool of millions of these home connections and charges you per gigabyte. What I wanted to know is whose homes those are.

Where the IPs come from

For a home device to work as a proxy exit, it needs software that accepts connections and forwards them. There are four ways that software ends up on a device. The FBI described most of them in a 2026 alert that is worth reading.

Pay-to-share apps. People install an app like Pawns.app or Honeygain, agree to share their unused bandwidth, and get paid a few dollars a month. The person knows what the app does, agreed to it, and gets something for it. This is the clean case.

SDK partnerships. The proxy company pays app developers to include a proxy SDK in their apps, a free game, a free VPN, a flashlight app. Users accept terms of service that mention bandwidth sharing somewhere. There is consent on paper. Whether the person understood what they accepted is another question. Most of the big pools are built this way.

Free VPNs with hidden terms. Same mechanism, with worse disclosure. The VPN is free because the company sells access to your IP.

Compromised devices. Malware on routers, TV boxes, digital picture frames, and Android devices sold already infected. Nobody consented to anything. This is a botnet sold as a proxy service.

The last one is real. In May 2024 the US Department of Justice took down 911 S5, a proxy service built on about 19 million infected computers in almost 200 countries. It was sold as a residential proxy network. Its customers used it for fraudulent claims, bomb threats and financial crimes, from IP addresses that belonged to people who had no idea.

Using a residential proxy is legal in most countries, as long as the network got its IPs with consent and you use it for something legal. Reading public search results through someone’s opted-in connection is not a crime anywhere I know of. I’m not a lawyer, and this varies by country, so check the rules where you live.

What can get you in trouble:

  • Buying botnet traffic. If the provider’s IPs come from malware, you’re paying to send your requests through hacked devices. In most places that’s the provider’s crime and not yours, but you’re funding it, and if the network is seized your customer records are seized with it.
  • What you do through the proxy. The proxy doesn’t change the nature of the request. Scraping personal data without a legal basis is a GDPR problem with or without a residential IP. Getting at content behind a login you don’t own is unauthorized access with or without one.
  • The target’s terms. Google’s terms forbid automated access. A residential IP makes you harder to detect. It doesn’t change the terms.

How to tell a clean provider from a bad one

Ask where the IPs come from. A good provider explains its sourcing model, shows you the opt-in screen users see, and offers a data processing agreement. If all you get is “millions of ethically sourced IPs” with nothing behind it, I’d move on.

Price says something too. The person sharing bandwidth gets paid, the app developer gets paid, and the provider keeps a margin. If a pool costs much less than the market, somewhere in that chain somebody isn’t getting paid, and it’s usually the person whose connection you’re using.

My take

I wouldn’t add residential proxies to one of my products without knowing how the pool was built. The $1 a month in the tweet is real. It’s also a dollar spent on someone else’s internet connection, and that person may or may not know about it. I don’t think that makes it wrong. I think it should be a conscious decision rather than something you skip past because the price is low.

If I needed more IPs than one server gives me, I’d start with ISP proxies. They cost more per IP, but they’re hosted in datacenters under consumer ISP registrations, and no home connection is involved. If those didn’t work, I’d pick a residential provider that runs a pay-to-share app, where the person on the other end signed up for exactly this. And I’d keep the resource blocking from the Google target, because every byte through that pool is billed to me and taken from someone’s connection.

The alternatives

We built the tool. Let’s also look at when you shouldn’t.

You send a URL and they return the HTML or parsed JSON. They deal with browsers, proxies and CAPTCHAs. Prices checked on September 9, 2026.

ScrapingBee sells credits. The Freelance plan is $49.99 for 250,000 credits and the Business plan is $249.99 for 3 million. A Google search costs 15 credits, so about 16,000 Google searches on the smallest plan and 200,000 on Business.

SerpApi specializes in search engines and returns structured JSON, with the knowledge panel, the local pack and the ads already parsed. You get 250 searches a month for free, then $25 for 1,000 and $275 for 30,000. Expensive per search, but you write no extraction code.

DataForSEO, Bright Data, Serper and many others sit in between, cheaper per query at volume and with more setup.

These win when you need many different sites rather than one, when your volume swings a lot, when your free time is worth more than $100 a month, or when you’re still figuring out what to scrape and want to be able to change your mind. Compare the cost per usable result and the maintenance time, not just the monthly price.

Search APIs

If you need search results and not Google’s page specifically, a search API is legal, stable, and often cheaper than a scraper once you count your time.

Brave Search API uses Brave’s own index, not Google’s. It costs $5 per 1,000 queries, and you get $5 of credits free every month, about 1,000 searches. You need a credit card to sign up. The results differ from Google’s, and for many uses that’s fine.

Google’s own Custom Search JSON API is closed to new customers and shuts down on January 1, 2027. Microsoft retired the Bing Search API in 2025. Brave is the only major independent index left with a self-serve API, and I think this is part of why Google scraping is so common.

Open source SERP projects

OpenSERP is a self-hosted SERP API written in Go. It drives a headless browser, and as we saw above, headless means a CAPTCHA on Google today. It still works with Bing, DuckDuckGo and the other engines it supports, which block much less.

ddgs is a Python metasearch library that queries Bing, Brave, DuckDuckGo, Google, Yandex and others with plain HTTP and TLS fingerprint impersonation, no browser. It includes an API server and an MCP server. It’s fast and light, and it works until an engine changes its endpoints, which happens. The README says it’s for educational purposes only, and that describes its stability well.

Scrapling is a Python scraping framework with stealth fetchers built in, from plain requests up to a patched browser, and selectors designed to survive layout changes. If you work in Python, it covers most of what we built here.

I’m not a lawyer, so this is the practical version.

Public pricing pages are the easy case. The data is public and it isn’t personal, and a request every few seconds costs the site nothing. Read robots.txt anyway and respect the Disallow rules for the paths you visit.

Google’s terms of service forbid automated access. Scraping search results violates the terms, but it isn’t a crime, and Google’s response is a CAPTCHA, not a lawsuit. Companies have done it at scale for twenty years. You decide how much risk you accept. Airbnb and Amazon are similar, with one difference: those pages contain other people’s listings and reviews, and storing that brings its own rules.

For any site, don’t collect personal data without a legal basis to keep it. In Europe GDPR applies as soon as you store a name. Don’t scrape behind a login that isn’t yours. And pace your requests so you never slow the site down for its real users. A request every few seconds is invisible to any server. A thousand a minute is an attack.

I wrote about the other side of this in how to handle AI crawlers in robots.txt. A good scraper behaves like the kind of bot you’d accept on your own site.

What I did, and what I would do

For this post I built and ran the Node.js tool from my laptop on September 9, 2026. The DigitalOcean target pulled seven plans in 228 ms on the first call and 7 ms from the cache. For Google I ran the four tests I described, curl, headless Chromium, stock Playwright with real Chrome, and Patchright with real Chrome. The first three failed and the fourth worked on every query I tried, so that’s what the target uses.

The pricing tool is the one I’m keeping. I’d add a target for each provider on hostingpicker.dev and paymentprocessor.dev, run it once a day from cron, and have it compare the new numbers with the stored ones and message me on Telegram when something changes. Most of those pages are plain HTML like DigitalOcean’s. A few, like Hetzner’s, need a headless browser. None of them needs Patchright, a persistent profile or a proxy.

I don’t run a Google scraper in production. If I needed search data for one of my sites I’d start with the Brave Search API. $5 a month covers a thousand queries and there’s nothing to maintain. I’d move to the Google target only if I needed Google specifically, or if the API bill grew too much, and I’d run it on a VPS rather than at home because I don’t want my own IP involved.

Where this tool doesn’t fit is anything that needs thousands of Google searches an hour. At that volume you’re running a proxy operation, which is a full-time job with a bill that grows with your success. Pay for a SERP API instead.

The maintenance question

“How long will this keep working?” was the top question under the tweet. Nobody knows, including me.

DigitalOcean will redesign its pricing page at some point and the selector will break. Google will change the results page. Chrome will update and Patchright will need a few days to catch up. Your IP will have a bad week. I still think the tool is worth building, as long as you build it knowing these things will happen, which is why we ended up with one small module per target, a check that turns empty results into an error, pacing set per target, a cache so most requests never leave the machine, a log, and an alert. I’d also keep a paid API key around for the week you don’t have time to fix things.

For sites that don’t fight back, you don’t need anything else. For Google, in my tests the difference came down to a real browser controlled without leaks, a profile that persists between runs, and slow enough pacing.

A prompt for your coding agent

The tweet that started this post was written as a prompt for a coding agent. Here is a more complete one, based on everything above. Paste it into Claude Code, Cursor or Codex, and compare what comes out with this post.

Build a self-hosted scraping tool in Node.js 24+.

- Each target is a module in targets/ exporting `scrape(params)` that returns
  data or throws, and an optional `pause()` returning milliseconds to wait
  after a run. Treat an empty result as an error.
- Target `digitalocean`: fetch https://www.digitalocean.com/pricing/droplets
  with a Chrome user-agent header, parse the first table with cheerio, return
  one object per row (memory, vcpu, transfer, ssd, hourly, monthly).
- Target `google`: use the `patchright` npm package (patched Playwright, same
  API) with `chromium.launchPersistentContext('./profile', { channel: 'chrome',
  headless: false, viewport: { width: 1366, height: 850 }, locale: 'en-US',
  timezoneId: '<my timezone>' })`, launched once and kept open. Never override
  the user agent. Block image, media and font requests with context.route().
  Navigate to https://www.google.com/search?q=<q>&hl=en, click an "Accept all"
  button if present, wait for #search. Treat a URL containing /sorry/ as a
  block. Extract `#search a:has(h3)`: title, href, the closest [data-hveid]
  block's `cite` text, and snippet. If href contains /goto?, resolve it with
  context.request.get(href, { maxRedirects: 0 }) and use the Location header.
  pause() returns a random 20–40 seconds.
- Expose GET /scrape?target=<name>&...params on port 3100, bound to
  127.0.0.1, protected by a Bearer token from API_TOKEN. Serialize all scrapes
  through a single promise queue, waiting the target's pause() (default 5 s)
  after each one.
- Cache results per target+params for 24 hours in SQLite using node:sqlite.
  Log every run (timestamp, key, ok, error) in the same database.
- After 3 consecutive failures, send a Telegram message via the Bot API using
  TELEGRAM_TOKEN and TELEGRAM_CHAT_ID.
- Provide a systemd unit that runs it as a non-root user with Restart=always
  and an EnvironmentFile for secrets, and a variant wrapped in
  `xvfb-run -a -s "-screen 0 1366x850x24"` for the browser target.

Then add a target for a site you actually need, look at the log after a week, and decide from there whether you need anything from the second half of this post.

Tagged: Node.js · All topics

Want me to talk about your product? You can sponsor this site.

~~~

Related posts about node: