Serving my site as Markdown to AI agents

AI agents read websites all day now. ChatGPT, Claude, Perplexity, and a hundred smaller tools fetch pages, parse them, and use what they find.

But they don’t really want my HTML. An HTML page is full of things they don’t care about: the header, the navigation, the footer, scripts, styling, the ebook box at the bottom of every post.

What they want is the content. And the cleanest way to hand someone content is Markdown.

So I made every post on this site available as Markdown. Append .md to the URL and you get it:

curl https://flaviocopes.com/npm.md

Here’s why I did it, and how.

Why Markdown

The first reason is tokens. LLMs read text in tokens, and tokens cost money. An HTML page is mostly noise: tags, classes, inline styles, scripts. The same content in Markdown is a fraction of the size, so it’s cheaper and faster to process. I built a free token cost calculator that counts the tokens in a piece of text and shows what it costs to send to each model.

The second is focus. Markdown contains the article without the menu, footer, or “subscribe to my newsletter” box.

If I want agents to use my writing, I should give it to them in the shape they like.

My first attempt: converting at the edge

My first version of this was fancier.

This site is hosted on Cloudflare Pages, and I wrote a middleware function that ran in front of every request. If the request carried an Accept: text/markdown header, the middleware took the HTML page and converted it back to Markdown on the fly, at the edge.

It worked. The same URL served two formats with proper content negotiation, and I was proud of it.

Then I looked at my usage stats.

A root middleware on Cloudflare Pages runs in front of everything. Every HTML page, every image, every CSS file. Each one of those became a metered Workers invocation. My fully static site, the one that cost nothing to serve, was suddenly burning through the free Workers quota just to serve images.

My posts are written in Markdown. Astro converts them to HTML at build time. I was paying to convert that HTML back to Markdown on every request.

That extra conversion made no sense, so I removed it.

The better way: just publish the source

The Markdown already exists, so I only have to publish it.

Astro can generate more than HTML pages. An endpoint is a .ts file in src/pages/ that returns whatever you want at build time. Name it [slug].md.ts and it generates one .md file per post, right next to the HTML version.

The endpoint itself is short:

import { getCollection } from 'astro:content'

export async function getStaticPaths() {
  const posts = await getCollection('posts')
  return posts.map((post) => ({
    params: { slug: post.id },
    props: { post },
  }))
}

export async function GET({ props }) {
  const { post } = props
  return new Response(post.body, {
    headers: { 'Content-Type': 'text/markdown; charset=utf-8' },
  })
}

post.body is the raw Markdown of the post, before any HTML conversion. I return it, and Astro writes it to dist/<slug>.md at build time.

The real file does a bit more. It adds the title, description, and canonical URL at the top. It also rewrites relative links to absolute ones so an agent can follow them.

The result is a folder of plain static files. Cloudflare serves them from the edge like any other asset, with no Workers invocations, conversion, CPU budget, or cost.

My advice is to publish the original when you already have it, instead of converting it back at request time.

Helping agents find it

Agents still need a way to discover these files, so I added three pointers.

First, a <link> tag in the head of every post:

<link rel="alternate" type="text/markdown" href="/npm.md" />

This is the standard way to say “this page also exists in another format”. It’s how browsers discover RSS feeds.

Second, an llms.txt file. It’s an emerging convention (see llmstxt.org): a curated Markdown index at the root of your site that tells LLMs what’s here, section by section. Mine lists the main topics, the courses, the books, and the tools. A second file, llms-full.txt, lists every single post with its description, each one linking to its .md version.

Third, robots.txt. Mine now explicitly welcomes the AI crawlers: GPTBot, ClaudeBot, PerplexityBot, and friends. A plain User-agent: * already allows them, but being explicit costs nothing and removes any doubt.

One warning if you’re on Cloudflare: check your dashboard. Cloudflare can block AI crawlers at the edge, and on newer zones it does so by default. If that switch is on, your robots.txt is irrelevant, because the bots never get through.

Try it

It works on every post. Pick any URL and add .md:

curl https://flaviocopes.com/javascript.md

You’ll get a clean Markdown document, title and all. The same file I wrote, more or less.

There’s a built-in way too

I’ll be honest with you. Cloudflare has a feature called Markdown for Agents that does this at the zone level, no code at all. On the Pro plan and above, you enable it in the dashboard under AI Crawl Control. When a request carries Accept: text/markdown, Cloudflare converts the HTML to Markdown at the edge and returns that instead.

That’s the same content negotiation I tried to build with middleware. Cloudflare runs it for you only when an agent asks for Markdown, not on every image and CSS file. I’m on the free plan, so I didn’t use it.

But I’m glad I didn’t. For a static site built from Markdown sources, the static approach is better anyway: no conversion step means nothing to get wrong, and no per-request cost means nothing to meter.

Want me to talk about your product? You can sponsor this site.