One URL, Two Formats: How Our Site Serves Markdown to AI Agents

This is Part 3 of our AI Search Experiment. Part 1 documented why 99francs.agency was invisible for the categories we wanted to own and what we restructured. Part 2 measured the first 31 days: impressions up 712%, three attributed leads, one of whom told us they found us through an LLM. Part 3 is the infrastructure piece — making every page readable, cheaply, by the machines that are starting to send us clients.

Why serve Markdown to AI agents at all?

Agents fetch pages to read them, not to render them. When ChatGPT, Claude or a browsing agent pulls one of our articles, everything except the words is overhead: the navigation, the footer, the React hydration payload, the inline styles. For a model with a context budget, a 130 KB HTML page is mostly noise wrapped around 13 KB of signal. Serve the signal alone and you are a cheaper, cleaner source to read — and, we suspect, an easier one to cite.

The mechanism is not new. Content negotiation shipped with HTTP/1.1 in 1999: a client states what format it prefers in the Accept header, and the server picks the best representation of the same resource. Browsers ask for text/html and get HTML. Agents that ask for text/markdown get Markdown. Same URL either way — which matters, because the URL in a citation is the URL a human clicks. A separate /markdown/ mirror would split that identity in two; negotiation keeps one canonical address for both audiences.

How it works: a proxy, a converter and a guard header

The implementation is three small pieces in a Next.js 16 app — the same app that runs this site, with Payload CMS underneath.

• The proxy intercept. Next's proxy (the middleware convention) checks three things on every request: it is a GET, the Accept header contains text/markdown, and the path is a public content page — not admin, not API routes, not files with extensions. When all three hold, it rewrites the request to an internal converter route and passes the original path along.
• The converter route. It fetches its own site — the same page, over HTTP, asking for text/html — and converts what it gets. Self-fetching sounds indirect, but it means the Markdown always reflects exactly what a browser would see: rendered CMS content, current prices, FAQ blocks, everything. There is no second content pipeline to keep in sync.
• The conversion. node-html-parser plus node-html-markdown, with script, style and noscript stripped. Each page marks its content zone with a data-md-root attribute, so the converter takes the article and leaves the chrome; it falls back to body when the marker is absent. The output gets the page title as an H1 and a Source line with the canonical URL.
• The advertisement. Every content page sends an RFC 8288 Link header announcing the canonical URL, the Markdown alternate at the same address, and /llms.txt — so an agent that lands on the HTML can discover the cheaper format on its own.

One obvious question: if the converter fetches its own site, and the proxy rewrites Markdown requests to the converter, what stops the loop? A guard header. The converter's self-fetch carries x-md-render: 1, and the proxy lets any request with that header pass through as normal HTML. Without it, the converter would fetch itself, which would rewrite to the converter, which would fetch itself — the kind of bug you only make once.

What broke in production

Now the honest part. We shipped this in late June. On August 2 we ran a routine check before writing this article — asked for a blog post as Markdown, from production, the way an agent would. We got the homepage. We asked for a services page. The homepage. Every negotiated request, for every path on the site, had been returning the homepage for about six weeks.

That table is the whole diagnosis. The converter worked when called directly, so the conversion was fine. It failed only behind the proxy rewrite — which meant the path was getting lost between the two. The converter read its target path from a query parameter that the proxy set on the rewrite URL. But after a rewrite, the route handler's request URL can reflect the original request — the one the client sent, which naturally carries no such parameter. The converter found nothing, fell back to its default of the homepage, and faithfully converted it. Every request, every path, the same wrong answer.

The fix is three lines: pass the path in a request header as well, because request headers survive a rewrite unconditionally, and keep the query parameter for direct converter calls. Shipped as commit 2cef139, verified in production ninety seconds later. The same audit surfaced a second, quieter problem: one URL now serves two formats, so any shared cache must key its entries on the Accept header — otherwise a Markdown body can be cached once and served to browsers, or the reverse. The negotiated response now sends Vary: Accept. Its HTML sibling still does not — Next.js overwrites middleware-set Vary headers on app-router pages — and since live cache behaviour tested correct in both directions, we chose to document that residue rather than fight the framework for a header.

The uncomfortable lesson is not the bug — it is that nothing caught it. Every browser test passed for six weeks, because browsers never send Accept: text/markdown. The only clients that saw the failure were the agents the feature was built for, and agents do not file bug reports; they just quietly read your homepage, conclude you are a design-subscription landing page with no articles, and move on. It even explained a mystery from July: an LLM fetch tool kept seeing our homepage wherever we pointed it, and we blamed the tool. If you ship an agent-facing surface, test it with the agent's request, not with yours.

What does llms.txt add on top?

Content negotiation answers one question: give me this page, in the format I read best. llms.txt answers the one before it: what pages exist, and which ones matter. Ours is a generated route, not a hand-maintained file — it queries the same content layer as the site, so every published page appears with a one-line description, and nothing goes stale when we publish. It opens with a short paragraph stating what the company does, the founding year, the location and the two engagement models — the facts an answer engine needs to describe us correctly — then maps the site: services, case studies, articles, each with its canonical URL.

Use this file as a concise map of the most important pages on 99francs.agency. For crawl permissions, use robots.txt. For full URL discovery, use sitemap.xml.

our llms.txt, on the division of labour between the three files

Do agents actually read any of this?

Honest answer: the evidence is early, and we will not dress it up. The strongest signal is in Part 2: of the three qualified leads attributed to the first 31 days of this experiment, one told us directly they found 99 Francs through an LLM. That proves the acquisition path exists; it does not prove which piece of plumbing carried it. Server logs show agent user-agents fetching llms.txt steadily, but our negotiated-Markdown numbers effectively restart from August 2 — for the six weeks before that, every agent that asked got the homepage, so whatever adoption curve existed, we cut it off ourselves. We are instrumenting negotiated requests now and will report real counts in a later part of this series, small or not.

Adding this to your own Next.js site

The build is an afternoon of work. The testing discipline is the actual deliverable.

• Intercept in middleware or proxy: GET requests whose Accept header contains text/markdown, on public content paths only — keep admin, API routes and anything with a file extension out.
• Rewrite to a converter route, and pass the original path in a request header, not only in the rewrite URL's query string. This is the six-week bug in one sentence: after a rewrite, your handler may see the original URL, and query parameters you set on the destination can be gone. Headers survive.
• Convert by self-fetching your own rendered HTML with a guard header that exempts the fetch from interception, then run an HTML-to-Markdown pass with scripts and styles stripped. Mark the content zone in your layout so the converter takes the article, not the chrome.
• Send Vary: Accept on the negotiated response, and check your cache in both directions after deploying: prime the HTML, request Markdown, then the reverse. Two curls, thirty seconds, and it rules out serving Markdown to your human visitors.
• Advertise the alternate with a Link header and an llms.txt route generated from your content layer. And verify the Link header with a GET — we spent a morning believing ours was missing because the check used a HEAD request, and the proxy only decorates GETs.

If there is one thing to take from this post, it is the last habit we picked up: every check on an agent-facing feature must impersonate the agent. Browsers will tell you the site works. They are not the audience this layer is for.