AI-Written Code: What Stops It Shipping Bugs

We use AI coding assistants every day, on client websites and on the six apps we build and run ourselves. The first thing that changed was not the quality of the code. It was where the work sits. Producing a change got cheap. Reading it did not. A diff that used to take an afternoon to write now arrives in minutes, and it still takes a careful person as long to read as it ever did.

A small studio cannot hire its way out of that. So we stopped trying to read harder and moved as much of the reading as we could into checks that run without us. Most of them live in the repository behind this website, which is why this post can be specific. Below is what they are, with real examples, and the rule we use to decide whether a new check is worth writing.

The rule: a check has to know something the code does not

Ask an assistant to test code it has just written and it will usually write a test that agrees with the code. That is less a flaw in the model than a limit of its evidence: the code is all it has. A check built that way passes by construction. The checks that have earned their place in our workflow take their expected answer from somewhere the generator could not see: something the client confirmed, a file the business owns, the compiler, the page a visitor actually receives. If the model could not have known the right answer, the check can catch it giving the wrong one.

What the client confirmed becomes an assertion

Every case study on this site has its own test file, and the most useful lines in them are negative. On one of our client projects, a WordPress-to-Webflow migration, the case page's test asserts that its text contains no percentage, no "traffic grew" and no "more leads". The client gave us no such figures, so the page may not have any. Our own product pages assert the same about audience, rating and revenue numbers. Those are exactly the numbers a model reaches for when it drafts case copy, because nearly every case study it has read contains them.

The Vines property case is the sharper example. The campaign we worked on sold all 10 homes, and the page says so. The client's video testimonial, quoted word for word on the same page, mentions 18 homes. Both are on the page for a reason, and an assistant tidying the copy would quite reasonably "fix" one number to match the other. The test counts every mention of "18 homes" and fails unless each one sits inside the quotation.

Each fact lives in one file, and every copy answers to it

Our prices appear in a lot of places: service pages, FAQ answers, the cost guides on this blog. They are stored once, in own-prices.ts. A test reads every service page and every blog sentence that speaks for us (it looks for "we", "our" or "99 Francs") and compares each of our prices it finds with that file. Change a price and the test prints every sentence that now disagrees. Competitor prices in the same articles are left alone, because they do not carry the marker.

This matters more with an assistant in the loop than without one. Ask a model to draft a cost guide and it will produce a confident, plausible price for a landing page. Plausible is the problem: a wrong number that looks right gets through a human skim. A regular expression does not skim.

Positioning copy works the same way. The homepage's short answer about what the studio does is pinned by a test, and the same test lists phrases that must never appear there, such as "unlimited design service" and "no contracts". They are the stock phrases of our category, and generated copy drifts back towards them the moment nobody is looking.

Gates that refuse a release before anyone sees it

• Types. Our unit test runner does not type-check, but the production build does, and a type error fails the deploy. So npx tsc --noEmit runs before every push. Generated code often works on the one path it was tried on and breaks a type somewhere else.
• Required CMS fields. Blog posts are written in TypeScript and mirrored into Payload CMS on every deploy, before the server starts. An empty FAQ answer or table header fails Payload's validation, the server never comes up and the healthcheck fails. A test mirrors every required field in the schema, so the empty string is caught on a laptop rather than on Railway.
• Dates. Each page's sitemap date is set by hand when its copy changes, and a test rejects dates in the future, measured in UTC. We work from Cyprus, three hours ahead, so a change made just after midnight and dated "today" fails until UTC catches up. Annoying, and correct.
• Generated files. Our llms.txt is built from the sitemap rather than written by hand, and a test checks that every service, industry, case study and series is listed and that the file stays under 30 KB.

When a gate does trip at deploy time, the previous release keeps serving. That is how our host behaves, not something we invented, but it turns a mistake from an outage into a failed-deploy notice and a fix.

Make the worst mistake impossible, not discouraged

On 23 July 2026 a deploy of this site hung and was killed. The cause was a local development server. The local environment file pointed at the production database, the CMS in development mode pushed its schema there and left a marker row, and the next production migration stopped to ask an interactive "data loss, proceed?" question that nobody was there to answer.

A line in the docs saying "don't run the dev server" would not have been enough. Assistants run commands as readily as people do, and a line in the docs is a suggestion. The fix was a separate script, dev:local, that sets a throwaway local database address before the app starts. Variables set inline win over the environment file, so production is unreachable from that process. We still document the rule; we just no longer rely on anyone, human or model, remembering it.

Check what the visitor actually receives

Tests run against the source. After each release we also run npm run seo:live, a script that requests key pages from production and checks status codes, canonical URLs, robots tags, structured data and the document counts in the public CMS API. The source can be right while the deployed page is wrong, and only a request to the live site shows which.

We learned why on the Markdown layer this site serves to AI agents: for six weeks, every agent request returned our homepage while every browser check passed, because browsers never send the request agents send. Since then, a check on any feature has to make the same request the feature's real users make.

Where the person sits

None of this replaces a reviewer. It changes what the reviewer reads. When the checks are green, the diff that reaches one of the founders is about intent: is this the right change, does the copy sound like us, would the client sign it off. The assistant drafts and a person decides. Nothing is pushed automatically, and files are added to a commit by name rather than all at once, because the working tree often holds unfinished work that is not ready to ship.

What this does not catch

Checks only know what we told them. A test that pins a price does nothing for a sentence that is accurate and badly argued. A layout that is consistent and dull passes every check we have. And content tests fail on purpose whenever copy changes, so every legitimate edit also means updating a test. We accept that friction. It costs less than a client finding a wrong price on their own page.

If you are hiring a studio, it almost certainly uses AI by now, so whether it does is not the useful question. Ask which checks its work has to pass that neither the person nor the model who wrote it could quietly bend. This is how we run web development and React and Next.js builds, and the same discipline applies when we take over a live site under website support. The client-facing side of the same rule is in where we use AI in client work, and where we don't, and if you are on the buying side, use our checklist for spotting AI slop from an agency.