I had Claude Code, an AI coding agent, move this blog from JavaScript-rendered pages to static HTML. It was a real job: new build tool, new templates, five posts converted, redirects for old links, a sitemap, a feed and structured data.
When it finished, it handed me a list of checks, and every one was green:
- The site built without errors.
- A link check over 209 internal links found nothing broken.
- Lighthouse scored 100 for SEO and accessibility on four pages.
- The sitemap, the feed and every block of structured data validated.
Then I opened the extensions page in a browser. All three of my extensions, already live in the Chrome Web Store, were sitting under a heading that said "Coming soon."
The bug
The extensions page splits extensions into two groups based on a status field. The template looked like this:
{% set live = extensions | selectattr("status", "equalto", "live") %}
{% set soon = extensions | selectattr("status", "equalto", "soon") %}That reads correctly. In Jinja, the Python template language Nunjucks is modeled on, it does exactly what it says. In the version of Nunjucks this site uses, selectattr ignored "equalto" and "soon" and kept every extension that had any status at all. All three do, so all three landed in "Coming soon."
The "live" list had the same bug. It came out right by luck, because every extension happens to be live.
The fix was a four-line filter that does an exact match:
// Nunjucks' selectattr ignores a comparison value and only tests truthiness
eleventyConfig.addFilter("where", (items = [], key, value) =>
items.filter((item) => item?.[key] === value)
);{% set soon = extensions | where("status", "soon") %}Why every check missed it
Each tool answered its own question correctly.
Lighthouse asked whether the page was accessible and readable by search engines. The "Coming soon" section had a proper heading, good contrast and clean markup, so it was. The link checker asked whether every link pointed somewhere real, and they all did. The structured data validator asked whether the JSON had the right shape, and it did.
No tool asked whether Pageful is coming soon. Answering that takes someone who knows Pageful is already in the Chrome Web Store. Automated checks tell you the code is well-formed. Whether it says the right thing about your content is a question only you can answer.
The bugs the agent caught itself
To be fair to the agent, it caught several problems on its own. In each case the tooling reported success, and the agent read the actual output instead of trusting the green light.
Relative canonical URLs. The build passed. When the agent opened the generated HTML, every canonical link read /blog/... instead of https://jarednewnam.com/blog/.... A plugin had registered a filter with the same name as one the agent wrote, and the plugin's version quietly replaced it.
A page it never touched. When it moved the site's header and footer into templates, it deleted the old includes.js script that used to draw them. A separate landing page still loaded that script. Nothing failed at build time, because the build never looks at that page's script tags. The link checker found the missing file.
Testing the wrong server. Stopping a local test server left its process running in the background, still holding port 5000. The next five experiments all sent requests to that old server, not the new ones. They "passed" and "failed" in ways that meant nothing. The agent noticed a log line saying port 5000 was taken, threw those results out and tested again on fresh ports.
In all three, a success message meant less than it seemed to.
What I check now
1. Look at the page
Open every page the change touched, in a real browser, and read it like a visitor. It takes two minutes and it's the only check that would have caught "Coming soon." If the change touched a component that appears on several pages, look at each one.
2. Read the output as well as the source
The source told me the template filtered by status. The output showed three live extensions under "Coming soon." For a static site, that means opening the built HTML. For an app, it means the rendered page or the API response. View source on one page and read the <head>: title, description, canonical link and share tags.
3. Ask what a passing check would miss
For each green check, ask: if this feature were broken in a way that matters, would this check fail? For the "Coming soon" bug, every answer was no. When the answer is no, the check isn't evidence for that part of the change.
You can turn the answer into a check. A few lines after the build would have caught this one:
// After the build: no live extension should be listed as coming soon
const html = fs.readFileSync("_site/extensions/index.html", "utf8");
const soonSection = html.split("<h2>Coming soon</h2>")[1] ?? "";
for (const ext of extensions.filter((e) => e.status === "live")) {
if (soonSection.includes(ext.name)) {
throw new Error(`${ext.name} is live but listed as coming soon`);
}
}4. Search before anything gets deleted
An agent will delete a file because its task says the file's job is done. Before you accept a deletion, search the whole project for the file name:
git grep -n "includes.js"Any result outside the files being changed needs an answer before the delete goes in.
5. Know what you're testing against
Which server answered the request? Which build is it serving? Is anything cached? A stale process, an old build folder or a cached service worker can make a broken change look fine, or a fixed one look broken. When results don't make sense, check the environment before you check the code.
6. Get a plan before any code
Before the agent wrote anything, I asked it to read the code and come back with how the blog worked, every post and its slug, and a proposed folder layout, then wait for my OK. That plan caught a conflict early: the folder I wanted for templates already held source code for another part of the site. Fixing it in the plan took one sentence. Fixing it after twenty files had been written would have meant moving them all.
7. Write the rules down before you start
Most of what made this migration reviewable was in the instructions I gave the agent before it touched anything:
- Check that the working tree is clean, and stop if it isn't.
- Work on a new branch, and don't merge, push or deploy.
- Read the existing code and report back before changing anything.
- Ask before adding any dependency beyond the one plugin I allowed.
- Don't change the homepage copy, the H1 or how I describe my role.
- Ask before guessing on anything the code doesn't make clear.
- Commit in logical chunks with clear messages.
Each rule exists because of a way an agent can quietly go wrong. The clean-tree check stopped it on step one: I had uncommitted changes, so it reported them and waited instead of mixing my work into its commits. The "ask before guessing" rule is how it ended up asking me which of two Chrome Web Store links for Pageful was correct, rather than picking one.
The copy rule matters too. Agents like to tidy, and a tidy rewrite of your homepage headline is still a rewrite you didn't ask for.
8. Review one idea at a time
The migration landed as a handful of commits, each with one job: build setup, blog, homepage and extensions page, sitemap. Small commits are easier to review and easier to undo. When the "Coming soon" fix came, it was one more small commit with a message that explained the cause.
If you're new to this
AI agents produce plausible code fast, and the plausible part is where the risk sits. Code that looks right and runs without errors invites you to stop looking. The selectattr line above would pass most human code reviews too, because it reads exactly like what it's supposed to do.
You don't need years of experience to catch this kind of bug. You need to know what the page is supposed to say, and to go look at it.
If you've been doing this a while
The checks that matter most can't be automated without knowing the content: whether a page says the right thing, whether a deleted file was still in use, whether a test ran against the code you think it did. Agents are good at the mechanical checks, and often better than people at running all of them every time. Put your review time where they can't help.
Before you merge the next change an agent hands you, open every page it touched and read them.