Test your AI agent before it touches your website

In July we started an internal project with a simple goal: write down how Noco builds and runs websites, clearly enough that a machine could follow the steps. Two months later, everything we called "our process" split into three groups. Work that was easy to hand over and now runs on its own. Work we chose to keep in human hands. And work that was simply missing when we went looking for it. It had never been written down anywhere. It was one person doing a good job, so consistently that everyone assumed a document existed behind them.

The project worked like a stress test. It showed us where the weight was actually being carried, and in a few places it was resting on one person. If you run marketing or ops at a B2B SaaS company and you are about to point an agent at your website, this is the check we wish someone had handed us first.

‍

Why does writing your process down matter now?

Every platform now gives you a blank box for your instructions, and what you put in it decides what the agent builds. The AGENTS.md standard became an open specification in August 2025. By December, more than 60,000 open-source projects had adopted it, and over twenty AI tools could read the format. Anthropic released its Skills format as an open standard the same year, and it now works across Claude Code, OpenAI Codex, Cursor, Gemini CLI and more. Website platforms ship the same layer under their own names.

The box is standard, and it starts empty. Fill it with your own rules and the agent follows your standard. Leave it empty and the agent falls back on the platform's defaults, which are the same defaults every competitor on that platform gets. Over time your website drifts toward the platform's house style. We made the platform side of this argument in our piece on Framer 3.0 agents: an agent picks up the build habits of whatever it runs on. The agency side is blunter. An agent also picks up the gaps in your documentation, and it does it quietly.

So we set out to close the gaps. Here is what happened.

‍

Which parts of the work were ready to automate?

The parts tied to sales and content were the easiest to hand over. Within weeks, these were running, each with a start date:

One result is now standing practice: from day one, we baseline and track every customer's organic traffic, AI-referred traffic, conversions and real-world load speed, using independent sources and an archive capture taken before we start. We can do that every month because the process behind it was written clearly enough for a machine to run it.

The most useful decision came out of this group. We store our pipeline prompts as pages in our knowledge base and load them at run time. Editing a prompt changes the next run. No deploy, no engineer in the middle, and every edit is saved in the page history. The people who own the craft edit the process themselves. That one decision did more for quality than any tool we bought, because it removed the step where judgment had to be translated into something the system could run.

‍

Which parts did we keep human on purpose?

We kept design and development work in human hands on purpose. Our internal rules say the automation layer "should not become the execution agent for design or development by accident." Publishing access is limited to our own site, draft only, and never touches a customer site. We parked one convenient piece of automation in July and wrote down why: shipping it would break a standing governance rule, and the rule has to be changed before the exception can exist.

From the outside this can look like moving slowly. It is the part we trust most. Every one of these choices has a name attached and a date. When we do write down the design and delivery work, it will happen as a decision with a paper trail.

‍

Which steps only lived in one person's head?

The steps that fell apart were the ones we were most sure about. Our own knowledge base has a page with a heading that reads, word for word, "Missing artifacts." Under it: a written QA and launch checklist. A design handoff standard. Build review standards. A CMS mapping standard. A launch readiness check. None of these existed in writing. Every one of those steps happened on every project anyway. The checklist lived in the heads of the people running it.

Design review was the same story. "What makes Noco quality" shows up in our records as missing. As of mid-August, our CRO measurement loop still runs by hand, once a month, which makes it the least documented part of the whole practice, at a company that sells CRO.

The clearest line in our own notes says it plainly. Writing about how we move from sales to delivery, our operating notes admit the real risk: this works because one person makes it work. "The workflow is AI-enabled but personal, not institutional."

We are publishing that line because every agency and every in-house team has one like it, and almost nobody has written theirs down. The person carrying it is usually very good, and that is why the gap stays hidden until you try to hand the work to something that follows instructions literally.

‍

What decides whether an agent can do the work?

One thing separated the work that converted from the work that fell apart: how you check the result. If you can check the output against a written rule, the task can become agent instructions. If you can only check it by taste, it resists. A meta description has a character count. A transcript import either ran or it did not. A hero section has a reviewer, and the reviewer knows quality when they see it.

This split was described decades ago. Research on tacit knowledge, from Polanyi through Nonaka, holds that a large share of what an organization knows is judgment and reading the situation, and that kind of knowledge resists being turned into a written procedure. We ran a two-month test on our own agency and that old prediction held. The sales and content work wrote down cleanly. The taste-based work stayed in people's heads, and the honest response is to give it to a named person on purpose.

For your team, the sort is simple and worth an afternoon. List every recurring website task. Mark each one as checked by a rule or checked by taste. Hand the first list to agents with written instructions. Staff the second list with people, on purpose.

‍

What can you copy on Monday?

These four written rules carried the most weight in our system. All four are cheap to copy, and each one prevents a specific, costly failure.

1. The honesty trap. When we set up a knowledge pack for a customer account, we only accept it after asking the agent five to ten set questions about that account. One is always a trap: a question whose correct answer is a known unknown or an open gap in our records. To pass, the agent has to get nine of ten right with sources, and answer the trap honestly. An agent that makes up a confident answer to the trap fails the whole test. Run this against any agent before it touches your website

2. The absence stub. Every empty page in our system ships with one line: "Nothing recorded yet. Do not treat absence of information as fact." Without it, a machine reading an empty page decides the emptiness means something. One sentence, built into the template, closes a whole class of failure.

3. The search-before-absence rule. Our rules state that the knowledge base's folder tree is only a partial map, and a missing folder is never proof that something does not exist. The agent has to search before concluding something is not there. This rule exists because the machine once drew the wrong conclusion from a tidy-looking file tree. Write yours before the same thing happens to you.

4. The tiered gate. One rule, stated the same way on three separate pages: AI prepares, humans approve. Anything visible outside the company is draft only. Low-stakes internal updates flow automatically into a nightly digest. Important pages keep a manual approval step. Anything the system cannot classify stays gated, to fail safe. The tiers were argued out over weeks, and those arguments are written down next to the tiers.

‍

Where does writing it all down stop working?

Four things pull against everything above, and leaving them out would make this article dishonest.

First, markdown copies perfectly. Anyone leaving can zip the whole folder. The lasting value is in the loop that produced the files: the dated corrections inside them, and the judgment about what to leave out. Our change log has an entry that is openly marked as a correction of the entry above it. The first blamed a bug on an iteration limit. The second reproduced the bug under a live trace and found the real cause: a check that was throwing away a complete item because one display field was missing. The limit was never the problem, because no run had ever hit it. A competitor who copied our prompts would get a snapshot, with no way to produce the next version. That is a real advantage, and a small one. Calling it a moat would oversell it.

Second, instructions get harder to follow as they pile up. Research on the curse of instructions finds that a model's ability to satisfy every directive in a prompt decays roughly exponentially with instruction count: even frontier models follow only a fraction of them once a prompt carries around ten. The IFScale benchmark pushes the count from 10 to 500 and finds the same decay in every model tested, with a consistent bias toward the earliest instructions. Multi-turn work is worse still: once instructions are spread across a conversation instead of stated up front, model performance drops by an average of 39%, and a model that takes a wrong turn early tends not to recover. One caveat: these benchmarks are synthetic, mostly on code and text tasks, so the direction is reliable but the exact size of the drop will differ for website work. The practical takeaway holds either way. Eight tight 300-word playbooks beat one 4,000-word master document. We learned this the expensive way, when one oversized pipeline stage quietly hit a runtime limit and we had to rebuild it as six named stages with a capped repair loop.

Third, written process goes stale. On August 12 we found stale duplicate data blocks with conflicting values in six of our nine evidence ledgers. The monthly refresh had been stacking a new block on top of the old one every run. A person caught it by reading, and the fix is still in progress. The rule we hold now: any change to how the work is done has to update the matching document before the change counts as finished, and anything that changes faster than you review it will be wrong in your docs.

Fourth, this system is only about two months old. The monthly refresh has run about twice, and the first run produced the ledger bug above. We can claim the capability. The compounding is a claim for next year.

‍

Where should you start?

The whole thing comes down to this. Write your website process down, clearly enough that an agent could follow it. Watch which parts convert and which fall apart. The parts that fall apart show you where one person is quietly carrying the load, and that map is worth more than the documents themselves. Then, before any agent touches a customer-facing page, run the honesty trap and read its answer to the question that has no clean answer.

To be clear about where we stand: this article tells buyers to build the thing we sell. If you would rather start from a written, gated process than spend two quarters building one, that is the system behind every Noco engagement. Talk to us about your website and we will show you the gates.