TL;DR: Two AI agents read the same 20 enquiries from a contact form. One was given a single sentence about what a good outcome is. The other was given a 40-line flowchart. Both sent all 20 buyers to the right plan and the right next step, in both runs. The same flowchart run as rules, with no model behind it, got 3 of them wrong, each time sending a small buyer to a sales call about the biggest plan. The test, the answer key and all 80 answers are free to take and run on your own enquiries.
Everything in the replay comes from the recorded runs. Each reply stands as Claude wrote it. Nothing new runs while you click.
One sentence against 40 lines
The idea is Anthropic's. On 30 September the company wrote about how its own sales team rebuilt inbound with Claude Managed Agents, and one line in that post is exactly this question: "Explaining the goal of the agent worked better than a detailed list of instructions."
Here the claim is tested on a company your size. The test is set at Vognlog, an invented Danish software firm that sells job planning to service companies on three plans: Start, Team and Business. Claude wrote 20 enquiries the way they arrive through a contact form: role, company size, what they need help with, what they actually wrote, and what an agent can see in the CRM. Are they a customer already? Do they have an account owner?
13 of the 20 carry a trap. The owner of a small plumbing firm asks for Business by name. An existing customer belongs with their account owner, not with sales. A company of 30 has only 3 people who plan. The answer key was written and saved before a single agent ran, so nobody can say it was fitted to the answers.
Both agents got the same knowledge base and the same answer format. Only the instructions differed. One got a single sentence: understand what this buyer actually needs, and recommend the plan and next step that are right for them and for Vognlog. The other got a 40-step flowchart, the kind of thing an inbound playbook can look like. Every enquiry ran in its own Claude agent, twice with each prompt. That makes 80 runs.
20, 20 and 17
Claude with one sentence got the plan and the next step right on all 20 buyers, in both runs. Claude with the flowchart did the same. Each agent made exactly the same choices in the first pass and the second. The two disagreed on one buyer only, a non-profit, and the key accepted both answers.
So the test cannot separate them. Claude with the flowchart used it as a guide: where a line clashed with the knowledge base, Claude followed the knowledge base.
Run literally, line by line with no model behind it, the flowchart got 3 of the 20 wrong. An agent built as a decision tree does exactly that: it follows the lines.
The replies also differed in length. With one sentence, Claude wrote 115 and 118 words per reply on average across the two passes. With the flowchart, 66 and 67. Partly because line 37 of the flowchart caps a reply at 90 words, and the single sentence says nothing about length.
The buyer who wrote SSO and meant passwords
Buyer 18 owns a painting firm. He wrote, in Danish: "We are 12 painters and we would like SSO via Google, because people keep forgetting their passwords. Which plan do we need?"
Line 19 of the flowchart says that if the message mentions SSO, the plan is Business. The rules did exactly that and sent 12 painters to a sales call about a plan that starts at 25 users. Claude had the same line in front of it and answered that Google sign-in is included in every plan, so his painters could sign in with their Google account instead of a password. It recommended Team, at 219 kroner per user per month, with a trial he could start himself.
The other two misses look the same. Buyer 1 has 6 fitters and asks for Business by name, and line 21 says that if a buyer asks for Business by name, they get Business. What he needs is a text to customers and today's jobs on a phone, and Team has both. Buyer 12 has 30 staff but only 3 in the office who plan; the carpenters just get a text. The rules read 30 and picked Business. Claude read 3 and picked Start, at 129 kroner per user.
In two of them the flowchart contradicts the knowledge base, and in the third the rules took the first number in the message as the number of users. All three times Claude went with the knowledge base. If your website AI agent is a decision tree, the lines you write are all it has. With a language model behind it, the knowledge base does most of the work: who counts as a user, what needs which plan, and what SSO means at your company.
Take the test home
All of it is free to copy and adapt:
- The whole test as a zip file: the 20 enquiries, the answer key, the knowledge base, both prompts, the script that builds each run's input and the scoring script
- The README, with exactly how the runs were made
- All 80 answers totted up, and every reply verbatim in the runs folder of the zip
To run it on your own enquiries, swap the knowledge base for what your agent should know: plans, prices, who handles what. Take 20 real enquiries, remove names and contact details, and keep what the agent would see in your CRM. Write the answer key before you run anything and save it with a date. Run each enquiry in its own fresh Claude conversation, twice, and let the script score them.
The test does not call Claude itself, so you need your own access. And the script says nothing about the tone of a reply. It checks the plan and the next step, nothing else.
What the test cannot see
The knowledge base answers every trap. It says who counts as a user, that Google sign-in is not SAML SSO, and that Business starts at 25 users. So this compares two ways of instructing on top of a good knowledge base, not a goal against a playbook.
Claude wrote both the flowchart and the traps. Lines 19 and 21 are the lines the traps hit, and a careful sales team might not write them that way. Line 21 sends anyone who asks for the top plan to a sales call, which is a perfectly reasonable way to sell. It is scored wrong here because the knowledge base sells Business from 25 users only. The rules are also only one literal reading: on enquiry 12 they took the first number in the message, the 30 staff, as the number of users.
The replies were read but not scored. Four of the flowchart replies carry a placeholder such as "[trial link]", because line 30 says to send the trial link and the knowledge base has none. Every price and every sum in all 80 replies is correct.
The first version used a company name that turned out to belong to a real Danish firm, and the name of a real university. The name check caught it before anything was published. Every company got a new, invented name, and all 80 runs were made again on 11 October. The result came out the same: 20, 20 and 17. It is still 20 invented enquiries of one message each, run twice with each prompt. Real enquiries are messier.
When your website AI agent has to sell
If someone is selling you a website agent set up as a flowchart, the first question is whether a model sits behind it or the lines are followed to the letter. The second is who writes the knowledge base. Here the knowledge base was what Claude followed every time the rules went wrong.
As an AI consultant in Copenhagen, I set up AI lead qualification agents on the websites of Nordic companies: an agent that qualifies enquiries against your own plans and prices and sends the right buyer to the right person. I start with the knowledge base, 20 of your own enquiries and an answer key, so you see what the agent does before it meets a real customer. See how a website AI agent with me works.
For the same kind of test on a different job, read how Claude agents handled the renewals in a sales team.
FAQ
Frequently asked questions
Start with the knowledge base: your plans, your prices and who on your side handles what. Then write a short goal for the agent and test it on 20 of your own enquiries against an answer key you wrote first. Only when it gets those right should it talk to real customers.
Not in this test. Claude got all 20 buyers right with either prompt. The difference came when the flowchart was followed as rules with no model behind it, and then it got 3 of 20 wrong.
A decision tree follows its lines as written. What many call an AI chatbot for websites is an agent with a language model behind it, and it can read what the buyer means and weigh it against the knowledge base. In this test that was the difference between 17 and 20 right.
Give the agent a knowledge base that says what each plan needs and who counts as a user, and test it before it goes live. The three misses here were all small buyers sent to a sales call about a plan that starts at 25 users.
Yes. The zip has everything you need and the README walks through it step by step. You need your own access to Claude, and you take personal data out of the enquiries before they are used.
Because a key written after the answers always fits the answers. This one was saved before the first agent ran and has not changed since. The README carries its fingerprint, so you can check it yourself.
No. Vognlog, the companies and the people are invented. The first version used a name that belonged to a real firm, so every company name was changed and the test was run again from scratch.
How this page was made. This work was produced together with AI. Overall: AI about 84 percent, Kim about 16 percent. Counting production alone, AI did about 95 percent. Claude wrote the 20 enquiries, the answer key and both prompts, made the 80 runs, scored them, built the replay and made the film. Kim wrote the working rules and the method in advance. The numbers are a qualified estimate, not a measured log.
