Who adds up the pipeline right?

Three Claude models get the same invented CRM and the same eight questions from a sales director. They may query the database as often as they like. Pick a question and see who lands on the true number.

The company, the deals and the numbers are invented. Each model answered every question four times at the same setting (effort medium): twice with the export's column documentation and twice without, and every query is shown at the moment it was sent, replayed four times faster. Cost is list price per million tokens in and out, times what the run actually used. An answer counts as right when it is within half a percent of the true number. The sales director asked in Danish; the questions are shown here in English.