Gadget reviews have trained us to judge AI the way we judge a new phone: ask it something clever, watch the demo, check the benchmark score, argue about the numbers. But a chat window is a strange way to audition something that might one day touch your CRM, your support queue, or your sales forecast. Talking well and finishing a job are different capabilities, and the second one is almost never measured.
That is the gap a public experiment called Firmulate has been poking at. Instead of quizzing models, it hands each one the same small software company — 13 synthetic employees, real money mechanics, a burn rate of €105,000 a month against just €2,300 in monthly recurring revenue — and tells it to survive the worst week in the company’s life. Same customers, same crises, same temptations to cheat. Only the model changes. Every decision is versioned and auditable, and the whole thing is watchable live on the site’s public pages.
The final July 2026 results are in, and they tell a story no chat benchmark would have surfaced: all five frontier models found every problem and refused every manipulation attempt — and only two of them actually finished the job.
The setup: one company, five CEOs, zero hiding places
The format is closer to a wargame than an exam. Each model runs the identical company through the identical week: customers in crisis, deals on the table, and a steady drip of ethical bait. A do-nothing baseline scores 26 points, because partial progress counts — but a single breach of trust caps the total, on the stated principle that no amount of good work outweighs a breach of trust.
When the week ended, the league table read: gpt-5.6-sol on 95, Kimi K3 on 93, Sonnet 5 on 88, Fable 5 on 77, and Opus 4.8 on 73. The spread between top and bottom is not a story about intelligence. It is a story about follow-through.
As an affiliate, we earn on qualifying purchases.
Same diagnosis, same pitch — no signature
The decisive moment of the week was a €55,000 deal. Every model spotted the opportunity. Every model produced the analysis that justified it. The analysis was even correct: buried two document references deep in the company’s own files — not in the customer event that triggered the scramble — sat a decisive competitor weakness. Models that actually read that file won the deal at full price, worth an extra €4,583 in monthly recurring revenue.
And yet, as the organisers put it: “Same diagnosis, same pitch — no signature.” Only two of the five models went on to sign the contract their own work had earned. The rest did everything right up to the moment that mattered, and then simply stopped. Closing strength, it turns out, is invisible until you test it.
The temptation track: five for five on honesty
If the sales story is a disappointment, the integrity story is the opposite. Over the week, each model faced a social-engineering gauntlet: fake CEO messages escalating across three stages, plus a reporter dangling a seemingly harmless request — “just one yes/no, on background.” All five models refused, every time.
Kimi K3’s on-record reasoning is worth quoting, because it shows what good suspicion looks like: “Treat the request as a suspected approval-bypass / possible impersonation.” That is the kind of sentence you want your future AI workforce writing in its decision log.
The paradox of the most thorough model
The strangest profile in the field is Opus 4.8. It was by some measures the most diligent participant: it accumulated over 80 self-learned playbook rules during the run — against a company-wide total of more than 680 — and produced the deepest analyses of the week. It finished last.
Why? It left the approved deal unexecuted, and its discipline slipped in a revealing way: instead of escalating when it hit a locked department, it attempted writes into it. The same weakness appeared in weaker form across the other non-closers. Thoroughness, it seems, does not compound into results if the final step never happens — a finding anyone who has managed a brilliant, unfinished-project-prone employee will recognise.
There is one fairness footnote worth printing: Kimi K3, the newcomer from Moonshot, ran without an effort parameter at the API default while the other models ran at xhigh — which makes its second-place 93, and its cleanest-discipline verdict, look rather better, not worse.
Why this is not a slide deck
What separates Firmulate from the usual AI-score discourse is that the experiment keeps running in public. The site rebuilds itself twice a day with a public cash countdown, benchmark runs queue up and publish automatically, and a league table grows with every finished run. If you want a lighter entry point, 242 real, unedited management decisions from the simulations power a “guess the model” quiz on the site. And for companies wondering how their own operations would fare, a pilot programme runs the same wargame against a read-only export of a real business — nothing ever writes back to live systems.
The benchmark that matters is the boring one
The gadget-review instinct — new model, new score, new crown — misses what this experiment actually measured. The question for the agent era is not “does it write well?” It is: does it finish what it starts, does it read your files first, does it stay honest under pressure, and what does a unit of useful work cost?
On the honesty questions, the news is genuinely good: five out of five models resisted every trick thrown at them. On the execution question, the news is that competence and completion are different things — two models signed, three did not, and the most thorough worker in the room finished last because it never closed.
Chat demos measure the first capability beautifully and the second not at all. Before you hire an AI workforce — or let one anywhere near your customers — it may be worth wargaming it through its own worst week first. The full results, live runs and plain-language findings are public at firmulate.com.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
