VigilSAR, a defense-ISR software product, has done something unusual for the AI beat: it published the public leaderboard, a scoring table for which large language models can be trusted with intelligence, surveillance, and reconnaissance work. The emphasis is on the reasoning, reporting, and restraint an analyst actually needs — not the general-knowledge trivia that dominates most benchmark coverage.
The setup is straightforward. 14 models were run against 300 tasks, with results scored on 2026-07-17. The aggregate results are public, but the task set itself is not — a deliberate choice so that model developers cannot train on it. On top of that sits a private held-out set, and the gap between each model’s public and held-out scores is published per model, which the operators say flags memorization rather than genuine capability.
The current standings come with a caveat the team is explicit about: band matters, not rank, because confidence intervals inside a band overlap. At the top, claude-fable-5 leads with 67.77, sitting in Band A as a pinned reference row. The headline for tech readers is the new entry: Moonshot’s Kimi K3 debuts at #3 with 64.65 in Band B — placing it ahead of every GPT and Gemini row on the board.
The rest of the table fills out as you might expect from that ordering. The GPT-5.x family occupies Bands C through D, while the Gemini rows sit in Bands E through F. One locally-runnable open model carries a “sovereign-deployable” score — a nod to the idea that deployment reality, not just raw accuracy, is part of what gets measured.
For a tech audience, the design choice worth understanding is why the task set stays private. Benchmark contamination — models quietly absorbing test material during training — has become a persistent credibility problem across the industry. By keeping the tasks unpublished and running a held-out set alongside, the evaluation makes training on the test both impossible from the outside and detectable from the score gap. The article you are reading deliberately respects that boundary: what the tasks look like is not disclosed, by design.
Why build this at all? The site’s own framing is blunt: “Vendor claims are not evidence.” The operators say they built the evaluation to decide which models get anywhere near their own product, to rank the models they themselves use, and that they are not paid by any vendor — “we would rather be measured than believed.” That motivation matters for readers trying to weigh how much stock to put in any single leaderboard.
The honesty features round out the picture: bands instead of pseudo-precise ranks, published confidence intervals, published held-out gaps, a pinned reference row, and per-model cost-per-correct-answer economics. Whether or not you care about the defense-ISR angle specifically, VigilSAR is making a case that domain-specific, contamination-resistant evaluation is where benchmarking has to go next — and K3’s debut above the GPT and Gemini rows is the kind of result that will get the whole industry watching.
AI Engineering: Building Applications with Foundation Models
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
