TL;DR
Get the latest gadgets delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
A report dated Nov. 3 says 16 large language models attempted all 55 puzzles in Zachtronics’ programming game TIS-100 and produced 133 new leaderboard frontier points across 18 puzzles. Opus 5.5 and GPT-6 Astra solved all 55; the report credits Opus with three outright cycle records, including a program that beat one community record by 29% in cycles.
A test of 16 large language models playing Zachtronics’ programming game TIS-100 produced 133 new leaderboard frontier points across 18 of the game’s 55 puzzles, according to a report published Nov. 3. The report says the models also set three outright cycle records, offering a measurable test of how well AI systems can write and optimize programs under tight constraints.
The researchers gave each model the game and its manual, but withheld the public leaderboard and community solutions. Every model attempted all 55 puzzles—47 from the game and eight community tournament challenges—in one scored attempt per puzzle. Depending on difficulty, models had 10 to 45 minutes to find a working solution and improve it.
TIS-100 asks players to write short programs for a network of small processors. Nodes have limited registers and code space, and must exchange values with neighboring nodes. A solution is scored on cycles, nodes and instructions; reducing one measure may worsen another, so the benchmark counts new points on a tradeoff frontier rather than treating one score as the sole best result.
The report says Opus 5.5 and GPT-6 Astra each solved all 55 puzzles and each found new frontier points on 11. Opus produced 72 new points, compared with 32 for GPT-6 Astra. GPT-6 Astra had a slightly higher record-proximity score, 78.0 to 77.2, but the report’s 95% bootstrap interval for the gap included zero, so it did not establish a clear difference on that measure.
What the Records Show About LLMs
The results show that language models can do more than produce plausible code: in this setup, they generated working programs that passed extensive testing and improved on solutions maintained by a player community. The puzzle format makes performance legible through executable outcomes and constrained scores, rather than relying only on subjective judgments of code quality.
The findings also caution against treating “solved the puzzle” as the same achievement as “optimized the solution.” Gemini 3.8 Flash solved 53 puzzles but found a new frontier point on only one; Fable 5.1 solved 52 and found new points on five. The report’s results suggest that reliability and optimization are distinct capabilities, even within a narrow programming task.
For readers following AI coding systems, the work provides a benchmark with competing objectives and visible evidence of execution. It does not, by itself, establish how the models would perform on ordinary software projects, where codebases, requirements and measures of success differ substantially.
Products Worth Considering
Apple AirPods Pro (2nd Generation) Wireless Earbuds with USB‑C Charging, Up to 2X More Active Noise Cancelling, Transparency Mode, Adaptive
Sonos Arc Ultra Soundbar with Dolby Atmos, Black
Apple MacBook Air 13.6-inch Laptop with M5 Chip, 16GB Memory, 512GB SSD, Midnight
As an affiliate, we earn on qualifying purchases.
How TIS-100 Scores Programs
TIS-100 is a Zachtronics game built around programming a fictional machine made up of 12 nodes. Its small instruction set and severe limits make even basic operations a sequence of coordinated steps. Players have spent years refining solutions on a public community leaderboard, giving the report a set of human-created records against which to compare model output.
For its benchmark, the team used text-based tools to present the manual, puzzle details, visible test inputs and game feedback. Models could run programs in the game and receive diagnostics for wrong outputs or deadlocks. Each passing program was saved to that model’s own frontier, allowing it to try changes without losing an earlier result. The harness also gave reminders to continue testing, but did not provide solution hints.
The report says a result counted toward its headline totals only after a fresh-game replay and 100,000 passing tests on the independent TIS-100-CXX simulator. That check is intended to provide stronger evidence that programs work beyond the limited cases checked in a typical game run.
““Solving and optimizing are also different skills.””
— The report’s authors
Limits of the Benchmark Results
The report describes performance on one programming game, not a general evaluation of software development. It does not establish whether the same systems would transfer their results to larger codebases or tasks with different constraints. The reported ranking also reflects one scored attempt per model per puzzle, with time limits that varied by difficulty.
The authors say Opus 5.5 and GPT-6 Astra are not separated by the record-proximity comparison: its 95% bootstrap interval includes zero. GPT-6 Astra led in all six tested hypervolume settings, which measure the area covered by different score tradeoffs, while Opus found more individual new points. Those measures capture different aspects of performance rather than resolving the models into a single definitive winner.
The report says players improved on most model programs within days, but the supplied material does not give a complete breakdown of those later changes or identify how many model-created results remained on the leaderboard at publication. The report also says research teams can request reinforcement-learning environments and traces; their availability and terms are not detailed in the material.
How Players Build on the Results
The report says community players began picking up and improving the models’ programs within days. That follow-up may show which model-generated ideas remain useful after human optimization, but the material does not specify a date for a final leaderboard comparison or a subsequent evaluation.
The authors say reinforcement-learning environments and model traces are available to research teams upon request. Further work could use those materials to study how systems arrive at solutions, how repeatable their scores are, and whether the strategies transfer beyond TIS-100. For now, the published results are a time-bounded comparison on 55 puzzles, with broader applicability still untested.
Key Questions
How many language models took part?
The report says 16 LLMs attempted all 55 puzzles: 47 from TIS-100 and eight community tournament puzzles.
Which models led the reported results?
Opus 5.5 and GPT-6 Astra each solved all 55 puzzles and found new frontier points on 11. Opus recorded 72 new points; GPT-6 Astra recorded 32.
What does a new frontier point mean?
It is a program that adds a new tradeoff among cycles, nodes and instructions, rather than simply improving every score at once. The report required a fresh-game replay and 100,000 passing simulator tests for a point to count.
Did the models beat every community record?
No. The report says they set three outright cycle records and produced 133 new frontier points, while players improved on most model programs within days. It does not claim that the models surpassed every community solution.
Do the results show that LLMs can program any software?
No. The study tests a specific puzzle game with limited instructions, fixed inputs and measurable score criteria. Its results do not establish performance on general software engineering tasks.
Source: tis-100.wincon.dev
