🔍 Read the full analysis: Pick An AI Harness: When CUGA Makes Sense Alongside Hermes And Deep Agents on ThorstenMeyerAI.com
Get the latest gadgets delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
In an IBM-reported AppWorld experiment, CUGA completed 73 of 90 tasks using GLM-5.3-flash, compared with 42 for Hermes and 27 for Deep Agents. The test suggests harness configuration can affect results on this benchmark, but it does not establish a general ranking or assess policy enforcement.
IBM reports that CUGA completed 73 of 90 AppWorld tasks, ahead of Hermes, with 42, and Deep Agents, with 27, in an experiment using the same open-weight model for all three agent harnesses. The results, discussed in the original analysis, point to harness design as a factor in agent performance on this particular multi-step benchmark, not a general verdict on which system is best.
The test used GLM-5.3-flash across all three setups and ran each on the same 90 tasks spanning 30 scenarios. IBM says the evaluation used the Harbor framework and one attempt per task. CUGA passed 81.1% of tasks, Hermes 46.7% and Deep Agents 30.0%. On a stricter scenario measure—counting a scenario only when all three of its variants passed—CUGA passed 20 of 30 scenarios, Hermes eight and Deep Agents two.
The harnesses did not have identical configurations. CUGA used task decomposition, app-focused tools, pre-execution verification and automatic reflection. Hermes ran Harbor’s default setup with AppWorld tools over MCP. Deep Agents used AppWorld tools over MCP with LangChain middleware, including tool selection, context summarization, a to-do list, retries and call limits. IBM also says it made two fixes to Deep Agents so malformed model replies would not end runs.
IBM says the experiment setup is publicly available in harbor-eval, an IBM Harbor fork. The supplied material does not include an independent replication or comparative figures for cost, latency or production-scale reliability. The scores are task-completion results from one benchmark run, not measurements of those other factors.
Why Harness Design Changes Results
The reported spread—from 30% to 81.1% task completion with the model held constant—shows why an agent score cannot always be read as a measure of the model alone. A harness organizes how an agent breaks down work, selects tools and checks actions. Those choices can affect whether a multi-step task succeeds, especially when it requires coordinating several applications.
For teams selecting an agent, the practical lesson is to test the model, harness and workload together, rather than assume a benchmark result transfers to other settings. The AppWorld outcome may be relevant to workflows that resemble its cross-app API tasks. It does not show that CUGA will lead on personal assistance, research, or other work for which Hermes and Deep Agents are designed.
The scores also do not evaluate policy enforcement or safeguards during execution. IBM describes CUGA as offering features for applying rules through the harness, but this experiment did not use policies and cannot establish how effectively those features work. Organizations would need separate tests for permissions, approvals and data boundaries.
Products Worth Considering
Bose SoundLink Flex Bluetooth Speaker (2nd Gen), Waterproof, Dustproof, USB‑C, Up to 12 hr Battery, Black
Apple Watch Series 12 GPS 46mm Smartwatch
Samsung 27-Inch Odyssey OLED G5 QHD Gaming Monitor
As an affiliate, we earn on qualifying purchases.
Three Harnesses, Different Intended Uses
The comparison concerns the software layer that coordinates an AI agent, rather than three different models. The source describes CUGA, or ConfigUrable Generalist Agent, as IBM’s open-source harness for enterprise tasks involving systems such as customer relationship management and payments. It presents Hermes from Nous Research as designed for always-on personal assistants, and LangChain’s Deep Agents as suited to extended, multi-step work such as research.
AppWorld tests tasks across nine apps and 457 APIs, including email, payments and file storage. IBM characterizes it as a proxy for some skills relevant to enterprise automation, such as reading API documentation, paging through results and changing state across applications. But the benchmark uses consumer apps; it does not directly establish how the agents perform on real enterprise systems.
The source also points to community directories tracking nearly 200 agent harnesses and cites a separate directory analysis in which GLM-5.2’s SWE-bench Pro score rose from 23% to 52% after a harness change. That is a separate reported comparison, not part of IBM’s AppWorld experiment, and should not be combined with its results.
“This is not a verdict on Hermes or Deep Agents.”
— IBM
Limits of the AppWorld Evidence
The result covers one model, 90 tasks and one attempt per task. IBM says the harness setups differed, so the test does not isolate every implementation difference. The supplied material does not show how the results would change with other models, repeated attempts, different task sets or production workloads.
It is also unclear whether the gap would persist on benchmarks or workflows outside AppWorld. The experiment did not test policy enforcement, cost, latency, or deployment outcomes, and no timeline is given for a broader comparison or independent replication. The reported numbers should therefore be treated as a result from a specific configuration and workload.
Testing Agents on Real Workflows
Teams interested in reproducing the comparison can consult the public harbor-eval setup identified by IBM. A useful next test would run the candidate systems on representative internal workflows, with the models and tool configurations a team actually expects to deploy, and examine both completion rates and consistency across repeated or varied tasks.
For enterprise use, evaluation would also need to cover access controls, approval steps, data boundaries and deployment conditions—areas not measured by these AppWorld scores. The supplied material does not specify when IBM or other researchers will publish a broader study.
Key Questions
What did the AppWorld experiment find?
IBM reports that CUGA completed 73 of 90 tasks, compared with 42 for Hermes and 27 for Deep Agents. All three used GLM-5.3-flash, but their harness configurations differed.
Does this establish CUGA as the best agent harness?
No. The result covers one model and one benchmark sample. It does not establish how the systems compare on other tasks, models or production workloads.
What is AppWorld measuring?
AppWorld tests multi-step tasks across nine apps and 457 APIs, including email, payments and file storage. IBM describes it as a proxy for some enterprise-automation skills, but it uses consumer apps.
Did the test evaluate safety policies or access controls?
No. IBM says the experiment did not use policies, so its scores do not show whether any harness effectively enforces rules, permissions or approvals.
Can teams inspect or reproduce the setup?
IBM says the experiment setup is available in harbor-eval, its Harbor fork. The supplied material does not report an independent replication or a schedule for further comparisons.
Primary source: Hugging Face · via ThorstenMeyerAI.com
