🔍 Read the full analysis: AutoSynthData: Generating Training Data For Enterprise Agents on ThorstenMeyerAI.com
Get the latest gadgets delivered free — and shop member deals
- Fast, free delivery on millions of items
- Access to Prime Big Deal Days deals on October 6–7
- Prime Video, Amazon Music and more included
TL;DR
ServiceNow CoreAI describes AutoSynthData, a system that uses an enterprise agent’s failures and a stronger model’s successful runs to generate new training tasks. The company points to the released EnterpriseOps Gym dataset as an example, but the supplied material provides no measured gains or comparison with other training methods.
ServiceNow CoreAI says it has built AutoSynthData, a system that turns an enterprise agent’s observed failures into new training tasks and checks those tasks in the target environment, as described in the original analysis. The company cites the released EnterpriseOps Gym dataset as an example of the approach, but the supplied description reports no performance results showing whether the training improves an agent.
In the process described by ServiceNow CoreAI, a target model first attempts diagnostic tasks in an enterprise environment. A stronger teacher model also attempts those tasks. The system uses the two sets of runs to identify the capability being tested, the tools and workflow involved, where the target model fails, how the teacher succeeds, and what conditions a valid result must meet.
AutoSynthData distills that information into sanitized capability specification cards. The company says task generators use those cards rather than the original prompts, entities, agent trajectories, or verifier details. They then create tasks with varied wording, starting states, entities, workflow combinations, tools, and difficulty. Each task includes an environment specification, a user prompt, and a verifier intended to check whether the request was completed within the environment’s constraints.
According to the description, generated tasks are checked in the environment, and accepted samples are used for post-training. The updated model can then be tested again, with remaining weaknesses informing another round of task generation. The supplied account does not state how many tasks were generated or accepted, or how much the model’s performance changed.
Training Agents on Local Workflows
Enterprise agents are judged not only on whether they produce plausible language, but also on whether they take permitted actions and leave systems in the required state. A model that performs well on broad evaluations may still mishandle a particular organization’s tools, access rules, or workflow steps. AutoSynthData is designed to focus training on those environment-specific gaps.
The verifier is central to that proposal. If it approves an incorrect outcome, training may reinforce faulty behavior; if it rejects a valid approach, it may penalize a sound solution. ServiceNow’s description says verifiers should match the request and environment, reject failures or policy violations, and accept valid solutions without requiring a single exact sequence of actions.
If the process works as intended, teams could produce more varied practice tasks without depending entirely on manually written examples. But the account describes a method, not evidence of improved reliability, lower costs, or successful transfer to other enterprise systems. Those outcomes matter because agents can change operational data, and the consequences of incorrect actions depend on the systems in which they run.
enterprise AI training data generation tools
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
From Evaluation Failures to Tasks
The system treats an enterprise environment as more than a prompt. Its description includes what the agent can observe and change, the tools or APIs available to it, and how its actions affect the system. A task combines that environment specification with a user-facing request and a verifier. Specifications may include policies, instructions, and task setup, such as a prepared database or knowledge articles.
This structure is meant to distinguish an action that is technically possible from a task that is realistic and allowed. A request can sound plausible but be impossible because the necessary tool or information is unavailable, a required state change cannot be made, or policy forbids it. Conversely, an executable task may not resemble normal work. The proposed generator aims to create tasks that are feasible and realistic while still testing a weakness in the target model.
For its example, ServiceNow CoreAI points to EnterpriseOps Gym and cites Malay et al. (2026), saying the released dataset is used in the demonstration. The supplied material does not give the dataset’s size, list the workflows tested, or report model scores. It also does not provide a publication date for the AutoSynthData account.
“A model may be broadly capable and still struggle with a particular environment.”
— ServiceNow CoreAI
Performance Evidence Is Missing
The supplied account gives no quantitative evidence that AutoSynthData improves an agent. It does not report before-and-after scores, the evaluation method, or comparisons with other ways of producing training data. It also omits the target and teacher model identities, training volume, task-generation and acceptance rates, and the time or cost of running the pipeline.
It remains unclear how well generated tasks generalize beyond EnterpriseOps Gym or whether the method works across different enterprise environments. The account says generators receive capability cards instead of original evaluation details, but provides no analysis of possible overlap between generated tasks and evaluation material. Without these details, readers cannot assess the size or robustness of the reported example.
Results Needed to Test the Method
The next useful evidence would be a reported evaluation of a post-trained model, including task counts, verifier acceptance rates, and before-and-after performance. A comparison with an appropriate baseline would help show whether the gains come from AutoSynthData rather than from additional training or other changes.
Testing across multiple workflows and environments would also clarify whether the approach addresses recurring operational weaknesses or is specific to the EnterpriseOps Gym example. The supplied material does not announce a results date or describe a planned release, so when those evaluations may appear is not clear.
Key Questions
What is AutoSynthData?
AutoSynthData is a system that ServiceNow CoreAI says generates enterprise-agent training tasks from observed failures and a stronger model’s successful attempts, then checks generated tasks in the target environment.
How does the system create training tasks?
It uses results from a target model and a teacher model to produce capability specification cards. A generator uses those cards to create varied tasks, each with an environment specification, a user prompt, and a verifier.
What is EnterpriseOps Gym’s role?
ServiceNow CoreAI cites the released EnterpriseOps Gym dataset as an example of the AutoSynthData pipeline. The supplied material does not give its size, tested workflows, or model scores.
Has AutoSynthData been shown to improve agent performance?
The supplied description reports no measured improvement, baseline comparison, or before-and-after evaluation. It explains the proposed method but does not establish its effectiveness.
What evidence would help evaluate the approach?
Useful evidence would include task volumes, verifier acceptance rates, model performance before and after training, a comparison with an appropriate baseline, and results across different enterprise workflows and environments.
Primary source: Hugging Face · via ThorstenMeyerAI.com
Halloween Picks
halloween
As an affiliate, we earn on qualifying purchases.
