🔍 Read the full analysis: Consultants And Data Science: The Skills Behind ChatGPT’s Boost on ThorstenMeyerAI.com
Get the latest gadgets delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
A randomized trial at Boston Consulting Group found that consultants given GPT-4 and brief training performed better than a control group on coding, statistics and prediction tasks. After ChatGPT was removed, researchers found no evidence that the consultants had gained an advantage on related technical questions.
A randomized trial involving Boston Consulting Group (BCG) consultants found that workers using ChatGPT performed better on data science tasks outside their usual skills, but showed no evidence of a lasting advantage when tested without the tool. The study, published October 26 in Nature Human Behaviour, bears on whether workplace AI assistance should be counted as employee upskilling or as temporary support for completing work.
The trial assigned consultants to use either GPT-4 after 15 to 20 minutes of training or conventional online resources, including Google, Stack Overflow and Khan Academy. Participants completed two of three 90-minute tasks: Python data cleaning, statistical analysis or prediction. Researchers designed the assignments so ChatGPT could help, but could not independently give correct answers to every question.
Among the 487 consultants who submitted work on both tasks, the ChatGPT group scored 49 percentage points higher than the control group on coding, 20 points higher on statistics and 17 points higher on prediction. The researchers also reported that the ChatGPT group completed coding and prediction tasks faster on average, by about nine and 14 minutes respectively. They found no significant time difference on statistics.
The comparison with 44 BCG data scientists showed that the AI-assisted consultants still scored below the expert group on average across all three tasks. After the work, participants answered technical questions without ChatGPT. The study authors reported no evidence that the treatment group performed better on this follow-up. The paper describes the task-performance benefit as temporary.
The trial in motion · BCG consultants · 2024
With ChatGPT: close to the experts. Without it: no measurable gain.
A randomized trial with a control group that used Google and Stack Overflow instead of ChatGPT.
- 1 · Training15–20 minutes on prompting ChatGPT
- 2 · TasksTwo 90-minute data science tasks, with ChatGPT
- 3 · ScoresGap to 44 data scientists closes, not fully
- 4 · Tool removedNo ChatGPT from here on
- 5 · TestFive technical questions
Coding task score
0 = data scientists’ average
ChatGPT on+49 pp on codingStill below the data scientists on average
Knowledge test without ChatGPT
ChatGPT group vs. control, five questions
ChatGPT off−7.7 to +4.8 ppNone significant · joint test p = 0.531
Source: Wiles, Krayer et al., “Generative AI and the temporary upskilling of knowledge workers”, Nature Human Behaviour (2026); authors’ manuscript, Sections 2.1 and 2.3, Tables B7 and B9. Bars scaled to the five published differences.
Graphic: Powered by Thorsten Meyer AI
Task Gains Did Not Equal Skill Gains
The findings distinguish between what employees can produce with AI and what they can do independently. For employers, that difference matters when evaluating training, hiring and the reliability of work completed with a chatbot. A higher task score can reflect effective use of assistance without showing that the worker has learned the underlying methods.
The experiment does not establish that AI never helps people learn. It tests a particular intervention: brief training, GPT-4 access and specific data science assignments, followed by a short assessment without the tool. Still, the results caution against treating AI-supported performance as proof of retained expertise. Organizations may need separate checks for output quality and unaided competence, especially where workers must explain, verify or reproduce technical results.
The study also reports a possible downside: participants who used ChatGPT became more confident that it could solve problems it could not. The authors classify this finding as exploratory, so it should not be read as definitive evidence of a general effect. It does, however, point to a practical concern: confidence in an AI answer may not match its correctness.
Products Worth Considering
Samsung Galaxy Tab A11 8.7-Inch Wi-Fi Tablet, 64GB, 4GB RAM, 90Hz, Gray, with 20W Charger
LG UltraGear 27G810A-B 27-Inch 4K UHD IPS Gaming Monitor
Logitech MX Mechanical Wireless Illuminated Keyboard, Clicky Switches
As an affiliate, we earn on qualifying purchases.
How the BCG Trial Was Run
The researchers preregistered the trial on March 13, 2024, before collecting data. The experiment took place at BCG in late March and early April 2024. Of 986 consultants allocated to the study, 573 began the survey and 487 submitted both assigned tasks; the main analysis was based on those 487 submissions. Participants had at least a bachelor’s degree, and the authors said most reported no coding experience.
Each participant completed two tasks, chosen from coding, statistics and prediction. The control group received instruction in using conventional web resources, giving researchers a comparison for access to information rather than a no-help condition. The expert benchmark came from BCG data scientists completing the same assignments without AI. The follow-up test covered related technical topics and prohibited ChatGPT use.
The paper’s title refers to “the temporary upskilling of knowledge workers.” The authors’ exoskeleton comparison captures their interpretation: AI can extend a worker’s capabilities while in use, but the study’s follow-up did not find that the extension remained when the tool was removed. This is the authors’ characterization of results from the trial, not a claim that every AI tool or training approach has the same effect.
““Generative AI and the temporary upskilling of knowledge workers.””
— The study authors
Limits of the Follow-Up Test
The study covers one consulting firm, a specific model and training session, and a small set of timed tasks. The follow-up consisted of five technical questions after the work, so the reported result does not show whether longer or more intensive instruction could produce durable learning. Nor does it establish how the findings transfer to other occupations, AI systems or workplace settings.
The phrase “no evidence” of an advantage should not be taken to mean researchers proved that participants learned nothing. The supplied study summary does not give enough detail to assess all uncertainty around the follow-up measure. The authors also label the confidence finding exploratory; its scale, persistence and effects on real work remain unclear.
The source material does not report a follow-up over weeks or months, or whether participants later performed similar tasks in their jobs. It also does not establish whether repeated AI use, different forms of instruction or human review would change results. Those questions remain open beyond this trial.
Testing Longer-Term Workplace Learning
The study reports a completed experiment and its publication, not a scheduled next trial or a new company policy. Further research would need to test whether longer training, repeated practice or instruction paired with AI helps workers retain skills after access to the tool ends. Longer follow-ups could also show whether short-term task gains translate into better work over time.
For employers applying the findings now, the immediate distinction is between measuring deliverables produced with AI and measuring skills employees can demonstrate independently. The trial does not prescribe a particular workplace process, but it supports evaluating those outcomes separately. Whether AI can reliably build durable expertise under different training conditions remains to be established.
Key Questions
What did the BCG study find?
Consultants given GPT-4 and brief training scored higher than a control group on coding, statistics and prediction tasks. Researchers found no evidence that the AI group did better on related technical questions answered later without ChatGPT.
Did the consultants with ChatGPT outperform data scientists?
No. The ChatGPT group performed better than the control group, but its average scores remained below those of 44 BCG data scientists who completed the tasks without AI.
Does the study prove that ChatGPT cannot teach skills?
No. It tested brief training and a particular follow-up assessment, not every possible teaching method. The authors reported no evidence of an advantage on the unaided questions; that is not proof that no learning occurred.
How large was the study?
Researchers allocated 986 consultants, but the main analysis included 487 who submitted both assigned tasks. The study also used a benchmark group of 44 BCG data scientists.
What remains unknown?
The trial does not answer whether longer training, repeated practice or different AI tools could lead to skills that persist over time. It also does not establish how well the results apply outside this firm and these data science tasks.
Source: ThorstenMeyerAI.com
