📊 Full opportunity report: Minerva. The opposite path. on ThorstenMeyerAI.com — validation score, market gap, and execution plan.
TL;DR
Italy’s Minerva project, which built a large European sovereign LLM from scratch, achieved impressive technical results but scored poorly on Italian academic tests. This highlights the challenges of scaling native-language models for complex tasks.
Italy’s Minerva-3B, a large language model trained from scratch primarily on Italian data, scored just 4.9% on the INVALSI Italian school-exam benchmark, despite significant investment and infrastructure. This performance raises questions about the effectiveness of native-language training at current scale levels and highlights ongoing debates about European sovereign AI strategies.
Minerva-3B was developed by Sapienza University of Rome’s NLP group, led by Roberto Navigli, with support from Italy’s national supercomputing consortium CINECA and funded through Italy’s PNRR AI strategy. It was trained on approximately 2.5 trillion tokens, with around 50% Italian content, making it one of the largest efforts to build a European sovereign LLM from scratch.
Despite these efforts, Minerva-3B’s performance on the INVALSI Italian exam benchmark was markedly poor—scoring only 4.9%, which is near chance level. Researchers noted that while dataset composition and size are important, the overall scale of parameters and data may still be insufficient for complex language tasks. This contrasts with earlier expectations that large native-language models would inherently perform well on academic assessments.
Comparatively, the European project AMÁLIA, which extended multilingual models with a small portion of European Portuguese data, has not published its weights but is part of a broader debate on whether continuation training or training from scratch better addresses language-specific needs. Minerva’s results suggest that even large-scale native-language training may not guarantee high performance in complex, academic language tasks.
Minerva.
The opposite
path.
Italy spent years building a European sovereign LLM from scratch. Then Minerva-3B scored 4.9% on the INVALSI Italian school exam.
Where AMÁLIA layered Portuguese specialization onto a multilingual foundation, Minerva trained from scratch on 2.5 trillion tokens with approximately 50% Italian content. Where AMÁLIA’s weights are not yet public, Minerva published weights, training data, and code as truly-open from day one. By every institutional measure, the Italian approach worked. But the empirical results contain a finding the press coverage has been quiet about — and it has implications that extend well beyond Italy.
Same problem. Opposite path.
European sovereign-LLM development has two primary architectural approaches. Italy chose from scratch with substantial native-language foundation. Portugal chose continuation pre-training of a multilingual model. The structural comparison surfaces what each commitment actually requires operationally.
The comparison is not “Italy did it better than Portugal.” Both projects respond to the same structural problem with different architectural strategies under different institutional and economic constraints. Italy’s national-AI investment is structurally larger by an order of magnitude — and Minerva is the visible artifact of that scale.
Large Language Models (LLMs)
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
4.9% on INVALSI. The bitter lesson surfaces.
In June 2024, researchers evaluated Minerva-3B on the Italian school-exam benchmark. The result was unambiguous. This is not a critique of Minerva — it is a critique of the public discourse around what Minerva’s empirical results actually demonstrate.
End-to-End AI Evaluation: Building Effective Metrics, Pipelines, and Monitoring for LLM Systems
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
350M to 7B. Four parameter scales, one architecture.
The Minerva model family covers four parameter tiers, each with specific training corpora. Each scale level reveals what the from-scratch path actually requires at different operating points.
Italian + English
100B English
~50% English
+ 200B code
Building AI-Powered Products: The Essential Guide to AI and GenAI Product Management
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Three answers. Same question.
Minerva, AMÁLIA, and OpenEuroLLM represent the three operational answers to the European sovereign-LLM question. Each makes different architectural and institutional bets. The strategic discourse benefits from treating all three as data points in the same empirical experiment.
Large-Scale AI Engineering: Design, Train, and Optimize Foundation Models on NVIDIA GPU Clusters
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Three standards the movement should adopt.
The structural critique generalizes beyond Minerva. The European sovereign-LLM movement benefits from internalizing these lessons across every subsequent national project. Italy modeled the openness standard; the movement should adopt it as norm.
Minerva is one valid answer to the European sovereign-LLM question. AMÁLIA is another. OpenEuroLLM is potentially a third. The strategic discourse benefits from treating all three as data points in the same empirical experiment rather than as competing national-prestige projects. More analysis like this is needed. Not less.
Implications for European Sovereign AI Investments
The results from Minerva-3B challenge assumptions that large-scale native-language training alone can produce models capable of handling complex academic and language-specific tasks. Despite Italy’s substantial investment and infrastructure, the model’s poor performance indicates that current parameter scales and training approaches may be insufficient. This raises important questions for European AI policy: how much native-language data and what scale of investment are truly needed to develop effective country-specific models? The findings suggest that European sovereign AI efforts may need to reconsider their scale and methodology to achieve meaningful language understanding and knowledge depth, especially for high-stakes applications like education.
European Sovereign LLM Strategies and the Minerva Case
Italy’s Minerva project represents a deliberate choice to build a large, native-language LLM from scratch, leveraging extensive computational resources and a significant dataset. This approach contrasts with models like Portugal’s AMÁLIA, which employs continuation training on multilingual foundations with smaller European-specific datasets. The debate centers on whether starting from scratch or extending multilingual models is more effective for capturing country-specific knowledge.
Prior to Minerva, European efforts have largely focused on multilingual models or smaller native-language adaptations, with mixed results. The Italian project’s large-scale investment aimed to demonstrate that native-language models could reach high performance, but the recent benchmark results complicate this narrative, suggesting that scale alone may not suffice for complex language understanding.
“The Italian approach demonstrates that large-scale native-language training is feasible but not necessarily effective at producing models capable of academic-level performance.”
— Thorsten Meyer
Unresolved Questions About Native-Language Model Scaling
It remains unclear whether increasing the scale of parameters and data will eventually lead to high performance in complex language tasks or if fundamentally different approaches are needed. The exact threshold of data and parameter scale required for effective country-specific models is still unknown, as is how to optimize training methodologies for these objectives.
Next Steps for European Sovereign AI Development
The Minerva team continues to refine their models, with ongoing research into different training regimes and dataset compositions. Future evaluations will likely explore larger models and alternative training strategies to better understand the scale-performance relationship. Policymakers and researchers will need to consider these findings when designing future investments, potentially emphasizing not just scale but also training methodology and data quality.
Key Questions
Why did Minerva-3B perform so poorly on the Italian exam benchmark?
Despite large-scale native-language training, the model’s limited performance suggests that current parameter scales and dataset size may still be insufficient for complex academic tasks.
Does this mean European efforts to build sovereign models are failing?
Not necessarily; it indicates that scale alone may not be enough, and that different approaches or larger investments might be needed to achieve desired performance levels.
What are the implications for future European AI policy?
Policymakers may need to reconsider the scale and methodology of native-language model investments, emphasizing the importance of data quality, parameter scale, and training approaches to meet complex language understanding goals.
Will increasing model size solve the performance issues?
It is uncertain; while larger models may improve results, the current findings suggest that scale alone might not be sufficient without methodological improvements.
Source: ThorstenMeyerAI.com
