🔍 Read the full analysis: How To Train An AI To Create Watercolour Paintings Using TRL And OpenEnv on ThorstenMeyerAI.com
TL;DR
A developer has released an open, end-to-end reproduction of Surya Narreddi’s watercolour-generating AI, including datasets, training code, and models, built with TRL and OpenEnv. This project tests reinforcement learning over aesthetic taste, not just correct answers, as detailed in the original analysis. The release aims to foster further research and transparency in AI art generation.
An independent engineer has released a comprehensive, open-source reproduction of Surya Narreddi’s viral watercolour AI model, built using the TRL framework and OpenEnv. This project includes all datasets, training scripts, and models, and runs end-to-end on Hugging Face infrastructure. The release aims to enable further exploration of reinforcement learning (RL) based on aesthetic preferences rather than verifiable correctness, marking a significant step toward transparent AI art generation.
The reproduction closely follows Narreddi’s original concept, which trained a language model to generate JavaScript code that produces watercolour-style paintings via the p5.brush library. For more context, see this detailed analysis. Unlike traditional models optimized for pixel accuracy, this approach uses reinforcement learning to optimize aesthetic qualities, guided by a reward function combining four terms: code correctness, length, style judged by a vision model, and human preference scores.
The training pipeline employs the Qwen3-VL-30B-A3B-Instruct model as a pairwise judge, comparing generated outputs against four reference images curated by the developer. The reward mix is designed to favor style, natural brushwork, and visual appeal, with the process running through 110 steps, 240 episodes per step, and generating multiple outputs per episode. All artifacts, including datasets, environment scripts, and trained models, are openly available on Hugging Face, promoting transparency and reproducibility. This project exemplifies the kind of work discussed in the original analysis.
This project tests whether reinforcement learning can effectively optimize for aesthetic taste, a departure from typical RL applications that focus on mathematical correctness or functional tests. The original viral watercolour video, viewed over 1.5 million times, demonstrated the model’s ability to produce loose, handmade-looking paintings that contrast with the more polished outputs of mainstream image generators. The open release removes barriers for researchers seeking to further develop or critique this approach, providing a foundation for experiments on aesthetic RL.
Implications for AI Art and Reinforcement Learning
This open reproduction represents a meaningful advance in AI art research by providing transparency and tools to explore reinforcement learning based on subjective aesthetic preferences. It challenges the prevailing paradigm of optimizing models for verifiable correctness, opening new avenues for creative AI applications. By releasing all datasets, code, and trained models, the project encourages community participation, validation, and further experimentation, potentially influencing future AI art tools and methodologies.
watercolor painting digital art tablet
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Historical and Technical Background of AI Watercolour Projects
The project builds on a lineage of early AI art experiments, from DeepDream (2015) to neural network portraits by Mario Klingemann and curated datasets by artists like Anna Ridler. Narreddi’s initial work focused on training models to generate floral images, later expanding to full compositions, with the viral watercolour video serving as a showcase of the aesthetic potential of AI-generated art. Prior efforts often relied on curated datasets and manual tuning; this project automates and opens the entire process, guided by reinforcement learning over human-like preferences.
The original model trained a language model to write JavaScript code that simulates watercolour strokes, enforcing style constraints by limiting the methods used. The recent reproduction extends this by openly sharing the datasets, environment, and training pipeline, allowing others to replicate or improve upon the work. The approach aligns with ongoing debates about whether AI can learn and reproduce subjective qualities like style and taste, rather than just functional correctness.
“This open reproduction aims to democratize access to aesthetic RL techniques, enabling researchers and artists to experiment with AI-generated watercolours using fully transparent pipelines.”
— Thorsten Meyer, lead developer
Unanswered Questions About Model Performance and Comparisons
It remains unclear how the outputs generated by the open reproduction compare quantitatively with Narreddi’s original models in terms of style, coherence, and artistic quality. The project presents visual median outputs for different reward mixes but does not provide a definitive evaluation or user study to determine which approach yields the most aesthetically pleasing results. Additionally, the precise influence of each reward component on the final paintings is still under investigation, and the full technical report from Narreddi has not yet been published, leaving some technical details and performance metrics unavailable.
Future Developments and Community Engagement Opportunities
The author plans to publish a comprehensive technical report from Narreddi, which will clarify the model’s architecture, training details, and evaluation results. The open artifacts serve as a foundation for other researchers and artists to experiment with aesthetic reinforcement learning, potentially leading to new models, datasets, and evaluation methods. Community feedback and comparative studies are expected to emerge as more users test the reproduction across different artistic styles and preferences. Further development may include integrating more sophisticated reward functions, expanding datasets, and exploring other art forms within the same framework.
Key Questions
What is the main goal of this open reproduction?
The goal is to enable transparent, reproducible experiments in reinforcement learning for aesthetic preferences, specifically in creating watercolour-style paintings with AI, and to foster community-driven advancements.
How does the reward system influence the generated art?
The reward combines criteria such as code correctness, style judged by a vision model, and human preferences, guiding the model to produce paintings that align with these aesthetic measures.
Can I use this pipeline to generate other art styles?
Yes, the open-source code and datasets can be adapted to different styles or artistic goals by modifying the reference pool and reward functions.
What remains unclear about the model’s effectiveness?
It is still uncertain how the outputs compare in quality to the original model, and whether the aesthetic rewards truly capture artistic beauty as perceived by humans.
What are the next steps for this project?
The next steps include publishing a detailed technical report from Narreddi, conducting user studies, and expanding the dataset and reward functions to refine artistic output.
Primary source: Hugging Face · via ThorstenMeyerAI.com
