Project Idea
For this project, we selected troll dolls from the Danish company Dam Things to train the neural network. These dolls are characterized by their prominent, spiky hair. The toys possess a highly recognizable aesthetic and expressive faces, which sparked our interest in seeing whether the neural network could accurately reproduce the facial features, proportions, and signature hairstyles in its generations.
A dataset of 18 toy troll images was collected for training and subsequent generation. Each image features the toy’s face, hairstyle, and proportions.


Image examples from the dataset
The images are compiled on the disk linked below. The disk also contains a document with links to the images and licenses for every image used.
Resulting Image Series

First (test) generation






Image Series Commentary
The final series, along with the initial test variation, comprises eight generated images of troll dolls. The test generation was used to check how well the LoRA absorbed and remembered the main visual characteristics of the trolls. Even at this stage, it was evident that the neural network could recognize the core features (spiky hair, small body, and distinctive face) based on a dataset of only 18 images.In subsequent images, the neural network demonstrated its ability to create variations, incorporate prompt-specific details, and so on. Crucially, the characteristic features of the trolls remained recognizable and coherent. Thus, this series of generations successfully verified the neural network’s capability to maintain the trolls' essential traits while integrating diverse details. This aligns perfectly with the project’s concept and objective—the hypothesis that the neural network could generate novel variations while preserving the recognizability of the dolls has been confirmed.
The neural network particularly captured the face and hairstyle while maintaining the characters' toy-like and plastic quality. Although specific details regarding clothing, accessories, etc., were described using separate prompts, the network successfully preserved the basic character design.
Google Colab and the Learning Process Description
First, Google Drive was connected with the images, the path was specified, and the count of source images was verified.Diffusers and a pre-made DreamBooth LoRA script for Stable Diffusion XL were used for training. The repository containing the source code was uploaded to Colab, after which the necessary dependencies were installed.
The Stable Diffusion XL Base 1.0 was used as the base model for the project, and the key phrase «trlltoy» (from Troll Toy) and the prompt «a photo of trlltoy troll doll» were set for the trained prompt. This keyword allows for a return to it in new prompts after training.Because training Stable Diffusion requires a large amount of memory, the additional libraries bitsandbytes and xformers were installed for operation.
During the launch of the course, a problem arose due to a conflict between the PEFT and torchao versions. To resolve this issue, it was necessary to update PEFT and remove the torchao library.
Next, the model was trained on the image dataset. Training was conducted over 500 steps, and upon completion, the resulting LoRA weights, training checkpoint, and logs were saved. Subsequently, these weights were integrated with the base SDXL model for image generation.The next step involved reloading Stable Diffusion XL Base 1.0 alongside the trained LoRA. The keyword (trlltoy) and details of each generation were incorporated into all prompts.
After reviewing the test generation, the final images were created, from which the final series was assembled.
Used Neural Networks
The project utilized the neural network ChatGPT to assist with library errors and conflicts, as well as to generate the cover image.












