Generative Neural Network Training

The project’s concept involves fine-tuning a generative model on images of a specific dog so that new, realistic images of the same subject can be generated using text prompts under various visual conditions. The core challenge was not merely to reproduce a similar-looking dog, but to make the subject controllable via a prompt. To achieve this, the project utilized a specially trained token, DOGTOK, which was associated with the dog’s image from the training set.

The final goal of the project is to demonstrate that, after fine-tuning, the model is capable of retaining the object’s key attributes while transferring it into novel scenes that differ in environment, lighting, and additional details. Consequently, the project illustrates not only the act of generation itself but also the principle of model personalization through LoRA.

As the training set, photographs of dogs from an open dataset were utilized. When selecting the images, considerations included realism, sufficient quality, and diversity of angles. The dataset contained both close-ups and wider shots, captured in various settings—outdoors, on the grass, and in a neutral environment. This allowed the model to learn not just a single specific image, but a comprehensive set of object attributes: body shape, muzzle structure, coat texture, and photographic characteristics of the dog.

For training, the same caption was applied to all images: «a photo of DOGTOK dog.» This strategy directs the model’s attention primarily to the subject itself, rather than the differences between individual scenes. On one hand, this simplifies the training and renders it more conceptually clean; on the other hand, the model receives less contextual information about each specific image. For an educational project, this approach is justifiable, as it allows for a clear demonstration of the principle behind the user token and LoRA fine-tuning.

The final image series consists of a set of test generations where the same trained token, DOGTOK, is placed in different settings. The prompts were given the following conditions: A dog in a park under natural light; A dog in a studio with soft lighting; A dog on a city street in a documentary style; A dog on grass with an emphasis on fur detail; A separate test adding an accessory—a red scarf. This set of prompts was chosen intentionally: it allows us to test how well the model can transfer the learned image of the dog into various environments and visual modes while maintaining the object’s recognizability.

The final series demonstrates the variability of a single trained subject across several photographic contexts. This directly relates to the project’s original concept: the training was not intended simply to reproduce a familiar image type, but to make the subject controllable using text. In other words, after fine-tuning, the user receives not «a dog in general, ” but realistic scenes that retain the key properties of the learned object.

Technically, the generation is built using a DiffusionPipeline connected to LoRA weights. The base model used was stabilityai/stable-diffusion-xl-base-1.0, and the VAE madebyollin/sdxl-vae-fp16-fix was connected separately, followed by loading the saved pytorch_lora_weights.safetensors. This indicates that the project relies not on complete model retraining, but on lightly adapting the base SDXL for a specific task. This approach is suitable for an academic project because it is resource-efficient while effectively demonstrating the principle of generative model personalization.

A crucial element of the generation is the negative prompt:

cartoon, painting, drawing, low quality, blurry, text, watermark.

With this, the model is further steered away from undesirable visual attributes. This pushes the generation toward realistic photography and reduces the likelihood of cartoonishness, painterly effects, blurriness, and artifacts like text or watermarks. This choice aligns well with the original dataset, which consisted of photographs rather than illustrations.

The generation parameters used—num_inference_steps=30 and guidance_scale=7—establish a balance between image quality and controllability. Thirty steps represent a reasonable compromise between generation speed and the level of detail in the image. The guidance_scale of 7 strengthens adherence to the text prompt without pushing the image toward excessive artificiality. For a demonstration project, this is a successful choice, yielding stable and convincing results.

The final series can be analyzed along several types of variability:

change of setting: park, studio, city street, grass; change of lighting: natural light versus soft studio light; change in an additional attribute: a red scarf as a new element added to an already learned image.

These differences form the basis of the visual analysis. If the model performed successfully, all images should maintain the general characteristics of the dog as an object: anatomical integrity, head shape, ears and body form, coat texture, and plausible composition within the frame. At the same time, the scenes themselves should not literally repeat each other. In one case, the dog is perceived as part of the natural environment; in another, as a studio portrait; and in a third, as an element of the urban visual environment. This variability demonstrates that the model has absorbed not an isolated image, but a broader representation of the object.

The test with the red scarf holds particular significance. It is necessary to check the model’s compositional flexibility: how well the LoRA retains the basic image of the dog when a new visual element—one absent from the original prompt—is added. If the result is convincing, it indicates that the model is capable not only of reproducing training patterns but also of combining a learned object with novel details.

Thus, the final image series illustrates how the fine-tuned SDXL LoRA translates the learned image of the dog into several new visual scenarios. The project’s main focus is not on stylization, but on realistic variability: different scenes, varying lighting, different shooting conditions, yet the same core object. This is precisely where the final series connects with the project’s initial concept.

In this project, a laptop running code for image training and generation was utilized. The code’s structure reveals that both the training and subsequent result generation are based on SDXL with the application of LoRA. After training, the pytorch_lora_weights.safetensors weights were saved and then loaded when generating the test series of images.

The workflow can be described as follows:

— A dataset of dog photographs was prepared; — A consistent text prompt, «a photo of DOGTOK dog, ” was applied to all images; — LoRA fine-tuning was conducted based on the stabilityai/stable-diffusion-xl-base-1.0 model; — The madebyollin/sdxl-vae-fp16-fix model was used to enhance decoding quality; — The saved LoRA weights were loaded following the training; — The final series was generated using a set of test prompts; — A negative prompt was additionally employed to boost the photorealism of the output. This process illustrates the entire project lifecycle: from preparing the dataset and training the custom token to verifying how the model generalizes the learned object to new scenarios.

In the project, the core AI model trained was a fine-tuned SDXL model utilizing LoRA, which was used to generate the final series of images.

If you used supplementary generative AI tools, you can list them separately, for example, using this phrasing:

ChatGPT was used to prepare the project description, exposition, and structure the materials. The trained SDXL neural network with attached LoRA weights was used for image generation.

If no generative AI other than the trained model was used, you can write:

No supplementary generative AI was utilized in the project besides the trained neural network used for image generation.