Creating Mosaic Art using AI

This repository includes the code (and other files) for creating mosaic art versions of a given image. Given an image (referred to as the original image), the main idea is to replace (square-shaped) patches of the original image with the most similar image from a given set of images (referred to as the tile images or simply tiles).

The implementation uses the Stable Diffusion model available from KerasCV submodule of Keras to create the tiles. It allows limitless creativity by creating mosaic art of the same image using different tiles created by changing the model's parameters and using different text prompts.

In the examples below, each row shows an (original) image and two mosaic art images created from it. The left-most image is the original image, and the middle image is the mosaic art created using 2,500 tiles, and the right-most image is the mosaic art created using 90,000 tiles.

Getting started

Main files and folders

The main files and folders included in this repository are:

1. images/canvases: This folder includes the (original) images. Place your image files in this folder to create mosaic art versions of them. Please note that the images must have the same dimensions (width and height).
2. images/tiles: This folder includes the tile images. There are already some tile images in this folder. You can use an instance of MosiacArtMaker to create your own custom set of tile images and put them in the images/tiles folder.
3. images/output: The mosaic art images are saved in this folder.
4. make_mosaic_art.ipynb: This notebook includes the code for creating mosaic art versions of the images in the images/canvases folder and also the code for creating the tile images.

Dependencies

The main dependencies are TensorFlow/Keras (at least version 2.9), Pillow, and Scipy. If you use Pip, you can install them by running the following command:

pip install -r requirements.txt

Moreover, the code is written in Python 3.9 and is tested on a computer running Ubuntu 22.04 LTS with Keras 2.9 (and KerasCV 0.3.4) on an NVIDIA RTX 3090 GPU with CUDA 11.6.

Running the code

Open Making Mosaic Art using KerasCV+StableDiffusion notebook to see the code.

Furthermore, you can modify the variables and parameters in the main function (shown below) to create your own mosaic art versions of the given images. You can place your images in the images/canvases folder if you want to create mosaic art versions of them.

def main(remake_tiles: bool) -> None:
    """Main function to pack everything together and run it"""

    # Extension to use for saving and loading the tile images
    tile_file_extension = "jpeg"

    # (Re)-make the tile images if the user wants to do so
    if remake_tiles:

        # Create a MosaicMaker object to make the tile images
        image_maker = MosaicMaker(img_width=400, img_height=400, jit_compile=False, seed=33)

        # The text prompts to be used to make the tile images
        prompt_seq = (("A laughing woman", ("realistic", "white background")),
                      ("A sad girl", ("realistic", "white background")),
                      ("An old man", ("realistic", "white background")),
                      ("Face of a sad man", ("realistic", "white background")),
                      ("Drawing of rings of Saturn", ("abstract", "white background")),
                      ("A watercolor painting of a puppy", ("detailed",)),
                      ("Drawing of a red rose", ("elegant", "detailed", "white background")),
                      ("View of a green forest with mountains in the background", ("elegant", "lush", "nature")),
                      ("A painting of four oranges in a bowl", ("elegant", "detailed", "white background")),
                      ("A ninja shuriken", ("realistic", "metal", "white background")),)

        # Make the tile images and save them
        for index, prompt_data in enumerate(prompt_seq):
            image_seq = image_maker.make_images(prompt_data[0], prompt_data[1], num_images=40)
            image_maker.save_images(img_seq=image_seq, path='images/tiles', prefix=f'p{index}',
                                    extension=tile_file_extension)

    # Use the images in the images/canvases and images/tiles directories to make mosaic arts
    for canvas_image_path in pathlib.Path("images/canvases").glob("*.png"):
        # Create a MosaicArtMaker object with about sqrt_num_tiles*sqrt_num_tiles tiles!
        art_maker = MosaicArtMaker(original_image_path=canvas_image_path, sqrt_num_tiles=300,
                                   tile_file_extension=tile_file_extension)

        # Make the mosaic art and save it in the images/outputs directory
        output_image = art_maker.make_mosaic_art(k=40)
        print(f"Created a mosaic art version of '{art_maker.original_image_path}' using "
              f"{art_maker.sqrt_num_tiles * art_maker.sqrt_num_tiles} smaller images created by a Stable Diffusion model")
        art_maker.save_images((output_image,), path='images/outputs',
                              prefix=f'{art_maker.original_image_name}_mosaic_art')

        # Display each original image and its mosaic art version
        art_maker.display_images((art_maker.original_image, output_image),
                                 (art_maker.original_image_name, art_maker.original_image_name + "_mosaic_art"))

Run the main function to execute the code. If you want to remake the tile images, set the remake_tiles parameter to True. Note that creating a new set of tile images might take a while, so it is recommended to use the current tile images in the beginning by setting remake_tiles to False.

main(remake_tiles=False)

Kaggle notebook

The notebook is available on Kaggle now. You can open it using this link.

Ideas for improvement

• Adding a link to a Kaggle or Colab version of the notebook.

- [ ] Using different shapes instead of squares (of the same size) for the tiles. For example, trapezoids or triangles.

- [ ] Using different tile images for different parts of the original image. For example, using a set of images containing faces for the faces in the original image, and using a set of images containing landscapes for the background in the original image.

- [ ] Using better similarity metrics for comparing the tiles with a patch from the original image. For example, using the SSIM metric instead of the Euclidean distance.

Credits

• KerasCV: The code for creating the tile images uses the StableDiffusion model available from KerasCV.
• The idea for creating mosaic art versions of images using a collection of tile images is inspired by this post.

License

This project is licensed under the terms of the Apache 2.0 license. See LICENSE for more details. Please note that the image files in images/canvases were downloaded from the Web and are not owned by the creator of this repository. Consequently, they may not be licensed under the Apache 2.0 license.