Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/RubenGres/Seg2Sat

Using StableDiffusion and ControlNet to generate synthetic aerial images
https://github.com/RubenGres/Seg2Sat

diffusion-models earth-observation remote-sensing stable-diffusion text-to-image

Last synced: about 1 month ago
JSON representation

Using StableDiffusion and ControlNet to generate synthetic aerial images

Host: GitHub
URL: https://github.com/RubenGres/Seg2Sat
Owner: RubenGres
Created: 2023-04-24T16:12:26.000Z (over 1 year ago)
Default Branch: main
Last Pushed: 2024-04-11T15:04:34.000Z (9 months ago)
Last Synced: 2024-04-13T00:24:15.021Z (9 months ago)
Topics: diffusion-models, earth-observation, remote-sensing, stable-diffusion, text-to-image
Language: Jupyter Notebook
Homepage:
Size: 55.9 MB
Stars: 67
Watchers: 2
Forks: 4
Open Issues: 0
Metadata Files:
- Readme: README.md

Awesome Lists containing this project

README

Seg2Sat - Segmentation to aerial view using pretrained diffuser models
======

Seg2Sat explores the potential of diffusion algorithms such as [StableDiffusion](https://github.com/CompVis/stable-diffusion) and [ControlNet](https://github.com/lllyasviel/ControlNet) to generate aerial images based on terrain segmentation data. The dataset is derived from IGN's [FLAIR (French Land cover from Aerospace Imagery)](https://ignf.github.io/FLAIR/), which provides land cover information for various regions in France. The goal is to create photorealistic synthetic aerial images or stylized aerial imagery.

This could have many uses like dataset creation/augmentation or making maps for your next ungeon and dragon session. Outpainting and inpainting are also possible and yield convincing results.

## Example photorealistic images

These images are derived from the same segmentation mask in the first column and a fixed prompt: `"Aerial view of 101 avenue de l'Espinet, Toulouse, Haute-Garonne, France"`. The different image are obtained by varying the seed in the random number generator.

Image Segmentation
Generated Images

Legend from the FLAIR dataset documentation

Class
Value
Freq.-train (%)
Freq.-test (%)

building
1
8.14
8.6

pervious surface
2
8.25
7.34

impervious surface
3
13.72
14.98

bare soil
4
3.47
4.36

water
5
4.88
5.98

coniferous
6
2.74
2.39

deciduous
7
15.38
13.91

brushwood
8
6.95
6.91

vineyard
9
3.13
3.87

herbaceous vegetation
10
17.84
22.17

agricultural land
11
10.98
6.95

plowed land
12
3.88
2.25

swimming pool
13
0.03
0.04

snow
14
0.15
-

clear cut
15
0.15
0.01

mixed
16
0.05
-

ligneous
17
0.01
0.03

greenhouse
18
0.12
0.2

other
19
0.14
-

## Example stylized images

Since this is derived from a general text-to-image model it is also possible to guide the generation further by modifying the prompt. Prompting can be used for guiding image style, seasonality, and architechture.

The following images were generated using the prompt `", aerial view of 31 Rue Molière, France` and varying the syle parameter. The examples were cherry picked with varying seed:

<table>
<tr>
<th>Image Segmentation</th>
<th>lego brick</th>
<th>pencil sketch</th>
<th>oil on canvas</th>
<th>stained glass</th>
</tr>
<tr>
<td><img src="images/fig2/mask.png" width="150" height="150"></td>
<td><img src="images/fig2/lego.png" width="150" height="150"></td>
<td><img src="images/fig2/pencil.png" width="150" height="150"></td>
<td><img src="images/fig2/oiloncanvas.png" width="150" height="150"></td>
<td><img src="images/fig2/stainedglass.png" width="150" height="150"></td>
</tr>
</table>

## How to use

### Drawing2Map web user interface

This project also comes with a webui to draw in the browser and generate images on the fly

- The webui can be used through this link: https://rubengr.es/Seg2Sat
- You can also run it locally: `cd ui && pip install -r requirements.txt && python start_webui.py`
- or with Google Colab: <a href="https://colab.research.google.com/github/RubenGres/Seg2Sat/blob/main/Drawing2Map_webui.ipynb" target="_parent"><img src="https://colab.research.google.com/assets/colab-badge.svg"/></a>

### Diffusers library

```python
from diffusers import StableDiffusionControlNetPipeline, StableDiffusionControlNetInpaintPipeline, ControlNetModel, UniPCMultistepScheduler
from diffusers.utils import load_image
from PIL import Image
import torch

dtype = torch.bfloat16 if torch.cuda.get_device_capability()[0] == 8 else torch.float16
device = "cuda:0" if torch.cuda.is_available() else "cpu"

controlnet = ControlNetModel.from_pretrained("rgres/Seg2Sat-sd-controlnet", torch_dtype=dtype)

pipe = StableDiffusionControlNetPipeline.from_pretrained(
"stabilityai/stable-diffusion-2-1-base", controlnet=controlnet, torch_dtype=dtype, safety_checker=None
).to("cuda")

segmentation = Image.open("segmentation.png")

image = pipe(
prompt="Aerial view of 31 Rue Molière, Paris, France.",
num_inference_steps=20, image=image
).images[0]

image.show()
```

## Download the model

The trained weights for the ControlNet model are available on HuggingFace [here](https://huggingface.co/rgres/Seg2Sat-sd-controlnet)

## About the dataset

To facilitate the use of StableDiffusion, a custom dataset has been derived from the FLAIR dataset. The TIFF images have been converted to PNG format for convenience, and the image segmentation process incorporates the recommended Look-Up Table (LUT) from the dataset documentation. Location data and descriptions have been gathered from the TIF files and the OpenStreetMap API to enrich the dataset, enabling the creation of meaningful labels for each image.

Each label is in the form `Aerial view of <OSM display_name>. <classes>`
*e.g. Aerial view of Pays de la Loire, France métropolitaine, 49260, France. brushwood, deciduous, herbaceous vegetation*

The complete dataset, consisting of over 61,000 images, can be accessed on HuggingFace [here](https://huggingface.co/datasets/IGNF/FLAIR_1_osm_clip).

## File Structure

This github repo consists of the following directories:

- `controlnet_training/`: Contains everything needed to train the ControlNet model with validation images.
- `dataset_preparation/`: Contains the code used to convert the FLAIR#1 dataset.
- `endpoint/`: Code for the hugging face endpoint
- `ui/`: Contains the code for the complete interface allowing you to draw from your browser
- `images/`: All images used in the README.md file
- `examples/`: Examples scripts for using this model with the diffusers library

## Limitations

One limitation of this project is that the FLAIR dataset covers France only. As a result, the learned representation and generated images may be biased towards French terrain. The model might struggle to accurately render other types of terrain outside of France. Future improvements could involve expanding the dataset to include a more diverse range of regions and landscapes.

Also theses results are obtained using a ControlNet on the base StableDiffusion 2.1 model. More details could be obtained in the generated images by finetuning a diffusion model on aerial imaging.