{"id":22417245,"url":"https://github.com/RubenGres/Seg2Sat","last_synced_at":"2025-08-01T02:30:53.524Z","repository":{"id":156133776,"uuid":"632058706","full_name":"RubenGres/Seg2Sat","owner":"RubenGres","description":"Using StableDiffusion and ControlNet to generate synthetic aerial images","archived":false,"fork":false,"pushed_at":"2024-04-11T15:04:34.000Z","size":58657,"stargazers_count":67,"open_issues_count":0,"forks_count":4,"subscribers_count":2,"default_branch":"main","last_synced_at":"2024-04-13T00:24:15.021Z","etag":null,"topics":["diffusion-models","earth-observation","remote-sensing","stable-diffusion","text-to-image"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/RubenGres.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2023-04-24T16:12:26.000Z","updated_at":"2024-04-14T20:45:32.082Z","dependencies_parsed_at":null,"dependency_job_id":"e22312d3-69ee-42ff-9382-5c32efe54a6a","html_url":"https://github.com/RubenGres/Seg2Sat","commit_stats":{"total_commits":91,"total_committers":4,"mean_commits":22.75,"dds":0.4065934065934066,"last_synced_commit":"79470a179380409885445197dbbcc9a42b3c3e9a"},"previous_names":["rubengres/drawing2map","rubengres/seg2sat"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RubenGres%2FSeg2Sat","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RubenGres%2FSeg2Sat/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RubenGres%2FSeg2Sat/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RubenGres%2FSeg2Sat/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/RubenGres","download_url":"https://codeload.github.com/RubenGres/Seg2Sat/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":228321218,"owners_count":17901604,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["diffusion-models","earth-observation","remote-sensing","stable-diffusion","text-to-image"],"created_at":"2024-12-05T15:17:19.221Z","updated_at":"2024-12-05T15:18:04.667Z","avatar_url":"https://github.com/RubenGres.png","language":"Jupyter Notebook","funding_links":[],"categories":["Remote Sensing \u003cins\u003eGenerative\u003c/ins\u003e Foundation Models","Generative networks"],"sub_categories":["Super-resolution - Miscellaneous"],"readme":"Seg2Sat - Segmentation to aerial view using pretrained diffuser models\n======\n\nSeg2Sat explores the potential of diffusion algorithms such as [StableDiffusion](https://github.com/CompVis/stable-diffusion) and [ControlNet](https://github.com/lllyasviel/ControlNet) to generate aerial images based on terrain segmentation data. The dataset is derived from IGN's [FLAIR (French Land cover from Aerospace Imagery)](https://ignf.github.io/FLAIR/), which provides land cover information for various regions in France. The goal is to create photorealistic synthetic aerial images or stylized aerial imagery.\n\nThis could have many uses like dataset creation/augmentation or making maps for your next ungeon and dragon session. Outpainting and inpainting are also possible and yield convincing results.\n\n## Example photorealistic images\n\nThese images are derived from the same segmentation mask in the first column and a fixed prompt: `\"Aerial view of 101 avenue de l'Espinet, Toulouse, Haute-Garonne, France\"`. The different image are obtained by varying the seed in the random number generator.\n\n\u003ctable\u003e\n  \u003ctr\u003e\n    \u003cth\u003eImage Segmentation\u003c/th\u003e\n    \u003cth colspan=\"4\"\u003eGenerated Images\u003c/th\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd\u003e\u003cimg src=\"images/fig1/0/mask.png\" width=\"150\" height=\"150\"\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003cimg src=\"images/fig1/0/0.png\" width=\"150\" height=\"150\"\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003cimg src=\"images/fig1/0/1.png\" width=\"150\" height=\"150\"\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003cimg src=\"images/fig1/0/2.png\" width=\"150\" height=\"150\"\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003cimg src=\"images/fig1/0/3.png\" width=\"150\" height=\"150\"\u003e\u003c/td\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd\u003e\u003cimg src=\"images/fig1/1/mask.png\" width=\"150\" height=\"150\"\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003cimg src=\"images/fig1/1/0.png\" width=\"150\" height=\"150\"\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003cimg src=\"images/fig1/1/1.png\" width=\"150\" height=\"150\"\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003cimg src=\"images/fig1/1/2.png\" width=\"150\" height=\"150\"\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003cimg src=\"images/fig1/1/3.png\" width=\"150\" height=\"150\"\u003e\u003c/td\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd\u003e\u003cimg src=\"images/fig1/2/mask.png\" width=\"150\" height=\"150\"\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003cimg src=\"images/fig1/2/0.png\" width=\"150\" height=\"150\"\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003cimg src=\"images/fig1/2/1.png\" width=\"150\" height=\"150\"\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003cimg src=\"images/fig1/2/2.png\" width=\"150\" height=\"150\"\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003cimg src=\"images/fig1/2/3.png\" width=\"150\" height=\"150\"\u003e\u003c/td\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd\u003e\u003cimg src=\"images/fig1/3/mask.png\" width=\"150\" height=\"150\"\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003cimg src=\"images/fig1/3/0.png\" width=\"150\" height=\"150\"\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003cimg src=\"images/fig1/3/1.png\" width=\"150\" height=\"150\"\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003cimg src=\"images/fig1/3/2.png\" width=\"150\" height=\"150\"\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003cimg src=\"images/fig1/3/3.png\" width=\"150\" height=\"150\"\u003e\u003c/td\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd\u003e\u003cimg src=\"images/fig1/4/mask.png\" width=\"150\" height=\"150\"\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003cimg src=\"images/fig1/4/0.png\" width=\"150\" height=\"150\"\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003cimg src=\"images/fig1/4/1.png\" width=\"150\" height=\"150\"\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003cimg src=\"images/fig1/4/2.png\" width=\"150\" height=\"150\"\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003cimg src=\"images/fig1/4/3.png\" width=\"150\" height=\"150\"\u003e\u003c/td\u003e\n  \u003c/tr\u003e\n\u003c/table\u003e\n\n\u003cdetails\u003e\n   \u003csummary\u003eLegend from the FLAIR dataset documentation\u003c/summary\u003e\n   \u003ctable style=\"width:80%;width:700px;\"\u003e\n      \u003cthead\u003e\n         \u003ctr\u003e\n            \u003cth width=\"7%\"\u003e\u003c/th\u003e\n            \u003cth\u003eClass\u003c/th\u003e\n            \u003cth style=\"text-align: center\" width=\"15%\"\u003eValue\u003c/th\u003e\n            \u003cth style=\"text-align: center\"\u003eFreq.-train (%)\u003c/th\u003e\n            \u003cth style=\"text-align: center\"\u003eFreq.-test (%)\u003c/th\u003e\n         \u003c/tr\u003e\n      \u003c/thead\u003e\n      \u003ctbody\u003e\n         \u003ctr\u003e\n            \u003ctd\u003e\u003cimg src=\"https://placehold.co/15x15/db0e9a/db0e9a.png\" alt=\"\"\u003e\u003c/td\u003e\n            \u003ctd\u003ebuilding\u003c/td\u003e\n            \u003ctd style=\"text-align: center\"\u003e1\u003c/td\u003e\n            \u003ctd style=\"text-align: center\"\u003e8.14\u003c/td\u003e\n            \u003ctd style=\"text-align: center\"\u003e8.6\u003c/td\u003e\n         \u003c/tr\u003e\n         \u003ctr\u003e\n            \u003ctd\u003e\u003cimg src=\"https://placehold.co/15x15/938e7b/938e7b.png\" alt=\"\"\u003e\u003c/td\u003e\n            \u003ctd\u003epervious surface\u003c/td\u003e\n            \u003ctd style=\"text-align: center\"\u003e2\u003c/td\u003e\n            \u003ctd style=\"text-align: center\"\u003e8.25\u003c/td\u003e\n            \u003ctd style=\"text-align: center\"\u003e7.34\u003c/td\u003e\n         \u003c/tr\u003e\n         \u003ctr\u003e\n            \u003ctd\u003e\u003cimg src=\"https://placehold.co/15x15/f80c00/f80c00.png\" alt=\"\"\u003e\u003c/td\u003e\n            \u003ctd\u003eimpervious surface\u003c/td\u003e\n            \u003ctd style=\"text-align: center\"\u003e3\u003c/td\u003e\n            \u003ctd style=\"text-align: center\"\u003e13.72\u003c/td\u003e\n            \u003ctd style=\"text-align: center\"\u003e14.98\u003c/td\u003e\n         \u003c/tr\u003e\n         \u003ctr\u003e\n            \u003ctd\u003e\u003cimg src=\"https://placehold.co/15x15/a97101/a97101.png\" alt=\"\"\u003e\u003c/td\u003e\n            \u003ctd\u003ebare soil\u003c/td\u003e\n            \u003ctd style=\"text-align: center\"\u003e4\u003c/td\u003e\n            \u003ctd style=\"text-align: center\"\u003e3.47\u003c/td\u003e\n            \u003ctd style=\"text-align: center\"\u003e4.36\u003c/td\u003e\n         \u003c/tr\u003e\n         \u003ctr\u003e\n            \u003ctd\u003e\u003cimg src=\"https://placehold.co/15x15/1553ae/1553ae.png\" alt=\"\"\u003e\u003c/td\u003e\n            \u003ctd\u003ewater\u003c/td\u003e\n            \u003ctd style=\"text-align: center\"\u003e5\u003c/td\u003e\n            \u003ctd style=\"text-align: center\"\u003e4.88\u003c/td\u003e\n            \u003ctd style=\"text-align: center\"\u003e5.98\u003c/td\u003e\n         \u003c/tr\u003e\n         \u003ctr\u003e\n            \u003ctd bgcolor=\"#194a26\"\u003e\u003cimg src=\"https://placehold.co/15x15/194a26/194a26.png\" alt=\"\"\u003e\u003c/td\u003e\n            \u003ctd\u003econiferous\u003c/td\u003e\n            \u003ctd style=\"text-align: center\"\u003e6\u003c/td\u003e\n            \u003ctd style=\"text-align: center\"\u003e2.74\u003c/td\u003e\n            \u003ctd style=\"text-align: center\"\u003e2.39\u003c/td\u003e\n         \u003c/tr\u003e\n         \u003ctr\u003e\n            \u003ctd bgcolor=\"#46e483\"\u003e\u003cimg src=\"https://placehold.co/15x15/46e483/46e483.png\" alt=\"\"\u003e\u003c/td\u003e\n            \u003ctd\u003edeciduous\u003c/td\u003e\n            \u003ctd style=\"text-align: center\"\u003e7\u003c/td\u003e\n            \u003ctd style=\"text-align: center\"\u003e15.38\u003c/td\u003e\n            \u003ctd style=\"text-align: center\"\u003e13.91\u003c/td\u003e\n         \u003c/tr\u003e\n         \u003ctr\u003e\n            \u003ctd bgcolor=\"#f3a60d\"\u003e\u003cimg src=\"https://placehold.co/15x15/f3a60d/f3a60d.png\" alt=\"\"\u003e\u003c/td\u003e\n            \u003ctd\u003ebrushwood\u003c/td\u003e\n            \u003ctd style=\"text-align: center\"\u003e8\u003c/td\u003e\n            \u003ctd style=\"text-align: center\"\u003e6.95\u003c/td\u003e\n            \u003ctd style=\"text-align: center\"\u003e6.91\u003c/td\u003e\n         \u003c/tr\u003e\n         \u003ctr\u003e\n            \u003ctd bgcolor=\"#660082\"\u003e\u003cimg src=\"https://placehold.co/15x15/660082/660082.png\" alt=\"\"\u003e\u003c/td\u003e\n            \u003ctd\u003evineyard\u003c/td\u003e\n            \u003ctd style=\"text-align: center\"\u003e9\u003c/td\u003e\n            \u003ctd style=\"text-align: center\"\u003e3.13\u003c/td\u003e\n            \u003ctd style=\"text-align: center\"\u003e3.87\u003c/td\u003e\n         \u003c/tr\u003e\n         \u003ctr\u003e\n            \u003ctd bgcolor=\"#55ff00\"\u003e\u003cimg src=\"https://placehold.co/15x15/55ff00/55ff00.png\" alt=\"\"\u003e\u003c/td\u003e\n            \u003ctd\u003eherbaceous vegetation\u003c/td\u003e\n            \u003ctd style=\"text-align: center\"\u003e10\u003c/td\u003e\n            \u003ctd style=\"text-align: center\"\u003e17.84\u003c/td\u003e\n            \u003ctd style=\"text-align: center\"\u003e22.17\u003c/td\u003e\n         \u003c/tr\u003e\n         \u003ctr\u003e\n            \u003ctd bgcolor=\"#fff30d\"\u003e\u003cimg src=\"https://placehold.co/15x15/fff30d/fff30d.png\" alt=\"\"\u003e\u003c/td\u003e\n            \u003ctd\u003eagricultural land\u003c/td\u003e\n            \u003ctd style=\"text-align: center\"\u003e11\u003c/td\u003e\n            \u003ctd style=\"text-align: center\"\u003e10.98\u003c/td\u003e\n            \u003ctd style=\"text-align: center\"\u003e6.95\u003c/td\u003e\n         \u003c/tr\u003e\n         \u003ctr\u003e\n            \u003ctd bgcolor=\"#e4df7c\"\u003e\u003cimg src=\"https://placehold.co/15x15/e4df7c/e4df7c.png\" alt=\"\"\u003e\u003c/td\u003e\n            \u003ctd\u003eplowed land\u003c/td\u003e\n            \u003ctd style=\"text-align: center\"\u003e12\u003c/td\u003e\n            \u003ctd style=\"text-align: center\"\u003e3.88\u003c/td\u003e\n            \u003ctd style=\"text-align: center\"\u003e2.25\u003c/td\u003e\n         \u003c/tr\u003e\n         \u003ctr\u003e\n            \u003ctd bgcolor=\"#3de6eb\"\u003e\u003cimg src=\"https://placehold.co/15x15/3de6eb/3de6eb.png\" alt=\"\"\u003e\u003c/td\u003e\n            \u003ctd\u003eswimming pool\u003c/td\u003e\n            \u003ctd style=\"text-align: center\"\u003e13\u003c/td\u003e\n            \u003ctd style=\"text-align: center\"\u003e0.03\u003c/td\u003e\n            \u003ctd style=\"text-align: center\"\u003e0.04\u003c/td\u003e\n         \u003c/tr\u003e\n         \u003ctr\u003e\n            \u003ctd bgcolor=\"#ffffff\"\u003e\u003cimg src=\"https://placehold.co/15x15/ffffff/ffffff.png\" alt=\"\"\u003e\u003c/td\u003e\n            \u003ctd\u003esnow\u003c/td\u003e\n            \u003ctd style=\"text-align: center\"\u003e14\u003c/td\u003e\n            \u003ctd style=\"text-align: center\"\u003e0.15\u003c/td\u003e\n            \u003ctd style=\"text-align: center\"\u003e-\u003c/td\u003e\n         \u003c/tr\u003e\n         \u003ctr\u003e\n            \u003ctd bgcolor=\"#8ab3a0\"\u003e\u003cimg src=\"https://placehold.co/15x15/8ab3a0/8ab3a0.png\" alt=\"\"\u003e\u003c/td\u003e\n            \u003ctd\u003eclear cut\u003c/td\u003e\n            \u003ctd style=\"text-align: center\"\u003e15\u003c/td\u003e\n            \u003ctd style=\"text-align: center\"\u003e0.15\u003c/td\u003e\n            \u003ctd style=\"text-align: center\"\u003e0.01\u003c/td\u003e\n         \u003c/tr\u003e\n         \u003ctr\u003e\n            \u003ctd bgcolor=\"#6b714f\"\u003e\u003cimg src=\"https://placehold.co/15x15/6b714f/6b714f.png\" alt=\"\"\u003e\u003c/td\u003e\n            \u003ctd\u003emixed\u003c/td\u003e\n            \u003ctd style=\"text-align: center\"\u003e16\u003c/td\u003e\n            \u003ctd style=\"text-align: center\"\u003e0.05\u003c/td\u003e\n            \u003ctd style=\"text-align: center\"\u003e-\u003c/td\u003e\n         \u003c/tr\u003e\n         \u003ctr\u003e\n            \u003ctd bgcolor=\"#c5dc42\"\u003e\u003cimg src=\"https://placehold.co/15x15/c5dc42/c5dc42.png\" alt=\"\"\u003e\u003c/td\u003e\n            \u003ctd\u003eligneous\u003c/td\u003e\n            \u003ctd style=\"text-align: center\"\u003e17\u003c/td\u003e\n            \u003ctd style=\"text-align: center\"\u003e0.01\u003c/td\u003e\n            \u003ctd style=\"text-align: center\"\u003e0.03\u003c/td\u003e\n         \u003c/tr\u003e\n         \u003ctr\u003e\n            \u003ctd bgcolor=\"#9999ff\"\u003e\u003cimg src=\"https://placehold.co/15x15/9999ff/9999ff.png\" alt=\"\"\u003e\u003c/td\u003e\n            \u003ctd\u003egreenhouse\u003c/td\u003e\n            \u003ctd style=\"text-align: center\"\u003e18\u003c/td\u003e\n            \u003ctd style=\"text-align: center\"\u003e0.12\u003c/td\u003e\n            \u003ctd style=\"text-align: center\"\u003e0.2\u003c/td\u003e\n         \u003c/tr\u003e\n         \u003ctr\u003e\n            \u003ctd bgcolor=\"#000000\"\u003e\u003cimg src=\"https://placehold.co/15x15/000000/000000.png\" alt=\"\"\u003e\u003c/td\u003e\n            \u003ctd\u003eother\u003c/td\u003e\n            \u003ctd style=\"text-align: center\"\u003e19\u003c/td\u003e\n            \u003ctd style=\"text-align: center\"\u003e0.14\u003c/td\u003e\n            \u003ctd style=\"text-align: center\"\u003e-\u003c/td\u003e\n         \u003c/tr\u003e\n      \u003c/tbody\u003e\n   \u003c/table\u003e\n\u003c/details\u003e  \n\n\u003cbr\u003e\n\n## Example stylized images\n\nSince this is derived from a general text-to-image model it is also possible to guide the generation further by modifying the prompt. Prompting can be used for guiding image style, seasonality, and architechture.\n\nThe following images were generated using the prompt `\"\u003cstyle\u003e, aerial view of 31 Rue Molière, France` and varying the syle parameter. The examples were cherry picked with varying seed:\n\n\u003ctable\u003e\n   \u003ctr\u003e\n      \u003cth\u003eImage Segmentation\u003c/th\u003e\n      \u003cth\u003elego brick\u003c/th\u003e\n      \u003cth\u003epencil sketch\u003c/th\u003e\n      \u003cth\u003eoil on canvas\u003c/th\u003e\n      \u003cth\u003estained glass\u003c/th\u003e\n   \u003c/tr\u003e\n   \u003ctr\u003e\n      \u003ctd\u003e\u003cimg src=\"images/fig2/mask.png\" width=\"150\" height=\"150\"\u003e\u003c/td\u003e\n      \u003ctd\u003e\u003cimg src=\"images/fig2/lego.png\" width=\"150\" height=\"150\"\u003e\u003c/td\u003e\n      \u003ctd\u003e\u003cimg src=\"images/fig2/pencil.png\" width=\"150\" height=\"150\"\u003e\u003c/td\u003e\n      \u003ctd\u003e\u003cimg src=\"images/fig2/oiloncanvas.png\" width=\"150\" height=\"150\"\u003e\u003c/td\u003e\n      \u003ctd\u003e\u003cimg src=\"images/fig2/stainedglass.png\" width=\"150\" height=\"150\"\u003e\u003c/td\u003e\n   \u003c/tr\u003e\n\u003c/table\u003e\n\n## How to use\n\n### Drawing2Map web user interface\n\nThis project also comes with a webui to draw in the browser and generate images on the fly\n\n\u003ccenter\u003e\n  \u003cimg src=\"images/webui_example.png\" alt=\"drawing\" style=\"width:450px;\"/\u003e\n\u003c/center\u003e\n\n- The webui can be used through this link: https://rubengr.es/Seg2Sat\n- You can also run it locally: `cd ui \u0026\u0026 pip install -r requirements.txt \u0026\u0026 python start_webui.py`\n- or with Google Colab: \u003ca href=\"https://colab.research.google.com/github/RubenGres/Seg2Sat/blob/main/Drawing2Map_webui.ipynb\" target=\"_parent\"\u003e\u003cimg src=\"https://colab.research.google.com/assets/colab-badge.svg\"/\u003e\u003c/a\u003e\n\n### Diffusers library\n\n```python\nfrom diffusers import StableDiffusionControlNetPipeline, StableDiffusionControlNetInpaintPipeline, ControlNetModel, UniPCMultistepScheduler\nfrom diffusers.utils import load_image\nfrom PIL import Image\nimport torch\n\ndtype = torch.bfloat16 if torch.cuda.get_device_capability()[0] == 8 else torch.float16\ndevice = \"cuda:0\" if torch.cuda.is_available() else \"cpu\"\n\ncontrolnet = ControlNetModel.from_pretrained(\"rgres/Seg2Sat-sd-controlnet\", torch_dtype=dtype)\n\npipe = StableDiffusionControlNetPipeline.from_pretrained(\n    \"stabilityai/stable-diffusion-2-1-base\", controlnet=controlnet, torch_dtype=dtype, safety_checker=None\n).to(\"cuda\")\n\nsegmentation = Image.open(\"segmentation.png\")\n\nimage = pipe(\n   prompt=\"Aerial view of 31 Rue Molière, Paris, France.\",\n   num_inference_steps=20, image=image\n).images[0]\n\nimage.show()\n```\n\n\u003ca href=\"https://colab.research.google.com/github/RubenGres/Seg2Sat/blob/main/Seg2Sat_inference_example.ipynb\" target=\"_parent\"\u003e\u003cimg src=\"https://colab.research.google.com/assets/colab-badge.svg\"/\u003e\u003c/a\u003e\n\n## Download the model\n\nThe trained weights for the ControlNet model are available on HuggingFace [here](https://huggingface.co/rgres/Seg2Sat-sd-controlnet)\n\n## About the dataset\n\nTo facilitate the use of StableDiffusion, a custom dataset has been derived from the FLAIR dataset. The TIFF images have been converted to PNG format for convenience, and the image segmentation process incorporates the recommended Look-Up Table (LUT) from the dataset documentation. Location data and descriptions have been gathered from the TIF files and the OpenStreetMap API to enrich the dataset, enabling the creation of meaningful labels for each image.  \n\nEach label is in the form `Aerial view of \u003cOSM display_name\u003e. \u003cclasses\u003e`  \n*e.g. Aerial view of Pays de la Loire, France métropolitaine, 49260, France. brushwood, deciduous, herbaceous vegetation*\n\nThe complete dataset, consisting of over 61,000 images, can be accessed on HuggingFace [here](https://huggingface.co/datasets/IGNF/FLAIR_1_osm_clip).\n\n## File Structure\n\nThis github repo consists of the following directories:\n\n- `controlnet_training/`: Contains everything needed to train the ControlNet model with validation images.\n- `dataset_preparation/`: Contains the code used to convert the FLAIR#1 dataset.\n- `endpoint/`: Code for the hugging face endpoint\n- `ui/`: Contains the code for the complete interface allowing you to draw from your browser\n- `images/`: All images used in the README.md file\n- `examples/`: Examples scripts for using this model with the diffusers library\n  \n## Limitations\n\nOne limitation of this project is that the FLAIR dataset covers France only. As a result, the learned representation and generated images may be biased towards French terrain. The model might struggle to accurately render other types of terrain outside of France. Future improvements could involve expanding the dataset to include a more diverse range of regions and landscapes.  \n\nAlso theses results are obtained using a ControlNet on the base StableDiffusion 2.1 model. More details could be obtained in the generated images by finetuning a diffusion model on aerial imaging.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FRubenGres%2FSeg2Sat","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FRubenGres%2FSeg2Sat","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FRubenGres%2FSeg2Sat/lists"}