{"id":18439176,"url":"https://github.com/wsh032/sd-webui-fast-dataset-maker","last_synced_at":"2025-04-07T21:32:14.259Z","repository":{"id":169383204,"uuid":"642078694","full_name":"WSH032/sd-webui-fast-dataset-maker","owner":"WSH032","description":"A funny extension that integrates image-browsing , downloader , deduplicate , cluster , can quickly collect, classify and process your images. | 一个有趣的扩展，整合了 图库，下载，去重，聚类 ，可以快速搜集、分类、处理你的图片。","archived":false,"fork":false,"pushed_at":"2023-08-08T13:45:09.000Z","size":127,"stargazers_count":23,"open_issues_count":1,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-23T01:01:39.534Z","etag":null,"topics":["cluster","dataset-maker","deduplicate","gelbooru","image-browsing","image-dataset","image-downloader","stable-duffusion","webui"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/WSH032.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-05-17T19:17:59.000Z","updated_at":"2025-02-26T00:46:33.000Z","dependencies_parsed_at":null,"dependency_job_id":"2cd5897b-663a-44b4-ac1b-a2c22ac487ce","html_url":"https://github.com/WSH032/sd-webui-fast-dataset-maker","commit_stats":null,"previous_names":["wsh032/sd-webui-fast-dataset-maker"],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/WSH032%2Fsd-webui-fast-dataset-maker","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/WSH032%2Fsd-webui-fast-dataset-maker/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/WSH032%2Fsd-webui-fast-dataset-maker/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/WSH032%2Fsd-webui-fast-dataset-maker/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/WSH032","download_url":"https://codeload.github.com/WSH032/sd-webui-fast-dataset-maker/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247732659,"owners_count":20986898,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cluster","dataset-maker","deduplicate","gelbooru","image-browsing","image-dataset","image-downloader","stable-duffusion","webui"],"created_at":"2024-11-06T06:23:39.096Z","updated_at":"2025-04-07T21:32:14.248Z","avatar_url":"https://github.com/WSH032.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# sd-webui-fast-dataset-maker\n一个有趣的扩展，整合了 `图库`，`下载`，`去重`，`聚类` ，可以快速搜集、分类、处理你的图片。\n\nA funny extension that integrates `image-browsing` , `downloader` , `deduplicate` , `cluster` , can quickly collect, classify and process your images.\n\n### Powered by\n\n- [zanllp/sd-webui-infinite-image-browsing](https://github.com/zanllp/sd-webui-infinite-image-browsing)\n- [WSH032/Gelbooru-API-Downloader](https://github.com/WSH032/Gelbooru-API-Downloader)\n- [WSH032/image-deduplicate-cluster-webui](https://github.com/WSH032/image-deduplicate-cluster-webui)\n- [toshiaki1729/dataset-tag-editor-standalone](https://github.com/toshiaki1729/dataset-tag-editor-standalone)\n\n\u003e 我只是把它们整合起来，并独立运行\n\n### Credit\nWebUI部分借鉴了[AUTOMATIC1111/stable-diffusion-webui](https://github.com/AUTOMATIC1111/stable-diffusion-webui)\n\n\n\n## 现在我们有什么？\n### 1. 强大的图片浏览器\n来自[zanllp/sd-webui-infinite-image-browsing](https://github.com/zanllp/sd-webui-infinite-image-browsing)\n![sd-webui-infinite-image-browsing](https://github.com/WSH032/sd-webui-fast-dataset-maker/assets/126865849/b98a293f-a3a0-4d59-a997-cae86e7f25b4)\n\n### 2. Gelbooru的图片下载器\n来自[WSH032/Gelbooru-API-Downloader](https://github.com/WSH032/Gelbooru-API-Downloader)\n![Gelbooru-API-Downloader](https://github.com/WSH032/sd-webui-fast-dataset-maker/assets/126865849/03dbd069-336c-40a1-9e14-5143eb915339)\n\n### 3. 基于imagededup，sklearn，WD14模型的图片去重与聚类\n来自[WSH032/image-deduplicate-cluster-webui](https://github.com/WSH032/image-deduplicate-cluster-webui)\n![deduplicate-webui](https://github.com/WSH032/sd-webui-fast-dataset-maker/assets/126865849/dccd5928-cdfc-4d58-b806-edc0ed2df9c9)\n\n**[cluster-webui 展示](https://github.com/WSH032/sd-webui-fast-dataset-maker/assets/126865849/c38200fa-78dc-4b91-a006-1784fb7059bb)**\n\n### 4. 强大的booru风格的tag编辑插件\n来自[toshiaki1729/dataset-tag-editor-standalone](https://github.com/toshiaki1729/dataset-tag-editor-standalone)\n![dataset-tag-editor-standalone](https://github.com/WSH032/sd-webui-fast-dataset-maker/assets/126865849/6761bbb6-b9bb-4463-a41c-0b0eceb1baab)\n\n**[dataset-tag-editor-standalone 完整展示](https://github.com/WSH032/sd-webui-fast-dataset-maker/assets/126865849/919a2d87-d399-41a9-86bf-b8c4e50973b0)**\n\n\n## 😀Development\n如果你觉得此项目有用💪，可以去 [![GitHub Repo stars](https://img.shields.io/github/stars/WSH032/sd-webui-fast-dataset-maker?style=social)](https://github.com/WSH032/sd-webui-fast-dataset-maker) 点一颗小星星🤤，非常感谢你⭐\n\n遇到问题可以在[Github上提issue ❓](https://github.com/WSH032/sd-webui-fast-dataset-maker/issues)\n\n\n## Change History\n\n### V 1.1.0\n08 Aug.2023 2023/08/08\n#### New Features:\n- Bump `image-deduplicate-cluster-webui` version to [`2.0.1`](https://github.com/WSH032/image-deduplicate-cluster-webui/tree/v2.0.1)\n  - refer to [v2.0.1#change-history](https://github.com/WSH032/image-deduplicate-cluster-webui/tree/v2.0.1#change-history)\n- Bump `sd-webui-infinite-image-browsing` version to 2023-8-4 [`fc1853c`](https://github.com/zanllp/sd-webui-infinite-image-browsing/tree/fc1853c1476324fa082a57d4e2e480e425bb7c7b)\n  - refer to [Change-log 2023-7-30](https://github.com/zanllp/sd-webui-infinite-image-browsing/wiki/Change-log)\n- Bump `dataset-tag-editor-standalone` version to 2023-8-8 [`2569280`](https://github.com/WSH032/dataset-tag-editor-standalone/tree/25692806ad64afe55b8c1eebfebc071a5529d9b5)\n  - 默认启用512分辨率缩略图\n- Update Colab Notebook\n  - 更新Colab notebook以匹配新版本\n\n#### Other Changes:\n- 只有在载入某个扩展时，才会临时修改sys.path，以防止载入各扩展时的潜在冲突\n- 添加`weibui.WebuiUtils._reload_script_modules`以供在Jupyter中使用时，重新载入、更新包与模块\n  - 实验性功能，未进行测试，可能会在未来版本中修改或移除\n  - 如果可行，建议直接依靠重启Jupyter内核来重新载入、更新包与模块\n\n#### Bug Fixes:\n- 载入扩展时的import将使用其所在文件夹做为顶级包\n  - 将修复：载入图库扩展时导入的包，与其内部导入的包的命名空间指向不同\n- 修复在Jupyter中多次重复运行时，错误修改`gradio.routes.templates.TemplateResponse`而造成的`\u003cscript scr\u003e`标签重复及顺序错误的问题\n\n## 更新 Update\n部署使用时更新方式：\n\n本项目带有子模块，请使用以下命令拉取新的更新，并再次重复 `安装 Install` ，以免有新的依赖要求\n```shell\ngit pull --recurse-submodules \n```\n如果你不会使用git命令，可以运行`update.ps1`完成更新\n\n## 安装 Install\n\n### （一）Colab使用\n#### 本项目链接 Fast-Dataset-Maker Colab\n| Notebook Name | Description | Link | Old-Version |\n| --- | --- | --- | --- |\n| [fast_dataset_maker](https://github.com/WSH032/sd-webui-fast-dataset-maker) `NEW` | 整合了 `图库`，`下载`，`去重`，`聚类` 的图片数据集WebUI | [![](https://img.shields.io/static/v1?message=Open%20in%20Colab\u0026logo=googlecolab\u0026labelColor=5c5c5c\u0026color=0f80c1\u0026label=%20\u0026style=flat)](https://colab.research.google.com/github/WSH032/sd-webui-fast-dataset-maker/blob/main/fast_dataset_maker.ipynb) |\n\n---\n\n#### 友情链接 SD-Lora-Training Colab\n| Notebook Name | Description | Link | Old-Version |\n| --- | --- | --- | --- |\n| [Colab_Lora_train](https://github.com/WSH032/lora-scripts/) | 基于[Akegarasu/lora-scripts](https://github.com/Akegarasu/lora-scripts)的定制化Colab notebook | [![](https://img.shields.io/static/v1?message=Open%20in%20Colab\u0026logo=googlecolab\u0026labelColor=5c5c5c\u0026color=0f80c1\u0026label=%20\u0026style=flat)](https://colab.research.google.com/github/WSH032/lora-scripts/blob/main/Colab_Lora_train.ipynb) | [![](https://img.shields.io/static/v1?message=Older%20Version\u0026logo=googlecolab\u0026labelColor=5c5c5c\u0026color=e74c3c\u0026label=%20\u0026style=flat)](https://colab.research.google.com/drive/1_f0qJdM43BSssNJWtgjIlk9DkIzLPadx) |\n| [kohya_train_webui](https://github.com/WSH032/kohya-config-webui) `NEW` | 基于[WSH032/kohya-config-webui](https://github.com/WSH032/kohya-config-webui)的WebUI版Colab notebook | [![](https://img.shields.io/static/v1?message=Open%20in%20Colab\u0026logo=googlecolab\u0026labelColor=5c5c5c\u0026color=0f80c1\u0026label=%20\u0026style=flat)](https://colab.research.google.com/github/WSH032/kohya-config-webui/blob/main/kohya_train_webui.ipynb) |\n\n### （二）部署使用\n**请尽量使用python \u003e= 3.10**\n\n运行以下git命令克隆仓库并初始化子模块\n```shell\ngit clone https://github.com/WSH032/sd-webui-fast-dataset-maker.git --recurse-submodules\n```\n安装依赖\n- Window用户可以运行 `install.ps1` 安装依赖\n- Linux用户请自行通过 `pip install -r requirements.txt` 安装依赖\n\n```shell\n# 对于通过 `requirements.txt` 安装的用户请注意\n# 如果你想使用 Torch Cuda 或者 GPU 进行加速\n# 请打开 `requirements.txt` 修改安装 torch=2.0.0 + cuda118\n\n# If you want to install with CUDA, please use the following command:\n\n--extra-index-url https://download.pytorch.org/whl/cu118\ntorch==2.0.0+cu118\ntorchvision==0.15.1+cu118\n```\n#### 注意\n\u003e 此WebUI使用了Gradio非官方文档的操作方法，并在 `gradio \u003c=3.35.2, \u003e=3.29.0` 进行了测试，但不保证未来Gradio官方不会修改相应接口\n\u003e\n\u003e `requirements.txt` 内只要求了 `gradio \u003e= 3.31.0` ，如果出现问题请尝试降级至 `gradio \u003c=3.35.2, \u003e=3.31.0`\n\u003e\n\u003e 或者使用 `requirements_versions.txt` 安装依赖\n\n\n### （三）做为[AUTOMATIC1111/stable-diffusion-webui](https://github.com/AUTOMATIC1111/stable-diffusion-webui)的扩展使用\n请参阅原作者各自的仓库，几乎所有扩展均可做为 SD-WebUI 的扩展使用\n- [zanllp/sd-webui-infinite-image-browsing](https://github.com/zanllp/sd-webui-infinite-image-browsing)\n- [WSH032/Gelbooru-API-Downloader](https://github.com/WSH032/Gelbooru-API-Downloader)\n- [WSH032/image-deduplicate-cluster-webui](https://github.com/WSH032/image-deduplicate-cluster-webui)\n- [toshiaki1729/dataset-tag-editor-standalone](https://github.com/toshiaki1729/dataset-tag-editor-standalone)\n\n![A1111-SD-WebUI](https://github.com/WSH032/sd-webui-fast-dataset-maker/assets/126865849/63caea91-42ce-4be5-b3c7-90ddaff17487)\n\n\n## 部署使用 - 使用方法\n\n### **注意**：\n**`dataset-tag-editor-standalone`扩展，在以原图方式，处理超过50张图片时候，会消耗巨大内存和时间，并且可能会出现图片过滤器不工作的情况**\n- 610张，总占用1.33G的图片；以512分辨率运行，耗时100s处理完毕\n- 本项目在`v1.1.0`更新中，已经默认为`dataset-tag-editor-standalone`设置启用512分辨率缩略图\n- 如果你需要更少的内存和更快的速度，可以自行调低分辨率；或者你也可以自行调高分辨率，用更长的处理时间换取更清晰的图片（不建议，因为512基本够用）\n\n![setting_resolution](https://github.com/WSH032/sd-webui-fast-dataset-maker/assets/126865849/a4780903-9b41-46db-9a0f-e740b60e57cf)\n\n---\n\nPlease refer to the original author's repository for the specific usage of each extension\n\n各插件具体的使用方法请查看原作者仓库自述文件，这里不再重复\n\n---\n\n### （一） 新手 请运行 `run_webui.ps1`\n\n### （二） 进阶 Shell中运行并设置参数\n```shell\npython webui.py # 使用 `--help` 查看可选参数\n\n# 例如\n# python webui.py --sd_webui_config=\"config.json\" --update_image_index --inbrowser\n\n```\n\n### （三） Python API 接口\n\n请参阅 Colab 示例 或者 `run_webui.ipynb`\n\n\n```python\nfrom webui import WebuiUtils\n\n# params for extension\n# 为各扩展设置参数，具体参数请参阅 `extensions/extensions_preload.py`\n# 或者运行 `WebuiUtils(help=True)` 查看帮助信息\n# Pyhton 接口参数 与 命令行参数一致\nwebui_utils = WebuiUtils(\n\n    # for image_browsing setting\n    sd_webui_config = \"config.json\",\n    extra_paths = [\"path\"],\n    update_image_index = True,\n    \n    # disable extension\n    disable_image_browsing = False,\n    disable_deduplicate_cluster = False,\n    disable_tag_editor = False,\n)\n\n\"\"\"\nThe args of `queue()` and `launch()` are the same as `gradio.Blocks.queue()` and `gradio.Blocks.launch()`\n\n接下来把它当成 `gradio.Blocks` 使用就好，所有接口与 `gradio.Blocks` 一致\n\"\"\"\nwebui_utils.queue(concurrency_count=2)  # equal to `gradio.Blocks.queue(concurrency_count=2)`\n\nwebui_utils.launch(debug=True, server_port=7860)  # equal to `gradio.Blocks.launch(debug=True, server_port=7860)`\n\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwsh032%2Fsd-webui-fast-dataset-maker","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fwsh032%2Fsd-webui-fast-dataset-maker","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwsh032%2Fsd-webui-fast-dataset-maker/lists"}