{"id":18439180,"url":"https://github.com/wsh032/gelbooru-api-downloader","last_synced_at":"2025-04-15T03:42:19.847Z","repository":{"id":168161604,"uuid":"643790607","full_name":"WSH032/Gelbooru-API-Downloader","owner":"WSH032","description":"A async coroutine script for gelbooru API and download.   一个采用异步协程，用于通过指定tags访问gelbooru公共API，并下载图片的脚本","archived":false,"fork":false,"pushed_at":"2024-07-03T20:30:55.000Z","size":659,"stargazers_count":8,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-28T15:21:18.158Z","etag":null,"topics":["api","asyncio","downloader","gelbooru","grabber","images","scraper"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/WSH032.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-05-22T06:59:48.000Z","updated_at":"2025-03-19T08:53:33.000Z","dependencies_parsed_at":null,"dependency_job_id":"b4854d74-3b8c-4ec1-bf9b-81bbd849f2df","html_url":"https://github.com/WSH032/Gelbooru-API-Downloader","commit_stats":null,"previous_names":["wsh032/gelbooru-api-downloader"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/WSH032%2FGelbooru-API-Downloader","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/WSH032%2FGelbooru-API-Downloader/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/WSH032%2FGelbooru-API-Downloader/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/WSH032%2FGelbooru-API-Downloader/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/WSH032","download_url":"https://codeload.github.com/WSH032/Gelbooru-API-Downloader/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":249003936,"owners_count":21196794,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["api","asyncio","downloader","gelbooru","grabber","images","scraper"],"created_at":"2024-11-06T06:23:39.486Z","updated_at":"2025-04-15T03:42:19.826Z","avatar_url":"https://github.com/WSH032.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Gelbooru-API-Downloader\n\nA async coroutine script for gelbooru API and download.  \n\n一个采用异步协程，用于通过指定tags关键词访问[Gelbooru](https://gelbooru.com/)公共API，并下载图片的脚本。\n\n对于[SD训练](https://github.com/kohya-ss/sd-scripts)很有用。\n\n## 现在我们有什么？\n\n输入你想要的tags关键词，自动查询Gelbooru的API，然后采用异步协程方式，并发下载图片，同时会把gelbooru上图片的详细tags一同保存\n特点：\n\n- Download automatically | 自动化下载\n- Download images with tag | 下载图片和tags\n- MD5, Say no to duplication | 重复文件md5校验\n- Asyncio and coroutine, fast | 协程异步并发写入和下载\n- httpx and aiofiles | 全部采用协程，不包含任何同步阻塞操作\n- Save detailed gelbooru tags txt concurrently | 下载图片同时保存详细的gelbooru tags\n- Get gelbooru tags without downloading duplicate images | 已经存在图片但无tags文本，会自动补全\n\n### powershell一键运行\n\n![use_in_powershell](./docs/use_in_powershell.png)\n\n### 并发下载，（用jupyter演示）\n\n![Scrape_images](./docs/Scrape_images.png)\n\n### API查询\n\n![GetAPI](./docs/GetAPI.png)\n\n## Credit\n\n**Attention! It's probably against [Gelbooru's TOS](https://gelbooru.com/tos.php)!**\n\n`scrape_images` 以自动的方式，httpx默认的UA头访问gelbooru的公共API，然后并发地异步下载图片\n\n**虽然我没找到gelbooru在这方面的限制，但这仍然可能违反了gelbooru的政策！**\n\n**不要滥用这个脚本，不要大量下载gelbooru的图片，保证下载频率和最大协程数不要过高！**\n\n**如果你觉得这个脚本有用，你应该感谢gelbooru的无私贡献\n你可以以[捐赠或者购买商品的形式](https://buymyshit.moneygrubbingwhore.com/index.php?page=products\u0026s=list)支持他们**\n\n## 安装\n\n**请保证 `python\u003e=3.9`**\n\n```shell\npip install -r requirements.txt\n```\n\n## 使用方式\n\n\u003e [!TIP]\n\u003e\n\u003e 中国用户访问gelbooru请自行使用系统代理.\n\u003e\n\u003e 或者通过 `$env:HTTPS_PROXY = http://127.0.0.1:7890` 的方式，以环境变量 `HTTPS_PROXY`指定代理服务器.\n\n\u003c!-- 此注释是为了避免markdown lint报错 --\u003e\n\n\u003e [!TIP]\n\u003e\n\u003e 这里有一些关于Gelbooru的tags/API规则:\n\u003e\n\u003e - [API](https://gelbooru.com/index.php?page=wiki\u0026s=view\u0026id=18780)\n\u003e - [tags](https://gelbooru.com/index.php?page=wiki\u0026s=\u0026s=view\u0026id=25921)\n\u003e - [cheatsheet](https://gelbooru.com/index.php?page=wiki\u0026s=\u0026s=view\u0026id=26263)\n\n### 脚本方式(推荐)\n\n在windows环境中，修改 [run_download_images_coroutine.ps1](run_download_images_coroutine.ps1) 中内容，powershell运行即可。\n\n```shell\n$tags = \"hifumi_(blue_archive)\"    # 符合gelbooru搜索规则的tags | tags for gelbooru\n$max_images_number = 200    # 需要下载的图片数 | the number of images you need to download\n$download_dir = \"images\"    # 下载图片的路径 | the folder path to download images\n$max_workers= 15    # 最大下载协程数 | maximum number of download coroutines\n$unit = 100    # 下载单位，下载图片数以此向上取一单位 | unit for download. eg: max_images_number=11, unit=10, then you get 20\n$timeout = 10    # 下载超时限制 | download connecting timeout limit\n```\n\n### API方式\n\n请查看[download_images_coroutine.py](download_images_coroutine.py)\n\n## Todo\n\n- [ ] 增加更多booru支持\n- [ ] ~~增加对pixiv的支持~~\n- [ ] 增加是否覆盖tags文本选项\n\n## Change History\n\n### 04 Jul.2024 2024/07/04\n\n- **使用 `pydantic` 代替 `pandas` 作为依赖项**\n- 重构代码，采用Google规范的Docstring\n- 移除被弃用的多线程下载器\n- 使用 `ruff` 进行格式化和lint修复\n- 微调部分函数签名\n\n\u003cdetails\u003e\n\n\u003csummary\u003eMore History\u003c/summary\u003e\n\n### 18 Jun.2023 2023/06/18\n\n- 增加了显示下载速度的功能\n- 基础类`class Downloader`的`.download()`方法输出类型已更改，现在会输出一个`class DownloadResult`的实例\n    - 因为实现了`__eq__`方法，所以仍然可以用`Downloader.download() == 1`等判断下载是否成功\n    - 具体请看[download_images_coroutine.py](download_images_coroutine.py)中的`class DownloadResult`定义\n\n### 08 Jun.2023 2023/06/08\n\n**新增了pillow库的要求.**\n\n- 增加检查下载目录中错误图片的功能[1#issue](https://github.com/WSH032/Gelbooru-API-Downloader/issues/1)\n    - 下载时候的使用方法请看[run_download_images_coroutine.ps1](run_download_images_coroutine.ps1)\n    - 你也做为单独的工具脚本使用，请看[utils/run_check_images.ps1](utils/run_check_images.ps1)\n    - API方式\n\n      ```python\n      await Scrape_images(*arg,**kwargs,\n      check_images_mode: Union[None, int]=None,  # 新增参数\n      )\n      # check_images_mode为是否在下载结束后检查图片是否正确\n      #   默认为None，不检查\n      #   0为检查，但只输出信息不做任何操作\n      #   1为尝试修复图片\n      #   2为尝试删除图片\n      ```\n\n\u003c/details\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwsh032%2Fgelbooru-api-downloader","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fwsh032%2Fgelbooru-api-downloader","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwsh032%2Fgelbooru-api-downloader/lists"}