{"id":20189457,"url":"https://github.com/jackfsuia/llm-data-cleaner","last_synced_at":"2025-07-06T11:33:17.172Z","repository":{"id":237534687,"uuid":"794665654","full_name":"jackfsuia/LLM-Data-Cleaner","owner":"jackfsuia","description":"用大模型批量处理数据，现支持各种大模型做OCR，支持通义千问, 月之暗面, 百度飞桨OCR, OpenAI 和LLAVA。Use LLM to generate or clean data for academic use. Support OCR with qwen, moonshot, PaddleOCR, OpenAI, Llava. ","archived":false,"fork":false,"pushed_at":"2024-09-15T18:47:58.000Z","size":625,"stargazers_count":13,"open_issues_count":0,"forks_count":3,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-10T09:05:49.798Z","etag":null,"topics":["dataset","llm","ocr"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jackfsuia.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-05-01T17:38:56.000Z","updated_at":"2025-03-02T15:25:37.000Z","dependencies_parsed_at":"2024-11-14T03:37:48.153Z","dependency_job_id":"a9033890-93ec-4fcf-908f-8d6dd25f4c67","html_url":"https://github.com/jackfsuia/LLM-Data-Cleaner","commit_stats":null,"previous_names":["jackfsuia/llm-data-cleaner"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/jackfsuia/LLM-Data-Cleaner","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jackfsuia%2FLLM-Data-Cleaner","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jackfsuia%2FLLM-Data-Cleaner/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jackfsuia%2FLLM-Data-Cleaner/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jackfsuia%2FLLM-Data-Cleaner/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jackfsuia","download_url":"https://codeload.github.com/jackfsuia/LLM-Data-Cleaner/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jackfsuia%2FLLM-Data-Cleaner/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":263893607,"owners_count":23526322,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["dataset","llm","ocr"],"created_at":"2024-11-14T03:37:19.492Z","updated_at":"2025-07-06T11:33:17.151Z","avatar_url":"https://github.com/jackfsuia.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cp align=\"left\"\u003e\n    \u003cimg src=\"logo/logo.png\" width=\"26%\" \u003e\n\u003c/p\u003e\n\n# LLM-Data-Cleaner\n简体中文 | [English](README_en.md)\n\n## 更新\n- 看来有人专门训练LLM来干这些活了。见[refuel-llm-2](https://www.refuel.ai/blog-posts/announcing-refuel-llm-2)。\n- 关于OCR，可以结合使用[Nougat](https://github.com/facebookresearch/nougat)、[Marker](https://github.com/VikParuchuri/marker)、或多模态大模型[MiniCPM-V](https://github.com/OpenBMB/MiniCPM-V)。\n## 背景\n未来人类会用大模型预加工所有数据。本项目旨在结合大模型来批量预处理数据，以支持科研目的。 现阶段支持OCR功能, 支持使用的大模型有 qwen(通义千问), moonshot(月之暗面), PaddleOCR(百度飞桨OCR), openai, Llava。\n## 启动\n克隆并且进入仓库\n```bash\ngit clone https://github.com/jackfsuia/LLM-Data-Cleaner.git \u0026\u0026 cd LLM-Data-Cleaner\n```\n进入仓库然后跑下面命令启动OCR\n```bash\npython start_ocr.py --model MODEL --key YOUR_API_KEY --img_path /path/to/images/ --outdir /path/to/output/ --lang language --batchsize batchsize\n```\n**MODEL** 的值可以是 [\"qwen\"(通义千问)](https://help.aliyun.com/zh/dashscope/developer-reference/activate-dashscope-and-create-an-api-key), [\"moonshot\"(月之暗面)](https://platform.moonshot.cn/console/api-keys), [\"paddle\"(百度飞桨OCR)](https://github.com/PaddlePaddle/PaddleOCR), [\"openai\"](https://platform.openai.com/docs/models/overview)和[llava](https://github.com/haotian-liu/LLaVA). **YOUR_API_KEY** 是你申请的API KEY，没有的话点上面相应的模型字体链接申请， paddle, llava不需要。 **/path/to/images/** 是图片目录, 里面所有图片都会被OCR, 结果保存在 **/path/to/output/** data.jsonl。 **language** 是识别的语言，值可以是 ch (中文), en (英文), fr (法语), german (德语), korean (韩语), japan (日语), 只有百度飞桨OCR可能会用到。**batchsize** 是每批量的大小，也是线程数，计算资源运行情况下，越大越好，默认是数据集大小。\n## 示例\n假如你要用通义千问的qwen-vl-plus模型做OCR，API密钥是sbadgassjda，图片数据所在目录是/images/，结果输出data.jsonl文件目录是/images/，无论是识别什么语言，你都应该跑下面的代码\n```bash\npython start_ocr.py --model qwen-vl-plus --key sbadgassjda --img_path /images/ --outdir /images/\n```\n假如你要用百度飞桨OCR做OCR，图片数据所在目录是/images/，希望结果输出data.jsonl文件目录是/images/，语言是中文，那你应该跑下面的代码\n```bash\npython start_ocr.py --model paddle --img_path /images/ --outdir /images/ --lang ch\n```\n假如你要用 `llava`, 跑下面的代码\n```bash\npython start_ocr --model LLAVA_PATH --img_path /images/ --outdir /images/\n```\n`LLAVA_PATH`` is 你的llava模型路径(HuggingFace类的模型路径).\n## 附录\nOCR的提示词存在文件[ocr.py](models/ocr.py)里。\n## 许可\n\n项目许可证是[LICENSE](LICENSE)。\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjackfsuia%2Fllm-data-cleaner","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjackfsuia%2Fllm-data-cleaner","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjackfsuia%2Fllm-data-cleaner/lists"}