{"id":13691439,"url":"https://github.com/showlab/Image2Paragraph","last_synced_at":"2025-05-02T15:31:56.603Z","repository":{"id":152825392,"uuid":"627059657","full_name":"showlab/Image2Paragraph","owner":"showlab","description":"[A toolbox for fun.] Transform Image into Unique Paragraph with ChatGPT, BLIP2, OFA, GRIT, Segment Anything, ControlNet.","archived":false,"fork":false,"pushed_at":"2023-04-28T06:41:05.000Z","size":22143,"stargazers_count":806,"open_issues_count":19,"forks_count":55,"subscribers_count":11,"default_branch":"main","last_synced_at":"2025-04-04T10:09:12.624Z","etag":null,"topics":["chatgpt","gpt4","segmentt-anything","toolbox"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/showlab.png","metadata":{"files":{"readme":"readme.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2023-04-12T17:44:00.000Z","updated_at":"2025-04-01T19:07:18.000Z","dependencies_parsed_at":null,"dependency_job_id":"e889be70-e108-41a7-bb4a-45cfbeab8301","html_url":"https://github.com/showlab/Image2Paragraph","commit_stats":{"total_commits":42,"total_committers":2,"mean_commits":21.0,"dds":"0.023809523809523836","last_synced_commit":"a24210a6dd4535a1a43af7a6170fb8ad1e3a6013"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/showlab%2FImage2Paragraph","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/showlab%2FImage2Paragraph/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/showlab%2FImage2Paragraph/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/showlab%2FImage2Paragraph/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/showlab","download_url":"https://codeload.github.com/showlab/Image2Paragraph/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252062906,"owners_count":21688618,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["chatgpt","gpt4","segmentt-anything","toolbox"],"created_at":"2024-08-02T17:00:45.107Z","updated_at":"2025-05-02T15:31:51.590Z","avatar_url":"https://github.com/showlab.png","language":"Python","funding_links":[],"categories":["\u003cspan id=\"head3\"\u003e *3. Related Project* \u003c/span\u003e","Recent Works","SDK, Libraries, Frameworks"],"sub_categories":["\u003cspan id=\"head2023\"\u003e *2023* \u003c/span\u003e","Python"],"readme":"\u003cp align=\"center\"\u003e\n  \u003cimg src=\"examples/logo.png\" alt=\"Your Image Description\" width=800\u003e\n\u003c/p\u003e\n\n\n# Image.txt: Transform Image Into Unique Paragraph \n\n\n[![Hugging Face Spaces](https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Spaces-blue)](https://huggingface.co/spaces/Awiny/Image2Paragraph), ![](https://img.shields.io/twitter/follow/awinyimgprocess?style=social), [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1RCzMgApzbwI6cp_NZGIbHBP_HDslZjeH?usp=sharing)\n\n[Project Website](https://zhaohengyuan1.github.io/image2paragraph.github.io)\n\n(huggingface sometimes may not work with safari, use chrome)\n\n**Demo**\n\n![](examples/demo.gif)\n\n\n\u003cp align=\"left\"\u003e\n  \u003cimg src=\"examples/icon/news.gif\" alt=\"Your Image Description\" width=100\u003e  \u003cstrong\u003e\u003cspan style=\"font-size: 24px;\"\u003eNews\u003c/span\u003e\u003c/strong\u003e\n\u003c/p\u003e\n\n- 17/April/2023. In addition to semantic segment anything, we use [Edit Anything](https://github.com/sail-sg/EditAnything) to get region-level semantic. Now all models takes less than 20s on 8G memory GPU card. (10times faster than previous version on cpu)\n- 17/April/2023. Our project is online on Huggingface. Have a try! [huggingface](https://huggingface.co/spaces/Awiny/Image2Paragraph/tree/main)\n- 14/April/2023. Our project is very popular in twitter. Looking [the posted twitter](https://twitter.com/awinyimgprocess/status/1646225454599372800?s=46\u0026t=HvOe9T2n35iFuCHP5aIHpQ) for details.\n\n\n\n**(Can run on 8GB memory GPU within 20S!)**\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"examples/gpu_memory.png\" alt=\"Your Image Description\" width=600\u003e\n\u003c/p\u003e\n\n\n\n**Main Pipeline**\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"examples/introduction.png\" alt=\"Your Image Description\" width=600\u003e\n\u003c/p\u003e\n\n\n\n\n**Reasoning Details**\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"examples/introduction_2.jpg\" alt=\"Your Image Description\" width=600\u003e\n\u003c/p\u003e\n\n\n### To Do List\n\n#### Done\n- [x] GRIT example.\n- [x] ControNet, BLIP2.\n- [x] Semantic Segment Anything.\n- [x] Segment Anything for fine-grained semantic.\n- [x] Gradio.\n- [x] Integrate GRIT into our code.\n- [x] Support GPT4 API.\n- [x] Notebook/Huggingface Space.\n- [x] Region Semantic Classification from Edit-Anything.\n- [x] Make the model lightweight.\n\n#### Doing\n- [ ] Replace ChatGPT with own trained LLM.\n- [ ] Other grounding text2image model as instead of Canny ControlNet.\n- [ ] Show retrieval result in gradio.\n\n## Visualization\n\nThe text to image model is conrolnet with canny from diffuser.\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"output/2_result.png\" alt=\"Your Image Description\" width=600\u003e\n\u003c/p\u003e\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"output/3_result.jpg\" alt=\"Your Image Description\" width=600\u003e\n\u003c/p\u003e\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"output/4_result.jpg\" alt=\"Your Image Description\" width=600\u003e\n\u003c/p\u003e\n\n\n\n##  Installation\n\nPlease find installation instructions in [install.md](install.md).\n\n\n## 2. Start\n\n### Simple visualization\n\n```bash\nexport OPENAI_KEY=[YOUR KEY HERE]\npython main.py  --image_src [image_path] --out_image_name [out_file_name]\n```\n\nIf your GPU memory smaller than 8 GPB.\n```bash\npython main.py --image_caption_device cpu --semantic_segment_device cpu\n```\n\nIf you have no GPU available.\n```bash\npython main.py --image_caption_device cpu --semantic_segment_device cpu --dense_caption_device cpu  --contolnet_device cpu\n```\n\nlike\n```bash\npython main.py --image_src \"examples/3.jpg\" --out_image_name \"output/3_result.jpg\"\n```\n\n**Note: If you have GPU card with larger memory than 15GB. Set all device to GPU for fast inference.**\n\nThe generated text and image are show in \"output/\".\n\n**Note: Use GPT4 for good result as GPT3.5 miss the position information sometime.**\n\n\n\n## Use gradio directly\n\n```bash\npython main_gradio.py\n```\n\n![](examples/gradio_UI.png)\n\nIf you have GPU Memory larger than 20GB.\nUse device='cuda' as default.\n\n\n\n\n## 3. Visualization\n\n\u003c!-- % a dog sitting on a porch with a bike --\u003e\n\n\u003ctable\u003e\n  \u003ctr\u003e\n    \u003ctd\u003e\n      \u003cimg src=\"examples/3.jpg\" alt=\"Your Image Description\" width=300\u003e\n    \u003c/td\u003e\n    \u003ctd\u003e\n      A dog sitting on a porch with a bike.\n    \u003c/td\u003e\n    \u003ctd\u003e\n      \u003cimg src=\"examples/3_grit.jpg\" alt=\"Your Image Description\" width=300\u003e\n    \u003c/td\u003e\n    \u003ctd\u003e\n      \u003cimg src=\"examples/3_semantic_segment_anything.png\" alt=\"Your Image Description\" width=300\u003e\n    \u003c/td\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd align=\"center\"\u003eInput\u003c/td\u003e\n    \u003ctd align=\"center\"\u003eBLIP2 Image Caption \u003c/td\u003e\n    \u003ctd align=\"center\"\u003eGRIT Dense Caption \u003c/td\u003e\n    \u003ctd align=\"center\"\u003eSemantic Segment Anything \u003c/td\u003e\n  \u003c/tr\u003e\n\u003c/table\u003e\n\nThe final generated paragraph with ChatGPT is:\n\n\u003e       This image depicts a black and white dog sitting on a porch beside a red bike. The dense caption mentions other objects in the scene, such as a white car parked on the street and a red bike parked on the side of the road. The region semantic provides more specific information, including the porch, floor, wall, and trees. The dog can be seen sitting on the floor beside the bike, and there is also a parked bicycle and tree in the background. The wall is visible on one side of the image, while the street and trees can be seen in the other direction. \n\n## 4. Retrieval Result on COCO\n\n| Method  | Trainable Parameter | Running Time  |  IR@1   | TR@1|\n|---|---|---|---|---|\n| Image-text  | 230M | 9H |  43.8 |  33.2 |\n|Generated Paragraph-text| 0 |5m|__49.7__|__36.1__|\n\n\nInteresting, we find compress image into paragraph. The retrieval result is even better than use source image.\n\n\n## Others\nIf you have more suggestions or functions need to be implemented in this codebase, feel free to drop me an email awinyimg dot gmail dot com or open an issue.\n\n## Acknowledgment\n\nThis work is based on [ChatGPT](http://chat.openai.com), [Edit_Anything](https://github.com/sail-sg/EditAnything), [BLIP2](https://huggingface.co/spaces/Salesforce/BLIP2), [GRIT](https://github.com/JialianW/GRiT),  [OFA](https://github.com/OFA-Sys/OFA),[Segment-Anything](https://segment-anything.com), [Semantic-Segment-Anything](https://github.com/fudan-zvg/Semantic-Segment-Anything), [ControlNet](https://github.com/lllyasviel/ControlNet).","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fshowlab%2FImage2Paragraph","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fshowlab%2FImage2Paragraph","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fshowlab%2FImage2Paragraph/lists"}