{"id":13645848,"url":"https://github.com/ttengwang/caption-anything","last_synced_at":"2025-05-15T18:09:40.680Z","repository":{"id":152581128,"uuid":"624860070","full_name":"ttengwang/Caption-Anything","owner":"ttengwang","description":"Caption-Anything is a versatile tool combining image segmentation, visual captioning, and ChatGPT, generating tailored captions with diverse controls for user preferences. https://huggingface.co/spaces/TencentARC/Caption-Anything https://huggingface.co/spaces/VIPLab/Caption-Anything","archived":false,"fork":false,"pushed_at":"2023-08-29T05:26:45.000Z","size":54466,"stargazers_count":1732,"open_issues_count":20,"forks_count":104,"subscribers_count":15,"default_branch":"main","last_synced_at":"2025-04-07T23:08:54.318Z","etag":null,"topics":["chatgpt","controllable-generation","controllable-image-captioning","image-captioning","segment-anything"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ttengwang.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2023-04-07T12:43:03.000Z","updated_at":"2025-04-07T06:47:28.000Z","dependencies_parsed_at":null,"dependency_job_id":"67873957-3490-4eac-b8e3-335c8c92191f","html_url":"https://github.com/ttengwang/Caption-Anything","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ttengwang%2FCaption-Anything","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ttengwang%2FCaption-Anything/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ttengwang%2FCaption-Anything/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ttengwang%2FCaption-Anything/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ttengwang","download_url":"https://codeload.github.com/ttengwang/Caption-Anything/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254394722,"owners_count":22063984,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["chatgpt","controllable-generation","controllable-image-captioning","image-captioning","segment-anything"],"created_at":"2024-08-02T01:02:43.500Z","updated_at":"2025-05-15T18:09:40.644Z","avatar_url":"https://github.com/ttengwang.png","language":"Python","funding_links":[],"categories":["Others"],"sub_categories":[],"readme":"\u003cdiv align=\"center\"\u003e\n    \u003cimg src=\"assets/caption_anything_logo.png\" height=\"160\" /\u003e\n\u003c/div\u003e\n\u003cdiv align=\"center\"\u003e\n\u003c!-- \u003ch1 align=\"center\"\u003e Caption Anything \u003c/h1\u003e --\u003e\n\u003ca src=\"https://img.shields.io/badge/arXiv-2305.02677-b31b1b.svg\" href=\"https://arxiv.org/abs/2305.02677\"\u003e\n    \u003cimg src=\"https://img.shields.io/badge/arXiv-2305.02677-b31b1b.svg\"\u003e\n\u003c/a\u003e\n\u003ca src=\"https://colab.research.google.com/assets/colab-badge.svg\" href=\"https://colab.research.google.com/github/ttengwang/Caption-Anything/blob/main/notebooks/tutorial.ipynb\"\u003e\n    \u003cimg src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open in Colab\"\u003e\n\u003c/a\u003e\n\u003ca src=\"https://img.shields.io/badge/%F0%9F%9A%80-SUSTech_VIP_Lab-important.svg\" href=\"https://zhengfenglab.com/\"\u003e\n\u003cimg src=\"https://img.shields.io/badge/%F0%9F%9A%80-SUSTech_VIP_Lab-important.svg\"\u003e\n\u003c/a\u003e\n\u003c/div\u003e\n\n[***Caption-Anything***](https://arxiv.org/abs/2305.02677) is a versatile image processing tool that combines the capabilities of [Segment Anything](https://github.com/facebookresearch/segment-anything), Visual Captioning, and [ChatGPT](https://openai.com/blog/chatgpt). Our solution generates descriptive captions for any object within an image, offering a range of language styles to accommodate diverse user preferences. It supports visual controls (mouse click) and language controls (length, sentiment, factuality, and language).\n* Visual controls and language controls for text generation\n* Chat about selected object for detailed understanding\n* Interactive demo\n\n\u003cdiv align=center\u003e\n\u003cimg src=\"./assets/qingming.gif\" /\u003e\n\u003cbr\u003e    \nAlong the River During the Qingming Festival (清明上河图)\n\u003c/div\u003e\n\u003cbr\u003e \n\n### :rocket: Updates\n* 2023/04/30: support caption everything in a paragraph\n* 2023/04/25: We are delighted to introduce [Track-Anything](https://github.com/gaomingqi/Track-Anything), an inventive project from our lab that offers a versatile and user-friendly solution for video object tracking and segmentation.\n* 2023/04/23: support langchain + VQA, better chatbox performance\n* 2023/04/20: add mouse trajectory as visual control (beta)\n* 2023/04/13: add Colab Tutorial \u003ca src=\"https://colab.research.google.com/assets/colab-badge.svg\" href=\"https://colab.research.google.com/github/ttengwang/Caption-Anything/blob/main/notebooks/tutorial.ipynb\"\u003e \u003cimg src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open in Colab\"\u003e \u003c/a\u003e\n* 2023/04/11: Release code\n\n### :joystick: Demo\nExplore the interactive demo of Caption-Anything, which showcases its powerful capabilities in generating captions for various objects within an image. The demo allows users to control visual aspects by clicking on objects, as well as to adjust textual properties such as length, sentiment, factuality, and language.\n\n---\n\n![](./assets/UI.png)\n\n---\n\n![](./assets/demo1.png)\n\n---\n\n![](./assets/demo2.png)\n\n### :hammer_and_wrench: Getting Started\n\n#### Linux\n```bash\n# Clone the repository:\ngit clone https://github.com/ttengwang/caption-anything.git\ncd caption-anything\n\n# Install dependencies (python version \u003e= 3.8.1):\npip install -r requirements.txt\n\n# Configure the necessary ChatGPT APIs\nexport OPENAI_API_KEY={Your_Private_Openai_Key}\n\n# Run the Caption-Anything gradio demo.\npython app_langchain.py --segmenter huge --captioner blip2 --port 6086  --clip_filter  # requires 13G GPU memory\n#python app_langchain.py --segmenter base --captioner blip2 # requires 8.5G GPU memory\n#python app_langchain.py --segmenter base --captioner blip # requires 5.5G GPU memory\n\n# (Optional) Use the pre-downloaded SAM checkpoints\nwget https://dl.fbaipublicfiles.com/segment_anything/sam_vit_h_4b8939.pth ./sam_vit_h_4b8939.pth\npython app_langchain.py --segmenter huge --captioner blip2 --port 6086 --segmenter_checkpoint ./sam_vit_b_01ec64.pth  # requires 11.7G GPU memory\n```\n\n#### Windows(powershell)\nTested in Windows11 using Nvidia 3070-8G.\n\n```shell\n# Clone the repository:\ngit clone https://github.com/ttengwang/caption-anything.git\ncd caption-anything\n\n# Install dependencies:\npip install -r requirements.txt\n\n# Download the [base SAM checkpoints](https://dl.fbaipublicfiles.com/segment_anything/sam_vit_b_01ec64.pth).\nInvoke-WebRequest https://dl.fbaipublicfiles.com/segment_anything/sam_vit_b_01ec64.pth -OutFile ./sam_vit_b_01ec64.pth\n\n# Configure the necessary ChatGPT APIs\n$env:OPENAI_API_KEY = '{Your_Private_Openai_Key}'\n\n# Run the Caption-Anything gradio demo.\npython app_langchain.py --captioner blip --port 6086 --segmenter base # better chatbox via langchain + VQA\npython app_langchain.py --captioner blip --port 6086 --segmenter base --segmenter_checkpoint ./sam_vit_b_01ec64.pth  # Use the pre-downloaded SAM checkpoints\npython app.py --captioner blip --port 6086 --segmenter base \n```\n\n## :computer: Usage\n```python\nfrom caption_anything import CaptionAnything, parse_augment\nargs = parse_augment()\nvisual_controls = {\n    \"prompt_type\":[\"click\"],\n    \"input_point\":[[500, 300], [1000, 500]],\n    \"input_label\":[1, 0], # 1/0 for positive/negative points\n    \"multimask_output\":\"True\",\n}\nlanguage_controls = {\n    \"length\": \"30\",\n    \"sentiment\": \"natural\", # \"positive\",\"negative\", \"natural\"\n    \"imagination\": \"False\", # \"True\", \"False\"\n    \"language\": \"English\" # \"Chinese\", \"Spanish\", etc.\n}\nmodel = CaptionAnything(args, openai_api_key)\nout = model.inference(image_path, visual_controls, language_controls)\n```\n## :book: Citation\nIf you find this work useful for your research, please cite our github repo:\n\n```bibtex\n@article{wang2023caption,\n  title={Caption anything: Interactive image description with diverse multimodal controls},\n  author={Wang, Teng and Zhang, Jinrui and Fei, Junjie and Ge, Yixiao and Zheng, Hao and Tang, Yunlong and Li, Zhe and Gao, Mingqi and Zhao, Shanshan and Shan, Ying and Zheng, Feng},\n  journal={arXiv preprint arXiv:2305.02677},\n  year={2023}\n}\n```\n## Acknowledgements\nThe project is based on [Segment Anything](https://github.com/facebookresearch/segment-anything), [BLIP/BLIP-2](https://github.com/salesforce/LAVIS), [ChatGPT](https://openai.com/blog/chatgpt), [Visual ChatGPT](https://github.com/microsoft/TaskMatrix), [GiT](https://github.com/microsoft/GenerativeImage2Text). Thanks for the authors for their efforts.\n## Contributor\nOur project wouldn't be possible without the contributions of these amazing people! Thank you all for making this project better.\n\n[Teng Wang](http://ttengwang.com/) @ Southern University of Science and Technology \u0026 HKU \u0026 Tencent ARC Lab \\\n[Jinrui Zhang](https://github.com/zjr2000) @ Southern University of Science and Technology \\\n[Junjie Fei](https://github.com/JunjieFei) @ Xiamen University \\\n[Zhe Li](https://github.com/memoryunreal) @ Southern University of Science and Technology \\\n[Yunlong Tang](https://github.com/yunlong10) @ Southern University of Science and Technology \\\n[Mingqi Gao](https://mingqigao.com/) @ Southern University of Science and Technology \u0026 University of Warwick \\\n[Hao Zheng](https://github.com/zh-plus) @ Southern University of Science and Technology\n\n\u003ca href=\"https://github.com/ttengwang/Caption-Anything/graphs/contributors\"\u003e\n  \u003cimg src=\"https://contrib.rocks/image?repo=ttengwang/Caption-Anything\" /\u003e\n\u003c/a\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fttengwang%2Fcaption-anything","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fttengwang%2Fcaption-anything","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fttengwang%2Fcaption-anything/lists"}