{"id":14964736,"url":"https://github.com/progamergov/vlm-captioning-tools","last_synced_at":"2025-10-12T17:09:39.689Z","repository":{"id":229501538,"uuid":"776869471","full_name":"ProGamerGov/VLM-Captioning-Tools","owner":"ProGamerGov","description":"Python scripts to use for captioning images with VLMs","archived":false,"fork":false,"pushed_at":"2024-08-01T16:03:19.000Z","size":22,"stargazers_count":36,"open_issues_count":2,"forks_count":0,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-01-31T06:51:10.985Z","etag":null,"topics":["cogvlm","image-captioning","llama3","llm","mistral","moondream","text-summarization","vision-language","vlm"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ProGamerGov.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":"CITATION.cff","codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-03-24T17:09:41.000Z","updated_at":"2024-12-23T16:14:32.000Z","dependencies_parsed_at":"2024-03-30T20:33:46.848Z","dependency_job_id":"38ca6d49-de6b-4de3-acfa-f47e98bafe42","html_url":"https://github.com/ProGamerGov/VLM-Captioning-Tools","commit_stats":{"total_commits":11,"total_committers":1,"mean_commits":11.0,"dds":0.0,"last_synced_commit":"ae859e9735a692b01937ea9afffc3ed85428c76d"},"previous_names":["progamergov/vlm-captioning-tools"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ProGamerGov%2FVLM-Captioning-Tools","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ProGamerGov%2FVLM-Captioning-Tools/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ProGamerGov%2FVLM-Captioning-Tools/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ProGamerGov%2FVLM-Captioning-Tools/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ProGamerGov","download_url":"https://codeload.github.com/ProGamerGov/VLM-Captioning-Tools/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":238104871,"owners_count":19417213,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cogvlm","image-captioning","llama3","llm","mistral","moondream","text-summarization","vision-language","vlm"],"created_at":"2024-09-24T13:33:42.551Z","updated_at":"2025-10-12T17:09:34.647Z","avatar_url":"https://github.com/ProGamerGov.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# VLM-Captioning-Tools\nPython scripts to use for captioning images with VLMs\n\n## CogVLM Captioning Tool\n\nThis script uses [CogVLM](https://github.com/THUDM/CogVLM) to create captions for one or more directories (including subdirectories) and lists of file paths in txt files.\nRepeating sequence caption failures caused by greedy search algorithms are automatically detected and resolved by changing model parameters when they arise.\n\nGenerated captions are saved in a parquet file with the following structure: `['image_name', 'hash', 'short_caption', 'long_caption', 'resolution']`.\n\n**Usage:**\n\nFirst adjust the `path_and_save_dir` parameter's input directories and backup directory in the script to your liking. Then run it like this:\n\n```\npython cogvlm_captioning_tool.py\n```\n\n### Prefixes\n\nCogVLM often uses common prefix substrings in output captions. This function provides a way to remove those if desired.\n\n```\ndef modify_caption(caption: str) -\u003e str:\n    \"\"\"\n    Removes common prefix substrings from CogVLM outputs.\n\n    Args:\n        caption (str): A string containing a cogvlm caption.\n\n    Returns:\n        str: The caption with the prefix substring removed\n            or altered if it was present.\n    \"\"\"\n    base_words = ['showcases ', 'portrays ', 'appears to be ', 'is ', 'depicts ', 'features ']\n    prefix_substrings = [(\"The image \" + s, '') for s in base_words] + [(\"This image \" + s, '') for s in base_words]\n    prefix_substrings += [(\"In this \" + s, '') for s in [\"picture, \", \"depiction, \",  \"piece, \", \"image, \", \"scene, \"]]\n    prefix_substrings  += [\n        ('In this artwork, ', 'Artwork of '),\n        ('In this illustration, ', 'Illustration of '),\n        ('In this art piece, ', 'Art of ')\n    ]\n    pattern = '|'.join([re.escape(opening) for opening, _ in prefix_substrings])\n    replacers = {opening: replacer for opening, replacer in prefix_substrings}\n    \n    def replace_fn(match):\n        return replacers[match.group(0)]\n    \n    return re.sub(pattern, replace_fn, caption, count=1, flags=re.IGNORECASE).capitalize()\n```\n\n## MoonDream Captioning Tool\n\nThis script uses [MoonDream](https://github.com/vikhyat/moondream) to create captions for one or more directories (including subdirectories) and lists of file paths in txt files.\nRepeating sequence caption failures caused by greedy search algorithms are automatically detected and resolved by changing model parameters when they arise.\n\nGenerated captions are saved in a parquet file with the following structure: `['image_name', 'hash', 'short_caption', 'long_caption', 'resolution']`.\n\n**Usage:**\n\nFirst adjust the `batch_size` and `path_and_save_dir` parameter's input directories and backup directory in the script to your liking. Then run it like this:\n\n```\npython moondream_batch_captioning_tool.py\n```\n\n## Dolphin Caption Summarization Tool\n\nThis script uses the [Dolphin 2.6 Mistral 7b - DPO](https://huggingface.co/cognitivecomputations/dolphin-2.6-mistral-7b-dpo) LLM model to create a shortened version of the `long_caption` value for every item in the provided parquet's `short_caption` column. \n\n**Usage:**\n\nFirst adjust the input parquet and output parquet parameters in the script to your liking. Then run it like this:\n\n```\npython dolphin_mistral_short_captioning_tool.py\n```\n\n## Meta Llama 3 Caption Summarization Tool\n\nThis script uses the [Meta Llama 3 8B Instruct](https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct) LLM model to create a shortened version of the `long_caption` value for every item in the provided parquet's `short_caption` column. \n\n**Usage:**\n\nFirst adjust the input parquet and output parquet parameters in the script to your liking. Then run it like this:\n\n```\npython meta_llama3_short_captioning_tool.py\n```\n\n\n## Bad Caption Finder Tool\n\nThis script provides the ability to determine the number of caption failures in a dataset due to greedy search induced failures. It is capable of detecting repeating characters, words, sentences, and more and thus is extremely robust. The same failed caption detection code is used in the CogVLM Captioning Tool script above to automatically detect and retry failed captions.\n\n**Usage:**\n\nFirst ensure the script loads your dataset captions (example code loads from parquet, but this can be changed). Then run it like this:\n\n```\npython bad_caption_finder.py\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fprogamergov%2Fvlm-captioning-tools","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fprogamergov%2Fvlm-captioning-tools","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fprogamergov%2Fvlm-captioning-tools/lists"}