{"id":13861677,"url":"https://github.com/gokayfem/ComfyUI_VLM_nodes","last_synced_at":"2025-07-14T09:33:07.308Z","repository":{"id":218964559,"uuid":"747807228","full_name":"gokayfem/ComfyUI_VLM_nodes","owner":"gokayfem","description":"Custom ComfyUI nodes for Vision Language Models, Large Language Models, Image to Music, Text to Music, Consistent and Random Creative Prompt Generation","archived":false,"fork":false,"pushed_at":"2024-05-22T01:58:24.000Z","size":292,"stargazers_count":235,"open_issues_count":20,"forks_count":14,"subscribers_count":7,"default_branch":"main","last_synced_at":"2024-05-22T18:24:12.205Z","etag":null,"topics":["comfyui","custom-nodes","image-captioning","img2sfx","img2text","joytag","llava","llm","mllm","nodes","phi15","siglip","vlm"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/gokayfem.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-01-24T17:18:35.000Z","updated_at":"2024-05-30T22:22:30.539Z","dependencies_parsed_at":"2024-05-30T22:22:25.008Z","dependency_job_id":"d928bad9-b224-4347-a431-0eb0574eeddd","html_url":"https://github.com/gokayfem/ComfyUI_VLM_nodes","commit_stats":null,"previous_names":["gokayfem/vlm_nodes","gokayfem/comfyui_vlm_nodes"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gokayfem%2FComfyUI_VLM_nodes","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gokayfem%2FComfyUI_VLM_nodes/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gokayfem%2FComfyUI_VLM_nodes/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gokayfem%2FComfyUI_VLM_nodes/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/gokayfem","download_url":"https://codeload.github.com/gokayfem/ComfyUI_VLM_nodes/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":225968853,"owners_count":17553147,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["comfyui","custom-nodes","image-captioning","img2sfx","img2text","joytag","llava","llm","mllm","nodes","phi15","siglip","vlm"],"created_at":"2024-08-05T06:01:27.807Z","updated_at":"2024-11-22T21:31:00.470Z","avatar_url":"https://github.com/gokayfem.png","language":"Python","funding_links":[],"categories":["Python","All Workflows Sorted by GitHub Stars","Workflows (3395) sorted by GitHub Stars","⭐ Community Favorites \u0026 Top Repos"],"sub_categories":["🎭 Specialized Extensions"],"readme":"\u003cdiv align=\"center\"\u003e\n  \u003ch1\u003e 👁️ VLM Nodes\u003c/h1\u003e\n  \u003cp align=\"center\"\u003e\n    \u003cb\u003e 🔽Examples below\u003c/b\u003e  •  \n    📙 \u003ca href=\"https://github.com/gokayfem/Awesome-VLM-Architectures\"\u003eVisit my other repo to learn more about Vision Language Models\u003c/a\u003e \n  \u003c/p\u003e\n\u003c/div\u003e\n\u003cbr/\u003e\n\n## Usage\n- For **Windows** and **Linux**\n```\ncd custom_nodes\ngit clone https://github.com/gokayfem/ComfyUI_VLM_nodes.git\n```\n## Acknowledgements\n\n - [JAGS](https://github.com/jags111) \n - [EnragedAntelope](https://github.com/EnragedAntelope)\n\n**If you get errors related to llama-cpp-python or if it is not using GPU.**  \n**I recommend installing it with the right arguments provided in this link [llama-cpp-python](https://github.com/abetlen/llama-cpp-python?tab=readme-ov-file#installation)**  \n\n## VLM Nodes\nUtilizes ```llama-cpp-python``` for integration of LLaVa models. You can load and use any VLM with LLaVa models in GGUF format with this nodes.   \nYou need to download the model similar to ```ggml-model-q4_k.gguf``` and it's clip projector similar to ```mmproj-model-f16.gguf``` from this repositories (in the files and versions).  \n```python=\u003e3.9``` is necessary.  \nPut all of the files inside ```models/LLavacheckpoints```  \nNote that every **model's clip projector** is different!  \n- [LlaVa 1.6 Mistral 7B](https://huggingface.co/cjpais/llava-1.6-mistral-7b-gguf/)\n- [Nous Hermes 2 Vision](https://huggingface.co/billborkowski/llava-NousResearch_Nous-Hermes-2-Vision-GGUF)\n- [LlaVa 1.5 7B](https://huggingface.co/mys/ggml_llava-v1.5-7b/)\n- [LlaVa 1.5 13B](https://huggingface.co/mys/ggml_llava-v1.5-13b)\n- [BakLLaVa](https://huggingface.co/mys/ggml_bakllava-1)  \netc..\n\n## Structured Output\nGetting structured outputs can be quite challenging through prompt engineering alone.  \nI've added the Structured Output node to VLM Nodes.  \nNow, you can obtain your answers reliably.  \nYou can extract entities, numbers, classify prompts with given classes, and generate one specific prompt. These are just a few examples.  \nYou can add additional descriptions to fields and choose the attributes you want it to return.  \n![structured](https://github.com/gokayfem/ComfyUI_VLM_nodes/assets/88277926/43b86ad4-0b91-499f-b2fd-d9771ee4acdd)\n\n## Image to Music\nUtilizes VLMs, LLMs and [AudioLDM-2](https://arxiv.org/abs/2308.05734) to make music from images.  \nUse SaveAudioNode to save the music inside ```output``` folder.  \nIt will automatically download the necessary files into ```models/LLavacheckpoints/files_for_audioldm2```  \n\nhttps://github.com/gokayfem/ComfyUI_VLM_nodes/assets/88277926/2c5bdcde-d637-49ad-b317-14ac0a12f7df\n\n## LLM to Music\nUtilizes Chat Musician, an open-source LLM that integrates intrinsic musical abilities.    \n[ChatMusician Demo Page](https://ezmonyi.github.io/ChatMusician/)  \nYou can try prompts from this demo page.\n\n**Download the GGUF file**  \n[ChatMusician GGUF Files](https://huggingface.co/MaziyarPanahi/ChatMusician-GGUF/tree/main)  \n**ChatMusician.Q5_K_M.gguf** or **ChatMusician.Q5_K_S.gguf** recommended  \n### BIG BIG BIG Warning: It **does NOT work perfectly**, if you got errors accept the error **queue prompt** again with the same settings!!\n\nhttps://github.com/gokayfem/ComfyUI_VLM_nodes/assets/88277926/7f22d4f2-b998-402e-88c8-c382a730d624\n\n## InternLM-XComposer2-VL Node\nUtilizes ```AutoGPTQ``` for integration of InternLM-XComposer2-VL Model. It will automatically download the necessary files into ```models/LLavacheckpoints/files_for_internlm```.\nThis is one of the best models for visual perception.   \n**Important Note : This model is heavy.**\n- [InternLM-XComposer2](https://huggingface.co/internlm/internlm-xcomposer2-vl-7b-4bit)\n\n## Automatic Prompt Generation and Suggestion Nodes\n**Get Keyword** node: It can take LLava outputs and extract keywords from them.    \n**LLava PromptGenerator** node: It can create prompts given descriptions or keywords using  (input prompt could be Get Keyword or LLava output directly).  \n**Suggester** node: It can generate 5 different prompts based on the original prompt using consistent in the options or random prompts using random in the options.  \n- Works best with **LLava 1.5** and **1.6**.  \n\n**Play with the ```temperature``` for creative or consistent results. Higher the temperature more creative are the results.**  \nIf you want to dive deep into [LLM Settings](https://www.promptingguide.ai/introduction/settings)  \n\nOutputs are JSON looking texts, you can see them as a text using JsonToText Node.  \nYou can see any string output with ViewText Node  \nYou can set any string input using SimpleText Node  \nUtilizes ```llama-cpp-agents``` for getting structured outputs.  \n## LLM Prompt Generation from text nodes\n\n**LLM PromptGenerator** node: \n[Qwen 1.8B Stable Diffusion Prompt](https://huggingface.co/hahahafofo/Qwen-1_8B-Stable-Diffusion-Prompt-GGUF)  \n[IF prompt MKR](https://huggingface.co/impactframes/IFpromptMKR-7b-L2-gguf-q4_k_m)  \nThis LLM's works best for now for prompt generation.  \n**LLMSampler** node: You can chat with any LLM in gguf format, you can use LLava models as an LLM also.  \n\n**API PromptGenerator** node: You can use ChatGPT and DeepSeek API's to create prompts. https://platform.deepseek.com/ gives 10m free tokens.\n- ChatGPT-4\n- ChatGPT-3.5\n- DeepSeek\nYou can use them for simple chat also there is an option in the node.\n\n## UForm-Gen2 Qwen Node\nUForm-Gen2 is an extremely fast small generative vision-language model primarily designed for Image Captioning and Visual Question Answering.  \n[UForm-Gen2 Qwen](https://huggingface.co/unum-cloud/uform-gen2-qwen-500m)  \nIt will automatically download the necessary files into ```models/LLavacheckpoints/files_for_uform_gen2_qwen```\n\n## Kosmos-2 Node\nKosmos-2: Grounding Multimodal Large Language Models to the World.\n[Kosmos-2](https://huggingface.co/microsoft/kosmos-2-patch14-224)\nIt will automatically download the necessary files into ```models/LLavacheckpoints/files_for_kosmos2```\n\n## moondream1 and moondream2 Node\nThis node is designed to work with the Moondream model, a powerful small vision language model built by @vikhyatk using SigLIP, Phi-1.5, and the LLaVa training dataset. \nThe model boasts 1.6 billion parameters and is made available for research purposes only; commercial use is not allowed.  \n\nmoondream2 is a small vision language model designed to run efficiently on edge devices.  \n\nIt will automatically download the necessary files into ```models/LLavacheckpoints/files_for__moondream``` and ```models/LLavacheckpoints/files_for_moondream2```\n\n## JoyTag Node\n@fpgamine's JoyTag is a state of the art AI vision model for tagging images, with a focus on sex positivity and inclusivity.  \nIt uses the Danbooru tagging schema, but works across a wide range of images, from hand drawn to photographic.\nIt will automatically download the necessary files into ```models/LLavacheckpoints/files_for_joytagger```\n\n## Qwen2-VL Node\nUtilizes the latest Qwen2-VL series of models, which are state-of-the-art vision language models supporting various resolutions, ratios, and languages. The models excel at:\n- Understanding images of various resolutions \u0026 ratios\n- Complex visual reasoning and decision making\n- Multilingual support (English, Chinese, European languages, Japanese, Korean, Arabic, Vietnamese, etc.)\n\nAvailable models include 2B, 7B, and 72B parameter versions, with standard, AWQ, and GPTQ quantized variants. It will automatically download the necessary files into `models/LLavacheckpoints/files_for_qwen2vl`.\n\n**Important Note**: Larger models (7B, 72B) require significant VRAM. Choose quantized versions (AWQ, GPTQ) for reduced memory usage.\n\n[Link to Qwen2-VL Models](https://huggingface.co/Qwen/Qwen2-VL-7B-Instruct)\n\n## Example LLaVa Nodes\n![image](https://github.com/gokayfem/ComfyUI_VLM_nodes/assets/88277926/c30b9599-fa14-4f1a-b023-65a3697892f2)\n\n## Example Image to Music\n![image](https://github.com/gokayfem/ComfyUI_VLM_nodes/assets/88277926/e216c299-c9ea-4227-aa85-9533cb6af260)\n\n## Example InternLM-XComposer Node\n![image](https://github.com/gokayfem/ComfyUI_VLM_nodes/assets/88277926/ff051e6c-5ad8-41fe-9d77-fdeea6eb2c5c)\n\n## Example Using Automatic Prompt Generation\n![image](https://github.com/gokayfem/ComfyUI_VLM_nodes/assets/88277926/bff68f6f-5f77-4cd6-ade3-6810a32500bf)\n\n## LLM Nodes\n![VLM + LLM](https://github.com/gokayfem/ComfyUI_VLM_nodes/assets/88277926/4897d11a-e818-4d7e-bf04-0cd7dd4102dc)\n\n## Example UForm-Gen2 Qwen Node\n![image](https://github.com/gokayfem/ComfyUI_VLM_nodes/assets/88277926/4531f8f2-94af-498f-b364-f9e07c826eb5)\n\n# Example Kosmos-2 Node\n![image](https://github.com/gokayfem/ComfyUI_VLM_nodes/assets/88277926/a28035dc-a0c4-4c4f-9c87-e8b284c3997d)\n\n## Example moondream\n![image](https://github.com/gokayfem/ComfyUI_VLM_nodes/assets/88277926/79ea61e9-60c6-406d-9e83-0d16128e30a6)\n\n## Example Joytag\n![image](https://github.com/gokayfem/ComfyUI_VLM_nodes/assets/88277926/df9da377-59e8-4b39-a31a-0e3b5071a8cc)\n\n## Example Prompt Generation \n![image](https://github.com/gokayfem/ComfyUI_VLM_nodes/assets/88277926/1c557f10-52ee-4e1f-ab8a-20932a07dd3b)\n\n## Example SimpleChat\n![image](https://github.com/gokayfem/ComfyUI_VLM_nodes/assets/88277926/057cfc2e-e772-43c0-972f-2916e6aeb03d)\n\n## Example LLava Sampler Advanced\n![image](https://github.com/gokayfem/ComfyUI_VLM_nodes/assets/88277926/32210c37-fe7d-479f-b0a6-2eb13ea0aac1)\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgokayfem%2FComfyUI_VLM_nodes","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgokayfem%2FComfyUI_VLM_nodes","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgokayfem%2FComfyUI_VLM_nodes/lists"}