{"id":26324598,"url":"https://github.com/groundlight/r1_vlm","last_synced_at":"2025-05-16T14:07:21.391Z","repository":{"id":275353575,"uuid":"925834268","full_name":"groundlight/r1_vlm","owner":"groundlight","description":"Build your own visual reasoning model","archived":false,"fork":false,"pushed_at":"2025-05-14T05:10:42.000Z","size":73618,"stargazers_count":362,"open_issues_count":16,"forks_count":20,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-05-14T06:06:36.804Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/groundlight.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-02-01T21:21:23.000Z","updated_at":"2025-05-12T20:07:21.000Z","dependencies_parsed_at":"2025-02-16T00:18:14.166Z","dependency_job_id":"0159f51d-458a-4a97-9fd4-2ebe2ce1d5f9","html_url":"https://github.com/groundlight/r1_vlm","commit_stats":null,"previous_names":["sunildkumar/r1_vlm","groundlight/r1_vlm"],"tags_count":16,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/groundlight%2Fr1_vlm","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/groundlight%2Fr1_vlm/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/groundlight%2Fr1_vlm/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/groundlight%2Fr1_vlm/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/groundlight","download_url":"https://codeload.github.com/groundlight/r1_vlm/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254544146,"owners_count":22088807,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-03-15T18:02:55.102Z","updated_at":"2025-05-16T14:07:16.382Z","avatar_url":"https://github.com/groundlight.png","language":"Jupyter Notebook","funding_links":[],"categories":["Projects","Jupyter Notebook"],"sub_categories":["Multimodal and Agents"],"readme":"\u003cp align=\"center\"\u003e\n\u003cimg src=\"images/gl_logo.png\"\u003e\n\u003c/p\u003e\n\n\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"https://opensource.org/licenses/MIT\"\u003e\n    \u003cimg src=\"https://img.shields.io/badge/License-MIT-yellow.svg\" alt=\"License: MIT\"\u003e\n  \u003c/a\u003e\n\u003c/p\u003e\n\n\nThis package makes it easy to train a VLM with GRPO. \n\nWe trained a small VLM to solve cryptograms. Use the buttons below to try the model using our demo on HuggingFace Spaces or read more about the technical details in our blog post.\n\n\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"https://huggingface.co/spaces/Groundlight/grpo-vlm-decoder\"\u003e\n    \u003cimg src=\"https://img.shields.io/badge/Try%20it-On%20HuggingFace-blue?style=for-the-badge\" alt=\"Try it on Hugging Face\"\u003e\n  \u003c/a\u003e\n  \u003ca href=\"https://www.groundlight.ai/blog/visual-reasoning-models\"\u003e\n    \u003cimg src=\"https://img.shields.io/badge/Read%20More-Technical%20Blog-orange?style=for-the-badge\" alt=\"Read More\"\u003e\n  \u003c/a\u003e\n\u003c/p\u003e\n\n\n\u003cp align=\"center\"\u003e\n\u003cimg src=\"images/demo.webp\" alt=\"Demo GIF\" width=\"800\" /\u003e\n\u003c/p\u003e\n\n\n# Installation\nThis project relies on forks for some dependencies. First clone this repo. Then clone the following repos adjacent to this one. The two forks are installed as editable dependencies into `r1_vlm`. For each fork, we checkout the relevant release branch. This process will be improved in the future. Please leave a github issue if you run into any issues.\n```\n# clone this repo\ngit clone git@github.com:groundlight/r1_vlm.git\n\n# clone forks at a specific release\ngit clone --branch release_2025_03_06 --single-branch git@github.com:groundlight/verifiers.git\ngit clone --branch release_2025_03_06 --single-branch git@github.com:groundlight/trl.git\n\n```\n\nAfterwards, your directory structure should look like this:\n```\nr1_vlm/\ntrl/\nverifiers/\n```\n\nThen install with the `uv` package manager. See the [uv docs](https://docs.astral.sh/uv/) for instructions if you don't have `uv` installed.\n```\ncd r1_vlm\nuv venv\nuv pip install hatchling editables torch==2.5.1 \nuv sync --no-build-isolation\n```\n\nSee our [Troubleshooting Installation](troubleshooting_installation.md) documentation if you run into issues.\n\n# Task 1: Message Decoding\nWe trained `Qwen2.5VL-3B-Instruct` to solve short cryptograms. A cryptogram is a message that has been encoded using a substitution cipher. The model is given a coded message and a decoder image, and it must recover the original message. This task has the nice property that it is very difficult to solve without engaging with both text and image modalities - so it forces the model to use all of its capabilities. Our model achieves 96% accuracy on our eval set.\n \n\u003cp align=\"center\"\u003e\n\u003cimg src=\"images/demo2.webp\" alt=\"Demo2 webp\" width=\"800\" /\u003e\n\u003c/p\u003e\n\n\nIn this demo, you can see our model solve the cryptogram: `groundlight loves ml`. We visualize the model's attention weights from an intermediate layer of the model. Red = low attention, green = high attention. You can see its attention to the image is relatively diffuse initially, and then becomes hyper focused on the relevant region of the decoder as it decodes each letter in sequence. In effect, the model has learned to “read” the relevant regions of the decoder as it needs them.\n\nWe put a reasonable amount of effort into the [reward function design](src/r1_vlm/environments/message_decoding_words_and_sequences_env/message_decoding_sequences_env.py) to make this possible, so it is worth checking this out if you're interested in our approach.\n\n\u003cp align=\"center\"\u003e\n\u003ca href=\"https://huggingface.co/datasets/sunildkumar/message-decoding-words-and-sequences-r1\"\u003e\n    \u003cimg src=\"https://img.shields.io/badge/See%20Dataset-On%20HuggingFace-yellow?style=for-the-badge\" alt=\"See Dataset\"\u003e\n  \u003c/a\u003e\n\u003c/p\u003e\n\n## Example Reasoning Trace:\nHere's an example where the message is `vision`.\n\u003cp align=\"center\"\u003e\n\u003cimg src=\"images/example_decoder_sequences.png\" alt=\"decoder image\"\u003e\n\u003c/p\u003e\n\n```\nsystem\nYou are a helpful assistant. You first think about the reasoning process in the mind and then provide the user with the answer.\n\nuser\nUse the decoder in the image to decode a coded message.The decoded message will be one or more words. Underscore characters (\"_\") in the coded message should be mapped to a space (\" \") when decoding. Show your work in \u003cthink\u003e \u003c/think\u003e tags and return the answer in \u003canswer\u003e \u003c/answer\u003e tags. While thinking, you must include a section with the decoded characters using \u003cchars\u003e\u003c/chars\u003e tags. The \u003cchars\u003e section should include the decoded characters in the order they are decoded. It should include the underscore character wherever there is a space in the decoded message. For example, if the coded message is a b c _ d e f, the chars section might be \u003cchars\u003e c a t _ d o g \u003c/chars\u003e. You can think about the problem for as long as you'd like. While thinking, you should robustly verify your solution. Once you are done thinking, provide your answer in the \u003canswer\u003e section, e.g. \u003canswer\u003e cat dog \u003c/answer\u003e. The coded message is: r h q h c t.\n\nassistant\nLet me solve this step by step.\n\u003cthink\u003e \nI will decode each character one by one using the decoder:\nr → v\nh → i\nq → s\nh → i\nc → o\nt → n\n\u003cchars\u003e v i s i o n \u003c/chars\u003e\n\u003c/think\u003e\n\u003canswer\u003evision\u003c/answer\u003e\n```\n\nYou can run training on 4 GPUs, 3 for training, one for completion generation with `vllm` using the following command. We've tested it on 4x A100 80GB GPUs. You can also get it running on two GPUs as well by tuning down the number of generations and running without `deepspeed`.\n```bash\n# 4 GPU training with deepspeed\nCUDA_VISIBLE_DEVICES=0,1,2,3 uv run accelerate launch --config_file src/r1_vlm/deepspeed_configs/multi_gpu_3only.yaml src/r1_vlm/environments/message_decoding_words_and_sequences_env/train.py\n\n# 2 GPU training without deepspeed, you'll need to adjust the number of generations in the train.py file.\nCUDA_VISIBLE_DEVICES=0,1 uv run src/r1_vlm/environments/message_decoding_words_and_sequences_env/train.py\n```\n\nTraining results:\n\n\u003cp align=\"center\"\u003e\n\u003cimg src=\"images/message_decoding_sequences_correctness_reward.png\" alt=\"Correctness Reward Message Decoding Sequences\" width=\"600\"/\u003e\n\u003c/p\u003e\n\n# Task 2: Digit Recognition\nAs a proof of concept, we trained `Qwen2.5VL-3B-Instruct` on a digit recognition task derived from MNIST. In each image, there are one, two or three digits. For each image, the model is either asked to return the list of digits in ascending order, or the sum of the digits.\n\n\n\u003cp align=\"center\"\u003e\n\u003ca href=\"https://huggingface.co/datasets/sunildkumar/digit-recognition-r1\"\u003e\n    \u003cimg src=\"https://img.shields.io/badge/See%20Dataset-On%20HuggingFace-yellow?style=for-the-badge\" alt=\"See Dataset\"\u003e\n  \u003c/a\u003e\n\u003c/p\u003e\n\nExample image from the dataset:\n\n\u003cp align=\"center\"\u003e\n\u003cimg src=\"images/digits_example.png\" alt=\"Example of digit recognition task\" width=\"300\"/\u003e\n\u003c/p\u003e\n\nYou can run training on 4 GPUs, 3 for training, one for completion generation with `vllm` using the following command. We've tested it on 4x A100 80GB GPUs. You can also get it running on two GPUs as well by tuning down the number of generations and running without `deepspeed`.\n```bash\n\n# 4 GPU training with deepspeed\nCUDA_VISIBLE_DEVICES=0,1,2,3 uv run accelerate launch --config_file src/r1_vlm/deepspeed_configs/multi_gpu_3only.yaml src/r1_vlm/environments/digit_recognition_env/train.py\n\n# 2 GPU training without deepspeed, you'll need to adjust the number of generations in the train.py file.\nCUDA_VISIBLE_DEVICES=0,1 uv run src/r1_vlm/environments/digit_recognition_env/train.py\n```\n\nTraining Results:\n\n\u003cp align=\"center\"\u003e\n\u003cimg src=\"images/digit_recognition_correctness_reward.png\" alt=\"Digit Recognition Correctness Reward\" width=\"600\"/\u003e\n\u003c/p\u003e\n\n# Task 3: Message Decoding - Single words\nA precursor to the message decoding task above. Only on single words. The reward design is significantly less sophisticated.\n\n\u003cp align=\"center\"\u003e\n\u003ca href=\"https://huggingface.co/datasets/sunildkumar/message-decoding-words-r1\"\u003e\n    \u003cimg src=\"https://img.shields.io/badge/See%20Dataset-On%20HuggingFace-yellow?style=for-the-badge\" alt=\"See Dataset\"\u003e\n  \u003c/a\u003e\n\u003c/p\u003e\n\n\n## Example Reasoning Trace:\nHere's an example where the message is `VISION`. You can see this model learned to \"repeat\" itself to verify its solution while thinking. \n\n\u003cp align=\"center\"\u003e\n\u003cimg src=\"images/example_decoder.webp\" alt=\"Decoder Image\"\u003e\n\u003c/p\u003e\n\n```\nsystem\nYou are a helpful assistant. You first think about the reasoning process in the mind and then provide the user with the answer.\n\nuser\nUse the decoder in the image to decode this coded message: \"K X I X R S\". The decoded message should be an english word. Show your work in \u003cthink\u003e \u003c/think\u003e tags and return the answer in \u003canswer\u003e \u003c/answer\u003e tags, for example \u003canswer\u003e \"CAT\" \u003c/answer\u003e.\n\nassistant\nLet me solve this step by step.\n\u003cthink\u003e \n1. First, I'll decode each character in the message \"K X I X R S\" using the provided decoder:\n\n   K → V\n   X → I\n   I → S\n   X → I\n   R → O\n   S → N\n\n2. Now, I'll Combine the results: \"V I S I O N\".\n\nAnswer: VISION\n\nAnswer: VISION\n\n\u003c/think\u003e\n\u003canswer\u003eVISION\u003c/answer\u003e\n```\n\nYou can run training on 4 GPUs, 3 for training, one for completion generation with `vllm` using the following command. We've tested it on 4x A100 80GB GPUs. You can also get it running on two GPUs as well by tuning down the number of generations and running without `deepspeed`.\n```bash \n# 4 GPU training with deepspeed\nCUDA_VISIBLE_DEVICES=0,1,2,3 uv run accelerate launch --config_file src/r1_vlm/deepspeed_configs/multi_gpu_3only.yaml src/r1_vlm/environments/message_decoding_env/train.py\n\n# 2 GPU training, you'll need to adjust the number of generations in the train.py file.\nCUDA_VISIBLE_DEVICES=0,1 uv run src/r1_vlm/environments/message_decoding_env/train.py\n```\n\nTraining results:\n\n\u003cp align=\"center\"\u003e\n\u003cimg src=\"images/message_decoding_correctness_reward.png\" alt=\"Message Decoding Single Word Correctness Reward\" width=\"600\"/\u003e\n\u003c/p\u003e\n\n# Acknowledgements\n- We thank @willccbb for his work on the `verifiers` [package](https://github.com/willccbb/verifiers). We loved his \"environment\" abstraction and take advantage of it in this project.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgroundlight%2Fr1_vlm","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgroundlight%2Fr1_vlm","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgroundlight%2Fr1_vlm/lists"}