{"id":27641261,"url":"https://github.com/shreydan/simplevlm","last_synced_at":"2025-04-23T23:42:21.928Z","repository":{"id":289044960,"uuid":"969945330","full_name":"shreydan/simpleVLM","owner":"shreydan","description":"building a simple VLM. Combining LlaMA-SmolLM2 + SigLip2","archived":false,"fork":false,"pushed_at":"2025-04-21T07:38:23.000Z","size":7686,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-21T08:36:37.918Z","etag":null,"topics":["computer-vision","deep-learning","finetuning-llms","finetuning-vision-models","huggingface","llm","multimodal","nlp","pytorch","transformers","vision-language-model","vlm"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/shreydan.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-04-21T07:36:17.000Z","updated_at":"2025-04-21T08:27:00.000Z","dependencies_parsed_at":"2025-04-21T08:52:26.608Z","dependency_job_id":null,"html_url":"https://github.com/shreydan/simpleVLM","commit_stats":null,"previous_names":["shreydan/simplevlm"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shreydan%2FsimpleVLM","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shreydan%2FsimpleVLM/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shreydan%2FsimpleVLM/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shreydan%2FsimpleVLM/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/shreydan","download_url":"https://codeload.github.com/shreydan/simpleVLM/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250535036,"owners_count":21446503,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["computer-vision","deep-learning","finetuning-llms","finetuning-vision-models","huggingface","llm","multimodal","nlp","pytorch","transformers","vision-language-model","vlm"],"created_at":"2025-04-23T23:42:19.952Z","updated_at":"2025-04-23T23:42:21.376Z","avatar_url":"https://github.com/shreydan.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Vision Language Model\n\n- Built by implementing LLaMA from scratch with sft-weights from SmolLM2, and siglip2 as vision encoder.\n- Vision projector is a LlaMA MLP style projection block (~3M params)\n- Total model size: 230M\n\n\u003e See `eval.ipynb` for inference\n\n# Backbone\n ```\ntext-backbone: HuggingFaceTB/SmolLM2-135M-Instruct\nvision-backbone: google/siglip2-base-patch16-224\n ```\n\n# Config\n```\nconfig\n    embed_dim = 576\n    intermediate_dim = 1536\n    max_position_embeddings = 8192\n    base_theta = 100000\n    num_q_heads = 9\n    num_kv_heads = 3\n    attn_dropout = 0.\n    num_layers = 30\n    vocab_size = 49152\n    eos_token_id = 2\n    dtype = torch.bfloat16\n    num_image_tokens = 196\n```\n\n# Datasets\n```\nstage1: theblackcat102/llava-instruct-mix\nstage2: openbmb/RLAIF-V-Dataset\n```\n\n- super basic training in bf16\n- the results are decent","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fshreydan%2Fsimplevlm","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fshreydan%2Fsimplevlm","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fshreydan%2Fsimplevlm/lists"}