{"id":19156503,"url":"https://github.com/kyegomez/fuyu","last_synced_at":"2025-05-07T07:45:01.429Z","repository":{"id":201036427,"uuid":"706833978","full_name":"kyegomez/Fuyu","owner":"kyegomez","description":"Implementation of Adepts Fuyu all-new Multi-Modality model in pytorch","archived":false,"fork":false,"pushed_at":"2024-11-11T09:52:24.000Z","size":402,"stargazers_count":25,"open_issues_count":1,"forks_count":3,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-04-19T20:17:09.809Z","etag":null,"topics":["ai","artificial-intelligence","gpt4","gpt5","machine-learning","multi-modal","multi-modality"],"latest_commit_sha":null,"homepage":"https://discord.gg/GYbXvDGevY","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/kyegomez.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null},"funding":{"github":["kyegomez"],"patreon":null,"open_collective":null,"ko_fi":null,"tidelift":null,"community_bridge":null,"liberapay":null,"issuehunt":null,"otechie":null,"lfx_crowdfunding":null,"custom":null}},"created_at":"2023-10-18T17:51:52.000Z","updated_at":"2025-01-19T21:57:24.000Z","dependencies_parsed_at":null,"dependency_job_id":"d225b316-afb3-4f1d-a836-ddeabb50220b","html_url":"https://github.com/kyegomez/Fuyu","commit_stats":null,"previous_names":["kyegomez/fuyu"],"tags_count":0,"template":false,"template_full_name":"kyegomez/Paper-Implementation-Template","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kyegomez%2FFuyu","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kyegomez%2FFuyu/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kyegomez%2FFuyu/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kyegomez%2FFuyu/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/kyegomez","download_url":"https://codeload.github.com/kyegomez/Fuyu/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252838460,"owners_count":21812078,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai","artificial-intelligence","gpt4","gpt5","machine-learning","multi-modal","multi-modality"],"created_at":"2024-11-09T08:34:44.748Z","updated_at":"2025-05-07T07:45:01.408Z","avatar_url":"https://github.com/kyegomez.png","language":"Python","funding_links":["https://github.com/sponsors/kyegomez"],"categories":[],"sub_categories":[],"readme":"[![Multi-Modality](agorabanner.png)](https://discord.gg/qUtxnK2NMf)\n\n# Fuyu\n![FUYU](/architecture.png)\n\nA implementation of Fuyu, the multimodal AI model from Adept in pytorch and zeta. The architecture is basically instead of using an encoder like VIT or CLIP they just patch the image then project it then feed it into the transformer decoder. The architecture is image patch embeddings -\u003e linear projection -\u003e decoder llm. \n\n**UPDATE**\n- [Fuyu-Heavy:](https://www.adept.ai/blog/adept-fuyu-heavy) proposes that scaling up the model architecture works but with some caveats. They need more stabilization during training. I have refactored the base Fuyu model implementation to include RMSNorm, LayerNorm, Swish, and a vast array of other techniques to radically increase multi-modal training such as normalizing the image during the shape rearrange and after.\n\n- DPO Confirmed [HERE](https://twitter.com/code_monet/status/1750218951832035580)\n\n\n\n[Blog paper code](https://www.adept.ai/blog/fuyu-8b)\n\n# Appreciation\n* Lucidrains\n* Agorians\n* Adept\n\n# Install\n`pip install fuyu`\n\n## Usage\n```python\nimport torch\nfrom fuyu import Fuyu\n\n# Initialize model\nmodel = Fuyu(\n    num_tokens=20342,\n    max_seq_len=4092,\n    dim=640,\n    depth=8,\n    dim_head=128,\n    heads=6,\n    use_abs_pos_emb=False,\n    alibi_pos_bias=True,\n    alibi_num_heads=3,\n    rotary_xpos=True,\n    attn_flash=True,\n    attn_kv_heads=2,\n    qk_norm=False,\n    attn_qk_norm=False,\n    attn_qk_norm_dim_scale=False,\n    patches=16,\n)\n\n# Text shape: [batch, seq_len, dim]\ntext = torch.randint(0, 20342, (1, 4092))\n\n# Img shape: [batch, channels, height, width]\nimg = torch.randn(1, 3, 256, 256)\n\n# Apply model to text and img\ny = model(text, img)\n\n# Output shape: [batch, seq_len, dim]\nprint(y)\n\n\n```\n\n\n# Citation\n\n\n\n# License\nMIT\n\n\n# Todo\n- [ ] Implement DPO with Fuyu","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkyegomez%2Ffuyu","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkyegomez%2Ffuyu","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkyegomez%2Ffuyu/lists"}