{"id":19156194,"url":"https://github.com/kyegomez/lfm","last_synced_at":"2025-04-12T18:51:01.912Z","repository":{"id":257807303,"uuid":"865473812","full_name":"kyegomez/LFM","owner":"kyegomez","description":"An open source implementation of LFMs from Liquid AI: Liquid Foundation Models","archived":false,"fork":false,"pushed_at":"2025-03-17T12:50:16.000Z","size":2299,"stargazers_count":161,"open_issues_count":5,"forks_count":27,"subscribers_count":8,"default_branch":"main","last_synced_at":"2025-04-03T23:11:18.418Z","etag":null,"topics":["agents","ai","genai","liquid","llms","ssm","swarms","transformers"],"latest_commit_sha":null,"homepage":"https://discord.com/servers/agora-999382051935506503","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/kyegomez.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null},"funding":{"github":["kyegomez"],"patreon":null,"open_collective":null,"ko_fi":null,"tidelift":null,"community_bridge":null,"liberapay":null,"issuehunt":null,"otechie":null,"lfx_crowdfunding":null,"custom":null}},"created_at":"2024-09-30T15:33:47.000Z","updated_at":"2025-04-03T17:42:22.000Z","dependencies_parsed_at":null,"dependency_job_id":"2dece375-0b2f-4fa4-9735-3340f799cdb1","html_url":"https://github.com/kyegomez/LFM","commit_stats":{"total_commits":17,"total_committers":2,"mean_commits":8.5,"dds":0.3529411764705882,"last_synced_commit":"0a6a50a78be86e444878a6cd3aa72b7326ace4db"},"previous_names":["kyegomez/lfm"],"tags_count":0,"template":false,"template_full_name":"kyegomez/Python-Package-Template","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kyegomez%2FLFM","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kyegomez%2FLFM/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kyegomez%2FLFM/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kyegomez%2FLFM/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/kyegomez","download_url":"https://codeload.github.com/kyegomez/LFM/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248618223,"owners_count":21134199,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["agents","ai","genai","liquid","llms","ssm","swarms","transformers"],"created_at":"2024-11-09T08:33:33.117Z","updated_at":"2025-04-12T18:51:01.890Z","avatar_url":"https://github.com/kyegomez.png","language":"Python","funding_links":["https://github.com/sponsors/kyegomez"],"categories":[],"sub_categories":[],"readme":"\n# Liquid Foundation Models [LFMs]\n\n[![Join our Discord](https://img.shields.io/badge/Discord-Join%20our%20server-5865F2?style=for-the-badge\u0026logo=discord\u0026logoColor=white)](https://discord.gg/agora-999382051935506503) [![Subscribe on YouTube](https://img.shields.io/badge/YouTube-Subscribe-red?style=for-the-badge\u0026logo=youtube\u0026logoColor=white)](https://www.youtube.com/@kyegomez3242) [![Connect on LinkedIn](https://img.shields.io/badge/LinkedIn-Connect-blue?style=for-the-badge\u0026logo=linkedin\u0026logoColor=white)](https://www.linkedin.com/in/kye-g-38759a207/) [![Follow on X.com](https://img.shields.io/badge/X.com-Follow-1DA1F2?style=for-the-badge\u0026logo=x\u0026logoColor=white)](https://x.com/kyegomezb)\n\nThis is an attempt to make an open source implementation of LFMs, this is obviously not the official repository because it's closed source. I link papers below which I am using as a referrence.\n[Discover more about the model from the original article](https://www.liquid.ai/liquid-foundation-models)\n\n## Installation\n```bash\n$ pip3 install -U lfm-torch\n```\n\n## Usage\n\n```python\nimport torch\nfrom lfm_torch.model import LFModel\nfrom loguru import logger\n\n# Instantiate and test the model\nif __name__ == \"__main__\":\n    batch_size, seq_length, embedding_dim = 32, 128, 512\n    token_dim, channel_dim, expert_dim, adapt_dim, num_experts = (\n        embedding_dim,\n        embedding_dim,\n        embedding_dim,\n        128,\n        4,\n    )\n    model = LFModel(\n        token_dim, channel_dim, expert_dim, adapt_dim, num_experts\n    )\n\n    input_tensor = torch.randn(\n        batch_size, seq_length, embedding_dim\n    )  # 3D text tensor\n    output = model(input_tensor)\n    logger.info(\"Model forward pass complete.\")\n```\n\n\n## Liquid Transformer \nA novel neural architecture combining Liquid Neural Networks, Transformer attention mechanisms, and Mixture of Experts (MoE) for enhanced adaptive processing and dynamic state updates. Very experimental and early! We're working on a training script [here](./liquid_transformer_train.py). It still needs an actual tokenizer like llama's tokenizer but it's getting there. If you can help with this then let me know.\n\n\n### Architecture Overview\n\n```mermaid\nflowchart TB\n    subgraph \"Liquid Transformer\"\n        Input[\"Input Sequence\"] --\u003e TL[\"Transformer Layer\"]\n        \n        subgraph \"Transformer Layer\"\n            direction TB\n            MHA[\"Multi-Head Attention\"] --\u003e LC[\"Liquid Cell\"]\n            LC --\u003e MOE[\"Mixture of Experts\"]\n            MOE --\u003e LN[\"Layer Norm + Residual\"]\n        end\n        \n        subgraph \"Liquid Cell Details\"\n            direction LR\n            HS[\"Hidden State\"] --\u003e WH[\"W_h Linear\"]\n            Input2[\"Input\"] --\u003e WI[\"W_in Linear\"]\n            WH --\u003e Add((+))\n            WI --\u003e Add\n            Add --\u003e Act[\"Activation\"]\n            Act --\u003e LN2[\"LayerNorm\"]\n            LN2 --\u003e DO[\"Dropout\"]\n        end\n        \n        subgraph \"MoE Details\"\n            direction TB\n            Input3[\"Input\"] --\u003e Gate[\"Gating Network\"]\n            Input3 --\u003e E1[\"Expert 1\"]\n            Input3 --\u003e E2[\"Expert 2\"]\n            Input3 --\u003e E3[\"Expert N\"]\n            Gate --\u003e Comb[\"Weighted Combination\"]\n            E1 --\u003e Comb\n            E2 --\u003e Comb\n            E3 --\u003e Comb\n        end\n        \n        TL --\u003e Output[\"Output Sequence\"]\n    end\n```\n\n\n\n```python\nimport torch\nfrom loguru import logger\n\nfrom lfm_torch.liquid_t_moe import LiquidTransformer\n\n# Example usage\nif __name__ == \"__main__\":\n    seq_len, batch_size, embed_size = 10, 2, 64\n    num_heads, num_experts, expert_size, num_layers = 8, 4, 64, 6\n\n    # Create the model\n    model = LiquidTransformer(embed_size, num_heads, num_experts, expert_size, num_layers)\n\n    # Example input tensor\n    x = torch.randn(seq_len, batch_size, embed_size)\n\n    # Forward pass\n    output = model(x)\n    logger.info(f\"Model output shape: {output.shape}\")\n```\n\n\n# Citations\n- All credit for the liquid transformer architecture goes to the original authors from liquid.ai\n- https://arxiv.org/abs/2209.12951\n- \n\n# License\nThis project is licensed under the MIT License. See the [LICENSE](LICENSE) file for details.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkyegomez%2Flfm","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkyegomez%2Flfm","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkyegomez%2Flfm/lists"}