{"id":19156487,"url":"https://github.com/kyegomez/jamba","last_synced_at":"2025-05-07T07:42:09.216Z","repository":{"id":230941663,"uuid":"780498063","full_name":"kyegomez/Jamba","owner":"kyegomez","description":"PyTorch Implementation of Jamba: \"Jamba: A Hybrid Transformer-Mamba Language Model\"","archived":false,"fork":false,"pushed_at":"2025-04-04T12:57:27.000Z","size":2273,"stargazers_count":166,"open_issues_count":0,"forks_count":11,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-04-14T17:02:50.833Z","etag":null,"topics":["ai","artificial-neural-networks","attention-is-all-you-need","attention-mechanism","attention-mechanisms","gpt","ml","ssm","transformers"],"latest_commit_sha":null,"homepage":"https://discord.gg/7VckQVxvKk","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/kyegomez.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null},"funding":{"github":["kyegomez"],"patreon":null,"open_collective":null,"ko_fi":null,"tidelift":null,"community_bridge":null,"liberapay":null,"issuehunt":null,"otechie":null,"lfx_crowdfunding":null,"custom":null}},"created_at":"2024-04-01T15:56:24.000Z","updated_at":"2025-04-10T20:28:18.000Z","dependencies_parsed_at":"2024-11-09T08:36:13.749Z","dependency_job_id":"310fd652-feb1-4e5f-938d-40128826cff5","html_url":"https://github.com/kyegomez/Jamba","commit_stats":null,"previous_names":["kyegomez/jamba"],"tags_count":0,"template":false,"template_full_name":"kyegomez/Python-Package-Template","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kyegomez%2FJamba","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kyegomez%2FJamba/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kyegomez%2FJamba/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kyegomez%2FJamba/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/kyegomez","download_url":"https://codeload.github.com/kyegomez/Jamba/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252836503,"owners_count":21811722,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai","artificial-neural-networks","attention-is-all-you-need","attention-mechanism","attention-mechanisms","gpt","ml","ssm","transformers"],"created_at":"2024-11-09T08:34:41.230Z","updated_at":"2025-05-07T07:42:09.194Z","avatar_url":"https://github.com/kyegomez.png","language":"Python","funding_links":["https://github.com/sponsors/kyegomez"],"categories":[],"sub_categories":[],"readme":"[![Multi-Modality](agorabanner.png)](https://discord.gg/qUtxnK2NMf)\n\n# Jamba\nPyTorch Implementation of Jamba: \"Jamba: A Hybrid Transformer-Mamba Language Model\"\n\n\n## install\n`$ pip install jamba`\n\n## usage\n\n```python\n# Import the torch library, which provides tools for machine learning\nimport torch\n\n# Import the Jamba model from the jamba.model module\nfrom jamba.model import Jamba\n\n# Create a tensor of random integers between 0 and 100, with shape (1, 100)\n# This simulates a batch of tokens that we will pass through the model\nx = torch.randint(0, 100, (1, 100))\n\n# Initialize the Jamba model with the specified parameters\n# dim: dimensionality of the input data\n# depth: number of layers in the model\n# num_tokens: number of unique tokens in the input data\n# d_state: dimensionality of the hidden state in the model\n# d_conv: dimensionality of the convolutional layers in the model\n# heads: number of attention heads in the model\n# num_experts: number of expert networks in the model\n# num_experts_per_token: number of experts used for each token in the input data\nmodel = Jamba(\n    dim=512,\n    depth=6,\n    num_tokens=100,\n    d_state=256,\n    d_conv=128,\n    heads=8,\n    num_experts=8,\n    num_experts_per_token=2,\n)\n\n# Perform a forward pass through the model with the input data\n# This will return the model's predictions for each token in the input data\noutput = model(x)\n\n# Print the model's predictions\nprint(output)\n\n```\n\n## Train\n`python3 train.py`\n\n\n# License\nMIT\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkyegomez%2Fjamba","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkyegomez%2Fjamba","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkyegomez%2Fjamba/lists"}