{"id":22703601,"url":"https://github.com/muditbhargava66/PyxLSTM","last_synced_at":"2025-08-07T10:31:31.743Z","repository":{"id":238962807,"uuid":"797967721","full_name":"muditbhargava66/PyxLSTM","owner":"muditbhargava66","description":"Efficient Python library for Extended LSTM with exponential gating, memory mixing, and matrix memory for superior sequence modeling.","archived":false,"fork":false,"pushed_at":"2024-06-28T04:52:49.000Z","size":123,"stargazers_count":269,"open_issues_count":1,"forks_count":21,"subscribers_count":5,"default_branch":"main","last_synced_at":"2024-11-21T12:49:35.931Z","etag":null,"topics":["language-modeling","lstm","sequence-modeling","xlstm"],"latest_commit_sha":null,"homepage":"https://pyxlstm.readthedocs.io/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/muditbhargava66.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-05-08T20:39:35.000Z","updated_at":"2024-11-21T07:53:09.000Z","dependencies_parsed_at":"2024-06-28T05:28:06.745Z","dependency_job_id":null,"html_url":"https://github.com/muditbhargava66/PyxLSTM","commit_stats":null,"previous_names":["muditbhargava66/pyxlstm"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/muditbhargava66%2FPyxLSTM","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/muditbhargava66%2FPyxLSTM/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/muditbhargava66%2FPyxLSTM/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/muditbhargava66%2FPyxLSTM/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/muditbhargava66","download_url":"https://codeload.github.com/muditbhargava66/PyxLSTM/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":229026412,"owners_count":18008325,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["language-modeling","lstm","sequence-modeling","xlstm"],"created_at":"2024-12-10T08:12:00.529Z","updated_at":"2024-12-10T08:12:01.856Z","avatar_url":"https://github.com/muditbhargava66.png","language":"Python","funding_links":[],"categories":["Implementations (Most of them are unofficial and under development)"],"sub_categories":[],"readme":"# PyxLSTM\n\n![Banner](assets/xlstm-logo-v2.png)\n\n![Python Version](https://img.shields.io/badge/python-3.6%20%7C%203.7%20%7C%203.8%20%7C%203.9%20%7C%203.10-blue)\n[![GitHub license](https://img.shields.io/github/license/muditbhargava66/PyxLSTM)](https://github.com/muditbhargava66/PyxLSTM/blob/main/LICENSE)\n[![Documentation Status](https://readthedocs.org/projects/pyxlstm/badge/?version=latest)](https://pyxlstm.readthedocs.io/en/latest/?badge=latest)\n![PRs Welcome](https://img.shields.io/badge/PRs-welcome-brightgreen.svg?style=flat-square)\n[![CodeQL](https://github.com/muditbhargava66/PyxLSTM/actions/workflows/github-code-scanning/codeql/badge.svg)](https://github.com/muditbhargava66/PyxLSTM/actions/workflows/github-code-scanning/codeql)\n[![GitHub stars](https://img.shields.io/github/stars/muditbhargava66/PyxLSTM)](https://github.com/muditbhargava66/PyxLSTM/stargazers)\n[![GitHub forks](https://img.shields.io/github/forks/muditbhargava66/PyxLSTM)](https://github.com/muditbhargava66/PyxLSTM/network/members)\n![GitHub Releases](https://img.shields.io/github/downloads/muditbhargava66/PyxLSTM/total)\n![Last Commit](https://img.shields.io/github/last-commit/muditbhargava66/PyxLSTM)\n[![Open Issues](https://img.shields.io/github/issues/muditbhargava66/PyxLSTM)](https://github.com/muditbhargava66/PyxLSTM/issues)\n[![Open PRs](https://img.shields.io/github/issues-pr/muditbhargava66/PyxLSTM)](https://github.com/muditbhargava66/PyxLSTM/pulls)\n\n\nPyxLSTM is a Python library that provides an efficient and extensible implementation of the Extended Long Short-Term Memory (xLSTM) architecture based on the research paper [\"xLSTM: Extended Long Short-Term Memory\"](https://arxiv.org/abs/2405.04517) by Beck et al. (2024). xLSTM enhances the traditional LSTM by introducing exponential gating, memory mixing, and a matrix memory structure, enabling improved performance and scalability for sequence modeling tasks.\n\n## Table of Contents\n\n- [Features](#features)\n- [Installation](#installation)\n- [Development Installation](#development-installation)\n- [Usage](#usage)\n- [Code Directory Structure](#code-directory-structure)\n- [Running and Testing the Codebase](#running-and-testing-the-codebase)\n- [Documentation](#documentation)\n- [Citation](#citation)\n- [Contributing](#contributing)\n- [License](#license)\n- [Acknowledgements](#acknowledgements)\n- [Contact](#contact)\n- [Star History](#star-history)\n- [TODO](#todo)\n\n## Features\n\n- Implements the sLSTM (scalar LSTM) and mLSTM (matrix LSTM) variants of xLSTM\n- Supports pre and post up-projection block structures for flexible model architectures\n- Provides high-level model definition and training utilities for ease of use\n- Includes scripts for training, evaluation, and text generation\n- Offers data processing utilities and customizable dataset classes\n- Lightweight and modular design for seamless integration into existing projects\n- Extensively tested and documented for reliability and usability\n- Suitable for a wide range of sequence modeling tasks, including language modeling, text generation, and more\n\n## Installation\n\nTo install PyxLSTM, you can use pip:\n\n```bash\npip install PyxLSTM\n```\n\n## Development Installation\n\nFor development installation with testing dependencies:\n\n```bash\npip install PyxLSTM[dev]\n```\n\nAlternatively, you can clone the repository and install it manually:\n\n```bash\ngit clone https://github.com/muditbhargava66/PyxLSTM.git\ncd PyxLSTM\npip install -r requirements.txt\npip install -e .\n```\n\n## Usage\n\nHere's a basic example of how to use PyxLSTM for language modeling:\n\n```python\nimport torch\nfrom xLSTM.model import xLSTM\nfrom xLSTM.data import LanguageModelingDataset, Tokenizer\nfrom xLSTM.utils import load_config, set_seed, get_device\nfrom xLSTM.training import train  # Assuming train function is defined in training module\n\n# Load configuration\nconfig = load_config(\"path/to/config.yaml\")\nset_seed(config.seed)\ndevice = get_device()\n\n# Initialize tokenizer and dataset\ntokenizer = Tokenizer(config.vocab_file)\ntrain_dataset = LanguageModelingDataset(config.train_data, tokenizer, config.max_length)\n\n# Create xLSTM model\nmodel = xLSTM(len(tokenizer), config.embedding_size, config.hidden_size,\n              config.num_layers, config.num_blocks, config.dropout,\n              config.bidirectional, config.lstm_type)\nmodel.to(device)\n\n# Train the model\noptimizer = torch.optim.Adam(model.parameters(), lr=config.learning_rate)\ncriterion = torch.nn.CrossEntropyLoss(ignore_index=tokenizer.pad_token_id)\ntrain(model, train_dataset, optimizer, criterion, config, device)\n```\n\nFor more detailed usage instructions and examples, please refer to the [documentation](docs/).\n\n## Code Directory Structure\n\n```\nxLSTM/\n│\n├── xLSTM/\n│   ├── __init__.py\n│   ├── slstm.py\n│   ├── mlstm.py\n│   ├── block.py\n│   └── model.py\n│\n├── utils/\n│   ├── config.py\n│   ├── logging.py\n│   └── utils.py\n│\n├── tests/\n│   ├── test_slstm.py  \n│   ├── test_mlstm.py\n│   ├── test_block.py\n│   └── test_model.py\n│\n├── docs/\n│   ├── slstm.md\n│   ├── mlstm.md\n│   └── training.md\n│\n├── examples/\n│   ├── language_modeling.py\n│   └── xLSTM_shape_verification.py\n│\n├── .gitignore\n├── pyproject.toml\n├── MANIFEST.in\n├── requirements.txt\n├── README.md\n└── LICENSE\n```\n\n- **xLSTM/**: The main Python package containing the implementation.\n  - slstm.py: Implementation of the sLSTM module.\n  - mlstm.py: Implementation of the mLSTM module.\n  - block.py: Implementation of the xLSTM blocks (pre and post up-projection).\n  - model.py: High-level xLSTM model definition.\n\n- **utils/**: Utility modules.\n  - `config.py`: Configuration management.\n  - `logging.py`: Logging setup.\n  - `utils.py`: Miscellaneous utility functions.\n\n- **tests/**: Unit tests for different modules.\n  - `test_slstm.py`: Tests for sLSTM module.  \n  - `test_mlstm.py`: Tests for mLSTM module.\n  - `test_block.py`: Tests for xLSTM blocks.\n  - `test_model.py`: Tests for the overall xLSTM model.\n\n- **docs/**: Documentation files.\n  - `README.md`: Main documentation file.\n  - `slstm.md`: Documentation for sLSTM.\n  - `mlstm.md`: Documentation for mLSTM.\n  - `training.md`: Training guide.\n\n- **.gitignore**: Git ignore file to exclude unnecessary files/directories.\n- **setup.py**: Package setup script.\n- **requirements.txt**: List of required Python dependencies.\n- **README.md**: Project README file.\n- **LICENSE**: Project license file.\n\n## Running and Testing the Codebase\n\nTo run and test the PyxLSTM codebase, follow these steps:\n\n1. Clone the PyxLSTM repository:\n   ```bash\n   git clone https://github.com/muditbhargava66/PyxLSTM.git\n   ```\n\n2. Navigate to the cloned directory:\n   ```bash\n   cd PyxLSTM\n   ```\n\n3. Install the required dependencies:\n   ```bash\n   pip install -r requirements.txt\n   ```\n\n4. Run the unit tests:\n   ```bash\n   python -m unittest discover tests\n   ```\n   This command will run all the unit tests located in the `tests` directory. It will execute the test files `test_slstm.py`, `test_mlstm.py`, `test_block.py`, and `test_model.py`.\n\nIf you encounter any issues or have further questions, please refer to the PyxLSTM documentation or reach out to the maintainers for assistance.\n\n## Documentation\n\nThe documentation for PyxLSTM can be found in the [docs](docs/) directory. It provides detailed information about the library's components, usage guidelines, and examples.\n\n## Citation\n\nIf you use PyxLSTM in your research or projects, please cite the original xLSTM paper:\n\n```bibtex\n@article{Beck2024xLSTM,\n  title={xLSTM: Extended Long Short-Term Memory},\n  author={Beck, Maximilian and Pöppel, Korbinian and Spanring, Markus and Auer, Andreas and Prudnikova, Oleksandra and Kopp, Michael and Klambauer, Günter and Brandstetter, Johannes and Hochreiter, Sepp},\n  journal={arXiv preprint arXiv:2405.04517},\n  year={2024}\n}\n```\n\nPaper link: [https://arxiv.org/abs/2405.04517](https://arxiv.org/abs/2405.04517)\n\n## Contributing\n\nContributions to PyxLSTM are welcome! If you find any issues or have suggestions for improvements, please open an issue or submit a pull request on the GitHub repository.\n\n## License\n\nPyxLSTM is released under the [MIT License](LICENSE). See the `LICENSE` file for more information.\n\n## Acknowledgements\n\nWe would like to acknowledge the original authors of the xLSTM architecture for their valuable research and contributions to the field of sequence modeling.\n\n## Contact\n\nFor any questions or inquiries, please contact the project maintainer:\n\n- Name: Mudit Bhargava\n- GitHub: [@muditbhargava66](https://github.com/muditbhargava66)\n\nWe hope you find PyxLSTM useful for your sequence modeling projects!\n\n## Star History\n\n\u003ca href=\"https://star-history.com/#muditbhargava66/PyxLSTM\u0026Date\"\u003e\n \u003cpicture\u003e\n   \u003csource media=\"(prefers-color-scheme: dark)\" srcset=\"https://api.star-history.com/svg?repos=muditbhargava66/PyxLSTM\u0026type=Date\u0026theme=dark\" /\u003e\n   \u003csource media=\"(prefers-color-scheme: light)\" srcset=\"https://api.star-history.com/svg?repos=muditbhargava66/PyxLSTM\u0026type=Date\" /\u003e\n   \u003cimg alt=\"Star History Chart\" src=\"https://api.star-history.com/svg?repos=muditbhargava66/PyxLSTM\u0026type=Date\" /\u003e\n \u003c/picture\u003e\n\u003c/a\u003e\n\n## TODO\n\n- [x] Add support for Python 3.10\n- [x] Add support for macOS MPS\n- [x] Add support for Windows MPS\n- [x] Add support for Linux MPS\n- [ ] Provide more examples on time series prediction\n- [ ] Include reinforcement learning examples\n- [ ] Add examples for modeling physical systems\n- [ ] Enhance documentation with advanced usage scenarios\n- [ ] Improve unit tests for new features\n- [ ] Add support for bidirectional parameter as it's not implemented in the current xLSTM model\n\n---","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmuditbhargava66%2FPyxLSTM","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmuditbhargava66%2FPyxLSTM","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmuditbhargava66%2FPyxLSTM/lists"}