{"id":51844982,"url":"https://github.com/banaankiamanesh/para-seq","last_synced_at":"2026-07-23T11:00:54.833Z","repository":{"id":363681657,"uuid":"1231213934","full_name":"BanaanKiamanesh/Para-Seq","owner":"BanaanKiamanesh","description":"Impl of Dyn Sys Parallelization Techniques on RNNs and ODE Solvers Using Torch.","archived":false,"fork":false,"pushed_at":"2026-06-09T23:02:44.000Z","size":1263,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-06-10T00:12:06.376Z","etag":null,"topics":["gru","linear-dynamical-systems","lstm","parallel-scan","parallelization","rnn","sequential-data","torch"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/BanaanKiamanesh.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-05-06T18:38:52.000Z","updated_at":"2026-06-09T23:02:48.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/BanaanKiamanesh/Para-Seq","commit_stats":null,"previous_names":["banaankiamanesh/para-seq"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/BanaanKiamanesh/Para-Seq","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BanaanKiamanesh%2FPara-Seq","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BanaanKiamanesh%2FPara-Seq/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BanaanKiamanesh%2FPara-Seq/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BanaanKiamanesh%2FPara-Seq/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/BanaanKiamanesh","download_url":"https://codeload.github.com/BanaanKiamanesh/Para-Seq/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BanaanKiamanesh%2FPara-Seq/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35800203,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-07-20T02:08:10.276Z","status":"online","status_checked_at":"2026-07-23T02:00:06.683Z","response_time":57,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["gru","linear-dynamical-systems","lstm","parallel-scan","parallelization","rnn","sequential-data","torch"],"created_at":"2026-07-23T11:00:46.828Z","updated_at":"2026-07-23T11:00:54.803Z","avatar_url":"https://github.com/BanaanKiamanesh.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Para-Seq\n\nPara-Seq is a research-oriented PyTorch codebase for parallelizing nonlinear recurrent and sequential models over the sequence length.\n\nThe project implements fixed-point and Newton-style algorithms such as **Jacobi, Picard, DEER, quasi-DEER, ELK, and quasi-ELK**, then wraps them into PyTorch-style recurrent layers. The goal is to keep the familiar RNN/GRU/LSTM interface while replacing the usual sequential unroll with scan-based parallel solvers whenever possible.\n\n## What is included\n\n```text\nsrc/\n├── algos/         # DEER, ELK, Jacobi, Picard, fixed-point solvers\n├── ode_solvers/   # scan-based linear and nonlinear fixed-step ODE solvers\n├── pararnn/       # PyTorch-style ParaRNN, ParaGRU, ParaLSTM layers\n└── utils/         # associative scans, accelerated_scan wrappers, adjoint scans\n\ntest/\n├── ode_solvers/  # ODE solver correctness tests\n├── pararnn/      # correctness, gradient, backend, and layer API tests\n├── utils/        # scan backend tests\n└── bench/        # benchmark suites, README files, scripts, and logs\n\nnotebooks/        # small demos and Sequential MNIST experiments\n```\n\n## Main features\n\n| Component         | Description                                                                                                  |\n| ----------------- | ------------------------------------------------------------------------------------------------------------ |\n| `ParaRNN`         | Vanilla tanh/relu RNN with sequential and DEER-style modes.                                                  |\n| `ParaGRU`         | Diagonal ParaGRU with quasi-DEER, optional adjoint backward, and optional `accelerated_scan` support.        |\n| `ParaLSTM`        | CIFG/peephole-style ParaLSTM with structured block and diagonal scan backends.                               |\n| `src.algos`       | Standalone solvers for sequential rollout, Jacobi, Picard, DEER, quasi-DEER, ELK, and quasi-ELK experiments. |\n| `src.ode_solvers` | Fixed-step linear simulation and nonlinear ODE solvers using scan-based parallel rollouts.                   |\n| `src.utils`       | Dense, diagonal, and 2x2 block associative scans, plus reverse adjoint scans.                                |\n\n## Installation\n\nThis repo currently uses a source-layout workflow, so run scripts from the repository root or make sure the repo root is on `PYTHONPATH`.\n\n```bash\ngit clone \u003crepo-url\u003e\ncd Para-Seq\n\npython -m venv .venv\nsource .venv/bin/activate\n\npip install -r requirements.txt\n```\n\nThe CUDA accelerated scan path requires a CUDA build of PyTorch and the `accelerated-scan` package. If the PyTorch version in `requirements.txt` does not match your machine, install the correct PyTorch wheel for your CUDA setup first.\n\n## Quick example\n\n```python\nimport torch\nfrom src.pararnn import ParaGRU\n\ndevice = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n\nrnn = ParaGRU(\n    input_size=16,\n    hidden_size=64,\n    batch_first=True,\n    mode=\"deer\",\n    backend=\"adjoint\",\n    scan_backend=\"torch\",\n    num_iters=4,\n    dtype=torch.float32,\n).to(device)\n\nx = torch.randn(8, 512, 16, device=device)\n\noutput, h_n = rnn(x)\n\nprint(output.shape)  # (8, 512, 64)\nprint(h_n.shape)     # (1, 8, 64)\n```\n\nFor the CUDA accelerated diagonal scan backend:\n\n```python\nrnn = ParaGRU(\n    input_size=16,\n    hidden_size=64,\n    batch_first=True,\n    mode=\"deer\",\n    backend=\"adjoint\",\n    scan_backend=\"accel_scan\",\n    accel_module=\"warp\",\n    num_iters=4,\n).to(\"cuda\")\n```\n\nFor fixed-step ODE experiments:\n\n```python\nimport torch\nfrom src.ode_solvers import lsim\n\nt = torch.linspace(0.0, 1.0, 128)\nU = torch.zeros(128, 1)\nA = torch.tensor([[-0.5]])\nB = torch.tensor([[1.0]])\nC = torch.tensor([[1.0]])\nD = torch.tensor([[0.0]])\n\ny, x = lsim(A, B, C, D, U, t, scan_backend=\"torch\")\n```\n\n## Running tests\n\n```bash\npytest test/pararnn test/utils test/ode_solvers -q\n```\n\nThe tests check forward correctness, gradient equivalence, layer API behavior, scan backends, adjoint scans, ODE solver correctness, and several DEER/ELK configurations.\n\n## Running benchmarks\n\nBenchmark documentation lives in `test/bench/README.md` and the README in each benchmark subfolder.\n\n```bash\nPYTHONPATH=. python test/bench/algos/benchmarking.py --max-seq-len 4096\nPYTHONPATH=. python test/bench/baseline/strong_baselines.py --repeats 2\npython test/bench/ode_solvers/heavy_benchmark.py --profile quick\n```\n\nThe benchmark folders also include bash entry points for larger runs:\n\n```bash\nbash test/bench/algos/benchmarking.sh\nODE_BENCH_PROFILE=quick bash test/bench/ode_solvers/run_ode_benchmarks.sh\n```\n\nBenchmark logs and CSV files are written under the corresponding `test/bench/*/logs/` folders.\n\n## Demos\n\nThe `notebooks/` folder contains small training and comparison demos, including:\n\n* synthetic drift classification,\n* synthetic memory with ParaLSTM,\n* linear and nonlinear ODE solver examples,\n* Sequential MNIST and permuted Sequential MNIST experiments,\n* ParaGRU and ParaLSTM examples using quasi-DEER and `accelerated_scan`.\n\n## Notes and limitations\n\nThis is research code. The implementation is useful for experimenting with parallel-in-time sequential model evaluation, but it is not yet packaged as a polished library.\n\nCurrent limitations include:\n\n* single recurrent layer only,\n* unidirectional models only,\n* dropout is not implemented for the ParaRNN/ParaGRU/ParaLSTM layers,\n* ODE solvers currently target fixed-step time grids,\n* full dense DEER is mainly practical for small hidden sizes,\n* `accelerated_scan` requires CUDA,\n* imports currently assume the repo root is available as a Python path.\n\nFor larger hidden sizes or long sequences, the diagonal, quasi, block-structured, or adjoint-backed paths are usually the most practical ones.\n\n## Background\n\nThe code follows the idea that a sequential model can be written as one large nonlinear residual system. Instead of unrolling the recurrence step by step, the solvers repeatedly linearize or approximate the system and solve the resulting linear recurrence with associative scan operations.\n\nThis connects nonlinear sequential model evaluation to parallel prefix scans and linear dynamical system solvers, making it possible to experiment with sequence-parallel versions of otherwise sequential recurrent models.\n\n## References\n\n[1] Y. H. Lim, Q. Zhu, J. Selfridge, and M. F. Kasim, “Parallelizing non-linear sequential models over the sequence length,” in *Proc. International Conference on Learning Representations (ICLR)*, 2024. [Online]. Available: https://arxiv.org/abs/2309.12252\n\n[2] Machine Discovery Ltd., “deer,” GitHub repository. [Online]. Available: https://github.com/machine-discovery/deer\n\n[3] X. Gonzalez, A. Warrington, J. T. H. Smith, and S. W. Linderman, “Towards scalable and stable parallelization of nonlinear RNNs,” in *Proc. Advances in Neural Information Processing Systems (NeurIPS)*, 2024. [Online]. Available: https://arxiv.org/abs/2407.19115\n\n[4] S. W. Linderman Lab, “elk,” GitHub repository. [Online]. Available: https://github.com/lindermanlab/elk\n\n[5] X. Gonzalez, E. K. Buchanan, H. D. Lee, J. W. Liu, K. A. Wang, D. M. Zoltowski, L. Kozachkov, C. Ré, and S. W. Linderman, “A unifying framework for parallelizing sequential models with linear dynamical systems,” *Transactions on Machine Learning Research*, 2026. [Online]. Available: https://arxiv.org/abs/2509.21716\n\n[6] S. W. Linderman Lab, “parallelizing_with_lds,” GitHub repository. [Online]. Available: https://github.com/lindermanlab/parallelizing_with_lds\n\n[7] F. Danieli, P. Rodríguez, M. Sarabia, X. Suau, and L. Zappella, “ParaRNN: Unlocking parallel training of nonlinear RNNs for large language models,” Apple, 2025. [Online]. Available: https://arxiv.org/abs/2510.21450\n\n[8] Apple, “ml-pararnn,” GitHub repository, 2025. [Online]. Available: https://github.com/apple/ml-pararnn\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbanaankiamanesh%2Fpara-seq","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbanaankiamanesh%2Fpara-seq","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbanaankiamanesh%2Fpara-seq/lists"}