{"id":21702395,"url":"https://github.com/aphp/foldedtensor","last_synced_at":"2025-04-12T14:51:29.672Z","repository":{"id":172720404,"uuid":"649667619","full_name":"aphp/foldedtensor","owner":"aphp","description":"PyTorch extension for handling deeply nested sequences of variable length","archived":false,"fork":false,"pushed_at":"2025-04-04T19:19:09.000Z","size":104,"stargazers_count":10,"open_issues_count":1,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-04-04T19:19:58.539Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/aphp.png","metadata":{"files":{"readme":"README.md","changelog":"changelog.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-06-05T11:29:30.000Z","updated_at":"2025-04-04T19:18:24.000Z","dependencies_parsed_at":null,"dependency_job_id":"d99a7830-9419-404f-8c9c-50d50c946738","html_url":"https://github.com/aphp/foldedtensor","commit_stats":null,"previous_names":["aphp/foldedtensor"],"tags_count":8,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aphp%2Ffoldedtensor","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aphp%2Ffoldedtensor/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aphp%2Ffoldedtensor/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aphp%2Ffoldedtensor/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/aphp","download_url":"https://codeload.github.com/aphp/foldedtensor/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248586270,"owners_count":21128995,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-25T21:15:12.054Z","updated_at":"2025-04-12T14:51:29.665Z","avatar_url":"https://github.com/aphp.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cdiv align=\"center\"\u003e\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"https://github.com/aphp/foldedtensor/raw/main/banner.png\" width=\"70%\"\u003e\n\u003c/p\u003e\n\u003c/div\u003e\n\u003cbr/\u003e\n\n---\n\n![Tests](https://img.shields.io/github/actions/workflow/status/aphp/foldedtensor/tests.yml?branch=main\u0026label=tests\u0026style=flat-square)\n[![PyPI](https://img.shields.io/pypi/v/foldedtensor?color=blue\u0026style=flat-square)](https://pypi.org/project/foldedtensor/)\n[![Coverage](https://raw.githubusercontent.com/aphp/foldedtensor/coverage/coverage.svg)](https://raw.githubusercontent.com/aphp/foldedtensor/coverage/coverage.txt)\n[![License](https://img.shields.io/github/license/aphp/foldedtensor?color=x\u0026style=flat-square)](https://github.com/aphp/foldedtensor/blob/main/LICENSE)\n![PyPI - Downloads](https://img.shields.io/pypi/dm/foldedtensor?style=flat-square\u0026color=purple)\n![Python versions](https://img.shields.io/pypi/pyversions/foldedtensor?style=flat-square)\n\n# FoldedTensor: PyTorch extension for handling deeply nested sequences of variable length\n\n`foldedtensor` is a PyTorch extension that provides efficient handling of tensors containing deeply nested sequences variable sizes. It enables the flattening/unflattening (or unfolding/folding) of data dimensions based on a inner structure of sequence lengths. This library is particularly useful when working with data that can be split in different ways and enables you to avoid choosing a fixed representation.\n\n## Installation\n\nThe library can be installed with pip:\n\n```bash\npip install foldedtensor\n```\n\n## Features\n\n- Support for arbitrary numbers of nested dimensions\n- No computational overhead when dealing with already padded tensors\n- Dynamic re-padding (or refolding) of data based on stored inner lengths\n- Automatic mask generation and updating whenever the tensor is refolded\n- C++ optimized code for fast data loading from Python lists and refolding\n- Flexibility in data representation, making it easy to switch between different layouts when needed\n\n## Examples\n\nAt its simplest, `foldedtensor` can be used to convert nested Python lists into a PyTorch tensor:\n\n```python\nfrom foldedtensor import as_folded_tensor\n\nft = as_folded_tensor(\n    [\n        [0, 1, 2],\n        [3],\n    ],\n)\n# FoldedTensor([[0, 1, 2],\n#               [3, 0, 0]])\n```\n\nYou can also specify names and flattened/unflattened dimensions at the time of creation:\n\n```python\nimport torch\nfrom foldedtensor import as_folded_tensor\n\n# Creating a folded tensor from a nested list\n# There are 2 samples, the first with 5 lines, the second with 1 line.\n# Each line contain between 1 and 2 words.\nft = as_folded_tensor(\n    [\n        [[1], [], [], [], [2, 3]],\n        [[4, 3]],\n    ],\n    data_dims=(\"samples\", \"words\"),\n    full_names=(\"samples\", \"lines\", \"words\"),\n    dtype=torch.long,\n)\nprint(ft)\n# FoldedTensor([[1, 2, 3],\n#               [4, 3, 0]])\n```\n\nOnce created, you can change the shape of the tensor by refolding it:\n\n```python\n# Refold on the lines and words dims (flatten the samples dim)\nprint(ft.refold((\"lines\", \"words\")))\n# FoldedTensor([[1, 0],\n#               [0, 0],\n#               [0, 0],\n#               [0, 0],\n#               [2, 3],\n#               [4, 3]])\n\n# Refold on the words dim only: flatten everything\nprint(ft.refold((\"words\",)))\n# FoldedTensor([1, 2, 3, 4, 3])\n```\n\nThe tensor can be further used with standard PyTorch operations:\n\n```python\n# Working with PyTorch operations\nembedder = torch.nn.Embedding(10, 16)\nembedding = embedder(ft.refold((\"words\",)))\nprint(embedding.shape)\n# torch.Size([5, 16]) # 5 words total, 16 dims\n\nrefolded_embedding = embedding.refold((\"samples\", \"words\"))\nprint(refolded_embedding.shape)\n# torch.Size([2, 5, 16]) # 2 samples, 5 words max, 16 dims\n```\n\n## Benchmarks\n\nView the comparisons of `foldedtensor` against various alternatives here: [docs/benchmarks](https://github.com/aphp/foldedtensor/blob/main/docs/benchmark.md).\n\n## Comparison with alternatives\n\nUnlike other ragged or nested tensor implementations, a FoldedTensor does not enforce a specific structure on the nested data, and does not require padding all dimensions. This provides the user with greater flexibility when working with data that can be arranged in multiple ways depending on the data transformation. Moreover, the C++ optimization ensures high performance, making it ideal for handling deeply nested tensors efficiently.\n\nHere is a comparison with other common implementations for handling nested sequences of variable length:\n\n| Feature                   | NestedTensor | MaskedTensor | FoldedTensor |\n|---------------------------|--------------|--------------|--------------|\n| Inner data structure      | Flat         | Padded       | Arbitrary    |\n| Max nesting level         | 1            | 1            | ∞            |\n| From nested python lists  | No           | No           | Yes          |\n| Layout conversion         | To padded    | No           | Any          |\n| Reduction ops w/o padding | Yes          | No           | No           |\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faphp%2Ffoldedtensor","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Faphp%2Ffoldedtensor","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faphp%2Ffoldedtensor/lists"}