{"id":22668367,"url":"https://github.com/lanl/pydntnk","last_synced_at":"2025-04-12T11:04:25.426Z","repository":{"id":49218583,"uuid":"378818934","full_name":"lanl/pyDNTNK","owner":"lanl","description":"Python Distributed Non Negative Tensor Networks","archived":false,"fork":false,"pushed_at":"2021-10-20T02:45:41.000Z","size":19516,"stargazers_count":13,"open_issues_count":0,"forks_count":5,"subscribers_count":5,"default_branch":"main","last_synced_at":"2024-06-27T00:19:55.928Z","etag":null,"topics":["compression","distributed-tensor-factorization","hierarchical-tensor-decompositions","hierarchical-tucker","non-negative-factorization","pydnmfk","tensor-networks","tensor-train"],"latest_commit_sha":null,"homepage":"https://lanl.github.io/pyDNTNK/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/lanl.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-06-21T05:42:00.000Z","updated_at":"2023-10-30T03:18:03.000Z","dependencies_parsed_at":"2022-09-16T03:22:06.573Z","dependency_job_id":null,"html_url":"https://github.com/lanl/pyDNTNK","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lanl%2FpyDNTNK","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lanl%2FpyDNTNK/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lanl%2FpyDNTNK/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lanl%2FpyDNTNK/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/lanl","download_url":"https://codeload.github.com/lanl/pyDNTNK/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":228911888,"owners_count":17990774,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["compression","distributed-tensor-factorization","hierarchical-tensor-decompositions","hierarchical-tucker","non-negative-factorization","pydnmfk","tensor-networks","tensor-train"],"created_at":"2024-12-09T15:14:53.808Z","updated_at":"2024-12-09T15:14:54.460Z","avatar_url":"https://github.com/lanl.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# [pyDNTNK: Python Distributed Non-Negative Tensor Networks](https://github.com/lanl/pyDNTNK)\n\n\n\u003cdiv align=\"center\", style=\"font-size: 50px\"\u003e\n\n[![Build Status](https://github.com/lanl/pyDNTNK/actions/workflows/ci_test.yml/badge.svg?branch=main)](https://github.com/lanl/pyDNTNK/actions/workflows/ci_test.yml/badge.svg?branch=main) [![License](https://img.shields.io/badge/License-BSD%203--Clause-blue.svg)](https://img.shields.io/badge/License-BSD%203--Clause-blue.svg) [![Python Version](https://img.shields.io/badge/python-v3.7.1-blue)](https://img.shields.io/badge/python-v3.7.1-blue) \n  [![DOI](https://zenodo.org/badge/378818934.svg)](https://zenodo.org/badge/latestdoi/378818934)\n\n\u003c/div\u003e\n\u003cp align=\"center\"\u003e\n  \u003cimg width=\"160\" height=\"200\" src=\"docs/RD100.png\"\u003e\n\u003c/p\u003e\n\n\u003cbr\u003e\n\n[pyDNTNK](https://github.com/lanl/pyDNTNK) is a software package for applying non-negative Hierarchical Tensor decompositions such as Tensor train and Hierarchical Tucker decompositons  in a distributed fashion to large datasets.\nIt is built on top of [pyDNMFk](https://github.com/lanl/pyDNMFk). Tensor train (TT) and Hierarchical Tucker(HT) are state-of-the-art tensor network introduced for factorization of high-dimensional tensors. These methods transform the initial high-dimensional tensor in a network of low dimensional tensors that requires only a linear storage. Many real-world data,such as, density, temperature, population, probability, etc., are non-negative and for an easy interpretation, the algorithms preserving non-negativity are preferred. Here, we introduce the  distributed non-negative Hierarchical tensor decomposition tools and demonstrate their scalability and the compression on synthetic and real world big datasets.\n\u003chr/\u003e\n\n\n## Features:\n\n* Utilization of MPI4py for distributed operation.\n* Distributed Reshaping and Unfolding operations with zarr and Dask.\n* Distributed Hierarchical Tensor decompositions such as Tensor train and Hierarchical Tucker.\n* Ability to perform both standard SVD based and NMF based decompositions. \n* Scalability to Tensors of very high dimensions.\n* Automated rank estimation with SVD for each stage of tensor decomposition.\n* Distributed Pruning of zero row and zero columns of the data. \n\n\n\n\n\n![plot](./docs/final_overview2.png)\n**Figure:Overview of the distributed Tensor Train  implementation.**\n\n## Installation:\n\nOn a desktop machine\n```\ngit clone https://github.com/lanl/pyDNTNK.git\ncd pyDNTNK\nconda create --name pyDNTNK python=3.7.1 openmpi mpi4py\nsource activate pyDNTNK\npython setup.py install\n```\n\n\u003chr/\u003e\n\nOn a server\n```\ngit clone https://github.com/lanl/pyDNTNK.git\ncd pyDNTNK\nconda create --name pyDNTNK python=3.7.1 \nsource activate pyDNTNK\nmodule load \u003copenmpi\u003e\npip install mpi4py\npython setup.py install\n```\n\n## Prerequisites:\n* pyDNMFk\n* conda\n* numpy\u003e=1.2\n* matplotlib\n* MPI4py\n* scipy\n* h5py\n* dask\n* zarr\n\n## Documentation\n\nYou can find the documentation [here](https://lanl.github.io/pyDNTNK/). \n\n\n## Usage\n```bash\nmpirun -n \u003cprocs\u003e python main.py usage: main.py [-h] [--p_grid P_GRID [P_GRID ...]] [--fpath FPATH]\n               [--model MODEL] [--routine ROUTINE] [--init INIT] [--itr ITR]\n               [--norm NORM] [--method METHOD] [--verbose VERBOSE]\n               [--results_path RESULTS_PATH] [--prune PRUNE]\n               [--precision PRECISION] [--err ERR] [--ranks RANKS [RANKS ...]]\n               [--save SAVE]\n\nArguments for pyDNTNK\n\noptional arguments:\n  -h, --help            show this help message and exit\n  --p_grid P_GRID [P_GRID ...]\n                        Processor Grid\n  --fpath FPATH         data path to read(eg: ../data/array.zarr)\n  --model MODEL         TN model (TT/TK) for tensor train/Tucker models\n  --routine ROUTINE     NMF for nTT/nTK and SVD for TT/TK\n  --init INIT           NMF initializations: rand/nnsvd\n  --itr ITR             NMF iterations, default:1000\n  --norm NORM           Reconstruction Norm for NMF to optimize:KL/FRO\n  --method METHOD       NMF update method:MU/BCD/HALS\n  --verbose VERBOSE\n  --results_path RESULTS_PATH\n                        Path for saving results\n  --prune PRUNE         Prune zero row/column.\n  --precision PRECISION\n                        Precision of the data(float32/float64/float16.\n  --err ERR             Error for rank estimation at each stage\n  --ranks RANKS [RANKS ...]\n                        Ranks for each stage of decomposition\n  --save SAVE           Store TN factors\n\n```\n\nWe provide a sample dataset that can be used for estimation of k:\n```python\n'''Imports block'''\nimport os\nos.environ[\"OMP_NUM_THREADS\"] = \"1\"\nfrom pyDNTNK import pyDNTNK\nfrom pyDNTNK import *\nfrom pyDNMFk.utils import *\nfrom pyDNMFk.dist_comm import *\nargs = parse()\n\n'''parameters initialization block'''\nargs.fpath = '../data/array.zarr'\nargs.p_grid = [2,1,1,1]\nargs.tt_ranks = [2,2,2,2]\nargs.model,args.routine = 'tt','nmf'\n\n'''Parameters prep block'''\nmain_comm = MPI.COMM_WORLD\nrank = main_comm.rank\nsize = main_comm.size\nargs.p_r, args.p_c = 1, size\ncomm = MPI_comm(main_comm, args.p_r, args.p_c)\nargs.rank = rank\nargs.main_comm = main_comm\nargs.comm1 = comm.comm\nargs.comm = comm\nargs.col_comm = comm.cart_1d_column()\nargs.row_comm = comm.cart_1d_row()\n\n'''Computations go here'''\nif main_comm.rank == 0: print('Starting ', args.model, ' Tensor Decomposition with ', args.routine)\ntt = pyDNTNK(args.fpath, args, model=args.model)\ntt.fit()\ntt.error_compute()\nfactors = tt.return_factors()\nassert len(factors)==4\nassert([i\u003c1e-2 for i in tt.rel_error])\n```\n\nAlternately, you can also run from test folder in command line as:\n```bash\nmpirun -n 2 python main.py   --fpath '../data/array.zarr'  --model 'tt' --routine 'nmf' --p_grid 2 1 1 1 --tt_ranks 2 2 2 2\n```\n\nSee the resources for more use cases.\n\u003chr/\u003e\n\n## Benchmarking\n\nStrong scaling (Overall)           |  Strong scaling (NMF) | Strong Scaling (Data Operations)\n:-------------------------:|:-------------------------:|:-------------------------:\n![](./docs/Strong_data_overall.png)  |  ![](./docs/Strong_data_nmf.png) |  ![](./docs/Strong_data_op.png) \n\nWeak scaling (Overall)         |  Weak scaling (NMF) | Weak scaling (Data Operations)\n:-------------------------:|:-------------------------:|:-------------------------:\n![](./docs/weak_data_overall.png)  |  ![](./docs/weak_data_nmf.png) |  ![](./docs/weak_data_op.png) \n\nScaling with TT-ranks (Overall)          |  Scaling with TT-ranks (NMF) | Scaling with TT-ranks (Data Operations)\n:-------------------------:|:-------------------------:|:-------------------------:\n![](./docs/k_data_overall.png)  |  ![](./docs/k_data_nmf.png) |  ![](./docs/k_data_op.png) \n\n**Figure: Scaling experiments for Tensor Train Decomposition.** \n\n## Scalability\n\nYale Face         |  Video dataset | Synthetic data(500GB)\n:-------------------------:|:-------------------------:|:-------------------------:\n![](./docs/compression_vs_error_vid_svd_nmf_final_loglog.png)  |  ![](./docs/compression_vs_error_vid_svd_nmf_final_loglog.png) |  ![](./docs/compression_vs_error_512gb_nmf_gr_mu.png) \n\n## Authors:\n\n* [Manish Bhattarai](mailto:ceodspspectrum@lanl.gov) - Los Alamos National Laboratory\n* [Erik Skau](mailto:ewskau@lanl.gov) - Los Alamos National Laboratory\n* [Phan Minh Duc Truong](mailto:dptruong@lanl.gov) - Los Alamos National Laboratory\n* [Maksim E. Eren](mailto:maksim@lanl.gov) - Los Alamos National Laboratory\n* [Namita Kharat](mailto:namita@lanl.gov) - Los Alamos National Laboratory\n* [Gopinath Chennupati](mailto:gchennupati@lanl.gov) - Los Alamos National Laboratory\n* [Raviteja Vangara](mailto:rvangara@lanl.gov) - Los Alamos National Laboratory\n* [Hristo Djidjev](mailto:djidjev@lanl.gov) - Los Alamos National Laboratory\n* [Boian Alexandrov](mailto:boian@lanl.gov) - Los Alamos National Laboratory\n\n## Citation:\n\n```latex\n@misc{bhattaraipyDNTNK,\n  author = {Manish Bhattarai,Erik Skau,Phan Minh Duc Truong,Maksim Eren,Namita Kharat,Gopinath Chennupati,Raviteja Vangara,Hristo Djidjev,Boian ALexandrov},\n  title = {pyDNTNK: Python Distributed Non Negative Tensor Networks},\n  year = {2021},\n  publisher = {GitHub},\n  journal = {GitHub repository},\n  doi = {10.5281/zenodo.5004490},\n  howpublished = {\\url{https://github.com/lanl/pyDNTNK}}\n}\n\n@misc{bhattaraipyDNMFk,\n  author = {Manish Bhattarai,Ben Nebgen,Erik Skau,Maksim Eren,Gopinath Chennupati,Raviteja Vangara,Hristo Djidjev,John Patchett,Jim Ahrens,Boian ALexandrov},\n  title = {pyDNMFk: Python Distributed Non Negative Matrix Factorization},\n  year = {2021},\n  publisher = {GitHub},\n  journal = {GitHub repository},\n  doi = {10.5281/zenodo.4722448},\n  howpublished = {\\url{https://github.com/lanl/pyDNMFk}}\n}\n\n@inproceedings{bhattarai2020distributed,\n  title={Distributed Non-Negative Tensor Train Decomposition},\n  author={Bhattarai, Manish and Chennupati, Gopinath and Skau, Erik and Vangara, Raviteja and Djidjev, Hristo and Alexandrov, Boian S},\n  booktitle={2020 IEEE High Performance Extreme Computing Conference (HPEC)},\n  pages={1--10},\n  year={2020},\n  organization={IEEE}\n}\n\n@inproceedings {s.20211055,\n  booktitle = {EuroVis 2021 - Short Papers},\n  editor = {Agus, Marco and Garth, Christoph and Kerren, Andreas},\n  title = {{Selection of Optimal Salient Time Steps by Non-negative Tucker Tensor Decomposition}},\n  author = {Pulido, Jesus and Patchett, John and Bhattarai, Manish and Alexandrov, Boian and Ahrens, James},\n  year = {2021},\n  publisher = {The Eurographics Association},\n  ISBN = {978-3-03868-143-4},\n  DOI = {10.2312/evs.20211055}\n}\n\n```\n\n## Acknowledgments:\nLos Alamos National Lab (LANL), T-1\n\n## Copyright Notice:\n\n© (or copyright) 2020. Triad National Security, LLC. All rights reserved.\nThis program was produced under U.S. Government contract 89233218CNA000001 for Los Alamos\nNational Laboratory (LANL), which is operated by Triad National Security, LLC for the U.S.\nDepartment of Energy/National Nuclear Security Administration. All rights in the program are\nreserved by Triad National Security, LLC, and the U.S. Department of Energy/National Nuclear\nSecurity Administration. The Government is granted for itself and others acting on its behalf a\nnonexclusive, paid-up, irrevocable worldwide license in this material to reproduce, prepare\nderivative works, distribute copies to the public, perform publicly and display publicly, and to permit\nothers to do so.\n\n## License:\n\nThis program is open source under the BSD-3 License.\nRedistribution and use in source and binary forms, with or without\nmodification, are permitted provided that the following conditions are met:\n\n1. Redistributions of source code must retain the above copyright notice, this\n   list of conditions and the following disclaimer.\n\n2. Redistributions in binary form must reproduce the above copyright notice,\n   this list of conditions and the following disclaimer in the documentation\n   and/or other materials provided with the distribution.\n\n3. Neither the name of the copyright holder nor the names of its\n   contributors may be used to endorse or promote products derived from\n   this software without specific prior written permission.\n\nTHIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\"\nAND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE\nIMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE\nDISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE\nFOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL\nDAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR\nSERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER\nCAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,\nOR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE\nOF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.\n\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flanl%2Fpydntnk","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flanl%2Fpydntnk","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flanl%2Fpydntnk/lists"}