{"id":17543090,"url":"https://github.com/iraikov/neuroh5","last_synced_at":"2025-07-28T18:32:08.140Z","repository":{"id":59427718,"uuid":"212163032","full_name":"iraikov/neuroh5","owner":"iraikov","description":"An HDF5-based library for parallel I/O operations on data structures for large-scale neural networks","archived":false,"fork":false,"pushed_at":"2024-08-28T21:19:37.000Z","size":20216,"stargazers_count":5,"open_issues_count":0,"forks_count":4,"subscribers_count":4,"default_branch":"master","last_synced_at":"2024-10-28T03:01:34.999Z","etag":null,"topics":["distributed-computing","hdf5","mpi","mpi-applications","mpi-io","neuron","neuronal-network","parallel-io"],"latest_commit_sha":null,"homepage":null,"language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/iraikov.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-10-01T17:55:25.000Z","updated_at":"2024-08-28T21:19:40.000Z","dependencies_parsed_at":"2023-02-09T21:15:53.677Z","dependency_job_id":"bf2ec08d-e07c-4507-b45a-003a99449597","html_url":"https://github.com/iraikov/neuroh5","commit_stats":null,"previous_names":[],"tags_count":9,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/iraikov%2Fneuroh5","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/iraikov%2Fneuroh5/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/iraikov%2Fneuroh5/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/iraikov%2Fneuroh5/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/iraikov","download_url":"https://codeload.github.com/iraikov/neuroh5/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":227944069,"owners_count":17845139,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["distributed-computing","hdf5","mpi","mpi-applications","mpi-io","neuron","neuronal-network","parallel-io"],"created_at":"2024-10-21T00:14:25.902Z","updated_at":"2025-07-28T18:32:08.126Z","avatar_url":"https://github.com/iraikov.png","language":"C++","funding_links":[],"categories":[],"sub_categories":[],"readme":"# neuroh5\n\nA parallel HDF5-based library for storage and processing of large-scale graphs and neural cell model attributes.\n\n## Introduction\n\nThe neuroh5 library implements an HDF5-based format for storing\nneuronal morphology information, synaptic and connectivity information\nof large neural networks, and perform parallel graph partitioning and\nanalysis.\n\nneuroh5 assumes that synaptic connectivity between neurons in\nneuronal network models is represented as directed graphs stored as\nadjacency lists, where the vertices represent the neurons in the\nnetwork and are identified by unsigned integers called unique global\nidentifiers (gid). \n\n## Installation\n\nBuilding and installing NeuroH5 \n\nNeuroH5 requires parallel HDF5, MPI, cmake. The Python module requires Python 3 and numpy.\n\nTo build the NeuroH5 C++ library and applications:\n\n```\ngit clone https://github.com/iraikov/neuroh5.git\ncd neuroh5\ncmake .\nmake \n```\n\n\nTo build the python module:\n\n```\ngit clone https://github.com/iraikov/neuroh5.git\ncd neuroh5\nCMAKE_BUILD_PARALLEL_LEVEL=8 \\\n  CMAKE_MPI_C_COMPILER=$(which mpicc) \\\n  CMAKE_MPI_CXX_COMPILER=$(which mpicxx) \\\n  pip install .\n```\n\n\n## Basic concepts and terminology\n\nConnectivity between neurons is described in terms of vertices and\ndirected edges, where each vertex has an integer id associated with\nit, which corresponds to the id of the respective neuron in the\nnetwork simulation. Each edge is identified by source and destination\nvertex, and a number of additional attributes, such as distance and\nsynaptic weight. In addition, the vertices are organized in\npopulations of neurons, where each population is comprised of the set\nof neurons that belong to the same biological type of neuron (such as\ngranule cell or basket cell). Connectivity is organized in\nprojections, where a projection is the set of connections between two\npopulations, or withing the same population. A projection is\nidentified by its source and destination populations, and all edges\nbetween those two populations.\n\n## Graph representation\n\nAn adjacency matrix is a square matrix where the elements of the\nmatrix indicate whether pairs of vertices are connected or not in the\ngraph. A sparse adjacency matrix representation explicitly stores only\nthe source or only the destination vertices, and uses range data\nstructures to indicate which vertices are connected. For example, in\none type of sparse format the source dataset will contain only how\nmany destination vertices are associated with a given source, but will\nnot explicitly store the source vertex ids.\n\nOur initial implementation of an HDF5-based graph representation\nincludes two types of representation. The first one is a direct\nedge-list-type representation where source and destination indices are\nexplicitly represented as HDF5 datasets. The second one is what we\nrefer to as Destination Block Sparse format, which is a type of sparse\nadjacency matrix representation where the connectivity is additionally\ndivided into blocks of contiguous indices to account for potential\ngaps in connectivity (i.e. ranges of indices that are not\nconnected). In the next section, we present the Destination Block\nSparse format, and present details of the implementation and initial\nperformance metrics.\n\n## Destination Block Sparse (DBS) Format\n\nThe Destination Block Sparse (DBS) format is a memory-efficient graph\nrepresentation designed for parallel processing of large-scale neural\nnetwork connectivity. This format optimizes for the common case where\neach destination (target neuron) connects to a relatively small subset\nof sources (input neurons).\n\n### Core Data Structures\n\nThe DBS format consists of four primary arrays:\n\n1. **Source Index Array (`src_idx`)**: \n   - Contains the indices of all source vertices in the projection\n   - Length equals the total number of edges (connections) in the projection\n   - Stores the actual connectivity information\n\n2. **Destination Block Pointer Array (`dst_blk_ptr`)**: \n   - Contains offsets into the Destination Pointer array\n   - Length equals the number of blocks plus one (includes a sentinel value)\n   - The difference between consecutive elements indicates the number of destinations in each block\n\n3. **Destination Index Array (`dst_idx`)**: \n   - Contains the first destination index in each block\n   - Length equals the number of blocks\n   - Destinations within a block have contiguous indices\n\n4. **Destination Pointer Array (`dst_ptr`)**: \n   - Contains offsets into the Source Index array\n   - Length equals the total number of destinations plus one (includes a sentinel value)\n   - Indicates where each destination's source connections begin and end\n\n### Key Properties and Relationships\n\n- **Block Structure**: Destinations are organized into blocks where each block contains contiguous destination indices\n- **Variable Block Size**: The number of destinations per block can vary\n- **Contiguous Destinations**: All destinations within a block have contiguous indices\n- **Efficient Edge Lookup**: To find all sources connected to a specific destination:\n   1. Locate the block containing the destination\n   2. Calculate the destination's offset within the block\n   3. Use the offset to find the appropriate pointers in `dst_ptr`\n   4. Access the source indices from `src_idx`\n\n### Formal Relationships\n\nFor a given block index `i`:\n- Number of destinations in block `i` = `dst_blk_ptr[i+1] - dst_blk_ptr[i]`\n- Destination index of the j-th destination in block `i` = `dst_idx[i] + j`\n- For the j-th destination in block `i`:\n  - Offset into `dst_ptr` = `dst_blk_ptr[i] + j`\n  - Source index range starts at `src_idx[dst_ptr[dst_blk_ptr[i] + j]]`\n  - Source index range ends at `src_idx[dst_ptr[dst_blk_ptr[i] + j + 1] - 1]`\n  - Number of sources = `dst_ptr[dst_blk_ptr[i] + j + 1] - dst_ptr[dst_blk_ptr[i] + j]`\n\n## Benefits for Parallel Processing\n\nThis format is particularly well-suited for parallel processing because:\n1. It clusters related destinations into blocks, improving cache locality\n2. It allows for balanced distribution of computational load across processors\n3. It minimizes communication overhead when distributing graph data\n4. It provides efficient access patterns for both forward and backward traversals\n\nThe format achieves memory efficiency by using index arrays and offset\npointers rather than storing a full adjacency matrix, making it ideal\nfor sparse connectivity patterns typical in neural networks.\n\n## Edge Attributes\n\nSeveral datasets are defined that hold the non-zero edge attributes of\na projection. Each edge attribute dataset is of the same length as the\nSource Index datasets (i.e. the number of edges in the projection).\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Firaikov%2Fneuroh5","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Firaikov%2Fneuroh5","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Firaikov%2Fneuroh5/lists"}