{"id":38522938,"url":"https://github.com/flyconnectome/hnf","last_synced_at":"2026-01-17T06:43:37.351Z","repository":{"id":153215425,"uuid":"339088525","full_name":"flyconnectome/hnf","owner":"flyconnectome","description":"Documentation for the hierarchical neuron format","archived":false,"fork":false,"pushed_at":"2021-02-15T13:59:07.000Z","size":17,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":5,"default_branch":"main","last_synced_at":"2024-01-27T12:09:54.423Z","etag":null,"topics":["annotations","data","dotprops","hdf5","mesh","neurons","skeleton","storage"],"latest_commit_sha":null,"homepage":"","language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/flyconnectome.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2021-02-15T13:35:41.000Z","updated_at":"2021-02-15T14:00:57.000Z","dependencies_parsed_at":null,"dependency_job_id":"e4317cd3-2a1f-4680-9815-eaca49d500b3","html_url":"https://github.com/flyconnectome/hnf","commit_stats":{"total_commits":2,"total_committers":2,"mean_commits":1.0,"dds":0.5,"last_synced_commit":"b950fb12575773f7c8527096f0d5947d13978c0d"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/flyconnectome/hnf","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/flyconnectome%2Fhnf","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/flyconnectome%2Fhnf/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/flyconnectome%2Fhnf/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/flyconnectome%2Fhnf/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/flyconnectome","download_url":"https://codeload.github.com/flyconnectome/hnf/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/flyconnectome%2Fhnf/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28502819,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-17T04:31:57.058Z","status":"ssl_error","status_checked_at":"2026-01-17T04:31:45.816Z","response_time":85,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["annotations","data","dotprops","hdf5","mesh","neurons","skeleton","storage"],"created_at":"2026-01-17T06:43:37.295Z","updated_at":"2026-01-17T06:43:37.343Z","avatar_url":"https://github.com/flyconnectome.png","language":null,"funding_links":[],"categories":[],"sub_categories":[],"readme":"## Hierarchical Neuron Format\nThe Hierarchical Neuron Format (HNF) is a schema for storing neuron morphologies\nand meta data in Hdf5 files.\n\nWe provide read/write implementations for R and Python:\n\n- for R see [nat.hdf5](https://github.com/schlegelp/nat.hdf5)\n- for Python see [navis](https://navis.readthedocs.io/en/latest/)\n\n## Preamble\nThere are a few file formats that can store neuron morphology. To name but a few:\n- [SWC](http://www.neuronland.org/NLMorphologyConverter/MorphologyFormats/SWC/Spec.html)\n  for simple skeletons\n- [NeuroML](https://en.wikipedia.org/wiki/NeuroML) is an XML-based format\n  primarily used for modelling but can store compartment models (i.e. skeletons)\n  of neurons and meta data\n- [NWB](https://www.nwb.org) (neurodata without borders) is an HDF5-based format\n  focused on physiological data\n- [NRRD](http://teem.sourceforge.net/nrrd/format.html) files can be used to\n  store dotprops\n\n_Why then start a new format?_\n\nBecause none of the existing formats tick all the boxes! We need a file format\nthat can hold:\n\n1. thousands of neurons\n2. multiple representations (mesh, skeleton, dotprops) of a given neuron\n3. annotations (e.g. synapses) associated with each neuron\n4. meta data such as names, soma positions, etc.\n\nEnter HDF5: basically a filesystem-in-a-file. The important thing is that we\ndon't have to worry about how data is en-/decoded because other libraries\n(like `h5py` for Python or `hdf5r` for R) take care of that. All we have to\ndo is come up with a schema.\n\n### Schema\nHDF5 knows \"groups\" (=folders), \"datasets\" and \"attributes\". The basic idea for\nour schema is this:\n\n- the `root` contains info about the format as attributes\n- each group in `root` represents a neuron and the group's name is the neuron's ID\n- a neuron group holds and meta data, and the neuron's representations (mesh,\n  skeleton and/or dotprops) and annotations in separate sub-groups\n\nTo illustrate the basic principle:\n\n```\n.\n├── attrs: format-related meta data\n├── group: neuron1\n│   ├── attrs: neuron-related meta data\n│   ├── group: skeleton\n│   |    ├── attrs: skeleton-related meta data\n|   |    └── datasets: node table, etc\n│   ├── group: dotprops\n│   |    ├── attrs: dotprops-related meta data\n|   |    └── datasets: points, tangents, alpha, etc\n│   ├── group: mesh\n│   |    ├── attrs: mesh-related meta data\n|   |    └── datasets: vertices, faces, etc\n|   └── group: annotations\n|       └── group: e.g. connectors\n|           ├── attrs: connector-related meta data\n|           └── datasets: connector data\n├── group: neuron2\n|   ├── ...  \n...\n```\n\n#### Root attributes\n\nThe root meta data must contain two attributes:\n- `format_spec` specifies format and version\n- `format_url` points to a library or format specifications\n\n```\n.\n├── attr['format_spec']: str = 'hnf_v1'   \n├── attr['format_url']: str = 'https://github.com/schlegelp/navis'\n...\n```\n\n#### Neuron base groups\n\nEach neuron group contains properties that apply to all the neuron's potential\nrepresentations - for example a `neuron_name`. Note that if an attribute is\ndefined at the neuron level and again at a deeper level (i.e. the skeleton,\nmesh or dotprops), the more proximal attribute takes precedence for a given\nrepresentation.\n\n```\n.\n└── group['123456']  # note that numeric IDs will be \"stringified\"\n    ├── attr[\"neuron_name\"]: str = \"some name\"\n...\n```\n\n#### Skeletons  \n\nAttributes:\n- `units_nm` (float | int | tuple, optional): specifies the units in\n  nanometer space - can be a tuple of `(x, y, z)` if units are\n  non-isotropic  \n- `soma` (int, optional): the node ID of the soma  \n\nDatasets:\n- `node_id` (int): IDs for the nodes\n- `parent_id` (int): for each node, the ID of it's parent; nodes with\n  out parents (i.e. roots) have `parent_id` of `-1`\n- `x`, `y`, `z` (float | int): node coordinates\n- `radius` (float | int, optional): radius for each node\n\n```\n└── group['123456']\n    ├── attr['neuron_name'] = \"example neuron with a skeleton\"\n    ├── attr['units_nm'] = (4, 4, 40)\n    └── grp['skeleton']\n         ├── attr['soma']: 1\n         ├── ds['node_id']: (N, ) array\n         ├── ds['parent_id']: (N, ) array\n         ├── ds['x']: (N, ) array\n         ├── ds['y']: (N, ) array\n         ├── ds['z']: (N, ) array\n         └── ds['radius']: (N, ) array, optional\n```\n\n#### Meshes  \nMeshes are principally represented as vertices + triangular faces (`navis`\nis using trimesh under the hood).\n\nAttributes:\n- `units_nm` (float | int | tuple, optional): specifies the units in\n  nanometer space - can be a tuple of `(x, y, z)` if units are\n  non-isotropic  \n- `soma` (tuple, optional): tuple of `(x, y, z)` coordinates of the soma\n\nDatasets:\n- `vertices` (int | float): (N, 3) array of vertex positions\n- `faces` (int): (M, 3) array of vertex indices forming the faces (indices start\n  at 0)\n- `skeleton_map` (int, optional): (N, ) array mapping each vertex to a\n  node ID in the skeleton\n\n```\n└── group['4353421']\n    ├── attr['neuron_name'] = \"example neuron with a mesh\"\n    ├── attr['units_nm'] = (4, 4, 40)\n    └── grp['mesh']\n         ├── attr['soma']: (1242, 6533, 400)\n         ├── ds['vertices']: (N, 3) array\n         ├── ds['faces']: (M, 3) array\n         └── ds['skeleton_map']: (N, ) array, optional\n\n```\n\n#### Dotprops  \n\nAttributes:\n- `k` (int): number of k-nearest neighbours used to calculate the tangent\n  vectors from the point cloud\n- `units_nm` (float | int | tuple, optional): specifies the units in\n  nanometer space - can be a tuple of `(x, y, z)` if units are\n  non-isotropic  \n- `soma` (tuple, optional): tuple of `(x, y, z)` coordinates of the soma\n\nDatasets:\n- `points` (int | float): (N, 3) array of x/y/z positions\n- `vect` (int | float, optional): (N, 3) array of tangent vectors -    \n  generated if not provided\n- `alpha` (int | float, optional): (N, ) array of alpha values for each\n  point in ``points`` generated if not provided\n\n```\n└── group['65432']    \n    ├── attr['neuron_name'] = \"example neuron with dotprops\"    \n    └── grp['dotprops']\n        ├── attr['k'] = 5\n        ├── attr['units_nm'] = (4, 4, 40)\n        ├── attr['soma']: (1242, 6533, 400)\n        ├── ds['points']: (N, 3) array\n        ├── ds['vect']: (N, 3) array\n        └── ds['alpha']: (N, ) array\n```\n\n#### Annotations\nAnnotations are meant to be flexible and are principally parsed into\npandas DataFrames. Because they won't follow a common format, it is\ngood practice to leave some (optional) meta data pointing to columns\ncontaining data relevant for e.g. plotting:\n\nAttributes:\n- `point_col` (str | list thereof): pointer to the column(s) containing\n   x/y/z positions\n- `type_col` (str): pointer to a column specifying types\n- `skeleton_map` (str): pointer to a column associating the row with\n   a node ID in the skeleton\n\nLet's illustrate this with a mock synapse table:\n\n```\n└── group['32434566']\n    ├── attr['neuron_name'] = \"example neuron with synapse annotations\"\n    ├── attr['units_nm'] = 1\n    └── grp['annotations']\n         └── grp['synapses']\n             ├── attr['points']: ['x', 'y', 'z']\n             ├── attr['types']: 'prepost'\n             ├── attr['skeleton_map']: 'node_id'\n             ├── ds['x']: (N, ) array\n             ├── ds['x']: (N, ) array\n             ├── ds['z']: (N, ) array\n             ├── ds['prepost']: (N, ) array of [0, 1, 2, 3, 4]\n             └── ds['node_id']: (N, )\n```\n\n#### \"Hidden\" attributes \u0026 datasets\n\nIt can be useful to have attributes and datasets that contain information that's\nonly pertinent for the reader/writer but does not directly relate to the neuron.\n\nFor this, we prefix the attribute/dataset with a `.`:\n\n```\n└── group['4353421']\n    ├── attr['neuron_name'] = \"example neuron with a mesh\"\n    ├── attr['units_nm'] = (4, 4, 40)\n    ├── attr['.hidden_attribute'] = \"typically ignored when reading\"\n    └── grp['mesh']\n         ├── attr['soma']: (1242, 6533, 400)\n```\n\nWe use hidden attributes to e.g. store a serialized version of a neuron instead/\nin addition to the raw data to speed up reading the data.\n\n### A final remark\nThe above schema describes a \"minimal\" layout - i.e. we expect no less\ndata than that. However, e.g. the `navis` implementations for reading/writing\nthe schema are flexible: you can add more attributes or datasets\nand `navis` will by default try to read and attach them to the neuron.\n\n### Is this stable?\nIsh? The format is versioned and I will maintain readers/writers for\npast versions in ``navis``. In other good news: the HDF5 backend is\nstable - so even if `navis` acts up when parsing your file, you can\nalways read it manually using `h5py`.\n\n### Changelog\n\nThe current version of the format is 1.0.\n\nChanges:\n- 2021/02/01: Version 1.0\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fflyconnectome%2Fhnf","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fflyconnectome%2Fhnf","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fflyconnectome%2Fhnf/lists"}