{"id":20956405,"url":"https://github.com/bluebrain/bbp-atlas-data-fetch","last_synced_at":"2026-02-08T07:31:13.429Z","repository":{"id":262800399,"uuid":"854582219","full_name":"BlueBrain/bbp-atlas-data-fetch","owner":"BlueBrain","description":"CLI to fetch datasets from Nexus","archived":false,"fork":false,"pushed_at":"2024-11-14T10:30:07.000Z","size":89,"stargazers_count":1,"open_issues_count":0,"forks_count":1,"subscribers_count":4,"default_branch":"develop","last_synced_at":"2025-03-29T20:04:37.044Z","etag":null,"topics":["atlas","database"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/BlueBrain.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.rst","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":"AUTHORS.txt","dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-09-09T12:37:41.000Z","updated_at":"2024-11-14T10:30:25.000Z","dependencies_parsed_at":"2024-11-14T11:25:57.467Z","dependency_job_id":"948348a4-71a2-4292-ae1f-e2e441d58f05","html_url":"https://github.com/BlueBrain/bbp-atlas-data-fetch","commit_stats":null,"previous_names":["bluebrain/bbp-atlas-data-fetch"],"tags_count":8,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BlueBrain%2Fbbp-atlas-data-fetch","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BlueBrain%2Fbbp-atlas-data-fetch/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BlueBrain%2Fbbp-atlas-data-fetch/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BlueBrain%2Fbbp-atlas-data-fetch/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/BlueBrain","download_url":"https://codeload.github.com/BlueBrain/bbp-atlas-data-fetch/tar.gz/refs/heads/develop","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250343865,"owners_count":21415035,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["atlas","database"],"created_at":"2024-11-19T01:25:51.248Z","updated_at":"2026-02-08T07:31:13.401Z","avatar_url":"https://github.com/BlueBrain.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"## Description\nThis module is a (Python) CLI in charge of fetching datasets from Nexus, one file (or \npayload) at the time. It can fetch payloads and save them as JSON files, or it can fetch \nbinaries (_distributions_) linked to resources.\n\nThere is mainly two ways of fetching a piece of data:\n- using the @id of a resource\n- using filters on the resource properties to narrow down the selection and eventually \nfind the relevant dataset\n\nWhen using filters (see below), a SPARQL query is dynamically generated, allowing graph \ntraversal.\n\n## Install\n```\npip install \"bba-data-fetch\"\n```\nFrom now on, the executable **bba-data-fetch** is in your PATH.\n\n## Usage\n### Inputs\nThere is no input apart from configuration and filter/id. See the [CLI arguments](#cli-arguments)\nsection for more info.\n\n### Outputs\nThis CLI writes on disc one of the two depending on the arguments provided:\n- a JSON file that corresponds to the targeted resource payload,\n- a copy of the distribution file linked in the targeted resource (can be any kind of file).\n\n### CLI arguments\n- **--version** - [flag] Display the version\n- **--help** - [flag] Display help\n- **--verbose** - [flag] Enables verbose mode to print the generated SPARQL query and response\n- **--nexus-token-file /path/to/token.txt** - [single string] The path to the text file that contains the Nexus token. Mandatory\n- **--nexus-env https://bbp.epfl.ch/nexus/v1/** - [single string] The URL to the Nexus environment. Mandatory\n- **--nexus-org bbp** - [single string] The name of the Nexu organization to look for a resource. Mandatory\n- **--nexus-id some_id_probably_uuid** - [single string] The @id of the Nexus resource to fetch. Optional, but necessary if **--filter** is not provided\n- **--payload** - [flag] Fetch the payload as a JSON file. Optional, the default behavior is to fetch the file linked by the _distribution.contentUrl_ property.\n- **--favor - [multiple string] Payload properties and values with the format \u003c'properties:value'\u003e (ex: 'name:1.json') which will be used to determine which file to choose when retrieving a distribution from a resource with multiple distributions. Optional\n- **--out /some/file.json** - [single string] Path to the output file to create. The extension has to be .json if the flag --payload is provided. Otherwise, the extension must be the same as the distant file. Mandatory\n- **--keep-meta** - [flag] if --payload is provided, the JSON file will not contain the Nexus/JSON-LD system properties. If this flag is provided, the system metadata are kept\n- **--rev n** - [number] The revision argument is mainly to be used along **--nexus-id** to fetch a specific revision of a given resource. Optional, fetches the last revision if not provided\n- **--tag some_tag** - [single string] The tag argument is mainly to be used along **--nexus-id** to fetch a specific tag of a given resource. Optional\n- **--filter prop1=1 prop2=20** - [multiple strings] Filters are to be used instead of --nexus-id if the @id is not known. Filters are applied on properties and can work with graph traversal. Optional but necessary of **--nexus-id** is not provided\n\n### Filters\nThe **--filter** argument is powerful and deserves its own paragraph.  \nUsing filters keeps only the resources that match the conditions. Each filter takes this\nshape:\n\nproperty name [ operator ] value to compare with\n\nWhere:\n- **property name** can be a direct root level property (eg. _name_), a subproperty (eg. _resolution.value_) or even a graph traversal property (eg. _atlasRelease.name_ when the _atlasRelease_ property is actually just an _@id_ to another resource that has a _name_ property).\n- **value** can be a number or a string\n- **operator** can be one of the following:\n  - **=** strictly equal for number and case-insensitive equal for strings\n  - **~=** contains, only for strings\n  - **!=** different, for numbers or does-not-contain for strings\n  - **\u003e=** greater than or equal to, for numbers only\n  - **\u003c=** lower than or equal to, for numbers only\n  - **\u003e** greater than, for numbers only\n  - **\u003c** lower than, for numbers only\n\nExample: _resolution.value=25_\n\nThen, multiple filters of this kind can be used, space separated, after a single **--filter** flag.\n\n⚠️ Warning: since this is used in a terminal, the symbol \"\u003e\" is natively made to redirect the output to a file. Hence, filters that use the operators \u003e or \u003e= must have their whole expression famed with double or simple quotes.\n\n⚠️ Warning 2: Like most CLI, multiword strings must also be framed in double or simple quotes.\n\nExample: --filter _\"resolution.value\u003e=25\" name=\"hello world\" \"another.prop=bip boop\"_\n\nℹ️ Info: property names such as resolution or name should not be preceded by a context prefix. For example: \"nsg:resolution.schema:value\" is not valid.\n\nℹ️ Info: if a property is a **@list**, then we can address an element from the list using square brackets. Example: _--filter dimension[0].name=intensity worldMatrix[0]=10_\n\nUnder the hood, this is using _rdf:first_ and _rdf:rest_.\n\n### Examples\n- Fetch a resource payload from its `@id`:\n```\nbba-data-fetch --nexus-env https://bbp.epfl.ch/nexus/v1/ \\\n               --nexus-token-file ./token.txt \\\n               --nexus-org bbp \\\n               --nexus-proj atlas \\\n               --payload \\                                       # \u003c-- to fetch the payload !\n               --out ./tmp/some_payload.json \\                   # \u003c-- needs a .json extension\n               --nexus-id 7f85cd66-d212-4799-bb4c-0732b8534442\n```\n\n- Fetch the distribution file linked a resource, from resource `@id`:\n```\nbba-data-fetch --nexus-env https://bbp.epfl.ch/nexus/v1/ \\\n               --nexus-token-file ./token.txt \\\n               --nexus-org bbp \\\n               --nexus-proj atlas \\\n               --out ./tmp/some_distribution.nrrd \\              # \u003c-- extension must be the same as distant file\n               --nexus-id 7f85cd66-d212-4799-bb4c-0732b8534442\n```\n\nFetch the distribution file corresponding to the favor argument among multiple distributions from a resource:\n```\nbba-data-fetch --nexus-env https://bbp.epfl.ch/nexus/v1/ \\\n               --nexus-token-file ./token.txt \\\n               --nexus-org bbp \\\n               --nexus-proj atlas \\\n               --out ./tmp/some_payload.json \\\n               --nexus-id http://bbp.epfl.ch/neurosciencegraph/ontologies/mba\\\n               --favor \"encodingFormat:application/ld+json\" \\\n               --verbose \\\n```\n\n- Fetch a resource payload properties (dynamic SPARQL query building):\n```\nbba-data-fetch --nexus-env https://bbp.epfl.ch/nexus/v1/ \\\n               --nexus-token-file ./token.txt \\\n               --nexus-org bbp \\\n               --nexus-proj atlas \\\n               --payload \\                                       # \u003c-- to fetch the payload !\n               --out ./tmp/some_payload.json \\                   # \u003c-- needs a .json extension\n               --filter \\\n                   type=BrainParcellationDataLayer \\\n                   resolution.value=10 \\\n                   atlasRelease.name=\"Allen Mouse CCF v2\" \\\n```\n\n- Some comparison operator don't play well with shell script and need to be framed with quote signs:\n```\nbba-data-fetch --nexus-env https://bbp.epfl.ch/nexus/v1/ \\\n               --nexus-token-file ./token.txt \\\n               --nexus-org bbp \\\n               --nexus-proj atlas \\\n               --out ./tmp/some_distribution.nrrd \\\n               --filter \\\n                   type=BrainParcellationDataLayer \\\n                   \"resolution.value\u003e=10\" \\                # \u003c-- if not framed with \"...\", the \u003e symbol redirects the output\n                  atlasRelease.name=\"Allen Mouse CCF v2\" \\\n```\n\n- Case insensitive for property and type names:\n```\nbba-data-fetch --nexus-env https://bbp.epfl.ch/nexus/v1/ \\\n               --nexus-token-file ./token.txt \\\n               --nexus-org bbp \\\n               --nexus-proj atlas \\\n               --out ./tmp/some_payload.json \\\n               --payload \\\n               --verbose \\\n               --filter \\\n                   type=BrainParcellationDataLayer \\\n                   resOlution.value=10 \\                      # \u003c-- wrong case spelling, still works!\n                  Bufferencoding=gzip \\                      # \u003c-- wrong case spelling, still works!\n                  atlasRelease.name=\"Allen Mouse CCF v2\" \\\n```\n\n- Fetch the point cloud file linked to the brain region _`mba:1048`_:\n```\nbba-data-fetch --nexus-env https://bbp.epfl.ch/nexus/v1/ \\\n               --nexus-token-file ./token.txt \\\n               --nexus-org bbp \\\n               --nexus-proj atlas \\\n               --out ./tmp/some_payload.raw \\\n               --verbose \\\n               --filter \\\n                   type=CellPositions \\\n                   brainLocation.brainRegion=\"mba:1048\" \\\n```\n\nNote that _mba:1048_ is the id of a brain region (Allen CCF 1048 is the \"gigantocellular\nreticular nucleus\") preceded by _mba_, which is the prefix in the graph database \n(_mba_ = mouse brain atlas). This means that prefixes can be used in values if need be.\n\n\n## Funding \u0026 Acknowledgment\nThe development of this software was supported by funding to the Blue Brain Project, a \nresearch center of the École polytechnique fédérale de Lausanne (EPFL), from the Swiss \ngovernment’s ETH Board of the Swiss Federal Institutes of Technology.\n\nCopyright © 2020-2024 Blue Brain Project/EPFL\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbluebrain%2Fbbp-atlas-data-fetch","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbluebrain%2Fbbp-atlas-data-fetch","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbluebrain%2Fbbp-atlas-data-fetch/lists"}