{"id":19721998,"url":"https://github.com/datajoint/djarchive-client","last_synced_at":"2025-10-11T03:05:18.675Z","repository":{"id":45329393,"uuid":"356339338","full_name":"datajoint/djarchive-client","owner":"datajoint","description":"DataJoint Archive Client","archived":false,"fork":false,"pushed_at":"2021-12-20T20:21:08.000Z","size":2600,"stargazers_count":0,"open_issues_count":2,"forks_count":8,"subscribers_count":9,"default_branch":"master","last_synced_at":"2025-10-11T03:04:44.961Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/datajoint.png","metadata":{"files":{"readme":"README","changelog":null,"contributing":null,"funding":null,"license":"COPYING","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-04-09T16:47:03.000Z","updated_at":"2021-12-20T20:21:11.000Z","dependencies_parsed_at":"2022-09-17T08:11:49.426Z","dependency_job_id":null,"html_url":"https://github.com/datajoint/djarchive-client","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/datajoint/djarchive-client","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/datajoint%2Fdjarchive-client","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/datajoint%2Fdjarchive-client/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/datajoint%2Fdjarchive-client/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/datajoint%2Fdjarchive-client/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/datajoint","download_url":"https://codeload.github.com/datajoint/djarchive-client/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/datajoint%2Fdjarchive-client/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":279006041,"owners_count":26084024,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-11T02:00:06.511Z","response_time":55,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-11T23:16:11.012Z","updated_at":"2025-10-11T03:05:18.635Z","avatar_url":"https://github.com/datajoint.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":".. -*- mode: rst -*-\n\n================\ndjarchive-client\n================\n\nThis is 'djarchive-client', a library for interfacing with DataJoint Neuro's\ndata publication service.\n\nStatus: WIP; currently for internal/prototyping use.\n\nInstallation \u0026 Setup\n====================\n\nInstallation \u0026 Setup instructions are as follows:\n\n  1) Install the package:\n\n     ``$ pip install .``\n\n  2) If applicable, configure datajoint with appropriate dj.config['custom'] \n     values:\n\n      Admin usage expects dj.config['custom'] values for:\n\n        - djarchive.access_key\n        - djarchive.secret_key\n\n      Client and admin usage allow overriding dj.config['custom']\n      defaults for:\n\n        - djarchive.bucket\n        - djarchive.endpoint\n\n      This should only be needed for development or custom server\n      configurations.\n\nUsage via 'djarchive' utility script\n====================================\n\nThe 'djarchive' utility script can be used for CLI interactions with the\ndata archive. Usage synopsys is described in the following subsections.\n\nBrowsing \u0026 Retrieving Datasets\n------------------------------\n\nThe following example shows a minimal usage synopsis to navigate the available\ndatasets and to download a single dataset.\n\n.. code-block:: sh\n\n  $ djarchive datasets # list datasets\n  some-set\n  $ djarchive revisions # list all datasets \u0026 revisions\n  some-set,000\n  other-set,001\n  $ djarchive revisions some-set # list revisions for dataset 'some-set'\n  some-set,000\n  $ djarchive download some-set 000 ./some-set-000 # download some-set rev. 000\n\nFile Integrity, Download Caching \u0026 Error Handling\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nThe djarchive client provides integrity protection in the form of a\nfile manifest, called 'djarchive-manifest.csv', which is a CSV list\nof file sizes, sha1 hashes, and file subpaths for a given dataset.\n\nBy default, files having path components beginning with '.' are not\nconsidered for inclusion within archives. This behavior can be overridden\nby setting the dj.config.filename_filter property to a regular expression\ncontaining the desired filtering pattern, or the special string \"''\",\nwhich is treated as matching the empty string ('^$') and so should never\nmatch a file.\n\nRetrieval of datasets begins with the download of this manifest, which\nis then used to direct the remaining download and to verify download\nactions complete successfullly.\n\nTo allow for resuming of partially downloaded datasets, local files in the\ntarget directory of the download are compared against the manifest. If a\nlocal file matches the manifest contents, it is not re-retrieved, and if\na local file does not match the manifest contents, it is re-retrieved\nwithout confirmation.\n\nUploading Datasets\n------------------\n\nDatasets can be uploaded using the djarchive client using the 'upload'\nfunctionality. In short:\n\n.. code-block:: sh\n\n  $ djarchive upload my-dataset 1 /path/to/my-dataset\n\nIf a djarchive manifest is not present in /path/to/my-dataset, it will\nbe generated as part of the upload process, with the manifest uploaded\nas the last step to indicate that the dataset is complete.\n\nIf a djarchive manifest is present in /path/to/my-dataset, files will be\nchecked against the manifest as part of the upload process, and an error\nwill be signalled if files exist in the dataset folder which are not\ntracked in the manifest, or if files do not match the expected manifest\nvalues.\n\nThe djarchive manifest file can also be generated as a separate step via\nthe 'manifest' command:\n\n.. code-block:: sh\n\n  $ djarchive manifest /path/to/my-dataset\n\nUsage via 'djarchive_client' python pacakge\n===========================================\n\nThe djarchive_client python package contains the logic for interacting\nwith the archive. Best reference is the code/docstrings; minimal usage\nexample is as follows:\n\n.. code-block:: python\n\n  \u003e\u003e\u003e from djarchive_client import client\n  \u003e\u003e\u003e c = client()\n  \u003e\u003e\u003e c.datasets()\n  \u003cgenerator object DJArchiveClient.datasets at 0x7faf65ec4ba0\u003e\n  \u003e\u003e\u003e list(c.datasets())\n  ['some-set']\n  \u003e\u003e\u003e c.revisions()\n  \u003cgenerator object DJArchiveClient.revisions at 0x7faf65ec4ba0\u003e\n  \u003e\u003e\u003e list(c.revisions())\n  [('some-set', '000'), ('other-set', '001')]\n  \u003e\u003e\u003e list(c.revisions('some-set'))\n  [('some-set', '000')]\n  \u003e\u003e\u003e c.download('some-set', '000', './some-set-000', create_target=True)\n\nLogging \u0026 Output\n================\n\nFunctions in the djarchive_client library and the djarchive utility\ncurrently do not produce output in the normal case.  In some cases,\nsuch as for viewing per-file download status messages, more detailed\noutput may be desired.\n\nIn the script case, setting ``dj.config['loglevel']`` or the environment\nvariable ``DJARCHIVE_LOGLEVEl`` to ``DEBUG`` will increase logging\noutput. Additionally, output can be logged to a file by setting\n``dj.config['custom']['logfile']`` to a string containing the path to a\ndesired logfile.\n\nFor library usage, setting the python logging for the ``djarchive_client``\nmodule to ``logging.DEBUG`` will enable more verbose output. See the function\n``logsetup`` in ``scripts/djarchive`` for more details.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdatajoint%2Fdjarchive-client","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdatajoint%2Fdjarchive-client","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdatajoint%2Fdjarchive-client/lists"}