{"id":18084986,"url":"https://github.com/jwodder/gamdam-rust","last_synced_at":"2025-03-28T23:32:46.788Z","repository":{"id":61997132,"uuid":"556924635","full_name":"jwodder/gamdam-rust","owner":"jwodder","description":"Git-Annex Mass Downloader and Metadata-er (in Rust!)","archived":true,"fork":false,"pushed_at":"2024-11-18T13:40:08.000Z","size":316,"stargazers_count":2,"open_issues_count":6,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-03-21T00:20:38.468Z","etag":null,"topics":["async","download","git-annex","rust","tokio"],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jwodder.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-10-24T19:21:16.000Z","updated_at":"2024-12-20T20:24:45.000Z","dependencies_parsed_at":"2023-02-16T03:16:06.033Z","dependency_job_id":"0a9affbe-68d2-4f0d-ad70-d752e7f4e676","html_url":"https://github.com/jwodder/gamdam-rust","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jwodder%2Fgamdam-rust","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jwodder%2Fgamdam-rust/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jwodder%2Fgamdam-rust/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jwodder%2Fgamdam-rust/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jwodder","download_url":"https://codeload.github.com/jwodder/gamdam-rust/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246117586,"owners_count":20726066,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["async","download","git-annex","rust","tokio"],"created_at":"2024-10-31T15:08:56.163Z","updated_at":"2025-03-28T23:32:46.522Z","avatar_url":"https://github.com/jwodder.png","language":"Rust","readme":"[![Project Status: Concept – Minimal or no implementation has been done yet, or the repository is only intended to be a limited example, demo, or proof-of-concept.](https://www.repostatus.org/badges/latest/concept.svg)](https://www.repostatus.org/#concept)\n[![CI Status](https://github.com/jwodder/gamdam-rust/actions/workflows/test.yml/badge.svg)](https://github.com/jwodder/gamdam-rust/actions/workflows/test.yml)\n[![codecov.io](https://codecov.io/gh/jwodder/gamdam-rust/branch/master/graph/badge.svg)](https://codecov.io/gh/jwodder/gamdam-rust)\n[![MIT License](https://img.shields.io/github/license/jwodder/gamdam-rust.svg)](https://opensource.org/licenses/MIT)\n\n`gamdam` is the Git-Annex Mass Downloader and Metadata-er (in Rust!).  It takes\na stream of JSON Lines describing what files to download and what metadata each\nfile has, downloads them in parallel to a\n[git-annex](https://git-annex.branchable.com) repository, attaches the metadata\nusing git-annex's metadata facilities, and commits the results.\n\n`gamdam` requires `git-annex` v10.20220222 or higher to be installed separately\nin order to run.\n\n\nUsage\n=====\n\n    gamdam [\u003coptions\u003e] [\u003cinput-file\u003e]\n\n`gamdam` reads a series of JSON entries from a file (or from standard input if\nno file is specified) following the [input format](#input-format) described\nbelow.  It feeds the URLs and output paths to `git-annex addurl`, and once each\nfile has finished downloading, it attaches any listed metadata and extra URLs\nusing `git-annex metadata` and `git-annex registerurl`, respectively.\n\nNote that the latter step can only be performed on files tracked by git-annex;\nif you, say, have configured git-annex to not track text files, then any text\nfiles downloaded will not have any metadata or alternative URLs registered.\n\nOptions\n-------\n\n- `--addurl-opts \u003cOPTIONS\u003e` — Extra options to pass to the `git-annex addurl`\n  command.  Multiple options \u0026 arguments need to be quoted as a single string,\n  which must also use proper shell quoting internally, e.g.,\n  `--addurl-opts=\"--user-agent 'gamdam via git-annex'\"`.\n\n- `-C \u003cDIR\u003e`, `--chdir \u003cDIR\u003e` — The directory in which to download files;\n  defaults to the current directory.  If the directory does not exist, it will\n  be created.  If the directory does not belong to a Git or git-annex\n  repository, it will be initialized as one.\n\n- `-F \u003cFILE\u003e`, `--failures FILE` — If any files fail to download or fail to\n  have their metadata/URLs set, write their input records back out to `FILE`.\n\n- `-J \u003cINT\u003e`, `--jobs \u003cINT\u003e` — Number of parallel jobs for `git-annex addurl`\n  to use; by default, the process is instructed to use one job per CPU core.\n\n- `-l \u003cLEVEL\u003e`, `--log-level \u003cLEVEL\u003e` — Set the log level to the given value.\n  Possible values are \"`OFF`\", \"`ERROR`\", \"`WARN`\", \"`INFO`\", \"`DEBUG`\", and\n  \"`TRACE`\" (all case-insensitive) [default: `INFO`]\n\n- `-m \u003cTEXT\u003e`, `--message \u003cTEXT\u003e` — The commit message to use when saving.\n  This may contain a `{downloaded}` placeholder which will be replaced with the\n  number of files successfully downloaded.\n\n- `--no-save-on-fail` — Don't commit the downloaded files if any files failed\n  to download\n\n- `--save`, `--no-save` — Whether to commit the downloaded files once they've\n  all been downloaded  [default: `--save`]\n\n\nInput Format\n------------\n\nInput is a series of JSON objects, one per line (a.k.a. \"JSON Lines\").  Each\nobject has the following fields:\n\n- `url` — *(required)* A URL to download\n\n- `path` — *(required)* A relative path where the contents of the URL should be\n  saved.  If an entry with a given path is encountered while another entry with\n  the same path is being downloaded, the later entry is discarded, and a\n  warning is emitted.\n\n  If a file already exists at a given path, `git-annex` will try to register\n  the URL as an additional location for the file, failing if the resource at\n  the URL is not the same size as the extant file.\n\n  Paths must be relative to the directory specified with `--chdir`, cannot\n  contain \"`..`\" as a path component, cannot end with a path separator, and\n  cannot be empty or contain only the path component \"`.`\".  Forward slashes\n  (`/`) are accepted as path separators on all platforms, while backslashes are\n  only treated as path separators on Windows.\n\n- `metadata` — A collection of metadata in the form used by `git-annex\n  metadata --json`, i.e., a mapping of key names to lists of string values.\n\n- `extra_urls` — A list of alternative URLs for the resource, to be attached to\n  the downloaded file with `git-annex registerurl`.\n\nIf a given input line is invalid, it is discarded, and a warning message is\nemitted.\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjwodder%2Fgamdam-rust","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjwodder%2Fgamdam-rust","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjwodder%2Fgamdam-rust/lists"}