{"id":46396223,"url":"https://github.com/minerva-archive/worker","last_synced_at":"2026-03-09T14:01:28.583Z","repository":{"id":341712327,"uuid":"1171112740","full_name":"minerva-archive/worker","owner":"minerva-archive","description":"Distributed Minerva Worker","archived":false,"fork":false,"pushed_at":"2026-03-07T10:40:45.000Z","size":15804,"stargazers_count":62,"open_issues_count":2,"forks_count":12,"subscribers_count":1,"default_branch":"master","last_synced_at":"2026-03-07T17:22:36.172Z","etag":null,"topics":["archive","distributed","downloader","erista","myrient","preservation","worker"],"latest_commit_sha":null,"homepage":"https://minerva-archive.org","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"cc0-1.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/minerva-archive.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-03-02T22:14:20.000Z","updated_at":"2026-03-07T12:10:05.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/minerva-archive/worker","commit_stats":null,"previous_names":["rlaphoenix/minerva","minerva-archive/worker"],"tags_count":10,"template":false,"template_full_name":null,"purl":"pkg:github/minerva-archive/worker","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/minerva-archive%2Fworker","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/minerva-archive%2Fworker/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/minerva-archive%2Fworker/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/minerva-archive%2Fworker/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/minerva-archive","download_url":"https://codeload.github.com/minerva-archive/worker/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/minerva-archive%2Fworker/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":30297781,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-09T13:46:43.843Z","status":"ssl_error","status_checked_at":"2026-03-09T13:46:42.821Z","response_time":61,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["archive","distributed","downloader","erista","myrient","preservation","worker"],"created_at":"2026-03-05T10:00:23.236Z","updated_at":"2026-03-09T14:01:28.485Z","avatar_url":"https://github.com/minerva-archive.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cp align=\"center\"\u003e\n  \u003ca href=\"https://github.com/minerva-archive/worker\"\u003eMinerva Worker\u003c/a\u003e\n  \u003cbr/\u003e\n  \u003csup\u003e\u003cem\u003ePreserving Myrient's legacy, one file at a time.\u003c/em\u003e\u003c/sup\u003e\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"https://github.com/minerva-archive/worker/blob/master/LICENSE\"\u003e\n    \u003cimg src=\"https://img.shields.io/:license-CC%201.0-blue.svg\" alt=\"License\"\u003e\n  \u003c/a\u003e\n  \u003ca href=\"https://pypi.org/project/minerva\"\u003e\n    \u003cimg src=\"https://img.shields.io/badge/python-3.10%2B-informational\" alt=\"Python version\"\u003e\n  \u003c/a\u003e\n  \u003ca href=\"https://github.com/astral-sh/uv\"\u003e\n    \u003cimg src=\"https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/Onyx-Nostalgia/uv/refs/heads/fix/logo-badge/assets/badge/v0.json\" alt=\"Manager: uv\"\u003e\n  \u003c/a\u003e\n  \u003ca href=\"https://github.com/astral-sh/ruff\"\u003e\n    \u003cimg src=\"https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json\" alt=\"Linter: Ruff\"\u003e\n  \u003c/a\u003e\n  \u003ca href=\"https://github.com/minerva-archive/worker/actions/workflows/ci.yml\"\u003e\n    \u003cimg src=\"https://github.com/minerva-archive/worker/actions/workflows/ci.yml/badge.svg\" alt=\"Build status\"\u003e\n  \u003c/a\u003e\n\u003c/p\u003e\n\n* * *\n\nMyrient is shutting down. Minerva is a volunteer-driven effort to archive its entire collection before it goes offline.\nRun a script, share your bandwidth, help preserve the archive.\n\n## Installation\n\nDownload and install the Python script from PIP/PyPI:\n\n```shell\n$ pip install minerva-worker\n```\n\n\u003e [!NOTE]\nIf pip gives you a warning about a path not being in your PATH environment variable then promptly add that path then\nclose all open command prompt Windows, or running `minerva` won't work as it will not be recognized as a program.\n\nYou now have the `minerva` package installed - Voilà 🎉!  \nGet started by running `minerva` in your Terminal or Windows Run.  \nFor configuration options, see help and options by running `minerva --help`.\n\n*Alternatively, a Windows EXE is available on the [Releases] page, simply run it to begin!*\n\n  [Releases]: \u003chttps://github.com/minerva-archive/worker/releases\u003e\n\n## Usage\n\nIt's very easy to use. Simply run the minerva.exe file, or run `minerva` in your Terminal/Command Prompt.\n\nYou can configure the worker settings by running `minerva run --help` to see what configuration options\nyou can change. If you have a great computer and network, it's recommended to bump up the -c and -b\noptions.\n\n\u003e [!TIP]\nIt's recommended to keep -c smaller or the same as -b.\n\n## How it works\n\nThe worker script asks the minerva server for jobs to download. The server gives active workers random\nmissing needed file URLs to download. When the worker is given a job, it temporarily downloads the file\nand uploads it to the minerva file servers. Once the file is downloaded, it is deleted from your machine.\n\n\u003e [!TIP]\nIf you also want a copy of the files, use the `--keep-files` setting.\n\nJobs are given exclusively to each worker, no two workers download the same file at the same time. However,\nto verify that uploads aren't corrupted, each job gets given (eventually) to a second worker. Both uploads\nare then confirmed and if both workers give back the same file to the minerva file server, then the job is\nmarked as complete and verified.\n\n\u003e [!NOTE]\nYou may see 409 Conflict errors on upload, this happens when either you or another worker had mismatching\nfiles uploaded. Just ignore these error's and let the worker continue. If you suspect every file has this\nissue, please verify your network connection is stable and verify your downloads arent corrupted.\n\n## Discord Authentication\n\nWhen using the minerva worker, you are prompted upon startup to login and authorize with Discord. This is\nto authenticate unique users on the minerva server, to know who jobs are given to, and to use your username\nand avatar in the worker dashboard leaderboards.\n\nThis does not give Minerva, this script, or anyone else access to your account, or any permissions.\n\n## Docker\n\nYou can run the Minerva Worker inside a headless Docker container.\nThe following steps assume some knowledge on git/docker.\n\nTo change the settings of the worker, you can set the following environment variables:\n\n- `--server`: `MINERVA_SERVER`\n- `--upload-server`: `MINERVA_UPLOAD_SERVER`\n- `-c/--concurrency`: `MINERVA_CONCURRENCY`\n- `-b/--batch-size`: `MINERVA_BATCH_SIZE`\n- `-a/--aria2c-connections`: `MINERVA_ARIA2C_CONNECTIONS`\n- `-p/--pre-allocation`: `MINERVA_ARIA2C_PRE_ALLOCATION`\n- `--temp-dir`: `MINERVA_TEMP_DIR`\n- `--keep-files`: `MINERVA_KEEP_FILES`\n\nThere are more advanced environment variables available, you can find them listed in\n[constants.py](/minerva/constants.py).\n\n### 1. Download a copy of the repository\n\n- Clone the Git Repository: `git clone https://github.com/minerva-archive/worker`  \n- Enter it: `cd minerva`\n\n### 2. Get an Authorization Token\n\nThere's two ways to go about this,\n\n- either run the normal python script to authenticate with it,\n- or, get and save the token to a specific location manually.\n\nTo get the token manually, go to \u003chttps://api.minerva-archive.org/auth/discord/login\u003e and you will get a\ntoken value once you authorize with Discord. Copy that token value and save it to `~/.minerva-dpn/token`\non Linux/macOS, or save it to `%USERPROFILE%/.minerva-dpn/token` on Windows.\n\nThe token file must stay there at all times for the Docker container to have it. The location where the\ntoken needs to be can be changed, just make sure you change the volume location and environment variable\nin the docker-compose config.\n\n### 3. Start the container\n\nBuild the Docker image and start the container with `docker compose up -d` to run it in the background.\nYou can later stop the container with `docker compose down`.\n\n\u003e [!TIP]\nIf you prefer to run it in the foreground (attached to your terminal), simply use `docker compose up`.\n\n## Development\n\n1. Install [uv]\n2. `uv sync --all-extras --all-groups`\n3. `.venv\\Scripts\\activate` (or `source .venv/bin/activate` on macOS and Linux)\n4. `uv tool install pre-commit --with pre-commit-uv --force-reinstall`\n5. `pre-commit install`\n\nNow feel free to work on the project however you like, all code will be checked before committing.\n\n  [uv]: \u003chttps://docs.astral.sh/uv\u003e\n\n## Licensing\n\nThis software is licensed under the terms of [CC0 1.0 Universal](LICENSE).\nYou can find a copy of the license in the LICENSE file in the root folder\n\nThis project uses the following software:\n\n- Aria2(c) licensed under GPL-v2, used for faster downloads: \u003chttps://github.com/aria2/aria2\u003e\n\nNo changes were made to any of the aforementioned software and copies of their licenses\ncan be found next to their binaries within the bin folder.\n\n* * *\n\n© minerva-archive 2026\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fminerva-archive%2Fworker","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fminerva-archive%2Fworker","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fminerva-archive%2Fworker/lists"}