{"id":41698206,"url":"https://github.com/grip-on-software/upload","last_synced_at":"2026-01-24T20:55:05.858Z","repository":{"id":171609892,"uuid":"648151550","full_name":"grip-on-software/upload","owner":"grip-on-software","description":"Encrypted file upload server","archived":false,"fork":false,"pushed_at":"2024-07-30T11:35:00.000Z","size":69,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-09-05T02:22:50.555Z","etag":null,"topics":["database-importer","gpg-encryption","secure-upload"],"latest_commit_sha":null,"homepage":"https://gros.liacs.nl","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/grip-on-software.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":"CITATION.cff","codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2023-06-01T10:18:55.000Z","updated_at":"2024-07-30T11:35:03.000Z","dependencies_parsed_at":null,"dependency_job_id":"2889e91e-4701-4910-8c8a-c0d88eb9e2db","html_url":"https://github.com/grip-on-software/upload","commit_stats":null,"previous_names":["grip-on-software/upload"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/grip-on-software/upload","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/grip-on-software%2Fupload","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/grip-on-software%2Fupload/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/grip-on-software%2Fupload/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/grip-on-software%2Fupload/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/grip-on-software","download_url":"https://codeload.github.com/grip-on-software/upload/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/grip-on-software%2Fupload/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28736791,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-24T19:23:36.361Z","status":"ssl_error","status_checked_at":"2026-01-24T19:23:28.966Z","response_time":89,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["database-importer","gpg-encryption","secure-upload"],"created_at":"2026-01-24T20:55:05.223Z","updated_at":"2026-01-24T20:55:05.842Z","avatar_url":"https://github.com/grip-on-software.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Encrypted file upload server\n\n[![PyPI](https://img.shields.io/pypi/v/gros-upload.svg)](https://pypi.python.org/pypi/gros-upload)\n[![Build \nstatus](https://github.com/grip-on-software/upload/actions/workflows/upload-tests.yml/badge.svg)](https://github.com/grip-on-software/upload/actions/workflows/upload-tests.yml)\n[![Coverage \nStatus](https://coveralls.io/repos/github/grip-on-software/upload/badge.svg?branch=master)](https://coveralls.io/github/grip-on-software/upload?branch=master)\n[![Quality Gate\nStatus](https://sonarcloud.io/api/project_badges/measure?project=grip-on-software_upload\u0026metric=alert_status)](https://sonarcloud.io/project/overview?id=grip-on-software_upload)\n[![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.12784820.svg)](https://doi.org/10.5281/zenodo.12784820)\n\nThis repository includes a service for running a HTTP server which accepts \nuploads of GPG-encrypted files. Although available as a package, it is mostly \nmeant to run as a standalone program or service. The application uses \na keychain to keep GPG passphrases which can be modified using a subcommand. \nCertain uploaded files can be used to import a database dump. Usual deployment \nsetups would host this service behind a reverse proxy such as NGINX or Apache \nwhich handles SSL termination and additional access control over the \nDigest-based user authentication in this server.\n\n## Installation\n\nThe [GPG exchange](https://github.com/lhelwerd/gpg-exchange) library is \nrequired to be in a working state. Follow the instructions there to install the \nGPG dependencies first. Then, to install the latest release version of the \npackaged program from PyPI, run the following command:\n\n```\npip install gros-upload\n```\n\nAnother option is to build the program from this repository, which allows using \nthe most recent development code. Run `make setup` to install the dependencies. \nThe upload server itself may then be installed with `make install`, which \nplaces the package in your current environment. We recommend using a virtual \nenvironment during development.\n\n## Configuration\n\nConfigure server settings in `upload.cfg` by copying `upload.cfg.example` or \nthe example file below:\n```ini\n[server]\nkey = $SERVER_KEY\nengine = $SERVER_ENGINE\nfiles = $SERVER_FILES\nsecret = $SERVER_SECRET\nkeyring = $SERVER_KEYRING\nrealm = $SERVER_REALM\n\n[import]\ndatabase = $IMPORT_DATABASE\ndump = $IMPORT_DUMP\npath = $IMPORT_PATH\nscript = $IMPORT_SCRIPT\n\n[client]\n$CLIENT_ID=$CLIENT_NAME\n\n[auth]\n$CLIENT_ID=$CLIENT_AUTH\n\n[symm]\n$CLIENT_ID=$CLIENT_PASSPHRASE\n```\nReplace the variables with actual values. The following configuration sections \nand items are known:\n\n- `server`: Configuration of the listener server.\n  - `key`: Fingerprint of the GPG key pair to be used by the server to identify \n    itself toward the uploaders.\n  - `engine`: Path to the GPG utility for GPG tasks, e.g. `/usr/bin/gpg2`.\n  - `files`: Space-separated list of file names that the server accepts in the\n    upload, for example `dump.sql.gz` to allow a database dump from the example\n    tasks of the `export-exchange` repository.\n  - `secret`: Secret string to use for the hash algorithm for the digest \n    authentication to challenge the uploader to provide a known username and \n    password in encrypted format.\n  - `keyring`: Name of the keyring in which authentication data is stored.\n  - `realm`: Name of the realm to use within the digest authentication.\n- `import`: Configuration of the file-specific import. Normally, uploaded files \n  are simply placed in subdirectories below the current working directory; in \n  nesting order 'upload', the login name of the client and the current date. \n  For a specific file name, an import script can be started to load a database \n  from scratch with data from an uploaded dump file.\n  - `database`: The name of the database to import dumps into. Provided as \n    a third parameter to the import script; ignored by the standard `import.sh` \n    script since the database name is determined by the organizational user.\n  - `dump`: Name of the uploaded file that is considered for the import script. \n    Other files do not trigger the import script. The standard `import.sh` \n    script expects there to be a file called `dump.tar.gz`.\n  - `path`: Path to the \n    [monetdb-import](https://github.com/grip-on-software/monetdb-import) \n    repository where further import scripts are located. The `Scripts` \n    directory within this repository is used as working directory for the \n    import script.\n  - `script`: Path to the script to run when a specific dump file is uploaded. \n    If the script is placed elsewhere than the currrent working directory, use \n    an absolute path. The standard `import.sh` script performs database \n    recreation, archive extraction, import, update and schema publication.\n- `client`: Accepted logins and public key names. Each item has a configuration \n  key which has the login name of a uploader client, and the value is the name \n  registered in the public key that the uploader must provide in order to be \n  accepted.\n- `auth`: Accepted logins with usernames as keys and passwords as values in the \n  configuration items. The usernames and passwords are imported to the keyring \n  if possible, so that they can be removed from the configuration once imported \n  (assuming the `secret` remains the same).\n- `symm`: Usernames and passphrases for symmetric decryption of uploaded data, \n  respectively as keys and values in the configuration items. The usernames and \n  passphrases are imported to the keyring if possible, so that they can be \n  removed from the configuration once imported.\n\n## Running\n\nThe upload server can be started directly using the following command:\n\n```\ngros-upload server\n```\n\nThe subcommand takes various options that can be reviewed by using the \n`gros-upload server --help` argument, including debugging instances and \ndifferent CGI deployment options. Uploads are stored beneath the \"upload\" \nsubdirectory structured of the current working directory.\n\nA `gros-uploader.service` file is provided for installing as a systemd service. \nOne can also use the `upload-session.sh` file to start the service within \na GNOME keyring context, preset to store uploads in `/home/upload/upload` and \nlogs in `/var/log/upload`, using a `virtualenv` setup shared with the \n[controller](https://gros.liacs.nl/data-gathering/api.html#controller-api) of \nthe agent-based data gathering setup. The script requires a password to unlock \nthe keyring. In combination with the service, a root user needs to input \na keyring password using a systemd Password Agent, for example by running the \n`systemd-tty-ask-password-agent` command, before the server actually starts \nunder the `upload` user. Some pointers on the advanced setup can be found in \n[installation](https://gros.liacs.nl/data-gathering/installation.html#controller) \nof the controller environment.\n\nIn order to adjust client authentication credentials, the subcommand \n`gros-upload auth [--add|--modify|--delete] --user ... [--password ...]` may be \nused. Additional arguments shown in `gros-upload auth --help` allow setting the \nsecret Digest token and the private key passphrase. Configuring credentials is \nalso possible for users and the Digest token using the `auth` section and \n`secret` option of the `server` section in the [configuration](#configuration) \nfile, respectively.\n\n## Development and testing\n\nTo run tests, first install the test dependencies with `make setup_test` which \nalso installs all dependencies for the upload server. Then `make coverage` \nprovides test results in the output and in XML versions compatible with, e.g., \nJUnit and SonarQube available in the `test-reports/` directory. If you do not \nneed XML outputs, then run `make test` to just report on test successes and \nfailures or `make cover` to also have the terminal report on hits and misses in \nstatements and branches.\n\n[GitHub Actions](https://github.com/grip-on-software/upload/actions) is used to \nrun the unit tests and report on coverage on commits and pull requests. This \nincludes quality gate scans tracked by \n[SonarCloud](https://sonarcloud.io/project/overview?id=grip-on-software_upload) \nand [Coveralls](https://coveralls.io/github/grip-on-software/upload) for \ncoverage history.\n\nThe Python module conforms to code style and typing standards which can be \nchecked using Pylint with `make pylint` and mypy with `make mypy`, after \ninstalling the pylint and mypy dependencies using `make setup_analysis`; typing \nreports are XML formats compatible with JUnit and SonarQube placed in the \n`mypy-report/` directory. To also receive the HTML report, use `make mypy_html` \ninstead.\n\nWe publish releases to [PyPI](https://pypi.org/project/gros-upload/) using \n`make setup_release` to install dependencies and `make release` which performs \nmultiple checks: unit tests, typing, lint and version number consistency. The \nrelease files are also published on \n[GitHub](https://github.com/grip-on-software/upload/releases) and from there \nare archived on [Zenodo](https://zenodo.org/doi/10.5281/zenodo.12784819).\n\n## License\n\nGROS encrypted file upload server is licensed under the Apache 2.0 License.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgrip-on-software%2Fupload","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgrip-on-software%2Fupload","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgrip-on-software%2Fupload/lists"}