{"id":13563778,"url":"https://github.com/jasonwhite/rudolfs","last_synced_at":"2025-04-03T20:31:44.808Z","repository":{"id":37580301,"uuid":"175289743","full_name":"jasonwhite/rudolfs","owner":"jasonwhite","description":"A high-performance, caching Git LFS server with an AWS S3 and local storage back-end.","archived":false,"fork":false,"pushed_at":"2024-12-29T01:04:09.000Z","size":349,"stargazers_count":412,"open_issues_count":15,"forks_count":41,"subscribers_count":6,"default_branch":"master","last_synced_at":"2025-03-01T23:11:21.932Z","etag":null,"topics":["aws-s3","cache","git","lfs","server"],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jasonwhite.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-03-12T20:16:22.000Z","updated_at":"2025-02-27T07:17:38.000Z","dependencies_parsed_at":"2023-10-19T05:30:36.749Z","dependency_job_id":"6d24390e-ae12-47e0-a917-5afcf1659b58","html_url":"https://github.com/jasonwhite/rudolfs","commit_stats":{"total_commits":121,"total_committers":12,"mean_commits":"10.083333333333334","dds":0.5041322314049587,"last_synced_commit":"30f76fed124b851f7884fbd0cb1dc1daa78bfe67"},"previous_names":[],"tags_count":21,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jasonwhite%2Frudolfs","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jasonwhite%2Frudolfs/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jasonwhite%2Frudolfs/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jasonwhite%2Frudolfs/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jasonwhite","download_url":"https://codeload.github.com/jasonwhite/rudolfs/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247074800,"owners_count":20879329,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws-s3","cache","git","lfs","server"],"created_at":"2024-08-01T13:01:23.181Z","updated_at":"2025-04-03T20:31:44.519Z","avatar_url":"https://github.com/jasonwhite.png","language":"Rust","funding_links":[],"categories":["Rust"],"sub_categories":[],"readme":"# Rudo*lfs*\n\n[![Cirrus CI - Specific Branch Build Status](https://img.shields.io/cirrus/github/jasonwhite/rudolfs/master?style=for-the-badge)](https://cirrus-ci.com/github/jasonwhite/rudolfs)\n[![Crates.io](https://img.shields.io/crates/v/rudolfs?style=for-the-badge)](https://crates.io/crates/rudolfs)\n[![AUR version](https://img.shields.io/aur/version/rudolfs?style=for-the-badge)](https://aur.archlinux.org/packages/rudolfs/)\n[![Docker Image Version (latest semver)](https://img.shields.io/docker/v/jasonwhite0/rudolfs?label=Docker\u0026sort=semver\u0026style=for-the-badge)](https://hub.docker.com/r/jasonwhite0/rudolfs)\n[![Docker Image Size (latest semver)](https://img.shields.io/docker/image-size/jasonwhite0/rudolfs?sort=semver\u0026style=for-the-badge)](https://hub.docker.com/r/jasonwhite0/rudolfs)\n\nA high-performance, caching Git LFS server with an AWS S3 back-end.\n\n## Features\n\n - Multiple backends:\n\n   1. AWS S3 backend with an optional local disk cache.\n   2. Local disk backend.\n\n - A configurable local disk cache to speed up downloads (and reduce your\n   S3 bill).\n\n - Corruption-resilient local disk cache. Even if the disk is getting\n   blasted by cosmic rays, it'll find corrupted LFS objects and purge them from\n   the cache transparently. The client should never notice this happening.\n\n - Encryption of LFS objects in both the cache and in permanent storage.\n\n - Separation of GitHub organizations and projects. Just specify the org and\n   project names in the URL and they are automatically created. If two projects\n   share many LFS objects, have them use the same URL to save on storage space.\n\n - A tiny (\u0026lt;10MB) Docker image ([jasonwhite0/rudolfs][]).\n\n[jasonwhite0/rudolfs]: https://hub.docker.com/r/jasonwhite0/rudolfs\n\nThe back-end storage code is very modular and composable. PRs for implementing\nother storage back-ends are welcome. If you begin working on this, please let us\nknow by submitting an issue.\n\n## Non-Features\n\n - There is no client authentication. This is meant to be run in an internal\n   network with clients you trust, not on the internet with malicious actors.\n\n## Running It\n\n### Generate an encryption key (optional)\n\nIf configured, all LFS objects are encrypted with the xchacha20 symmetric stream\ncipher. You must generate a 32-byte encryption key before starting the server.\n\nGenerating a random key is easy:\n\n    openssl rand -hex 32\n\nKeep this secret and save it in a password manager so you don't lose it. We will\npass this to the server below via the `--key` option. If the `--key` option is\n**not** specified, then the LFS objects are **not** encrypted.\n\n**Note**:\n - If the key ever changes (or if encryption is disabled), all existing LFS\n   objects will become garbage. When the Git LFS client attempts to download\n   them, the SHA256 verification step will fail.\n - Likewise, if encryption is later enabled after it has been disabled, all\n   existing unencrypted LFS objects will be seen as garbage.\n - LFS objects in both the cache and in permanent storage are encrypted.\n   However, objects are decrypted before being sent to the LFS client, so take\n   any necessary precautions to keep your intellectual property safe.\n\n### Development\n\nFor testing during development, it is easiest to run it with Cargo. Create\na file called `test.sh` (this path is already ignored by `.gitignore`):\n\n```bash\n# Your AWS credentials.\nexport AWS_ACCESS_KEY_ID=XXXXXXXXXXXXXXXXXXXX\nexport AWS_SECRET_ACCESS_KEY=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\nexport AWS_DEFAULT_REGION=us-west-1\n\n# Change this to the output of `openssl rand -hex 32`.\nKEY=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\n\ncargo run -- \\\n  --cache-dir cache \\\n  --host localhost:8080 \\\n  --max-cache-size 10GiB \\\n  --key $KEY \\\n  s3 \\\n  --bucket foobar\n```\n\nIf you just need to use the local disk as the backend, use the following bash.\n\n```bash\n# Change this to the output of `openssl rand -hex 32`.\nexport RUDOLFS_KEY=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\n\ncargo run -- --port 8080 local --path=/data\n```\n\n**Note**: Always use a different S3 bucket, cache directory, and encryption key\nthan what you use in your production environment.\n\n**Warning**: *This server may not be accessible from other machines.* Specifying\n`--host localhost:8080` will often bind the server to an internal-only loopback\nnetwork interface (i.e., if `localhost` resolves to `127.0.0.1` or `[::1]`).\nThus, to make the server accessible from the outside world, specify `--host\n0.0.0.0:8080` or just `--port 8080` (the default IP the server will bind to is\n`0.0.0.0`). IP `0.0.0.0` means the server shall try to bind to all available\nnetwork interfaces, both internal and external. See\nhttps://github.com/jasonwhite/rudolfs/issues/38#issuecomment-973721511 for more\ninformation.\n\n### Production\n\nTo run in a production environment, it is easiest to use `docker-compose`:\n\n 1. Create a `.env` file next to `docker-compose.yml` with the configuration\n    variables:\n\n    ```\n    AWS_ACCESS_KEY_ID=XXXXXXXXXXXXXXXXXXXQ\n    AWS_SECRET_ACCESS_KEY=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\n    AWS_DEFAULT_REGION=us-west-1\n    LFS_ENCRYPTION_KEY=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\n    LFS_S3_BUCKET=my-bucket\n    LFS_MAX_CACHE_SIZE=10GB\n    ```\n\n 2. Use the provided `docker-compose.yml` file to run a production environment:\n\n    ```bash\n    docker-compose up -d\n\n    # use minio yml\n    docker-compose -f ./docker-compose.minio.yml up -d\n    \n    # use local disk yml\n    docker-compose -f ./docker-compose.local.yml up -d\n    ```\n\n 3. **[Optional]**: It is best to use nginx as a reverse proxy for this server.\n    Use it to enable TLS. How to configure this is better covered by other\n    tutorials on the internet.\n\n**Note**:\n - A bigger cache is (almost) always better. Try to use ~85% of the available\n   disk space.\n - The cache data is stored in a Docker volume named `rudolfs_data`. If you\n   want to delete it, run `docker volume rm rudolfs_data`.\n\n## AWS Credentials\n\nAWS credentials must be provided to the server so that it can make requests to\nthe S3 bucket specified on the command line (with `--bucket`).\n\nYour AWS credentials will be searched for in the following order:\n\n 1. Environment variables: `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`\n 2. AWS credentials file. Usually located at `~/.aws/credentials`.\n 3. IAM instance profile. Will only work if running on an EC2 instance with an\n    instance profile/role.\n\nThe AWS region is read from the `AWS_DEFAULT_REGION` or `AWS_REGION` environment\nvariable. If it is malformed, it will fall back to `us-east-1`. If it is not\npresent it will fall back on the value associated with the current profile in\n`~/.aws/config` or the file specified by the `AWS_CONFIG_FILE` environment\nvariable. If that is malformed or absent it will fall back to `us-east-1`.\n\n## Client Configuration\n\nAdd a file named `.lfsconfig` to the root of your Git repository and commit it\nso everyone is using the same LFS server:\n\n```\n[lfs]\nurl = \"http://gitlfs.example.com:8080/api/my-org/my-project\"\n              ─────────┬──────── ──┬─ ─┬─ ───┬── ─────┬────\n                       │           │   │     │        └ Replace with your project's name\n                       │           │   │     └ Replace with your organization name   \n                       │           │   └ Required to be \"api\"\n                       │           └ The port your server started with\n                       └ The host name of your server\n```\n\nOptionally, I also recommend changing these global settings to speed things up:\n\n``` bash\n# Increase the number of worker threads\ngit config --global lfs.concurrenttransfers 64\n\n# Use a global LFS cache to make re-cloning faster\ngit config --global lfs.storage ~/.cache/lfs\n```\n\n## License\n\n[MIT License](/LICENSE)\n\n## Thanks\n\nThis was developed at [Environmental Systems Research\nInstitute](http://www.esri.com/) (Esri) who have graciously allowed me to retain\nthe copyright and publish it as open source software.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjasonwhite%2Frudolfs","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjasonwhite%2Frudolfs","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjasonwhite%2Frudolfs/lists"}