{"id":15152901,"url":"https://github.com/ipfs/rainbow","last_synced_at":"2025-04-06T06:12:19.946Z","repository":{"id":46360547,"uuid":"419460912","full_name":"ipfs/rainbow","owner":"ipfs","description":"A specialized IPFS HTTP gateway","archived":false,"fork":false,"pushed_at":"2024-10-25T16:01:46.000Z","size":544,"stargazers_count":83,"open_issues_count":18,"forks_count":12,"subscribers_count":13,"default_branch":"main","last_synced_at":"2024-10-29T12:11:32.968Z","etag":null,"topics":["http","ipfs","ipfs-gateway"],"latest_commit_sha":null,"homepage":"https://docs.ipfs.tech/reference/http/gateway/","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ipfs.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE-APACHE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-10-20T19:17:38.000Z","updated_at":"2024-10-29T09:58:44.000Z","dependencies_parsed_at":"2023-10-12T23:41:32.697Z","dependency_job_id":"e33debb2-9caf-453b-b60e-a45711ca7247","html_url":"https://github.com/ipfs/rainbow","commit_stats":{"total_commits":170,"total_committers":16,"mean_commits":10.625,"dds":0.6764705882352942,"last_synced_commit":"f66ecf7e2eff4941fb3be5cdbdffb5a2dd366e7a"},"previous_names":[],"tags_count":14,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ipfs%2Frainbow","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ipfs%2Frainbow/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ipfs%2Frainbow/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ipfs%2Frainbow/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ipfs","download_url":"https://codeload.github.com/ipfs/rainbow/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247061882,"owners_count":20877176,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["http","ipfs","ipfs-gateway"],"created_at":"2024-09-26T16:41:22.812Z","updated_at":"2025-04-06T06:12:19.919Z","avatar_url":"https://github.com/ipfs.png","language":"Go","readme":"\u003ch1 align=\"center\"\u003e\n  \u003cbr\u003e\n  \u003ca href=\"https://github.com/ipfs/rainbow/assets/157609/fd1bed0f-2055-468e-93e7-0aea158aa953\"\u003e\u003cimg src=\"https://github.com/ipfs/rainbow/assets/157609/8bf5b727-a360-4906-b965-826823c37aa3\" alt=\"Rainbo logo\" title=\"Rainbow logo\" width=\"200\"\u003e\u003c/a\u003e\n  \u003cbr\u003e\n  Rainbow\n  \u003cbr\u003e\n\u003c/h1\u003e\n\n\u003cp align=\"center\" style=\"font-size: 1.2rem;\"\u003eA to-be-released production-grade IPFS HTTP Gateway written in Go (using \u003ca href=\"https://github.com/ipfs/boxo\"\u003eBoxo\u003c/a\u003e).\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"https://ipfs.tech\"\u003e\u003cimg src=\"https://img.shields.io/badge/project-IPFS-blue.svg?style=flat-square\" alt=\"Official Part of IPFS Project\"\u003e\u003c/a\u003e\n  \u003ca href=\"https://discuss.ipfs.tech\"\u003e\u003cimg alt=\"Discourse Forum\" src=\"https://img.shields.io/discourse/posts?server=https%3A%2F%2Fdiscuss.ipfs.tech\"\u003e\u003c/a\u003e\n  \u003ca href=\"https://matrix.to/#/#ipfs-space:ipfs.io\"\u003e\u003cimg alt=\"Matrix\" src=\"https://img.shields.io/matrix/ipfs-space%3Aipfs.io?server_fqdn=matrix.org\"\u003e\u003c/a\u003e\n  \u003ca href=\"https://github.com/ipfs/rainbow/actions\"\u003e\u003cimg src=\"https://img.shields.io/github/actions/workflow/status/ipfs/rainbow/go-test.yml?branch=main\" alt=\"ci\"\u003e\u003c/a\u003e\n  \u003ca href=\"https://codecov.io/gh/ipfs/rainbow\"\u003e\u003cimg src=\"https://codecov.io/gh/ipfs/rainbow/branch/main/graph/badge.svg?token=9eG7d8fbCB\" alt=\"coverage\"\u003e\u003c/a\u003e\n  \u003ca href=\"https://github.com/ipfs/rainbow/releases\"\u003e\u003cimg alt=\"GitHub release\" src=\"https://img.shields.io/github/v/release/ipfs/rainbow?filter=!*rc*\"\u003e\u003c/a\u003e\n  \u003ca href=\"https://godoc.org/github.com/ipfs/rainbow\"\u003e\u003cimg src=\"https://img.shields.io/badge/godoc-reference-5272B4.svg?style=flat-square\" alt=\"godoc reference\"\u003e\u003c/a\u003e\n\u003c/p\u003e\n\n\u003chr /\u003e\n\n## About\n\nRainbow is an implementation of the [IPFS HTTP Gateway API](https://specs.ipfs.tech/http-gateways),\nbased on [boxo](https://github.com/ipfs/boxo) which is the tooling that powers [Kubo](https://github.com/ipfs/kubo) IPFS implementation.\nIt uses the same Go code as the [HTTP gateway](https://specs.ipfs.tech/http-gateways/) in Kubo,\nbut is fully specialized to just be a gateway:\n\n  * Rainbow acts as [Amino DHT](https://blog.ipfs.tech/2023-09-amino-refactoring/)\n    and [Bitswap](https://specs.ipfs.tech/bitswap-protocol/) client only.\n  * Rainbow does not pin, or permanently store any content. It is just meant\n    to act as gateway to content present in the network.\n  * Rainbow settings are optimized for production deployments and streamlined\n    for specific choices (flatfs datastore, writethrough uncached blockstore\n    etc.)\n  * [Denylist](https://specs.ipfs.tech/compact-denylist-format/) and denylist subscription support is included.\n  * And more to come...\n\n\n## Building\n\n```\ngo build\n```\n\n## Running\n\n```\nrainbow\n```\n\nUse `rainbow --help` for documentation.\n\n### Docker\n\nAutomated Docker container releases are available from the [Github container registry](https://github.com/ipfs/rainbow/pkgs/container/rainbow):\n\n- 🟢 Releases\n  - `latest` always points at the latest stable release\n  - `vN.N.N` point at a specific [release tag](https://github.com/ipfs/rainbow/releases)\n- 🟠 Unreleased developer builds\n  - `main-latest` always points at the `HEAD` of the `main` branch\n  - `main-YYYY-DD-MM-GITSHA` points at a specific commit from the `main` branch\n- ⚠️ Experimental, unstable builds\n  - `staging-latest` always points at the `HEAD` of the `staging` branch\n  - `staging-YYYY-DD-MM-GITSHA` points at a specific commit from the `staging` branch\n  - This tag is used by developers for internal testing, not intended for end users\n\nWhen using Docker, make sure to pass necessary config via `-e`:\n```console\n$ docker pull ghcr.io/ipfs/rainbow:main-latest\n$ docker run --rm -it --net=host -e RAINBOW_SUBDOMAIN_GATEWAY_DOMAINS=dweb.link ghcr.io/ipfs/rainbow:main-latest\n```\n\nSee [`/docs/environment-variables.md`](./docs/environment-variables.md).\n\n\n## Configuration\n\n### CLI and Environment Variables\n\nRainbow can be configured via command-line arguments or environment variables.\n\nSee `rainbow --help` and [`/docs/environment-variables.md`](./docs/environment-variables.md) for information on the available options.\n\nRainbow uses a `--datadir` (or `RAINBOW_DATADIR` environment variable) as\nlocation for persisted data. It defaults to the folder in which `rainbow` is\nrun.\n\n### Peer Identity\n\n**Using a key file**: By default generates a `libp2p.key` in its data folder if none exist yet. This\nfile stores the libp2p peer identity.\n\n**Using a seed + index**: Alternatively, random can be initialized with a\n32-byte, b58 encoded seed and a derivation index. This allows to use the same\nseed for multiple instances of rainbow, and only change the derivation index.\n\nThe seed and index can be provided as command line arguments or environment\nvars (`--seed` , `--seed-index`). The seed can also be provided as a `seed`\nfile in the datadir folder. A new random seed can be generated with:\n\n    rainbow gen-seed \u003e seed\n\nTo facilitate the use of rainbow with systemd\n[`LoadCredential=`](https://www.freedesktop.org/software/systemd/man/systemd.exec.html#LoadCredential=ID:PATH)\ndirective, we look for both `libp2p.key` and `seed` in\n`$CREDENTIALS_DIRECTORY` first.\n\n### Denylists\n\nRainbow can subscribe to append-only denylists using the `--denylists` flag. The value is a comma-separated list of URLs to subscribe to, for example: `https://denyli.st/badbits.deny`. This will download and update the denylist automatically when it is updated with new entries.\n\nDenylists can be manually placed in the `$RAINBOW_DATADIR/denylists` folder too.\n\nSee [NoPFS](https://github.com/ipfs-shipyard/nopfs) for an explanation of the denylist format. Note that denylists should only be appended to while Rainbow is running. Editing differently, or adding new denylist files, should be done with Rainbow stopped.\n\n## Blockstores\n\nRainbow ships with a number of possible blockstores for the purposes of caching data locally.\nBecause Rainbow, as a gateway-only IPFS implementation, is not designed for long-term data storage there are no long\nterm guarantees of support for any particular backing data storage.\n\nSee [Blockstores](./docs/blockstores.md) for more details.\n\n## Garbage Collection\n\nOver time, the datastore can fill up with previously fetched blocks. To free up this used disk space, garbage collection can be run. \n\n### Automatic Garbage Collection\n\nAutomatic GC is enabled by default, and configurable with [`RAINBOW_GC_INTERVAL`](https://github.com/ipfs/rainbow/blob/main/docs/environment-variables.md#rainbow_gc_interval) and [`RAINBOW_GC_THRESHOLD`](https://github.com/ipfs/rainbow/blob/main/docs/environment-variables.md#rainbow_gc_threshold).\n\n### Manual Garbage Collection\n\nGarbage collection can also be manually triggered. This process can be automated by using a cron job.\n\nThe API route to trigger GC is `http://$RAINBOW_CTL_LISTEN_ADDRESS/mgr/gc`. The `BytesToFree` parameter must be passed as JSON in the POST request body to specify the upper limit of how much disk space should be cleared. GC will try to clear as much space as needed, up to `BytesToFree`, to create `RAINBOW_GC_THRESHOLD` of free space. Setting this parameter to a very high value will GC the entire datastore.\n\nExample cURL commmand to run GC:\n\n    curl -v --data '{\"BytesToFree\": 1099511627776}' http://127.0.0.1:8091/mgr/gc\n\n## Logging\n\nWhile the logging can be controlled via [environment variable](./docs/environment-variables.md#logging) it is also\npossible to dynamically modify the logging at runtime.\n\n- `http://$RAINBOW_CTL_LISTEN_ADDRESS/mgr/log/level?subsystem=\u003csystem name or * for all system\u003e\u0026level=\u003clevel\u003e` will set the logging level for a subsystem\n- `http://$RAINBOW_CTL_LISTEN_ADDRESS/mgr/log/ls` will return a comma separated list of available logging subsystems\n\n## Purging Peer Connections\n\nConnections to a specific peer, or to all peers, can be closed and the peer information removed from the peer store. This can be useful to help determine if the presence/absence of a connection to a peer is affecting behavior. Be aware that purging a connection is inherently racey as it is possible for the peer to reestablish a connection at any time following a purge.\n\nIf `RAINBOW_DHT_SHARED_HOST=false` this endpoint will not show peers connected to DHT host, and only list ones used for Bitswap.\n\n- `http://$RAINBOW_CTL_LISTEN_ADDRESS/mgr/purge?peer=\u003cpeer_id\u003e` purges connection and info for peer identifid by peer_id\n- `http://$RAINBOW_CTL_LISTEN_ADDRESS/mgr/purge?peer=all` purges connections and info for all peers\n- `http://$RAINBOW_CTL_LISTEN_ADDRESS/mgr/peers` returns a list of currently connected peers\n\nExample cURL commmand to show connected peers and purge peer connection:\n\n    curl http://127.0.0.1:8091/mgr/peers\n    curl http://127.0.0.1:8091/mgr/purge?peer=QmQzqxhK82kAmKvARFZSkUVS6fo9sySaiogAnx5EnZ6ZmC\n\n## Tracing\n\nSee [docs/tracing.md](docs/tracing.md).\n\n## Deployment\n\nSuggested method for self-hosting is to run a [prebuilt Docker image](#docker).\n\nAn ansible role to deploy Rainbow is available within the ipfs.ipfs collection in Ansible Galaxy (https://github.com/ipfs-shipyard/ansible). It includes a systemd service unit file.\n\n## Release\n\n1. Create a PR from branch `release-vX.Y.Z` against `main` that:\n   1. Tidies the [`CHANGELOG.md`](CHANGELOG.md) with the changes for the current release\n   2. Updates the  [`version.json`](./version.json) file\n2. Once the release checker creates a draft release, copy-paste the changelog into the draft\n3. Merge the PR, the release will be automatically created once the PR is merged\n\n## License\n\nDual-licensed under [MIT](https://github.com/filecoin-project/lotus/blob/master/LICENSE-MIT) + [Apache 2.0](https://github.com/filecoin-project/lotus/blob/master/LICENSE-APACHE)\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fipfs%2Frainbow","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fipfs%2Frainbow","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fipfs%2Frainbow/lists"}