{"id":28126116,"url":"https://github.com/cidgravity/gw","last_synced_at":"2026-03-05T23:42:29.422Z","repository":{"id":65643747,"uuid":"596232185","full_name":"CIDgravity/gw","owner":"CIDgravity","description":"Next Gen Filecoin Client","archived":false,"fork":false,"pushed_at":"2025-05-06T08:05:28.000Z","size":7831,"stargazers_count":28,"open_issues_count":34,"forks_count":2,"subscribers_count":6,"default_branch":"main","last_synced_at":"2025-05-06T09:24:46.171Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/CIDgravity.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE-APACHE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2023-02-01T18:42:31.000Z","updated_at":"2025-05-06T08:05:34.000Z","dependencies_parsed_at":"2023-10-16T03:47:56.535Z","dependency_job_id":"39fc800f-493b-4ab3-a1a1-8ba2cf7ea44e","html_url":"https://github.com/CIDgravity/gw","commit_stats":{"total_commits":76,"total_committers":1,"mean_commits":76.0,"dds":0.0,"last_synced_commit":"fcdc76828c6b0d4fdaa700b4963ba3d30384518a"},"previous_names":["filcat/ribs","lotus-web3/ribs","filozone/ribs","cidgravity/gw"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CIDgravity%2Fgw","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CIDgravity%2Fgw/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CIDgravity%2Fgw/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CIDgravity%2Fgw/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/CIDgravity","download_url":"https://codeload.github.com/CIDgravity/gw/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254120395,"owners_count":22017981,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-05-14T10:23:18.127Z","updated_at":"2026-03-05T23:42:24.380Z","avatar_url":"https://github.com/CIDgravity.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# RIBS\n\n\u003e Reasonable Interplanetary BlockStore (RIBS)\n\nRIBS is a Filecoin-native IPFS/IPLD blockstore designed for seamless integration\nwith the Filecoin tech stack. It provides users with a scalable blockstore\ncapable of handling almost arbitrary amounts of data. RIBS aims to be an\nauto-dagstore-like blockstore with fully automated Filecoin data offload support.\n\n# WORK IN PROGRESS\n\n**RIBS is a work-in-progress project. Most features aren't finished, and on-disk\nformat is not stable. DO NOT USE FOR STORING CIRITICAL DATA, OR ANY OTHER DATA**\n\n**Status:**\n* Data layer is mostly implemented, but needs a lot of hardening to gain\n  confidence that it never loses data.\n* API Implementation is mostly complete, but there is a lot of space for optimization.\n* Filecoin dealmaking process is mostly implemented.\n* RIBSWeb is mostly complete and covers existing functionality.\n* Kubo integration (KuboRIBS - KuRI) works, but needs some UX improvements.\n* Retrieval probing / retrieval functionality is not implemented yet.\n* Multi-node support is not implemented yet.\n\n# Key Features\n\n**Filecoin-native IPFS nodes**: RIBS can be integrated into most IPFS nodes\nseamlessly, thanks to a layer that implements the standard Blockstore interface.\nRIBS also provides a high-performance block interface for more efficient data\nmanagement.\n\n**Scalable blockstore**: RIBS is designed to support 100PiB+ deployments, providing\na scalable solution for distributed storage systems that can grow alongside user\nrequirements.\n\n**Improved data locality and parallelism**: RIBS groups blocks into log-files,\nwhich in many scenarios provide good data locality, and are very easy to convert\ninto Filecoin-deal-friendly data format.\n\n**Fully automated Filecoin data offloading**: When block groups become \"full\",\nRIBS automatically backs them up to Filecoin, by converting them into a .car\nfile, computing deal CID, selecting SPs, and making deals. RIBS will also provide\nan offloading mechanism, which will make it possible to free up space on the\nlocal storage by removing the backed-up blocks, and fetching them from Filecoin\nwhen needed.\n\n**RIBSWeb**: RIBSWeb is a web-based UI for RIBS, which provides an advanced, but\neasy-to-use interface for managing RIBS nodes, and monitoring their status.\n\n![architecture](./doc/ribsweb.png)\n\n# Design Overview / Roadmap\n\n![architecture](./doc/architecture.png)\n\n**Grouping blocks**: RIBS groups blocks into log-files, which provides better\nperformance on all drives, including HDDs and SSDs. This approach leads to\nimproved data locality, allowing for better parallelism and easier offloading to\nFilecoin data chunks.\n\n**Local data index**: RIBS maintains a local index of all blocks, which allows\nfor efficient access to all block data, including offloaded blocks. Multiple\nbackends will be supported for multi-node deployments, such as FoundationDB.\n\n**Filecoin dealmaking functionality**: RIBS automates all steps of the Filecoin dealmaking process:\n\n![architecture](./doc/deals.png)\n\n* **Storage Provider crawler**: Discovers and monitors storage providers in the network.\n* **Efficient on-the-fly .car file generation**: RIBS can generate .car files in one \n  sequential scan of a group log.\n* **Fast DataCID computation**: RIBS Uses all available cores to comput DataCID as fast as possible.\n* **Advanced SP selection process**: Utilizes a built-in reputation system to select\n  the most suitable storage providers based on their performance and reliability.\n* **Retrieval probing**: Attempts retrievals for a random sample of data, to ensure\n  that SPs provide the promised retrieval service. Unretrievable deals will not\n  count torwards the redundancy factor, and SPs who fail to provide retrievals\n  will be selected for deals much less frequently.\n* **Automatic deal repair**: Maintains a user-defined redundancy factor by\n  automatically repairing and recreating deals as needed.\n\n**Arbitrary scalability**: All parts of RIBS are designed to scale horizontally, \nallowing for almost arbitrary amounts of data to be stored.\n\n*multi-node support is not implemented yet*\n\n* Group files can be stored on any storage backend, including distributed\n  filesystems, and can be managed by a fleet of \"Group workers\" which can run\n  tasks such as DataCID computation, car file creation, etc.\n* Local data index (\"Top Level Index\") can be backed by any scalable KV store\n* \"Group Manager\" processes can run redundantly to provide high availability\n* (Future) Users can deploy additional car file caching servers to improve\n  efficiency of making redundant deals.\n* (Future) Retrievals can be served by a fleet of \"Retrieval workers\".\n* (Future) Access to storage is provided by a smart, session-aware driver, which can talk\n  directly to the relevant parts of the cluster.\n\n# Usage\n\n### Integrating as a blockstore\n\n*Interface not stable yet*\n\n* Main interface definition [here](https://github.com/lotus-web3/ribs/blob/main/interface.go)\n* Can be wrapped into a standard IPFS blockstore using [Blockstore layer](https://github.com/lotus-web3/ribs/blob/main/integrations/blockstore/ribsbs.go)\n* Example Kubo plugin [here](https://github.com/lotus-web3/ribs/blob/main/integrations/kuri/ribsplugin/kuboribs.go)\n\n### Running (demo) Kubo-Ribs (KuRI) Node\n\n* Install Golang\n* Clone this repo\n\n```\ngit clone https://github.com/lotus-web3/ribs.git\ncd ribs\ngo build -o kuri ./integrations/kuri/cmd/kuri\n```\n\n* **backup / move away .ipfs / set IPFS_PATH to an alternative directory if you have a local IPFS node**\n* Init KuRI node and start the daemon\n\n```\n./kuri init\n\n# By default a new wallet will be generated.\n# Send Filecoin funds or DataCap before the next\n# starting the node daemon.\n\n./kuri daemon\n```\n\n* Use the node, like any Kubo node!\n\n```\n./kuri add README.md\n```\n\n* Kuri CLI/API is the same as Kubo\n* IPFS WebUI is available at http://localhost:5001/webui. The files tab can be\n  used as essentially an infinite-storage file browser.\n* RIBSWeb is served at http://127.0.0.1:9010\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcidgravity%2Fgw","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcidgravity%2Fgw","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcidgravity%2Fgw/lists"}