{"id":19294573,"url":"https://github.com/web3-storage/gendex-consumer","last_synced_at":"2025-04-22T08:30:33.639Z","repository":{"id":176390998,"uuid":"655705346","full_name":"web3-storage/gendex-consumer","owner":"web3-storage","description":"🗂️ Cloudflare queue consumer that submits blocks to gendex to build block indexes","archived":true,"fork":false,"pushed_at":"2023-06-29T18:23:07.000Z","size":1198,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-02-24T00:28:04.266Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/web3-storage.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-06-19T12:35:41.000Z","updated_at":"2024-04-18T16:23:58.000Z","dependencies_parsed_at":null,"dependency_job_id":"d42994c3-5bd9-423a-85f3-612d29b01468","html_url":"https://github.com/web3-storage/gendex-consumer","commit_stats":null,"previous_names":["web3-storage/gendex-consumer"],"tags_count":23,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/web3-storage%2Fgendex-consumer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/web3-storage%2Fgendex-consumer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/web3-storage%2Fgendex-consumer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/web3-storage%2Fgendex-consumer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/web3-storage","download_url":"https://codeload.github.com/web3-storage/gendex-consumer/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250205971,"owners_count":21392157,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-09T22:38:44.651Z","updated_at":"2025-04-22T08:30:33.634Z","avatar_url":"https://github.com/web3-storage.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# gendex-consumer\n\n\u003cp\u003e\n  \u003ca href=\"https://github.com/web3-storage/gendex-consumer/actions/workflows/release.yml\"\u003e\u003cimg alt=\"GitHub Workflow Status\" src=\"https://img.shields.io/github/actions/workflow/status/web3-storage/gendex-consumer/test.yml?branch=main\u0026style=for-the-badge\" /\u003e\u003c/a\u003e\n  \u003ca href=\"https://github.com/web3-storage/gendex-consumer/blob/main/LICENSE.md\"\u003e\u003cimg alt=\"License: Apache-2.0 OR MIT\" src=\"https://img.shields.io/badge/LICENSE-Apache--2.0%20OR%20MIT-yellow?style=for-the-badge\" /\u003e\u003c/a\u003e\n\u003c/p\u003e\n\nCloudlfare queue consumer that submits blocks to gendex in order to build block indexes for uploaded DAGs.\n\n## Usage\n\nSend a message to the queue with the format: \n\n```ts\ninterface Message {\n  /**\n   * Base encoded string CIDs of CAR shards the block (and it's descendents)\n   * can be found.\n   */\n  shards: string[]\n}\n```\n\nPassing shard CIDs allows gendex to not require a root CID and thus allows it to not require a DUDEWHERE index (to lookup shard CIDs). It also removes the hard dependency on SATNAV, in the event that there are no indexes in SATNAV for the passed shards, they can be materialized from the source data as needed.\n\nIn current NFT.Storage and web3.storage APIs we know the shards because linkdex returns them (and we'll only submit to gendex when linkdex says that a DAG is \"Complete\" in structure). In the new w3up APIs our `upload/add` invocation includes `root` and `shards` allowing us to submit to gendex when this call is made.\n\nSetting `recursive: false` allows individual blocks requested by bitswap that are not yet indexed in blockly to be added from information available in the current Elastic IPFS DynamoDB.\n\n### Manually add a message to the queue\n\ne.g.\n\n```sh\ncurl --header \"Content-Type: application/json\" \\\n  --request POST \\\n  --data '{\"block\":\"bafybeigwaxpc64gtjs25brjozyktwhdnr3hirrbnjyq746gd26mhe7jllm\",\"shards\":[\"bagbaierarysgidzym55qcdud66parpzxo6jb2wj3vpzax6lhtw7vaxmy224a\"],\"root\":\"bafybeigwaxpc64gtjs25brjozyktwhdnr3hirrbnjyq746gd26mhe7jllm\",\"recursive\":true}' \\\n  https://gendex-consumer-env.org.workers.dev/send\n```\n\n## Contributing\n\nFeel free to join in. All welcome. Please [open an issue](https://github.com/web3-storage/gendex-consumer/issues)!\n\n## License\n\nDual-licensed under [MIT + Apache 2.0](https://github.com/web3-storage/gendex-consumer/blob/main/LICENSE.md)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fweb3-storage%2Fgendex-consumer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fweb3-storage%2Fgendex-consumer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fweb3-storage%2Fgendex-consumer/lists"}