{"id":15451953,"url":"https://github.com/willscott/http-cid-data","last_synced_at":"2026-03-18T03:57:47.125Z","repository":{"id":140831855,"uuid":"557755171","full_name":"willscott/http-cid-data","owner":"willscott","description":"example of an http provider of content addressed data","archived":false,"fork":false,"pushed_at":"2022-10-29T13:08:55.000Z","size":58,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-03-31T06:13:20.031Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/willscott.png","metadata":{"files":{"readme":"readme.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-10-26T08:40:17.000Z","updated_at":"2022-10-29T14:34:34.000Z","dependencies_parsed_at":"2023-07-07T00:01:46.013Z","dependency_job_id":null,"html_url":"https://github.com/willscott/http-cid-data","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/willscott/http-cid-data","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/willscott%2Fhttp-cid-data","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/willscott%2Fhttp-cid-data/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/willscott%2Fhttp-cid-data/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/willscott%2Fhttp-cid-data/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/willscott","download_url":"https://codeload.github.com/willscott/http-cid-data/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/willscott%2Fhttp-cid-data/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":30646391,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-18T02:48:56.676Z","status":"ssl_error","status_checked_at":"2026-03-18T02:48:55.747Z","response_time":104,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-01T21:40:34.459Z","updated_at":"2026-03-18T03:57:47.088Z","avatar_url":"https://github.com/willscott.png","language":null,"funding_links":[],"categories":[],"sub_categories":[],"readme":"# HTTP CID Data\n\nA static website that acts as a provider of content addressed data.\n\nHow to replicate this:\n\n1. Get blocks representing some data:\n\n```\ncar create -f my.car files\ncar ls lw.car\ncar gb lw.car bafybeifgz2z4kghtvo2iy6xxo3uil2yu5iils5ffvjzgn4znpbjpas7lw4 \u003e bafybeifgz2z4kghtvo2iy6xxo3uil2yu5iils5ffvjzgn4znpbjpas7lw4\n```\n\nExtract each block from `ls` into it's own block named by it's CID.\n\n2. make the http index provider records.\n\nFor this example the command used is:\n```\ngo run ./cmd/provider e --ctxid helloworld --metadata 4AMA --cid bafkreiefztf2jizqd7qsgu4xhdjqbi3tkcrputiqbrjrtu7ywfgnsirgsu --addr /dns4/willscott.github.io/tcp/443/https/httpath/L2h0dHAtY2lkLWRhdGEv --outDir to/ -i cid.contact\n```\n\nfrom the modified version of the `index-provider` in the feat/http-provider branch:\nhttps://github.com/filecoin-project/index-provider/tree/feat/http-provider\n\n3. take the resulting files (an advertisement, and entries chunk, and a `head`, and put them with the blocks at the hosted location.)\n\n4. Announce to an indexer.\n\n```js\nimport { Duplex } from 'stream'\nimport fetch from 'node-fetch'\nimport { BufferList } from 'bl'\nimport { encode, Token, Type } from 'cborg'\nimport { CID } from 'multiformats/cid'\nimport { sha256 } from 'multiformats/hashes/sha2'\n\nfunction _base64ToArrayBuffer(base64) {\n    var binary_string = atob(base64);\n    var len = binary_string.length;\n    var bytes = Buffer.alloc(len);\n    for (var i = 0; i \u003c len; i++) {\n        bytes[i] = binary_string.charCodeAt(i);\n    }\n    return bytes;\n}\n\nconst ma64 = \"NhN3aWxsc2NvdHQuZ2l0aHViLmlvBgG7uwOAhMABDWh0dHAtY2lkLWRhdGGlAyYAJAgBEiChXJ2VTZmsm3DKn1oxKYkuKL9vpC/gE9N5huKrM7BVZw==\"\nconst mab = _base64ToArrayBuffer(ma64)\n\nconst none = Buffer.alloc(0)\nconst advertisementCid = CID.parse(\"baguqeeradhg7qdq6dpizzd3e552m5dsbdz5jmeu7fgapduz5ly3sk5b4wseq\")\n//const advertisementCid = CID.create(1, 0x0129, await sha256.digest(none))\nconsole.log(\"advert is \", advertisementCid)\nconst cbor = encode([advertisementCid, [mab], Buffer.alloc(0)],\n{\n typeEncoders: {\n   Object: function (cid) {\n     // CID must be prepended with 0 for historical reason - See: https://github.com/ipld/cid-cbor\n     const bytes = new BufferList(Buffer.alloc(1))\n     bytes.append(cid.bytes)\n\n     return [new Token(Type.tag, 42), new Token(Type.bytes, bytes.slice())]\n   }\n }\n})\n\nconst url = 'https://dido.prod.cid.contact/ingest/announce';\n                    let stream = new Duplex();\n                    stream.push(cbor);\n                    stream.push(null);\n\n                    const response = await fetch(url, {\n                        method: 'PUT',\n                        mode: 'same-origin',\n                        headers: {\n                            'Content-Type': 'application/cbor'\n                        },\n                        body: stream\n                    })\n```\n\nnotes:\n* the cid is the cid of your advertisement\n* the multiaddr needs to be generated in golang, becuase the js multiaddr library can't support multiaddr extensions. the extend command in the index-provider library has an example of how to generate these bytes.\n\n5. You can confirm the records become available, at `cid.contact/providers`, and for the cid you used in the extend command, e.g. https://dido.prod.cid.contact/cid/bafkreiefztf2jizqd7qsgu4xhdjqbi3tkcrputiqbrjrtu7ywfgnsirgsu\n\n6. The [w3rc library](https://github.com/ipfs-shipyard/w3rc/tree/feat/http) is an experimental retrieval client with support for these HTTP providers, and can `w3rc get -r bafkreiefztf2jizqd7qsgu4xhdjqbi3tkcrputiqbrjrtu7ywfgnsirgsu`","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwillscott%2Fhttp-cid-data","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fwillscott%2Fhttp-cid-data","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwillscott%2Fhttp-cid-data/lists"}