{"id":29219793,"url":"https://github.com/protofire/filecoin-cid-checker","last_synced_at":"2025-08-24T10:14:30.545Z","repository":{"id":38424893,"uuid":"266746476","full_name":"protofire/filecoin-CID-checker","owner":"protofire","description":"Filecoin CID checker and Storage Oracle","archived":false,"fork":false,"pushed_at":"2025-02-27T07:09:51.000Z","size":3042,"stargazers_count":17,"open_issues_count":26,"forks_count":12,"subscribers_count":9,"default_branch":"calibration","last_synced_at":"2025-07-03T02:07:32.101Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/protofire.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-05-25T10:07:15.000Z","updated_at":"2025-02-24T10:47:29.000Z","dependencies_parsed_at":"2023-11-13T10:30:10.303Z","dependency_job_id":"f27db8e5-f436-46f3-a1a8-23eaf1ec34d3","html_url":"https://github.com/protofire/filecoin-CID-checker","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/protofire/filecoin-CID-checker","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/protofire%2Ffilecoin-CID-checker","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/protofire%2Ffilecoin-CID-checker/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/protofire%2Ffilecoin-CID-checker/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/protofire%2Ffilecoin-CID-checker/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/protofire","download_url":"https://codeload.github.com/protofire/filecoin-CID-checker/tar.gz/refs/heads/calibration","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/protofire%2Ffilecoin-CID-checker/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":263245316,"owners_count":23436515,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-07-03T02:07:32.039Z","updated_at":"2025-07-03T02:07:33.493Z","avatar_url":"https://github.com/protofire.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Filecoin CID checker and Storage Oracle\n  \n- [Introduction](#Introduction)\n- [User Starting Guide](#User-starting-guide)\n- [Deployment](#Deployment)\n- [Docker Images](#Docker-images)\n- [App structure](#App-structure)\n\n## Introduction\n\nInitial RFP: \"A website and API service that can list all CIDs along with their current status in the latest state tree. \nThe page could also support queries by CID or miner. \nOne option would be to build 1 long table that shows each miner x sectors they are storing x state as a colored indicator: green - good | grey - capacity | red - failing.\"\n\nThe benefits of the CID checker and Storage Oracle:\n\n- for a User storing own files:\nthe CID checker is the only service that allows you to quickly check information about your file by its CID or the Deal ID. In addition, you can find information about other CIDs stored by a chosen Miner.\n\n- for a Developer:\nyou can use this service as a Storage Oracle providing the extended range of data related to a chosen CID, a Deal, or a Miner from the latest state tree.\n\nWhat it brings to you as a Developer in addition to the existing Lotus API:\n\n1. A simple REST API service.\nThe CID Checker as a service provides two REST endpoints, where you need to specify a single search criteria to get the full CID summary (when the Lotus JSON RPC requires complicated JSON requests). \n\n2. It combines responses of several JSON RPC methods (StateMarketDeals, StateMinerSectors, StateGetActor, ChainReadObj) into a single convenient summary (database).\n\n3. The back-end automates the process of fetching the new data, which requires thousands of JSON RCP calls for every loop.\n\n4. Effective and reliable Mongo DB database with indexes provides fast paginated search by Deal ID, Piece CID, Miner ID.\n\n5. UI provides rich details on deals.\nFor the end user it doesn't require any developer skills to quickly get information on deals. \n\n**Software requirements specification (the SRS) is** [here](https://hackmd.io/RMpGnE3YQm607jl0QevCoQ?view)\n\n**Project Roadmap is** [here](https://github.com/protofire/filecoin-CID-checker#workspaces/filecoin-cid-checker-5ecbabcb812f8965b13d94cb/roadmap?repos=266746476)\n\n**Project management board is** [here](https://github.com/protofire/filecoin-CID-checker#workspaces/filecoin-cid-checker-5ecbabcb812f8965b13d94cb/board?repos=266746476)\n\n## User starting guide\n\nIf you are a total beginner to this, start here!\n\n**Use remote CID checker service:**\n1. Navigate to the website: [https://old.filecoin.tools](https://old.filecoin.tools)\n2. See the list of all piece CIDs and related information:\n- Pieced CID\n- Status\n- Deal ID\n- Payload CID\n3. Search a record by a Piece CID, Deal ID, or Miner ID or Payload CID\n4. Click on a chosen record and see other related details\n\n**If you are running an own Filecoin (Lotus) node:**\n\nInstall the CID checker to your Filecoin (Lotus) node: see deployment instructions below\n\n**If you are an application developer:**\n\nTwo API endpoints available to be used as a Storage Oracle (see the API section below) \n\n\n## Deployment\n\nThe simplest way to deploy the CID checker is doing it with docker-compose.\n\nThe CID checker is supposed to query the data from a running Lotus node.\nTo connect the CID checker to a Lotus node specify the Lotus node's address as an environment variable and the corresponding JWT token in the docker-compose-js.yaml:\n- CID_LOTUS_RPCURL - URL available through the network and fully synced Lotus node.\n- CID_LOTUS_JWT_TOKEN - JWT token with at least read permissions\n\nInstructions on how to run Lotus node - https://docs.filecoin.io/get-started/lotus/installation/\n\n\nBuild all docker images of the different components (the `package.json` files has the comands to build the images individually):\n\n```\nyarn run dc:build\n```\n\nRun app with docker-compose:\n\n```\ndocker-compose -f docker-compose.yaml up\n# or\nyarn start:dev\n```\n\n### ENV variables\n\nList of available environment variables for app configuration.\n\n##### API component\n\nSee [packages/backend/.env-example](packages/backend/.env-example) \n\n##### Watcher component\n\nSee [packages/watcher/.env-example](packages/watcher/.env-example)\n\n##### Frontend component \n\nSee [packages/frontend/.env-example](packages/frontend/.env-example)\n\n## Docker images\n\nDeployed application contains a number of docker images.\n\n#### cid-checker-watcher\n\nRuns the loops that retrieves Deals' data from the Lotus node and feeds the Mongo DB.\n\n#### cid-checker-backend\n\nAPI that queries and searches through the DB to serve the UI\n\n#### cid-checker-frontend\n\nWeb UI created via [create-react-app](https://reactjs.org/docs/create-a-new-react-app.html) and typescript; image used build files via Nginx (see `./.config/nginx.conf`).\n\n### How to run\n\n1. Prepare `.env` file in root directory\n\n2. Build images:\n\n```bash\ndocker build -t cid-checker-frontend:$(cat ./packages/frontend/version.txt) -f Dockerfile.frontend . \u0026\u0026 \\\ndocker build -t cid-checker-backend:$(cat ./packages/frontend/version.txt) -f Dockerfile.backend . \u0026\u0026 \\\ndocker build -t cid-checker-watcher:$(cat ./packages/frontend/version.txt) -f Dockerfile.watcher .\n```\n3. Run docker compose:\n\n```bash\ndocker-compose up -d\n```\n\n## App structure\n\nThe main components of the CID checker are:\n\n### Watcher (packages/watcher, typescript)\n\nPeriodically checks the network for new chain height. \nEvery time a new height is detected, the watcher runs the _processors_ (originally, more than one). \nCurrently the only processor that runs ins the `DealsProcessor` that calls Lotus StateMarketDeals() method and saves all deals into the \"deals\" collection.\n\n### API (packages/backend, nodejs)\n\nUsed in frontend for remote REST calls\nHas openapi interactive UI to make calls ( route /docs )\n\n### Frontend (packages/frontend typescript)\n\ncreate-react-app based application that queries the API\n\n## Run app components\n\n### With docker-compose\n\n#### Dependencies\n\n1. docker-compose [https://docs.docker.com/compose/](https://docs.docker.com/compose/)\n2. yarn - [https://yarnpkg.com/](https://yarnpkg.com/)\n \n#### Process\n\n1. Got to package root dir\n2. Fill .env file - as an example - .env-example in packages/backend, packages/frontend, packages/wqtcher \n3. Build containers(once, after each changes in code)\n\n```\nyarn dc:build # all\n\n# or as separated\nyarn run dc:build:api\nyarn run dc:build:ui\nyarn run dc:build:watcher\n\n```\n\n4. Run\n\n```\n    yarn install\n    yarn start # will run all app components, app will be available on url http://localhost\n    # or separated\n    yarn run start:mongo:dc\n    yarn run start:api:dc\n    yarn run start:ui:dc\n    yarn run start:watcher:dc\n        \n    yarn stop # stop all containers\n    # or separated\n    yarn run stop:mongo:dc\n    yarn run stop:caddy:dc\n    yarn run stop:api:dc\n    yarn run stop:ui:dc\n    yarn run stop:watcher:dc\n    \n```\n\n#### Dependencies\n\n1. yarn - [https://yarnpkg.com/](https://yarnpkg.com/)\n\n#### Process\n\n1. install dependencies\n```\nyarn install\n```\n2. run services in different terminals\n\n```\nyarn run start:ui\nyarn run start:api\nyarn run start:watcher\n```\n\n## CI/CD\n\nThe image uses as part of the solution for the cid-checker collects data from the endpoint and provides it to the external DB.\n\n### CI Logic\nCI verifies that the Dockerfile can build successfully.\n\n### CD Logic\nCD has mandatory requirements to bump a new image version in the buildspec.yaml file.\n\nCD push image to DockerHub repository, and it is publicly available.\n\nURL for DockerHub repository: [https://hub.docker.com/r/protofire/cid-checker](https://hub.docker.com/r/protofire/cid-checker)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fprotofire%2Ffilecoin-cid-checker","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fprotofire%2Ffilecoin-cid-checker","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fprotofire%2Ffilecoin-cid-checker/lists"}