{"id":22551521,"url":"https://github.com/crustio/crust-smanager","last_synced_at":"2025-07-25T22:41:57.658Z","repository":{"id":42069754,"uuid":"308536876","full_name":"crustio/crust-smanager","owner":"crustio","description":"A storage manager integrated with Crust, IPFS and sWorker(storage inspector) of Crust protocol","archived":false,"fork":false,"pushed_at":"2024-10-17T02:41:49.000Z","size":610,"stargazers_count":6,"open_issues_count":4,"forks_count":18,"subscribers_count":7,"default_branch":"mainnet","last_synced_at":"2025-03-24T04:43:43.916Z","etag":null,"topics":["crust","ipfs","polkadot-js"],"latest_commit_sha":null,"homepage":"https://crust.network","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/crustio.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-10-30T05:47:36.000Z","updated_at":"2024-11-12T14:14:00.000Z","dependencies_parsed_at":"2024-06-24T05:38:53.780Z","dependency_job_id":"3f010a70-9e3b-46f5-9fe6-c2aeb2230299","html_url":"https://github.com/crustio/crust-smanager","commit_stats":null,"previous_names":[],"tags_count":3,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/crustio%2Fcrust-smanager","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/crustio%2Fcrust-smanager/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/crustio%2Fcrust-smanager/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/crustio%2Fcrust-smanager/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/crustio","download_url":"https://codeload.github.com/crustio/crust-smanager/tar.gz/refs/heads/mainnet","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248150841,"owners_count":21055988,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["crust","ipfs","polkadot-js"],"created_at":"2024-12-07T17:12:58.577Z","updated_at":"2025-04-10T03:33:10.935Z","avatar_url":"https://github.com/crustio.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# crust-smanager\nsManager (Storage Manager) is a file picking bot which continuously picking and handling files from Crust Network. Node operators can customize this module to implement their own file handling strategy. sManager maintains a local database to help making decision on pulling files.\n\n## Local Database\nThe local database stores below information:\n1. **File Records**: The files metadata(tips, size, replicas count, expire time etc.) on Crust Network.\n2. **File and Owner Relationship**: sManager also maintains the relationship between a file and an on-chain account. This information will help making better pulling decision.\n3. **Chain Metadata**: e.g. the block and time on chain.\n4. **Pin Records**: The pin history of files.\n5. **Cleanup Records**: The files needs to removed from local filesystem, normally this is triggered when a file expires on Crust Network.\n\nCheckout [Db Schema](db-schema.md) for the schema details.\n\n## Components\nsManager was designed to have serveral tasks running independently. Tasks are either scheduled by the block event or by configured intervals. Each task plays as an actor which consumes/produces some information and communicate with other tasks through the db or applicaion context.\n\nsManager follows the **Fails Early** priciple which means it will shutdown on any unexpected error. To support this priciple, tasks are designed to be recoverable after application restarts.\n\nBelow are a list of components that sManager has implemented.\n### Indexers\nIndexers extract information into the local database from various data sources. Currently sManager has implemented below indexers:\n1. **Chain Database Indexer**: indexes file records from the Crust Network on-chain database.\n2. **Chain Event Indexer**: indexes file records by listening latest chain event.\n3. **Chain Time Indexer**: a simple indexer which push the latest block height and it's timestamp to the config table.\n\n### Simple Tasks\nSimple tasks are speciualized tasks which runs periodly. Currently sManager has implemented below tasks:\n1. **Group Info Updater**: Update sworker identity information from sworker api.\n2. **Ipfs Gc**: Schedule ipfs gc periodly.\n3. **Telemetry Reporting**: Report smanager statistics information to the telemetry server.\n4. **Pull Scheduler**: Schedule file pulling based on configured strategey.\n5. **Seal Status Updater**: Update sealing status periodly.\n6. **File Retry Task**: Retry pulling if possible.\n7. **File Cleanup Task**: Cleanup deleted files from local filesystem.\n\n## Usage\n\n1. Clone repo\n\n```shell\ngit clone https://github.com/crustio/crust-smanager.git\n```\n\n2. Installing\nIt's recommended to use `volta` as the node version manager. Please follow the [volta docs](https://docs.volta.sh/guide/getting-started) to install it.\n\n```shell\ncd crust-smanager \u0026\u0026 npm i\n```\n\n3. Debug\n\n```shell\nnpm run dev\n```\n\n4. Run in Prod\n```shell\nnpm run build\nnpm start\n```\n\nIt's recommended to run sManager using Docker with the `restart=always` restart policy.\n\nA daemon guard should be configured if you want to run sManager natively without docker. Tools like `pm2` and `nodemon` could be used.\n\n## Configuration\n\nCheckout [smanager-config.example.json](data/smanager-config.example.json)\n\nThose config items will be configured in the sManager configuration setup process. The meaning of each item is as follows:\n\n* chain.account: your member account\n* chain.endPoint: your chain endpoint\n* sworker.endPoint: your sWorker endpoint\n* ipfs.endPoint: your IPFS endpoint\n* dataDir: the directory of the database of sManager\n* scheduler.minSrdRatio: a minimum ratio of SRD that one node can start to accept storage orders.\n\n    \u003e For example, if the ratio is 30, then your node will start to accept storage order once the ratio of SRD capacity is higher than 30%\n\n* scheduler.strategy.dbFilesWeight: how much bandwidth of this node will be used to fetch and store the history storage orders(Storage orders in the past four months).\n* scheduler.strategy.newFilesWeight: how much bandwidth of this node will be used to fetch and store the newest storage orders.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcrustio%2Fcrust-smanager","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcrustio%2Fcrust-smanager","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcrustio%2Fcrust-smanager/lists"}