https://github.com/joystream/swg-files-comparer
this tool allows SWG workers to check if the local files are identical to the files listed under their bucket in storage node
https://github.com/joystream/swg-files-comparer
Last synced: 25 days ago
JSON representation
this tool allows SWG workers to check if the local files are identical to the files listed under their bucket in storage node
- Host: GitHub
- URL: https://github.com/joystream/swg-files-comparer
- Owner: Joystream
- Created: 2023-12-20T16:24:48.000Z (over 2 years ago)
- Default Branch: main
- Last Pushed: 2024-07-11T19:18:40.000Z (almost 2 years ago)
- Last Synced: 2025-01-06T22:52:56.937Z (over 1 year ago)
- Language: TypeScript
- Size: 42 KB
- Stars: 0
- Watchers: 2
- Forks: 1
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# SWG-files-comparer
This tool allows to check if some of the files for collosus were lost and if they can be found on other nodes.
- [Installation](#installation)
- [Usage](#usage)
# Installation
```shell
# Install dependencies
yarn install
# Build the project
yarn build
```
# Usage
Run the program using yarn start. You will see the command seleciton menu.
1. Index local files
```sh-session
localFiles
@param
```
This command will index the files in the directory and create a file `localFiles.json` in the root directory of the project. It will be used
for comparison later.
2. Index objects from bucket
```sh-session
bucketObjects
@param
@param
```
This will get all the bags from the bucket and index all the objects in them. It will create a file `remote.json` in the root directory of the project.
With provided `bagId` it will get all the objects from the bag and index them. It will create a file `remote-${bagId}.json` in the root directory of the project.
3. Compare the files
```sh-session
diff
@param
```
This will compare your bags from `local.json` with the bags from `remote.json` and will output the results in `diff.json` in the root directory of the project and in the console.
With provided `bagId` it will compare your bags from `local.json` with the bags from `remote-${bagId}.json` and will output the results in `diff-${bagId}.json` in the root directory of the project and in the console.
4. Check missing files
```sh-session
checkMissing
@param
@param
```
This will check the `diff.json` file and will show if some of the other SP has the missing files.
You can specify the custom path to diff file and include the SP that you want to ignore in the search.
e.g
5. Download missing files
```sh-session
downloadMissing
```
This should be used after the checkMissing command. It will download the missing files from the remote node containing missing files.
```sh-session
$ yarn start restoreMissing ./jsons/diff.json 1,2,5
```
6. Manual HEAD request
```sh-session
head
@param
```
This will execute a HEAD request to the specified url and will indicate a presence of the file.
7. Check Remote node
```sh-session
remotenode
@param
```
This will check the remote node and will return the same output as the localFiles command.
8. Check single operator
```sh-session
checknode
@param
```
This is a combination of remoteNode, bucketObjects and diff commands for a single operator. It will check the operator, check all files for its bags in QN and will generate a diff.
8. Check all operators
```sh-session
checkalloperators
```
This is a combination of remoteNode, bucketObjects and diff commands. It will check all the operators, check all files in QN and will generate a diff.