Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/cybercongress/crawler
A toolchain for bringing web2 to web3
https://github.com/cybercongress/crawler
cosmos-sdk crawler cyber cyberd ipfs web3 wiki
Last synced: 2 months ago
JSON representation
A toolchain for bringing web2 to web3
- Host: GitHub
- URL: https://github.com/cybercongress/crawler
- Owner: cybercongress
- License: mit
- Archived: true
- Created: 2019-01-14T18:45:28.000Z (over 5 years ago)
- Default Branch: master
- Last Pushed: 2019-11-11T14:23:19.000Z (over 4 years ago)
- Last Synced: 2024-02-12T20:04:24.833Z (5 months ago)
- Topics: cosmos-sdk, crawler, cyber, cyberd, ipfs, web3, wiki
- Language: Go
- Homepage:
- Size: 54.7 KB
- Stars: 10
- Watchers: 6
- Forks: 3
- Open Issues: 18
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Lists
- awesome-cyber - Wikipedia crawler - by cyberCongress (archived). (Crawlers)
README
[](https://github.com/cybercongress/cyber-wiki-index/releases/latest)
[](https://circleci.com/gh/cybercongress/cyber-wiki-index/tree/master)
[](https://github.com/cybercongress/cyber-wiki-index/blob/master/LICENSE)
[](https://github.com/cybercongress/cyber-wiki-index)
[](https://github.com/cybercongress/cyber-wiki-index/graphs/contributors)
[](https://t.me/fuckgoogle)
[](https://github.com/cybercongress/cyber-wiki-index/blob/master/CONTRIBUTING.md)
[://cyber](https://github.com/cybercongress/cyberd) wiki index
==================
- [Installation](#installation)
- [Usage](#usage)
- [Issues](#issues)
- [Contributing](#contributing)
- [Changelog](#changelog)
## Installation
Note: Requires Go 1.12+
```
git clone https://github.com/cybercongress/crawler
cd crawler
go build -o crawler
```
## Preparation
1. IPFS daemon should be launched
2. Download enwiki-latest-all-titles to crawler root dir:
```
ipfs get QmddV5QP87BZGiSUCf9x9hsqM73b83rsPC6AYMNqkjKMGx -o enwiki-latest-all-titles
```
3. Add account to cyberdcli:
```
docker exec -ti cyberd cyberdcli keys add --recover
```
## Usage
### Submit links
Basically, there are two main functions provided by `crawler` tool.
The first one is to parse wiki titles and submit links between keywords and wiki pages.
```
./crawler submit-links-to-cyber ./enwiki-latest-all-titles --home= --address= --passphrase= --chunk=100
```
> Note: Uses only local cyberd node.
> Note: Submit links do not add duras to IPFS.
> Note: Chunk - how many links messages added to one tx
> Note: There is --help command, for example
```
./crawler submit-links-to-cyber --help
```
Here, **enwiki-latest-all-titles** is titles file obtained from
[official Wiki dumps](https://dumps.wikimedia.org/enwiki/latest/enwiki-latest-all-titles-in-ns0.gz).
### Uploading duras to IPFS
Also, `crawler` has separate command `upload-duras-to-ipfs` to upload files to local IPFS node.
All DURAs are collected under single root unixfs directory.
```
./crawler upload-duras-to-ipfs enwiki-latest-all-titles
```
## Issues
If you have any problems with or questions about search, please contact us through a
[GitHub issue](https://github.com/cybercongress/crawler/issues).
## Contributing
You are invited to contribute new features, fixes, or updates, large or small; We are always thrilled to receive pull
requests, and do our best to process them as fast as We can. You can find detailed information in our
[contribution guide](./docs/contributing/contributing.md).
## Changelog
Stay tuned with our [Changelog](./CHANGELOG.md).