https://github.com/elastos/elastos.carrierclassic.crawler
https://github.com/elastos/elastos.carrierclassic.crawler
Last synced: about 2 months ago
JSON representation
- Host: GitHub
- URL: https://github.com/elastos/elastos.carrierclassic.crawler
- Owner: elastos
- License: gpl-3.0
- Created: 2018-10-23T04:34:53.000Z (over 6 years ago)
- Default Branch: master
- Last Pushed: 2019-07-22T02:12:06.000Z (almost 6 years ago)
- Last Synced: 2025-02-12T20:38:49.104Z (4 months ago)
- Language: C
- Size: 64.5 KB
- Stars: 1
- Watchers: 29
- Forks: 3
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
Carrier Crawler
===============
[](https://travis-ci.com/elastos/Elastos.NET.Carrier.Crawler)
## Summary
elacrawler is a Elastos Carrier network crawler.
## How crawler works
The carrier crawler crawls the Carrier network with multiple concurrent instances (default 5 crawlers), allowing the connection by sending **get_nodes** requests to (and get responses from) the number of active Carrier nodes on the network.
When a cralwer instance comletes it's mission, a log file containing all space separated IP addresses that it found is created in {currentdate} under your config data directory, with the name {timestamp}.lst.
## IP2Location
IP2Location supports to reveal the geological location for crawled carrier nodes. With IP2Location, the geological depoloyment of almost all active nodes of whole carrier network can vividly demenstrated.
Please refer to [ip2location.com](https://www.ip2location.com/) to download ip2location database before IP2Location enabled.
## Config Profile
The list of config items and their description is briefly profiled below:
#### 1. internval
The allowing least interval value in munites for the next cralwer to begin crawling carrier nodes.
#### 2. timeout
The timeout value in seconds for the **get nodes** request to crawler.
#### 3. data_dir
This data directory stores carrier node cached data itself and all dumped files of cralwed carrier nodes information.
#### 4. log\_level and log\_file
The level value and the file path to for logging.
#### 5. ip2location_database
The location of ip2location database relative to current config file.
#### 6. bootstraps
The bootstrap nodes list.
## Build from source
With CMake installed, use the following commands to conduct the whole build steps:
```shell
$ git clone https://github.com/elastos/Elastos.NET.Carrier.Cralwer
$ mkdir build
$ cd build
$ cmake -DCMAKE_INSTALL_PREFIX=dist ..
$ make install
```
## Run
Use commands to run crawler after installation:
```shell
$ cd dist/bin
$ ./elacrawler -c ../etc/carrier/crawler.conf
```
## Contribution
We welcome contributions to the Elastos Carrier Project.
## Acknowledgments
A sincere thank you to all teams and projects that we rely on directly or indirectly.
## License
GPLv3