https://github.com/qubistlabs/jticker-grabber
Jticker grabber app
https://github.com/qubistlabs/jticker-grabber
backend jticker kafka
Last synced: about 2 months ago
JSON representation
Jticker grabber app
- Host: GitHub
- URL: https://github.com/qubistlabs/jticker-grabber
- Owner: qubistlabs
- Created: 2018-11-22T09:51:18.000Z (almost 7 years ago)
- Default Branch: develop
- Last Pushed: 2023-03-21T22:38:45.000Z (over 2 years ago)
- Last Synced: 2025-08-04T15:33:52.695Z (3 months ago)
- Topics: backend, jticker, kafka
- Language: Python
- Size: 1.82 MB
- Stars: 0
- Watchers: 0
- Forks: 0
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
Awesome Lists containing this project
README
# Jticker grabber
Grab live candle streams from exchanges.
## Configuration
All arguments can be passed as cli arguments or via environment variables. Environment variables
naming are same as arguments but uppercased and `-` replaced with `_`. E.g. `--log-level INFO` cli argument
or `LOG_LEVEL=INFO` environment variable. Environment variabes have **higher** priorty.
### cli spec (version 2.0.0)
```
$ python -m jticker_grabber --help
usage: jticker-grabber [-h] [--log-level LOG_LEVEL]
[--sentry-dsn SENTRY_DSN]
[--io-network-max-workers IO_NETWORK_MAX_WORKERS]
[--io-file-max-workers IO_FILE_MAX_WORKERS]
[--web-host WEB_HOST] [--web-port WEB_PORT]
[--providers-enabled PROVIDERS_ENABLED]
[--kafka-bootstrap-servers KAFKA_BOOTSTRAP_SERVERS]
[--disable-streaming]
optional arguments:
-h, --help show this help message and exit
--log-level LOG_LEVEL
Python logging level [default: INFO]
--sentry-dsn SENTRY_DSN
Sentry DSN [default: None]
--io-network-max-workers IO_NETWORK_MAX_WORKERS
Max workers for network io executor [default: 100]
--io-file-max-workers IO_FILE_MAX_WORKERS
Max workers for file io executor [default: 10]
--web-host WEB_HOST Web server host [default: 0.0.0.0]
--web-port WEB_PORT Web server port [default: 8080]
--providers-enabled PROVIDERS_ENABLED
Comma separated exchanges [default: None]
--kafka-bootstrap-servers KAFKA_BOOTSTRAP_SERVERS
Comma separated kafka bootstrap servers [default:
kafka:9092]
--disable-streaming Disable initial streaming [default: False]
```
# Jticker local grabber
Grab big™ historical data from supported exchanges with proxy support. **Local grabber have no deployment.**
**It is only for by hand usage.**
### cli spec (version 2.0.0)
```
$ python -m jticker_grabber.local --help
usage: jticker-grabber-local [-h] [--log-level LOG_LEVEL]
[--sentry-dsn SENTRY_DSN]
[--io-network-max-workers IO_NETWORK_MAX_WORKERS]
[--io-file-max-workers IO_FILE_MAX_WORKERS]
[--web-host WEB_HOST] [--web-port WEB_PORT]
[--storage STORAGE]
[--request-timeout REQUEST_TIMEOUT]
[--interval INTERVAL]
{dump,list,upload} ...
positional arguments:
{dump,list,upload}
optional arguments:
-h, --help show this help message and exit
--log-level LOG_LEVEL
Python logging level [default: INFO]
--sentry-dsn SENTRY_DSN
Sentry DSN [default: None]
--io-network-max-workers IO_NETWORK_MAX_WORKERS
Max workers for network io executor [default: 100]
--io-file-max-workers IO_FILE_MAX_WORKERS
Max workers for file io executor [default: 10]
--web-host WEB_HOST Web server host [default: 0.0.0.0]
--web-port WEB_PORT Web server port [default: 8080]
--storage STORAGE Output directory [default: storage]
--request-timeout REQUEST_TIMEOUT
Request timeout [default: 5]
--interval INTERVAL Candles interval [default: 60]
```
## List
List available historical exchanges
```
$ python -m jticker_grabber.local list --help
usage: jticker-grabber-local list [-h]
List available exchanges for historical grabbing
optional arguments:
-h, --help show this help message and exit
```
## Dump
Dump candles from selected exchanges. After dump finished storage will have `{exchange_name}`
directories with stored candles in sqlite dbs (file per month).
```
$ python -m jticker_grabber.local dump --help
usage: jticker-grabber-local dump [-h] [--proxies-file PROXIES_FILE]
[--exchanges EXCHANGES]
[--from-time FROM_TIME] [--to-time TO_TIME]
[--disable-automatic-start-timestamp]
[--ignore-zero-volume-candles]
optional arguments:
-h, --help show this help message and exit
--proxies-file PROXIES_FILE
File with proxies [default: None]
--exchanges EXCHANGES
Comma separated exchanges names [default: None means
`all`]
--from-time FROM_TIME
Start timestamp or isoformat datetime for grabbing
[default: 2009-01-01T00:00:00]
--to-time TO_TIME End timestamp or isoformat datetime for grabbing
[default: 2019-10-22]
--disable-automatic-start-timestamp
Disable automatic start timestamp
--ignore-zero-volume-candles
Ignore candles with base volume equal to zero
```
## Upload
Upload selected exchanges data to kafka via jticker-controller.
```
$ python -m jticker_grabber.local upload --help
usage: jticker-grabber-local upload [-h] [--controller-host CONTROLLER_HOST]
[--controller-port CONTROLLER_PORT]
[--exchange EXCHANGE]
[--from-time FROM_TIME]
[--to-time TO_TIME]
Upload dumped candles to kafka via controller
optional arguments:
-h, --help show this help message and exit
--controller-host CONTROLLER_HOST
Target controller host [default: localhost]
--controller-port CONTROLLER_PORT
Target controller port [default: 8080]
--exchange EXCHANGE Exchange name
--from-time FROM_TIME
Start timestamp or isoformat datetime for upload
[default: 2009-01-01T00:00:00]
--to-time TO_TIME End timestamp or isoformat datetime for upload
[default: 2019-10-22]
```