{"id":31040466,"url":"https://github.com/qubistlabs/jticker-grabber","last_synced_at":"2025-09-14T08:51:51.191Z","repository":{"id":308152105,"uuid":"158677315","full_name":"qubistlabs/jticker-grabber","owner":"qubistlabs","description":"Jticker grabber app","archived":false,"fork":false,"pushed_at":"2023-03-21T22:38:45.000Z","size":1912,"stargazers_count":0,"open_issues_count":1,"forks_count":0,"subscribers_count":0,"default_branch":"develop","last_synced_at":"2025-08-04T15:33:52.695Z","etag":null,"topics":["backend","jticker","kafka"],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/qubistlabs.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2018-11-22T09:51:18.000Z","updated_at":"2025-06-15T19:31:44.000Z","dependencies_parsed_at":"2025-08-04T15:34:09.653Z","dependency_job_id":"d78311c9-8cbe-4893-a6ee-81764f7947a8","html_url":"https://github.com/qubistlabs/jticker-grabber","commit_stats":null,"previous_names":["qubistlabs/jticker-grabber"],"tags_count":5,"template":false,"template_full_name":null,"purl":"pkg:github/qubistlabs/jticker-grabber","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/qubistlabs%2Fjticker-grabber","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/qubistlabs%2Fjticker-grabber/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/qubistlabs%2Fjticker-grabber/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/qubistlabs%2Fjticker-grabber/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/qubistlabs","download_url":"https://codeload.github.com/qubistlabs/jticker-grabber/tar.gz/refs/heads/develop","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/qubistlabs%2Fjticker-grabber/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":275082356,"owners_count":25402339,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-14T02:00:10.474Z","response_time":75,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["backend","jticker","kafka"],"created_at":"2025-09-14T08:51:48.808Z","updated_at":"2025-09-14T08:51:51.179Z","avatar_url":"https://github.com/qubistlabs.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Jticker grabber\nGrab live candle streams from exchanges.\n\n## Configuration\nAll arguments can be passed as cli arguments or via environment variables. Environment variables\nnaming are same as arguments but uppercased and `-` replaced with `_`. E.g. `--log-level INFO` cli argument\nor `LOG_LEVEL=INFO` environment variable. Environment variabes have **higher** priorty.\n\n### cli spec (version 2.0.0)\n```\n$ python -m jticker_grabber --help\nusage: jticker-grabber       [-h] [--log-level LOG_LEVEL]\n                             [--sentry-dsn SENTRY_DSN]\n                             [--io-network-max-workers IO_NETWORK_MAX_WORKERS]\n                             [--io-file-max-workers IO_FILE_MAX_WORKERS]\n                             [--web-host WEB_HOST] [--web-port WEB_PORT]\n                             [--providers-enabled PROVIDERS_ENABLED]\n                             [--kafka-bootstrap-servers KAFKA_BOOTSTRAP_SERVERS]\n                             [--disable-streaming]\n\noptional arguments:\n  -h, --help            show this help message and exit\n  --log-level LOG_LEVEL\n                        Python logging level [default: INFO]\n  --sentry-dsn SENTRY_DSN\n                        Sentry DSN [default: None]\n  --io-network-max-workers IO_NETWORK_MAX_WORKERS\n                        Max workers for network io executor [default: 100]\n  --io-file-max-workers IO_FILE_MAX_WORKERS\n                        Max workers for file io executor [default: 10]\n  --web-host WEB_HOST   Web server host [default: 0.0.0.0]\n  --web-port WEB_PORT   Web server port [default: 8080]\n  --providers-enabled PROVIDERS_ENABLED\n                        Comma separated exchanges [default: None]\n  --kafka-bootstrap-servers KAFKA_BOOTSTRAP_SERVERS\n                        Comma separated kafka bootstrap servers [default:\n                        kafka:9092]\n  --disable-streaming   Disable initial streaming [default: False]\n```\n\n# Jticker local grabber\nGrab big™ historical data from supported exchanges with proxy support. **Local grabber have no deployment.**\n**It is only for by hand usage.**\n\n### cli spec (version 2.0.0)\n```\n$ python -m jticker_grabber.local --help\nusage: jticker-grabber-local [-h] [--log-level LOG_LEVEL]\n                             [--sentry-dsn SENTRY_DSN]\n                             [--io-network-max-workers IO_NETWORK_MAX_WORKERS]\n                             [--io-file-max-workers IO_FILE_MAX_WORKERS]\n                             [--web-host WEB_HOST] [--web-port WEB_PORT]\n                             [--storage STORAGE]\n                             [--request-timeout REQUEST_TIMEOUT]\n                             [--interval INTERVAL]\n                             {dump,list,upload} ...\n\npositional arguments:\n  {dump,list,upload}\n\noptional arguments:\n  -h, --help            show this help message and exit\n  --log-level LOG_LEVEL\n                        Python logging level [default: INFO]\n  --sentry-dsn SENTRY_DSN\n                        Sentry DSN [default: None]\n  --io-network-max-workers IO_NETWORK_MAX_WORKERS\n                        Max workers for network io executor [default: 100]\n  --io-file-max-workers IO_FILE_MAX_WORKERS\n                        Max workers for file io executor [default: 10]\n  --web-host WEB_HOST   Web server host [default: 0.0.0.0]\n  --web-port WEB_PORT   Web server port [default: 8080]\n  --storage STORAGE     Output directory [default: storage]\n  --request-timeout REQUEST_TIMEOUT\n                        Request timeout [default: 5]\n  --interval INTERVAL   Candles interval [default: 60]\n```\n\n## List\nList available historical exchanges\n```\n$ python -m jticker_grabber.local list --help\nusage: jticker-grabber-local list [-h]\n\nList available exchanges for historical grabbing\n\noptional arguments:\n  -h, --help  show this help message and exit\n```\n## Dump\nDump candles from selected exchanges. After dump finished storage will have `{exchange_name}`\ndirectories with stored candles in sqlite dbs (file per month).\n```\n$ python -m jticker_grabber.local dump --help\nusage: jticker-grabber-local dump [-h] [--proxies-file PROXIES_FILE]\n                                  [--exchanges EXCHANGES]\n                                  [--from-time FROM_TIME] [--to-time TO_TIME]\n                                  [--disable-automatic-start-timestamp]\n                                  [--ignore-zero-volume-candles]\n\noptional arguments:\n  -h, --help            show this help message and exit\n  --proxies-file PROXIES_FILE\n                        File with proxies [default: None]\n  --exchanges EXCHANGES\n                        Comma separated exchanges names [default: None means\n                        `all`]\n  --from-time FROM_TIME\n                        Start timestamp or isoformat datetime for grabbing\n                        [default: 2009-01-01T00:00:00]\n  --to-time TO_TIME     End timestamp or isoformat datetime for grabbing\n                        [default: 2019-10-22]\n  --disable-automatic-start-timestamp\n                        Disable automatic start timestamp\n  --ignore-zero-volume-candles\n                        Ignore candles with base volume equal to zero\n```\n## Upload\nUpload selected exchanges data to kafka via jticker-controller.\n```\n$ python -m jticker_grabber.local upload --help\nusage: jticker-grabber-local upload [-h] [--controller-host CONTROLLER_HOST]\n                                    [--controller-port CONTROLLER_PORT]\n                                    [--exchange EXCHANGE]\n                                    [--from-time FROM_TIME]\n                                    [--to-time TO_TIME]\n\nUpload dumped candles to kafka via controller\n\noptional arguments:\n  -h, --help            show this help message and exit\n  --controller-host CONTROLLER_HOST\n                        Target controller host [default: localhost]\n  --controller-port CONTROLLER_PORT\n                        Target controller port [default: 8080]\n  --exchange EXCHANGE   Exchange name\n  --from-time FROM_TIME\n                        Start timestamp or isoformat datetime for upload\n                        [default: 2009-01-01T00:00:00]\n  --to-time TO_TIME     End timestamp or isoformat datetime for upload\n                        [default: 2019-10-22]\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fqubistlabs%2Fjticker-grabber","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fqubistlabs%2Fjticker-grabber","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fqubistlabs%2Fjticker-grabber/lists"}