{"id":13725280,"url":"https://github.com/kruglov-dmitry/crypto_crawler","last_synced_at":"2026-02-01T19:32:22.511Z","repository":{"id":40392311,"uuid":"208153459","full_name":"kruglov-dmitry/crypto_crawler","owner":"kruglov-dmitry","description":"Crypto arbitrage bot","archived":false,"fork":false,"pushed_at":"2019-09-17T06:23:50.000Z","size":998,"stargazers_count":128,"open_issues_count":1,"forks_count":27,"subscribers_count":8,"default_branch":"master","last_synced_at":"2025-06-15T17:09:11.437Z","etag":null,"topics":["arbitrage-bot","asynchronous-programming","cryptotrading","cryptotradingbot","microservices-architecture","python27"],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/kruglov-dmitry.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-09-12T22:00:20.000Z","updated_at":"2025-04-07T14:49:13.000Z","dependencies_parsed_at":"2022-09-06T08:10:42.488Z","dependency_job_id":null,"html_url":"https://github.com/kruglov-dmitry/crypto_crawler","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/kruglov-dmitry/crypto_crawler","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kruglov-dmitry%2Fcrypto_crawler","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kruglov-dmitry%2Fcrypto_crawler/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kruglov-dmitry%2Fcrypto_crawler/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kruglov-dmitry%2Fcrypto_crawler/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/kruglov-dmitry","download_url":"https://codeload.github.com/kruglov-dmitry/crypto_crawler/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kruglov-dmitry%2Fcrypto_crawler/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28987278,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-01T18:17:03.387Z","status":"ssl_error","status_checked_at":"2026-02-01T18:16:57.287Z","response_time":56,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["arbitrage-bot","asynchronous-programming","cryptotrading","cryptotradingbot","microservices-architecture","python27"],"created_at":"2024-08-03T01:02:18.136Z","updated_at":"2026-02-01T19:32:22.493Z","avatar_url":"https://github.com/kruglov-dmitry.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"# Project structure\n\nSystem consist from multiple independent processes that may be run on different node.   \nCommunication is implemented on top of various message queues on top of redis.    \nRedis is also used as distributed caching layer - to reflect actual balance across all exchanges.    \nData in queues usually pickled.  \nPersistence for historical data - postgres.    \nAlerting is implemented on top of Telegram messenger.   \nCLI management is done manually using scripts within this project.    \n\nNOTE: Web based UI and data scrapping framework from various resources are NOT part of this repo.\n\n## Terminology\n- _**bot**_ - arbitrage process that trade single commodity between exactly two exchanges\n- _**order**_ - what bot is placed to markets\n- _**trade**_ - what was actually executed at markets, single order may be closed by multiple trades\n\n\n## Key services:\n- _**telegram notifier**_ - watch message queues and forward messages based on their severity \nto corresponding Telegram channels\n- _**balance_monitoring**_ - update cached values of balances for all currencies for all exchanges in portfolio\nAll trading processes validate time of last update and will immediately stopped if it expired.\n- _**expired order processing**_ - due to many reasons order placed by bot may be not closed in time,\nthis service re-process them trying to minimise loss \n- _**failed order processing**_ - timeout, exchange errors and ill fate may lead to situation that \nwe got errors as result of order placement request. This service carefully check current state of order:\nwhether it was registered within exchange or not, whether it was fulfilled or not.\n\n## Data analysis:\n- _**order saving**_ - save all orders that were placed by arbitrage processes into Postgres db\n- _**bot trade retrieval**_ - retrieve information from all exchanges in regards to recently executed trades. \n- _**arbitrage monitoring**_ - read tickers for all supported currency across all exchanges and \nissue a telegram notification about direct arbitrage opportunities\n\n\n\nYou may find all available services under services packages.\n\n# DEPLOY - HOWTO\n\n## Prerequisites:\nNode with data and cache:\n* redis in place\n```bash\nsudo service docker start\ncd ~/crypto_deploy/redis\n# NOTE: you may want to edit path to mounted volume for data persistence \nsudo docker-compose -f redis_compose.yml up\n# NOTE: redis IP is hardcoded there:\npython redis_queue.py\n```\n* postgres in place with proper schema and data:\n```bash\nsudo service docker start\ncd ~/crypto_deploy/postgres\n# NOTE: you may want to edit path to mounted volume for data persistence\nsudo docker-compose -f docker-compose-postgres.yml up\npsql -h 127.0.0.1 -Upostgres -f schema/schema.sql\npsql -h 127.0.0.1 -Upostgres -f schema/data.sql\n```\n* data retrieval \u0026 nodes with bot processes:\n```bash\nyum groupinstall \"Development Tools\"\npip install -r requirements.txt\n``` \n* copy `common_sample.cfg` to `common.cfg` and update it with proper public IP addresses and domain names\n* make sure that firewall rules at aws allow incoming connections from bot nodes to data node\n\n## Deploying data retrieval services\n\nWill deploy: \n* order_book, history, candles, \n* arbitrage notifications based on tickers\n* telegram notifications\n```bash\npython deploy_data_retrieval.py\n```\n\n## Deploying arbitrage bots\n1. verify settings at config file:\nmore deploy/deploy.cfg\n2. Initiate deployment processes\npython deploy_arbitrage_bots.py deploy/deploy.cfg\n\n### How to run dedicated services from subfolder:\n``` bash\npython -m services.telegram_notifier\n```\n## Kill ALL processes\n``` bash\nps -ef | grep arbitrage | awk '{print $2}' | xargs kill -9 $1\n```\nor just\n``` bash\npkill python\n```\n\n## Kill ALL screens with all session MacOs\n``` bash\nscreen -ls | awk '{print $1}' | xargs -I{} screen -S {} -X quit\n```\nscreen -ls | grep -v deploy | awk '{print $1}' | xargs -I{} screen -S {} -X quit\nbased on  https://stackoverflow.com/questions/1509677/kill-detached-screen-session\n\n``` bash\nalias cleanscreen=\"screen -ls | tail -n +2 | head -n -1|cut -d'.' -f 1 |xargs kill -9 ; screen -wipe\"\n\nalias bot_count='ps -ef | grep python | wc -l'\nalias bot_kill='pkill python'\nalias bot_stop_screen=\"screen -ls | tail -n +2 | head -n -1|cut -d'.' -f 1 |xargs kill -9 ; screen -wipe\"\n```\n\n## Rename existing screen session\n``` bash\nscreen -S old_session_name -X sessionname new_session_name\n```\n\n### REMOTE_ACCESS:\n``` bash\nssh -v -N -L 7777:192.168.1.106:5432 86.97.142.164 -i .ssh/crptdb_sec_openssh -l dima -p 8883\nssh -i .ssh/crptdb_sec_openssh -v dima@86.97.142.164 -p 8883\nssh dima@86.96.108.235 -p 8883\n```\n\n### MacOs dependencies:\n``` bash\npip install python-telegram-bot --user\n```\n\n### redis\ntype \u003ckey\u003e\nand depending on the response perform:\n\nhttps://lzone.de/cheat-sheet/Redis\nfor \"string\": get \u003ckey\u003e\nfor \"hash\": hgetall \u003ckey\u003e\nfor \"list\": lrange \u003ckey\u003e 0 -1\nfor \"set\": smembers \u003ckey\u003e\nfor \"zset\": zrange \u003ckey\u003e 0 -1 withscores\n\n### POSTGRES\nhttps://wiki.postgresql.org/wiki/Deleting_duplicates\n\nPostgres backups:\n``` bash\npg_dump -h 192.168.1.106 -p 5432 -U postgres -F c -b -v -f \"/home/dima/full_DDMMYYYY\"\npg_dump -h 192.168.1.106 -p 5432 -U postgres -s public\n-- How to do full dump without particular tables\npg_dump -h 192.168.1.106 -p 5432 -U postgres -F c -b -v --exclude-table=alarams --exclude-table=tmp_binance_orders --exclude-table=tmp_history_trades --exclude-table=tmp_trades --exclude-table=trades -f \"/home/dima/full_DDMMYYYY\"\n```\n\nAWS:\n``` bash\npsql --host=orders.cervsj06c8zw.us-west-1.rds.amazonaws.com --port=5432 --username=postgres --password --dbname=crypto\n```\n\n\n### TELEGRAM BOT\nHow to get ID of telegram chat:\nhttps://api.telegram.org/bot\u003cYourBOTToken\u003e/getUpdates\n\n\"\"\"\nHow to check what the fuck is happening with the bot:\nhttps://api.telegram.org/bot438844686:AAE8lS3VyMsNgtytR4I1uWy4DLUaot2e5hU/getUpdates\n\n\"\"\"\n\n\n### Get all tradable pairs\nhttps://www.binance.com/api/v1/ticker/allBookTickers\nhttps://bittrex.com/api/v1.1/public/getmarkets\n\nhttps://api.kraken.com/0/public/Assets\nhttps://api.kraken.com/0/public/AssetPairs\n\nhttps://poloniex.com/public?command=returnTicker\n\nhttp://api.huobi.pro/v1/common/symbols\n\n\n### Socket subscriptions endpoints:\nwraping out websocket into class with callbacks:\nhttps://github.com/websocket-client/websocket-client\n\nexchanges API:\nhttps://bittrex.github.io/#ws-api-overview\nhttps://github.com/binance-exchange/binance-official-api-docs/blob/master/web-socket-streams.md\nhttps://github.com/huobiapi/API_Docs_en/wiki/Huobi-API\nhttps://poloniex.com/support/api/ - not too much info\nkraken - na\n\nExamples of implementation:\nhttps://github.com/s4w3d0ff/python-poloniex\nhttps://github.com/absortium/poloniex-api\nhttps://github.com/czheo/huobi-client-python\nhttps://github.com/sammchardy/python-binance\nhttps://github.com/slazarov/python-bittrex-websocket\n\n\n### Rounding rules\n\nhttps://support.binance.com/hc/en-us/articles/115000594711-Trading-Rule\n\n\n### Setup balance monitoring from the scratch\n``` bash\nsudo curl -L https://github.com/docker/compose/releases/download/1.18.0/docker-compose-`uname -s`-`uname -m` -o /usr/local/bin/docker-compose\nsudo yum install docker, mc, git\nsudo service docker start\nsudo /usr/local/bin/docker-compose -f docker_compose.yml up\nscp -i wtf.pem -r crypto_crawler/secret_keys/ ec2-user@ec2-54-183-153-123.us-west-1.compute.amazonaws.com:/tmp/\n```\n\n\n### sysops\n``` bash\nsudo logrotate -s /var/log/logstatus /etc/logrotate.conf\n/home/ec2-user/crypto_crawler/logs/*.log {\n    size 10M\n    compress\n    rotate 10\n    nodateext\n    missingok\n    notifempty\n}\n\nsudo vim /etc/crontab\n*/5 * * * * root logrotate -f /etc/logrotate.conf\n\nsudo service crond restart\n```\n\nhttps://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_VPC.Scenarios.html#USER_VPC.Scenario3\n\n\n### Logs analysis\n\nHow to find last modified files recursively:\n``` bash\nfind $1 -type f -print0 | xargs -0 stat --format '%Y :%y %n' | sort -nr | cut -d: -f2- | head\n```\n\nHow to merge all files in single file sorted by numerical indexes:\n```bash\nls socket_errors.log* | sort -Vr | xargs cat \u003e history.log\n```\n\nHow to find all lines in log containing PID and sort entries by time:\n```bash\ngrep 'PID: 9848' *.log* | sed 's/:/ : /'  | sort -k 3 \u003e 9848_sorted_by_time.log\n```\n\nHow to select log entries that are within particular time range:\n```bash\nawk '($3 \u003e= 1553066553) \u0026\u0026 ($3\u003c=1553066599)' 9848_1.log \u003e suspect.log\n```\n\nHow to build processing histogram:\n```bash\nhead all_profile.log\n1553052602 :  PID: 19399 Start: 1553052602200 ms End: 1553052602201 ms Runtime: 1 ms\n1553052602 :  PID: 19115 Start: 1553052602187 ms End: 1553052602201 ms Runtime: 14 ms\n1553052602 :  PID: 18629 Start: 1553052602201 ms End: 1553052602202 ms Runtime: 1 ms\n\nmore all_profile.log | awk '{ print $12 }' | sort -n | uniq -c\n\n```\n\n\n### Anaconda profit report How-TO Windows\n1. Install https://www.anaconda.com/download/ for 2.7 Python\n2. Run  Start-\u003ePrograms-\u003eAnaconda Prompt\n3. Install necessary dependencies using pip:\n``` bash\n    pip install redis tqdm\n```\n4. Run Start-\u003ePrograms-\u003eJupiter Notebook\n5. Open Notebook from ipython_notebooks/iPython_local_Input.ipynb\n6. Adjust following parameters:\n* CRYPTO_MODULE\n* should_fetch_data\n* time_end\n* time_start\n* api_key_full_path\n7. Sequentially execute all sells\n8. Profit report should be under your %HOME%/logs folder\n\n\n\n### How to setup dynuiuc domain name update\n```bash\nmore /usr/lib/systemd/system/dynuiuc.service\n[Unit]\nDescription=Dynu\n\n[Service]\nType=forking\nPIDFile=/var/run/dynuiuc.pid\nExecStart=/usr/bin/dynuiuc --conf_file /etc/dynuiuc/dynuiuc.conf --log_file /var/log/dynuiuc.log --pid_file /var/run/dynuiuc.pid --daemon\nExecReload=/bin/kill -HUP $MAINPID\n# DK manually\nRestart=always\n\n[Install]\nWantedBy=multi-user.target\n```\nsudo systemctl enable dynuiuc.service\nsudo service dynuiuc start\n\n\n###\npython -m services.arbitrage_between_pair_subscription --threshold 1.2 --reverse_threshold 0.71 --balance_threshold 15 --sell_exchange_id 4 --buy_exchange_id 4 --pair_id 1 --deal_expire_timeout 15 --cfg deploy/deploy.cfg\n\n### Postgres various\n```sql\n-- avg amount of records per table\nSELECT schemaname,relname,n_live_tup \n  FROM pg_stat_user_tables \n  ORDER BY n_live_tup DESC;\n```","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkruglov-dmitry%2Fcrypto_crawler","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkruglov-dmitry%2Fcrypto_crawler","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkruglov-dmitry%2Fcrypto_crawler/lists"}