{"id":20486212,"url":"https://github.com/ivopetiz/crypto-database","last_synced_at":"2025-04-13T15:20:49.199Z","repository":{"id":144313995,"uuid":"122538050","full_name":"ivopetiz/crypto-database","owner":"ivopetiz","description":"Database for crypto data, supporting several exchanges. Can be used for TA, bots, backtest, realtime trading, etc.","archived":false,"fork":false,"pushed_at":"2019-10-04T13:06:19.000Z","size":40,"stargazers_count":85,"open_issues_count":1,"forks_count":31,"subscribers_count":5,"default_branch":"master","last_synced_at":"2023-03-21T20:46:49.827Z","etag":null,"topics":["binance","bitcoin","bittrex","crypto","cryptocoins","cryptocurrency-exchanges","cryptopia","database","ethereum","fundamental-analysis","hft","hft-trading","high-frequency-trading","poloniex","ripple","technical-analysis","trading","trading-bot"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ivopetiz.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-02-22T21:34:11.000Z","updated_at":"2024-06-19T06:22:28.563Z","dependencies_parsed_at":null,"dependency_job_id":"e5d4ebe3-7f81-4bf6-a6a7-a663e0590107","html_url":"https://github.com/ivopetiz/crypto-database","commit_stats":null,"previous_names":[],"tags_count":0,"template":null,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ivopetiz%2Fcrypto-database","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ivopetiz%2Fcrypto-database/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ivopetiz%2Fcrypto-database/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ivopetiz%2Fcrypto-database/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ivopetiz","download_url":"https://codeload.github.com/ivopetiz/crypto-database/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248733080,"owners_count":21152954,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["binance","bitcoin","bittrex","crypto","cryptocoins","cryptocurrency-exchanges","cryptopia","database","ethereum","fundamental-analysis","hft","hft-trading","high-frequency-trading","poloniex","ripple","technical-analysis","trading","trading-bot"],"created_at":"2024-11-15T16:35:45.069Z","updated_at":"2025-04-13T15:20:49.157Z","avatar_url":"https://github.com/ivopetiz.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![Codacy Badge](https://api.codacy.com/project/badge/Grade/9243d193bbe34717978b72b0477df4d2)](https://app.codacy.com/app/ivopetiz/crypto-database?utm_source=github.com\u0026utm_medium=referral\u0026utm_content=ivopetiz/crypto-database\u0026utm_campaign=Badge_Grade_Dashboard)\n\n# Crypto-database\n\nDatabase to store all data from crypto exchanges, currently working with Binance, Bittrex, Cryptopia and Poloniex. \n\nCan be used for technical analysis, bots, backtest, realtime trading, etc.\n\n## Installation\n\n\n-   [BD Instalation](#bd-installation)\n    -   [Docker](##docker)\n\n    -   [Native](##native)\n        -   [BD Configuration](#BD-Configuration)\n\n-   [Market Prices To DB](#market-prices-to-db)\n    -   [Data Interval](##data-interval)\n\n-   [Balance To DB](#balance-to-db)\n\n-   [Using Chronograf](#using-chronograf)\n\n-   [TODO](#todo)\n\nThis install guide was made for **Ubuntu 16.04+**. Will need some adjustments to work with other distros.\n\n## BD Installation\n\n### Docker\n\nDocker directory has a default configuration that allows users to implement an pre configured database, ready to receive data from Exchanges and use it.\nIn order to use Influxdb Docker container is only necessary to follow the steps bellow.\n\n```bash\ngit clone https://github.com/ivopetiz/crypto-database.git\ncd crypto-database\n. docker/build\n. docker/start\n```\nAnd your Influxdb crypto database throw Docker container should be ready to be used.\n\n### Native\n\nStart by installing Golang, to build the applications responsible for populate Crypto-database.\n\n```bash\nsudo apt-get install golang\n```\n\nAfter install Golang, you will need to install InfluxDB. Chronograf is also recomended.\n\n```bash\nsudo apt-get install influxdb influxdb-client chronograf\n```\n\nYou can also get InfluxDB last versions from Influx website. \nThis version of Crypto-database where tested with Influxdb 1.5.3 and Chronograf 1.5.0. You can install these packages by running:\n\n```bash\nwget https://dl.influxdata.com/influxdb/releases/influxdb_1.5.3_amd64.deb\nsudo dpkg -i influxdb_1.5.3_amd64.deb \nwget https://dl.influxdata.com/chronograf/releases/chronograf_1.5.0.0_amd64.deb\nsudo dpkg -i chronograf_1.5.0.0_amd64.deb\n```\n\n### BD Configuration\n\nStart Influxdb and then run Influxdb prompt.\n\n```bash\nsudo systemctl enable influxdb.service\nsudo systemctl start influxdb.service\ninflux\n```\n\nCreate databases and user with privileges to use databases:\n\n```influx\nCREATE DATABASE altcoin\nCREATE DATABASE balance\nUSE altcoin\nCREATE USER \u003cusername\u003e WITH PASSWORD '\u003cpassword\u003e' WITH ALL PRIVILEGES\nGRANT ALL PRIVILEGES TO \u003cusername\u003e\n```\n\n## Market Prices To DB\n\nBefore build crypto markets and balance applications is necessary to clone this repository.\n\n```bash\ngit clone https://github.com/ivopetiz/crypto-database.git\n```\n\n After cloning this rep, is necessary to configure Influxdb user, password and server on system. One option is to define variables on system in order to hide it from git. Variables can also be defined on **consts.go**.\n\n```bash\nDBUSER=\u003cyour-db-user\u003e\nDBPASS=\u003cyour-db-password\u003e\nSERVERDB=\u003cyour-db-server\u003e\n```\n\n---\n### Data Interval\n\n*By default, data is recorded every 10 seconds but this value can be changed. Timeout can be defined on **consts.go**. A timeout too big won't present fast changes on prices. By the other hand, a timeout too small will make your IP address blocked on crypto exchanges, that only allows a certain number of request per minute. All exchanges have different limits and you can consult these values on exchanges API official websites.*\n\n---\n\nLog file will be stored on **/log/altdb_coin.log**. This path can be changed on **main.go**. Make sure you have the right privileges to write in **/log/**.\n\n```bash\nsudo mkdir /log\nsudo touch /log/altdb_coin.log\nsudo chown \u003cuser\u003e:\u003cuser\u003e /log/altdb_coin.log\nsudo chmod 644 /log/altdb_coin.log\n```\n\nCompile market data getter executable.\n\n```bash\ncd markets\ngo get\ngo build -o markets -ldflags=\"-s -w\" main.go consts.go\n```\n\nGo lang will return an executable file called **markets**. Now you need to run in order to populate database. One option is to use **markets** as a service, in order to keep tracking of it. To run **markets** as a service is necessary to create **/etc/systemd/system/cryptomarket.service** with the following content:\n\n```bash\n[Unit]\nDescription=Service running a crypto market data getter.\nAfter=network.target\n\n[Service]\nType=simple\nUser=user\nWorkingDirectory=/home/user/crypto-database\nExecStart=/home/user/crypto-database/markets\n\nRestart=always\n\n[Install]\nWantedBy=multi-user.target\n```\n\nAfter save **cryptomarket.service** file, run the commands bellow:\n\n```bash\nsudo systemctl daemon-reload\nsudo systemctl enable cryptomarket.service\nsudo systemctl start cryptomarket.service\n```\n\nNow **markets** will run as a service, starting when OS initializes and recovers in case of failure.\n\n## Balance To DB\n\nIf you want to add your balance to DB, you will need to generate your API key on crypto exchanges, in order to validate your login and get your data. Currently working only with Bittrex exchange.\n\nTo generate an API key and secret, you can access crypto exchanges website, on definitions part. Make sure you keep your API key and API secret **secret** and give this key minimum permissions, in order to obtain just balance info. With bad permissions, any person with this key and secret can consult, trade or withdraw your coins.\n\nAfter get API key and secret from exchanges, you need to add it to **balance.go**. One option is to define system variables:\n\n```bash\nBITTREX_API_KEY=\u003cyour-bittrex-api-key\u003e\nBITTREX_API_SECRET=\u003cyour-bittrex-api-secret\u003e\n```\n\nAfter define **API_KEY** and **API_SECRET** in **balance.go** is necessary to build **balance** program:\n\n```bash\ncd balance\ngo get\ngo build -o balance -ldflags=\"-s -w\" balance.go\n```\n\nOnce **balance** is not supposed to run every minute it wasn't built with a timeout, so user need to run it everytime. This can be done automatically with a Crontab rule or as a service, like was done with **markets.service**.\n\nTo add new balance info to DB every hour, add a crontab rule by running **crontab -e** and add the following line:\n\n```bash\n0 * * * *   /home/user/crypto-database/balance\n```\n\nThe above rule will run **balance** every hour at :00.\n\n## Using Chronograf\n\nChronograf presents crypto data from Influxdb. Can be particularly useful to plot data or to quick check market prices. Chronograf is easy to use and it's only necessary to configure with your Influxdb definitions.\n\nIn order to start service, after installation will need to run:\n\n```bash\nsudo systemctl start chronograf.service\n```\n\nIf you use Influxdb in the same machine as Chronograf service and have used the default configs, Influxdb will be in https://localhost:8086.\n\nYou will also need to input your Influxdb user and password in order to give Chronograf access to DB.\n\nIf Influxdb and Chronograf services are in diferent machines, you will need to change localhost by Influxdb machine IP address.\n\nChronograf service uses port **8888**. Port can be changed on **/etc/default/chronograf**.\n\nAnyone can access Chronograf but is possible to block access from other machines. In config file you can change all configs.\n\nBy default, Chronograf config file will be looking like this:\n\n```bash\nHOST=0.0.0.0\nPORT=8888\n```\n\n**HOST** will define who can access Chronograf website. It can be blocked to localhost machine or other specific IP address which can be a good option in terms of security.\n\n---\n## TODO\n\n-   add more exchanges to Balance\n-   makefile\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fivopetiz%2Fcrypto-database","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fivopetiz%2Fcrypto-database","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fivopetiz%2Fcrypto-database/lists"}