https://github.com/serhatci/web-scraping-from-cryptoexchanges

A script that collects real-time cryptocurrency price, bid-ask spread, and trade history data from online trade platforms of several different crypto-exchanges and saves it to the MySQL database.
https://github.com/serhatci/web-scraping-from-cryptoexchanges

bitcoin crypto-exchange real-time selenium web-scraping

Last synced: about 1 month ago
JSON representation

A script that collects real-time cryptocurrency price, bid-ask spread, and trade history data from online trade platforms of several different crypto-exchanges and saves it to the MySQL database.

• Host: GitHub
• URL: https://github.com/serhatci/web-scraping-from-cryptoexchanges
• Owner: serhatci
• License: mit
• Created: 2020-12-01T12:47:31.000Z (about 4 years ago)
• Default Branch: main
• Last Pushed: 2020-12-16T01:19:22.000Z (about 4 years ago)
• Last Synced: 2024-08-25T14:28:02.730Z (5 months ago)
• Topics: bitcoin, crypto-exchange, real-time, selenium, web-scraping
• Language: Python
• Homepage:
• Size: 5.85 MB
• Stars: 3
• Watchers: 3
• Forks: 1
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

        
# Web Scraping from Crypto-exchange Websites with Python & Selenium

This python script uses selenium browser to collect Price, Bid-Ask spread and Trade history data from online trade platforms of several crypto-exchanges. Data is being collected continually and stored in mySQL database.

![CodeFactor](https://www.codefactor.io/repository/github/serhatci/web-scraping-from-cryptoexchanges/badge)

## Demonstration of the Running Script

![Running Script Demo](web-scraping.gif)

##  Installation

You can clone below repository:  

`git clone https://github.com/serhatci/web-scraping-from-cryptoexchanges.git`

install the requirements:  

`pip install -r requirements.txt`

Be sure following chrome driver file is in the script folder:  

`chromedriver.exe`

You have to provide your own SQL database connection information in the run.py file:

```

    #### BELOW INFORMATION SHOULD BE SUBMITTED ###########

    pair = 'your_downloaded_trade_pair' (For example: btc_usd)

    db_name = 'your_database_name' (For example: Bitcoin)

    host = 'your_host'

    user = 'your_username'

    password = 'your_passwords'

    ######################################################

```

and run the application:  

`python script/run.py`

##  Customization

Script uses a JSON file to learn target data for web scraping. Xpath of price, spread and trade history data are declared for each exchange in the script/exchanges.json file as below. You can update the JSON file by adding new exchanges or overwriting the existing ones. Script will handle opening necessary number of the browser windows&tabs and collecting&saving the data.      

```

"FTX": {  

        "name" : "FTX",  

        "web": "https://ftx.com/trade/USDT/USD",  

        "homePageClick": {  

            "clickXpath":"//*[@id='root']/div/div[2]/div/header/div[1]/div[2]/button"},

        "price": {

            "clickXpath": "",

            "xpath": "//*[@id='root']/div/div[2]/div/main/div[1]/div[2]/div/div[3]"},

        "spread": {

            "clickXpath": "",

            "calculate": "Yes",

            "xpathBuy": "/html/body/div/div/div[2]/div/main/div[3]/div[4]/div/div/div[2]/div[2]/table/tbody/tr[1]/td[2]",

            "xpathSell": "//*[@id='root']/div/div[2]/div/main/div[3]/div[4]/div/div/div[2]/div[3]/div/table/tbody/tr[1]/td[1]"},

        "trade": {

            "clickXpath": "",

            "xpath": "/html/body/div/div/div[2]/div/main/div[3]/div[5]/div/table/tbody",

            "order": ["Price","Amount","Time"]}}