{"id":21901454,"url":"https://github.com/serhatci/web-scraping-from-cryptoexchanges","last_synced_at":"2026-05-07T05:39:07.820Z","repository":{"id":254703596,"uuid":"317538959","full_name":"serhatci/web-scraping-from-cryptoexchanges","owner":"serhatci","description":"A script that collects real-time cryptocurrency price, bid-ask spread, and trade history data from online trade platforms of several different crypto-exchanges and saves it to the MySQL database. ","archived":false,"fork":false,"pushed_at":"2020-12-16T01:19:22.000Z","size":6134,"stargazers_count":2,"open_issues_count":0,"forks_count":1,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-10-03T16:47:42.002Z","etag":null,"topics":["bitcoin","crypto-exchange","real-time","selenium","web-scraping"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/serhatci.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-12-01T12:47:31.000Z","updated_at":"2024-12-22T01:39:37.000Z","dependencies_parsed_at":"2024-08-25T14:28:06.405Z","dependency_job_id":"dcf0e7ab-aab9-4f5f-a109-7d6b745924f5","html_url":"https://github.com/serhatci/web-scraping-from-cryptoexchanges","commit_stats":null,"previous_names":["serhatci/web-scraping-from-cryptoexchanges"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/serhatci/web-scraping-from-cryptoexchanges","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/serhatci%2Fweb-scraping-from-cryptoexchanges","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/serhatci%2Fweb-scraping-from-cryptoexchanges/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/serhatci%2Fweb-scraping-from-cryptoexchanges/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/serhatci%2Fweb-scraping-from-cryptoexchanges/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/serhatci","download_url":"https://codeload.github.com/serhatci/web-scraping-from-cryptoexchanges/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/serhatci%2Fweb-scraping-from-cryptoexchanges/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32725145,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-07T02:14:30.463Z","status":"ssl_error","status_checked_at":"2026-05-07T02:14:29.405Z","response_time":62,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bitcoin","crypto-exchange","real-time","selenium","web-scraping"],"created_at":"2024-11-28T15:13:38.344Z","updated_at":"2026-05-07T05:39:07.804Z","avatar_url":"https://github.com/serhatci.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Web Scraping from Crypto-exchange Websites with Python \u0026 Selenium\n\nThis python script uses selenium browser to collect Price, Bid-Ask spread and Trade history data from online trade platforms of several crypto-exchanges. Data is being collected continually and stored in mySQL database.\n\n![CodeFactor](https://www.codefactor.io/repository/github/serhatci/web-scraping-from-cryptoexchanges/badge)\n\n## Demonstration of the Running Script\n![Running Script Demo](web-scraping.gif)\n\n##  Installation\nYou can clone below repository:  \n`git clone https://github.com/serhatci/web-scraping-from-cryptoexchanges.git`\n\ninstall the requirements:  \n`pip install -r requirements.txt`\n\nBe sure following chrome driver file is in the script folder:  \n`chromedriver.exe`\n\nYou have to provide your own SQL database connection information in the run.py file:\n```\n    #### BELOW INFORMATION SHOULD BE SUBMITTED ###########\n    pair = 'your_downloaded_trade_pair' (For example: btc_usd)\n    db_name = 'your_database_name' (For example: Bitcoin)\n    host = 'your_host'\n    user = 'your_username'\n    password = 'your_passwords'\n    ######################################################\n```\n\nand run the application:  \n`python script/run.py`\n\n##  Customization\nScript uses a JSON file to learn target data for web scraping. Xpath of price, spread and trade history data are declared for each exchange in the script/exchanges.json file as below. You can update the JSON file by adding new exchanges or overwriting the existing ones. Script will handle opening necessary number of the browser windows\u0026tabs and collecting\u0026saving the data.      \n\n```\n\"FTX\": {  \n        \"name\" : \"FTX\",  \n        \"web\": \"https://ftx.com/trade/USDT/USD\",  \n        \"homePageClick\": {  \n            \"clickXpath\":\"//*[@id='root']/div/div[2]/div/header/div[1]/div[2]/button\"},\n        \"price\": {\n            \"clickXpath\": \"\",\n            \"xpath\": \"//*[@id='root']/div/div[2]/div/main/div[1]/div[2]/div/div[3]\"},\n        \"spread\": {\n            \"clickXpath\": \"\",\n            \"calculate\": \"Yes\",\n            \"xpathBuy\": \"/html/body/div/div/div[2]/div/main/div[3]/div[4]/div/div/div[2]/div[2]/table/tbody/tr[1]/td[2]\",\n            \"xpathSell\": \"//*[@id='root']/div/div[2]/div/main/div[3]/div[4]/div/div/div[2]/div[3]/div/table/tbody/tr[1]/td[1]\"},\n        \"trade\": {\n            \"clickXpath\": \"\",\n            \"xpath\": \"/html/body/div/div/div[2]/div/main/div[3]/div[5]/div/table/tbody\",\n            \"order\": [\"Price\",\"Amount\",\"Time\"]}}\n   \n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fserhatci%2Fweb-scraping-from-cryptoexchanges","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fserhatci%2Fweb-scraping-from-cryptoexchanges","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fserhatci%2Fweb-scraping-from-cryptoexchanges/lists"}