https://github.com/alincode/etherscan-scraper

Scrape smart contract data off Etherscan and import into BlockScout
https://github.com/alincode/etherscan-scraper

Last synced: 11 days ago
JSON representation

Scrape smart contract data off Etherscan and import into BlockScout

• Host: GitHub
• URL: https://github.com/alincode/etherscan-scraper
• Owner: alincode
• License: apache-2.0
• Created: 2019-07-06T17:52:18.000Z (over 5 years ago)
• Default Branch: master
• Last Pushed: 2019-07-06T17:55:14.000Z (over 5 years ago)
• Last Synced: 2024-11-06T07:27:19.426Z (about 2 months ago)
• Language: TSQL
• Homepage:
• Size: 18.1 MB
• Stars: 18
• Watchers: 1
• Forks: 11
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE-APACHE

Awesome Lists containing this project

README

        
# Etherscan Contract Scraper

## Verified Contract Stats

| Verified      | Failed    | Total Contracts Checked  |

| ------------- |:---------:| ------------------------:|

| 56,251        | 19,030    | 227,715                  |

## Project Goals

The goal of this project is to scrape verified contracts from Etherscan.io and import them onto Blockscout.com. Without the ability to scrape more than 1,000 records from the verified smart contracts page (https://etherscan.io/contractsVerified) on Etherscan, we need to check every contract address from every block. 

After the initial catch-up period, the script will check https://etherscan.io/contractsVerified and monitor this page every hour looking for new verified contracts. 

The script will also check all new incoming blocks for any verified contracts. 

## Requirements

* MYSQL (`brew install mysql`)

* NODEJS 10+ (`brew install node`)

## Installation

* Setup MYSQL Database

  * From the command line, `CREATE DATABASE etherscan_contracts;`

  * `USE etherscan_contracts;`

  * Create the [addresses](https://github.com/acravenho/etherscan-scraper/blob/master/mysql/addresses.sql) and [blocks](https://github.com/acravenho/etherscan-scraper/blob/master/mysql/bblocks.sql) tables

  * Edit the MYSQL connection configuration located [here](https://github.com/acravenho/etherscan-scraper/blob/master/resources/database.js) 

* Run the script, `node app.js`

## How it Works

1. First, the script will check the DB for any pending addresses. 

  * Checks Etherscan.io to obtain the source code. If the address is not verified, the `checked` column in the DB is updated to `1`. This address will not be checked again (can be changed in the future). 

  * If the contract is verified on Etherscan, the source code, name, compiler version, and optimizations are extracted from the page.

  * The verified contract is then imported into BlockScout (https://github.com/acravenho/etherscan-scraper/issues/2)

2. The script will do an initial indexing from the last block that was indexed. All smart contract addresses are entered into the DB by each block.

3. All 40 pages of the Verified Contracts pages are indexed

4. Every hour the Verified Smart Contracts page is monitored

5. Every minute new blocks are checked for smart contracts.