Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/ademchaoua/walletscrapper
A Puppeteer scraper that collects top trader data from gmgn.ai, processes coin rankings, filters duplicates, and extracts trader details like address and profit.
https://github.com/ademchaoua/walletscrapper
crypto crypto-bot javascript nodejs puppeteer scraper scraping scrapper scrapping wallet
Last synced: about 22 hours ago
JSON representation
A Puppeteer scraper that collects top trader data from gmgn.ai, processes coin rankings, filters duplicates, and extracts trader details like address and profit.
- Host: GitHub
- URL: https://github.com/ademchaoua/walletscrapper
- Owner: ademchaoua
- Created: 2025-01-21T10:05:18.000Z (2 days ago)
- Default Branch: main
- Last Pushed: 2025-01-21T10:22:56.000Z (2 days ago)
- Last Synced: 2025-01-21T11:27:58.542Z (2 days ago)
- Topics: crypto, crypto-bot, javascript, nodejs, puppeteer, scraper, scraping, scrapper, scrapping, wallet
- Language: JavaScript
- Homepage:
- Size: 6.41 MB
- Stars: 1
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Puppeteer Web Scraper for Gmgn AI
This project is a web scraper built using Puppeteer and Puppeteer Extra with the Stealth Plugin to scrape data from the Gmgn AI platform. The scraper collects information about the top Solana (SOL) token swaps and associated trades, processes it, and saves the results into a JSON file.
## Features
- Scrapes the top Solana token swaps data.
- Fetches associated trade data for each token.
- Deduplicates the trade data by ensuring only unique addresses are included.
- Saves the collected data into a `smarts_data.json` file.
## Requirements
- Node.js (v14 or higher recommended)
- npm (Node Package Manager)
- Puppeteer and Puppeteer Extra plugins
## Setup
1. **Clone the repository:**
```bash
git clone https://github.com/ademchaoua/WalletScrapper
cd WalletScrapper
```
2. **Install the required dependencies:**
Make sure Node.js and npm are installed. If not, download and install them from [nodejs.org](https://nodejs.org/).
Then, run the following command to install the necessary dependencies:
```bash
npm install
```
3. **Run the scraper:**
After installing the dependencies, you can run the script with:
```bash
node gmgn.js
```
This will start the scraping process and the data will be saved in the `smarts_data.json` file.
## How it Works
1. The script opens the Gmgn AI website for the specified token swaps page.
2. It fetches a list of top tokens by swap volume.
3. For each token, it collects trading data about the top traders.
4. It processes the data and deduplicates it by eliminating trades with repeated addresses.
5. Finally, the processed data is saved in a `smarts_data.json` file.
## Error Handling
The scraper is designed to handle errors gracefully:
- If a coin's page cannot be opened or processed, it will log the error and move on to the next token.
- At the end of the process, any errors that occurred during the execution will be logged.
## Output
- The data is saved to a file named `smarts_data.json`.
- The file contains a JSON structure with the following format:
```json
[
{
"coin": "coin_address",
"trades": [
{
"address": "trade_address",
"solAddress": "native_transfer_from_address",
"profit": "realized_profit",
"timestamp": "trade_timestamp"
}
]
}
]
```
## License
This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.