https://github.com/arkorty/booststats-scraper
A Go application that scrapes profile information from a list of URLs, processes user data, and exports it to JSON. This tool is designed for efficient data collection and structured output for further analysis or reporting.
https://github.com/arkorty/booststats-scraper
golang goquery
Last synced: over 1 year ago
JSON representation
A Go application that scrapes profile information from a list of URLs, processes user data, and exports it to JSON. This tool is designed for efficient data collection and structured output for further analysis or reporting.
- Host: GitHub
- URL: https://github.com/arkorty/booststats-scraper
- Owner: arkorty
- License: mit
- Created: 2024-11-01T15:05:36.000Z (over 1 year ago)
- Default Branch: master
- Last Pushed: 2024-11-16T07:41:06.000Z (over 1 year ago)
- Last Synced: 2025-02-01T02:43:30.451Z (over 1 year ago)
- Topics: golang, goquery
- Language: Go
- Homepage:
- Size: 23.4 KB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Scraper
A Go application that scrapes profile information from a list of URLs, processes user data, and exports it to JSON. This tool is designed for efficient data collection and structured output for further analysis or reporting.
---
### Features
- **Automated Web Scraping**: Collects profile data from multiple URLs with a single command.
- **CSV Input, JSON Output**: Accepts a CSV file with user information and outputs structured JSON data.
- **Data Validation**: Ensures only valid records are processed and logs any issues encountered.
- **Detailed User Profiles**: Extracts essential details such as completed assignments, badges, and counts.
- **Configurable Assignments List**: Allows customization of assignment tracking based on user needs.
- **Error Handling & Logging**: Provides clear logging of errors and successes for each profile processed.
### Installation
1. Clone the repository:
```bash
git clone https://github.com/arkorty/booststats-scraper.git
```
2. Navigate to the project directory:
```bash
cd booststats-scraper
```
3. Build the project:
```bash
go build ./cmd/scraper
```
### Usage
To run the scraper, use the following command:
```bash
./scraper
```
- ``: Path to the CSV file containing user names, emails, and profile URLs.
- ``: Path where the JSON output will be saved.
Example:
```bash
./scraper input.csv output.json
```
### Contributing
We welcome contributions! Please fork this repository and submit a pull request if you have any improvements or fixes to suggest.
### License
This project is licensed under the MIT License. See [LICENSE](LICENSE) for details.