https://github.com/marigarey/devpost-hackathon-scraper
Collects the project submissions from a specific hackathon
https://github.com/marigarey/devpost-hackathon-scraper
devpost devpost-hackathon webscraper webscraping
Last synced: about 1 month ago
JSON representation
Collects the project submissions from a specific hackathon
- Host: GitHub
- URL: https://github.com/marigarey/devpost-hackathon-scraper
- Owner: marigarey
- License: mit
- Created: 2026-04-21T18:29:53.000Z (3 months ago)
- Default Branch: main
- Last Pushed: 2026-04-22T15:38:11.000Z (3 months ago)
- Last Synced: 2026-04-22T17:35:30.986Z (3 months ago)
- Topics: devpost, devpost-hackathon, webscraper, webscraping
- Language: Python
- Homepage:
- Size: 59.6 KB
- Stars: 2
- Watchers: 0
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Devpost Hackathon Scraper
Collects the project submissions from a specific hackathon
## Setting Up Dependencies
This project uses uv to help with dependencies, to install all the proper dependencies on your local version run the following:
```zsh
uv venv
# macOS/Linux
source .venv/bin/activate
# windows
.venv\Scripts\activate
# installing dependencies
uv pip install -r pyproject.toml
# if needed also run
uv sync
```
## Running the Script
Give the hackathon you want to scrape, find the subdomain of the hackathon on devpost. For example: the subdomain of `madhacks-fall-2025.devpost.com` is `madhacks-fall-2025`.
Once the subdomain is found we can run the script!
```zsh
uv run src/main.py ""
# example
uv run src/main.py "madhacks-fall-2025"
```
The output of the data will be found in `/data/.json`
## Testing
To verify the scraper is working correctly, you can run the included test files:
```zsh
# Test parsing a single project
uv run test/test_single.py
# Test parsing multiple projects
uv run test/test_multiple.py
# Test with full data extraction logic
uv run test/test_full.py
```
These tests will fetch and parse sample projects to confirm the scraper is functioning properly.