Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/den-is/gh-stars-scraper
Scrape your starred repositories
https://github.com/den-is/gh-stars-scraper
github python selenium
Last synced: about 1 month ago
JSON representation
Scrape your starred repositories
- Host: GitHub
- URL: https://github.com/den-is/gh-stars-scraper
- Owner: den-is
- License: mit
- Created: 2024-06-23T16:48:06.000Z (6 months ago)
- Default Branch: main
- Last Pushed: 2024-06-26T05:28:52.000Z (6 months ago)
- Last Synced: 2024-06-26T06:46:56.144Z (6 months ago)
- Topics: github, python, selenium
- Language: Python
- Homepage:
- Size: 7.81 KB
- Stars: 1
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# GitHub Stars scraper
Web-scraper for your own GitHub Stars for further analysis or to recall long-time ago forgotten repos.
For now, script just outputs a JSON file `output.json` with results of scraping, in the same directory with the script.
The script is extremely plain and primitive, and utilizes [Selenium](https://www.selenium.dev/) web-browser automation library.
Contributions are welcome.
## Requirements
- Chrome Browser
- chromedriver
- python 3.9+
## Installation
Instructions shown are for Unix based Operating Systems (specifically MacOS)
Download stable [chromedriver](https://googlechromelabs.github.io/chrome-for-testing/).
Better if version matches to version of the currently installed Chrome Browser.
```sh
cd ~/Downloads
curl -OL https://storage.googleapis.com/chrome-for-testing-public/126.0.6478.63/mac-x64/chrome-mac-x64.zip
unzip chrome-mac-x64.zip
# on MacOS you might need to run
cd chromedriver-mac-x64
xattr -d com.apple.quarantine chromedriver
```
Scraper setup
```sh
git clone https://github.com/den-is/gh-stars-scraper.git
cd gh-stars-scraper
python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt
```
## Running scraper
- Create `.env` file with correct values, from provided `.env.example`.
- Run script `python3 main.py`
- Wait script to open browser window and open login page
- Provide OTP code (I hope you have protected your github account with 2FA authentication)
- Watch magic to happen
- Do not interact with the web page
## Known-issues
- For some reason, rarely script is not able to fetch lists some specific repo belongs to.
- Web pages evolve. New features and versions are constantly released. That means that scraper's logic might and will degrade over time, and will require maintenance.