https://github.com/umaar/export-github-stars
A Node.js webapp to scrape the stars of specified GitHub users
https://github.com/umaar/export-github-stars
Last synced: 6 days ago
JSON representation
A Node.js webapp to scrape the stars of specified GitHub users
- Host: GitHub
- URL: https://github.com/umaar/export-github-stars
- Owner: umaar
- Created: 2019-03-23T10:59:34.000Z (about 6 years ago)
- Default Branch: master
- Last Pushed: 2023-01-11T03:12:55.000Z (over 2 years ago)
- Last Synced: 2025-04-08T03:13:02.140Z (about 1 month ago)
- Language: JavaScript
- Homepage:
- Size: 5.2 MB
- Stars: 40
- Watchers: 0
- Forks: 3
- Open Issues: 12
-
Metadata Files:
- Readme: readme.md
Awesome Lists containing this project
README
### GitHub Star Export
[](https://github.com/umaar/export-github-stars/actions)
Export _all_ your GitHub stars.
This webapp queries the GitHub API to query for the stars of a GitHub user, and stores them in an SQLite database. 10,000 stars will occupy ~6mb worth of data.
This repo uses persistent queues, and supports resumable downloading of stars. You can fetch stars of many users.
#### To run
1. Get a GitHub token from https://github.com/settings/tokens
2. Create a `.env` file:
```
# Contents of .env
GITHUB_TOKEN=your token here
# Any name for your new database
DATABASE_NAME=output
```
3. Run these commands:
```sh
npm i
make migrate-db-dev
make start
```
### TODO:
- Send emails using an email service about database size changes
- Automatically update user stars every ~ few hours
- Insert stars/repos using transactions in bulk
- Implement ajax for pagination
- Cache users and their star counts
- Use http://nvd3.org/ to visualise a users stars over time
- For many stars, should I use something like this https://github.com/tbranyen/hyperlist
- Configure pooling correctly: https://github.com/tgriesser/knex/issues/2820#issuecomment-481710112 e.g.
```
"pool": {
"min": 2,
"max": 6,
"createTimeoutMillis": 3000,
"acquireTimeoutMillis": 30000,
"idleTimeoutMillis": 30000,
"reapIntervalMillis": 1000,
"createRetryIntervalMillis": 100,
"propagateCreateError": false // <- default is true, set to false
},
```