https://github.com/umaar/export-github-stars

A Node.js webapp to scrape the stars of specified GitHub users
https://github.com/umaar/export-github-stars

Last synced: 29 days ago
JSON representation

A Node.js webapp to scrape the stars of specified GitHub users

• Host: GitHub
• URL: https://github.com/umaar/export-github-stars
• Owner: umaar
• Created: 2019-03-23T10:59:34.000Z (over 5 years ago)
• Default Branch: master
• Last Pushed: 2023-01-11T03:12:55.000Z (almost 2 years ago)
• Last Synced: 2024-11-08T12:43:21.209Z (about 1 month ago)
• Language: JavaScript
• Homepage:
• Size: 5.2 MB
• Stars: 40
• Watchers: 1
• Forks: 3
• Open Issues: 12
• Metadata Files:
  • Readme: readme.md

Awesome Lists containing this project

• project-awesome - umaar/export-github-stars - A Node.js webapp to scrape the stars of specified GitHub users (JavaScript)

README

        
### GitHub Star Export

[![Actions Status](https://github.com/umaar/export-github-stars/workflows/Node%20CI/badge.svg)](https://github.com/umaar/export-github-stars/actions)

Export _all_ your GitHub stars.

![preview.png](preview.png)

This webapp queries the GitHub API to query for the stars of a GitHub user, and stores them in an SQLite database. 10,000 stars will occupy ~6mb worth of data.

This repo uses persistent queues, and supports resumable downloading of stars. You can fetch stars of many users.

![preview-jobs.png](preview-jobs.png)

#### To run

1. Get a GitHub token from https://github.com/settings/tokens

2. Create a `.env` file:

```

# Contents of .env

GITHUB_TOKEN=your token here

# Any name for your new database

DATABASE_NAME=output

```

3. Run these commands:

```sh

npm i

make migrate-db-dev

make start

```

### TODO:

- Send emails using an email service about database size changes

- Automatically update user stars every ~ few hours

- Insert stars/repos using transactions in bulk

- Implement ajax for pagination

- Cache users and their star counts

- Use http://nvd3.org/ to visualise a users stars over time

- For many stars, should I use something like this https://github.com/tbranyen/hyperlist

- Configure pooling correctly: https://github.com/tgriesser/knex/issues/2820#issuecomment-481710112 e.g. 

```

"pool": {

  "min": 2,

  "max": 6,

  "createTimeoutMillis": 3000,

  "acquireTimeoutMillis": 30000,

  "idleTimeoutMillis": 30000,

  "reapIntervalMillis": 1000,

  "createRetryIntervalMillis": 100,

  "propagateCreateError": false // <- default is true, set to false

},

```