https://github.com/hooopo/oh-my-github-pipeline
🔄 A flexible open-source data pipeline for seamlessly syncing data from any github user to your database.
https://github.com/hooopo/oh-my-github-pipeline
dashboard data-pipeline github tidb
Last synced: 8 days ago
JSON representation
🔄 A flexible open-source data pipeline for seamlessly syncing data from any github user to your database.
- Host: GitHub
- URL: https://github.com/hooopo/oh-my-github-pipeline
- Owner: hooopo
- Created: 2023-03-23T11:05:51.000Z (over 2 years ago)
- Default Branch: main
- Last Pushed: 2023-08-04T08:39:09.000Z (about 2 years ago)
- Last Synced: 2025-08-16T20:46:16.572Z (2 months ago)
- Topics: dashboard, data-pipeline, github, tidb
- Language: Ruby
- Homepage: https://github-dashboard-v3.vercel.app/
- Size: 66.4 KB
- Stars: 6
- Watchers: 2
- Forks: 4
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
## Oh My GitHub Pipeline
## GitHub Data Sync
This repository provides a powerful data pipeline that can sync all historical data for a GitHub user, as well as data for other users, to a free MySQL-compatible cloud database, TiDB Cloud. It can be used as a standalone data pipeline and combined with [Oh My GitHub Dashboard](https://github.com/hooopo/oh-my-github-dashboard) to create a personal dashboard.
In addition to generating a dashboard, this GitHub user data can also be used for the following purposes:
* Generating a resume
* Providing seed data for your frontend app demo
* Feeding LLM to generate a digital persona
* Syncing to your social networking site
* Generating a personal branding website
* Generating a weekly report
### Setup Data Pipeline
To use this repository as a standalone data pipeline, simply fork this repo, and set the environment variables and the GitHub action will run automatically every hour. This will sync the specified user's GitHub data to TiDB Cloud.
Environment Secrets
To use this repository, you will need to set the following secrets on GitHub:
| Secret Name | Description |
| --- | --- |
| **DATABASE_URL** | The MySQL connection information in URI format for TiDB Cloud. You will need to register and create a serverless cluster on [https://tidb.cloud](https://tidb.cloud/), and the URI format should contain the necessary information for connecting to the cluster. An example of the DATABASE_URL format is: `mysql2://user:password@host:port/dbname`  |
| **USER_LOGIN** | The user login of the account you want to sync. This can be your own GitHub account's user login or the user login of other users you want to sync. Note that if you choose to sync other users, you won't be able to sync private repositories and author_association information. |
| ACCESS_TOKEN | Optional, default is `GITHUB_TOKEN`. A personal access token provided by GitHub, which can be obtained from [Sign in to GitHub · GitHub](https://github.com/settings/tokens). |
| OPENAI_TOKEN | Optional, use it to generate story for your OSS contribution|
⚠️ Make sure you enable GitHub Action for this forked repo.
Enable GitHub Action:
For the first time, you can manually run GitHub Action:
### Table Schema
This project involves the following table structures and their relationships, as illustrated in the diagram below.
The curr_user table contains only one record and is used by BI tools to directly locate the synchronized target user through SQL queries. If you are using other programming languages, you can directly query the users table using the login parameter: where login = 'user_login'.
Please refer to the table below for a brief description of each table's structure.
```sql
+------------------+--------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+------------------+--------------+------+-----+---------+----------------+
| id | bigint(20) | NO | PRI | | auto_increment |
| login | varchar(255) | NO | | | |
| company | varchar(255) | YES | | | |
| location | varchar(255) | YES | | | |
| twitter_username | varchar(255) | YES | | | |
| followers_count | int(11) | YES | | 0 | |
| following_count | int(11) | YES | | 0 | |
| region | varchar(255) | YES | | | |
| created_at | varchar(255) | NO | | | |
| updated_at | varchar(255) | NO | | | |
+------------------+--------------+------+-----+---------+----------------+
```
```sql
+--------------------+--------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+--------------------+--------------+------+-----+---------+----------------+
| id | bigint(20) | NO | PRI | | auto_increment |
| name | varchar(255) | YES | | | |
| owner | varchar(255) | YES | | | |
| user_id | bigint(20) | YES | | | |
| license | varchar(255) | YES | | | |
| is_private | tinyint(1) | YES | | | |
| disk_usage | int(11) | YES | | | |
| language | varchar(255) | YES | | | |
| description | text | YES | | | |
| is_fork | tinyint(1) | YES | | | |
| parent_id | bigint(20) | YES | | | |
| fork_count | int(11) | YES | | | |
| stargazer_count | int(11) | YES | | | |
| pushed_at | datetime(6) | YES | | | |
| topics | json | YES | | | |
| created_at | datetime(6) | NO | | | |
| updated_at | datetime(6) | NO | | | |
| is_in_organization | tinyint(1) | YES | | 0 | |
+--------------------+--------------+------+-----+---------+----------------+
```
```sql
+--------------------+--------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+--------------------+--------------+------+-----+---------+----------------+
| id | bigint(20) | NO | PRI | | auto_increment |
| repo_id | bigint(20) | YES | | | |
| locked | tinyint(1) | YES | | | |
| title | varchar(255) | YES | | | |
| closed | tinyint(1) | YES | | | |
| closed_at | datetime(6) | YES | | | |
| state | varchar(255) | YES | | | |
| number | int(11) | YES | | | |
| user_id | bigint(20) | YES | MUL | | |
| author | varchar(255) | YES | MUL | | |
| author_association | varchar(255) | YES | | | |
| created_at | datetime(6) | NO | MUL | | |
| updated_at | datetime(6) | NO | MUL | | |
+--------------------+--------------+------+-----+---------+----------------+
```
```sql
+--------------------+--------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+--------------------+--------------+------+-----+---------+----------------+
| id | bigint(20) | NO | PRI | | auto_increment |
| repo_id | bigint(20) | YES | | | |
| locked | tinyint(1) | YES | | | |
| title | varchar(255) | YES | | | |
| closed | tinyint(1) | YES | | | |
| closed_at | datetime(6) | YES | | | |
| state | varchar(255) | YES | | | |
| number | int(11) | YES | | | |
| author | varchar(255) | YES | MUL | | |
| user_id | bigint(20) | YES | MUL | | |
| author_association | varchar(255) | YES | | | |
| is_draft | tinyint(1) | YES | | | |
| additions | int(11) | YES | | 0 | |
| deletions | int(11) | YES | | 0 | |
| merged_at | datetime(6) | YES | | | |
| merged_by | varchar(255) | YES | | | |
| changed_files | int(11) | YES | | 0 | |
| merged | tinyint(1) | YES | | | |
| comments_count | int(11) | YES | | 0 | |
| created_at | datetime(6) | NO | MUL | | |
| updated_at | datetime(6) | NO | MUL | | |
+--------------------+--------------+------+-----+---------+----------------+
```
```sql
+--------------------+--------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+--------------------+--------------+------+-----+---------+----------------+
| id | bigint(20) | NO | PRI | | auto_increment |
| repo_id | bigint(20) | YES | | | |
| user_id | bigint(20) | YES | | | |
| author_association | varchar(255) | YES | | | |
| issue_id | bigint(20) | YES | | | |
| created_at | datetime(6) | NO | | | |
| updated_at | datetime(6) | NO | | | |
+--------------------+--------------+------+-----+---------+----------------+
```
```sql
+--------------------+--------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+--------------------+--------------+------+-----+---------+----------------+
| id | bigint(20) | NO | PRI | | auto_increment |
| repo_id | bigint(20) | YES | | | |
| user_id | bigint(20) | YES | | | |
| author_association | varchar(255) | YES | | | |
| created_at | datetime(6) | NO | | | |
| updated_at | datetime(6) | NO | | | |
+--------------------+--------------+------+-----+---------+----------------+
```
```sql
+----------------+------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+----------------+------------+------+-----+---------+-------+
| user_id | bigint(20) | NO | MUL | | |
| target_user_id | bigint(20) | NO | | | |
+----------------+------------+------+-----+---------+-------+
```
```sql
+----------------+------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+----------------+------------+------+-----+---------+-------+
| user_id | bigint(20) | NO | MUL | | |
| target_user_id | bigint(20) | NO | | | |
+----------------+------------+------+-----+---------+-------+
```
```sql
+------------+-------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+------------+-------------+------+-----+---------+-------+
| user_id | bigint(20) | NO | MUL | | |
| repo_id | bigint(20) | NO | | | |
| starred_at | datetime(6) | NO | | | |
+------------+-------------+------+-----+---------+-------+
```
```sql
+----------------------------+--------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+----------------------------+--------------+------+-----+---------+----------------+
| id | bigint(20) | NO | PRI | | auto_increment |
| login | varchar(255) | NO | | | |
| company | varchar(255) | YES | | | |
| location | varchar(255) | YES | | | |
| twitter_username | varchar(255) | YES | | | |
| followers_count | int(11) | YES | | 0 | |
| following_count | int(11) | YES | | 0 | |
| region | varchar(255) | YES | | | |
| created_at | varchar(255) | NO | | | |
| updated_at | varchar(255) | NO | | | |
| last_issue_cursor | varchar(255) | YES | | | |
| last_pr_cursor | varchar(255) | YES | | | |
| last_follower_cursor | varchar(255) | YES | | | |
| last_following_cursor | varchar(255) | YES | | | |
| last_starred_repo_cursor | varchar(255) | YES | | | |
| last_repo_cursor | varchar(255) | YES | | | |
| last_commit_comment_cursor | varchar(255) | YES | | | |
| last_issue_comment_cursor | varchar(255) | YES | | | |
| bio | text | YES | | | |
| story | text | YES | | | |
+----------------------------+--------------+------+-----+---------+----------------+
```
## Related repos
* [Oh My GitHub Circles](https://github.com/hooopo/oh-my-github-circles)
* [Oh My GitHub Pipeline](https://github.com/hooopo/oh-my-github-pipeline)
* [Repo Contributor Circles](https://github.com/hooopo/repo-contributor-circles)
* [Repo Track Pipeline](https://github.com/hooopo/repo-track-pipeline)
* [OSSInsight Lite](https://github.com/pingcap/ossinsight-lite)
* [OSSInsight](https://github.com/pingcap/ossinsight)
* [Hackernews Insight](https://github.com/hooopo/hackernews-insight)
* [Oh My GitHub Dashboard](https://github.com/hooopo/oh-my-github-dashboard)
### Personal Dashboard
See the [Oh My GitHub Dashboard](https://github.com/hooopo/oh-my-github-dashboard) repository for more information on how to create a personal dashboard.
