{"id":28501806,"url":"https://github.com/mu-majid/github-data-scrapper-be","last_synced_at":"2026-04-13T20:02:39.223Z","repository":{"id":297081529,"uuid":"995584026","full_name":"mu-majid/github-data-scrapper-be","owner":"mu-majid","description":"Backend built with Nodejs/Expressjs and Mongodb to handle Github OAuth2 and scrapes the logged in user data.","archived":false,"fork":false,"pushed_at":"2025-06-09T12:59:53.000Z","size":149,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-07-05T01:38:19.864Z","etag":null,"topics":["expressjs","github-api","mongodb","nodejs","oauth2"],"latest_commit_sha":null,"homepage":"","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mu-majid.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-06-03T17:53:11.000Z","updated_at":"2025-06-09T12:59:57.000Z","dependencies_parsed_at":"2025-08-07T05:30:55.400Z","dependency_job_id":null,"html_url":"https://github.com/mu-majid/github-data-scrapper-be","commit_stats":null,"previous_names":["mu-majid/github-data-scrapper-be"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/mu-majid/github-data-scrapper-be","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mu-majid%2Fgithub-data-scrapper-be","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mu-majid%2Fgithub-data-scrapper-be/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mu-majid%2Fgithub-data-scrapper-be/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mu-majid%2Fgithub-data-scrapper-be/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mu-majid","download_url":"https://codeload.github.com/mu-majid/github-data-scrapper-be/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mu-majid%2Fgithub-data-scrapper-be/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31768649,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-13T15:25:13.801Z","status":"ssl_error","status_checked_at":"2026-04-13T15:25:09.162Z","response_time":93,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["expressjs","github-api","mongodb","nodejs","oauth2"],"created_at":"2025-06-08T16:07:53.207Z","updated_at":"2026-04-13T20:02:39.218Z","avatar_url":"https://github.com/mu-majid.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Backend API Endpoints Summary\n\n## How to Run it:\n- Run `npm install`\n- Make sure you have Docker installed\n- make sure you set up the `.env` and `.env.docker` files properly\n- open a terminal in the project's directory, then run `make build`\n- then run `make up` to spin up the app and mongo\n- OR run `make tools` to spin up the app, mongo and mongo-express which is a GUI for mongo (localhost:8081).\n\n- .env\n```sh\nPORT=3000\nNODE_ENV=development\n\nMONGODB_URI=mongodb://localhost:27017/integrations\n\nJWT_SECRET=asd12345\n\nGITHUB_CLIENT_ID=Ov23liiejBNFsbBp7Trp\nGITHUB_CLIENT_SECRET=1ab3761ac5bf717134c12da2d8022ae1f3ad66f7\nGITHUB_CALLBACK_URL=http://localhost:3000/api/auth/github/callback\n\nFRONTEND_URL=http://localhost:4200\n```\n- .env.docker\n```sh\nNODE_ENV=production\n\nMONGO_ROOT_USERNAME=admin\nMONGO_ROOT_PASSWORD=password123\nMONGO_DATABASE=integrations\n\nJWT_SECRET=asd12345\n\nGITHUB_CLIENT_ID=Ov23liiejBNFsbBp7Trp\nGITHUB_CLIENT_SECRET=1ab3761ac5bf717134c12da2d8022ae1f3ad66f7\nGITHUB_CALLBACK_URL=http://localhost:3000/api/auth/github/callback\n\nFRONTEND_URL=http://localhost:4200\n\nMONGO_EXPRESS_USERNAME=admin\nMONGO_EXPRESS_PASSWORD=admin123\n```\n\n## Future enhancements and improvements:\n- Input validation and sanitization (Joi, ajv, ...)\n- Better logging for tracing the operations on the server.\n- Central Error Handling (semi-exists now), with custom business-domain errors\n- Using Typescript for proper typing.\n- Enhancing the sync endpoint:\n    - Idea-1: Use stream pipelines \n    - Idea-2: Fire and Forget endpoint and notify the client when done / client would long-poll the server.\n- Add testing (unit, integrations, e2e)\n\n## Authentication Endpoints (`/api/auth`)\n\n| Endpoint | Method | Purpose | Requirement Served |\n|----------|--------|---------|-------------------|\n| `/github` | GET | Initiate GitHub OAuth flow, returns auth URL | OAuth (2) authentication setup |\n| `/github/callback` | GET | Handle OAuth callback, exchange code for token | Complete OAuth flow, redirect after success |\n| `/status` | GET | Check if user is authenticated and get integration details | Display connection status and user info |\n| `/logout` | POST | Logout user (client discards JWT token) | Session management |\n| `/remove` | DELETE | Remove GitHub integration from database | \"Remove Integration\" button functionality |\n| `/verify` | POST | Verify JWT token validity | Token validation |\n\n## GitHub Operations (`/api/github`)\n\n| Endpoint | Method | Purpose | Requirement Served |\n|----------|--------|---------|-------------------|\n| `/sync` | POST | Sync all GitHub data (orgs, repos, commits, pulls, issues, users) | Fetch and store all required GitHub data |\n| `/sync-status` | GET | Get last sync time and data counts | Display sync status and data statistics |\n| `/sync-jupyter` | POST | Runs a Test Sync against Jupyter org | Testing Purposes |\n\n\n## Data Management (`/api/data`)\n\n| Endpoint | Method | Purpose | Requirement Served |\n|----------|--------|---------|-------------------|\n| `/collections` | GET | Get list of available collections with counts | Populate \"Entity Dropdown\" with collections |\n| `/collection/:name` | GET | Get paginated data from specific collection | Display data in AG Grid table |\n| `/collection/:name/fields` | GET | Get field definitions for AG Grid columns | Dynamic column setup for AG Grid |\n\n## Requirements Mapping\n\n**Core OAuth Requirements:**\n- Connect button → `/api/auth/github`\n- OAuth callback → `/api/auth/github/callback` \n- Success status display → `/api/auth/status`\n- Remove integration → `/api/auth/remove`\n\n**Data Fetching Requirements:**\n- All GitHub data sync → `/api/github/sync`\n- Organizations, repos, commits, pulls, issues, users → Stored via sync endpoint\n\n**Database Storage:**\n- Integration details → `github-integration` collection\n- All GitHub data → Separate collections per data type\n\n**Data Viewing Requirements:**\n- Active Integrations dropdown → `/api/data/collections`\n- Entity dropdown → `/api/data/collections`\n- AG Grid data display → `/api/data/collection/:name`\n- Dynamic columns → `/api/data/collection/:name/fields`\n- Search functionality → `/api/data/collection/:name +  searchQuery` (On a diff branch there a functionality for search across all collections)\n- Filters and pagination → Built into collection endpoint\n\nAll endpoints require JWT authentication except the initial OAuth endpoints, ensuring stateless operation as requested.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmu-majid%2Fgithub-data-scrapper-be","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmu-majid%2Fgithub-data-scrapper-be","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmu-majid%2Fgithub-data-scrapper-be/lists"}