{"id":25391497,"url":"https://github.com/estnafinema0/github-trends-aggregator","last_synced_at":"2025-04-10T00:20:04.913Z","repository":{"id":275482061,"uuid":"926194034","full_name":"estnafinema0/Github-Trends-Aggregator","owner":"estnafinema0","description":"A real-time GitHub trends aggregator that collects and visualizes popular repositories by language and topic, with WebSocket updates and analytics.","archived":false,"fork":false,"pushed_at":"2025-02-04T12:48:39.000Z","size":425,"stargazers_count":2,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-02-15T15:48:59.523Z","etag":null,"topics":["data-aggregation","full-stack","go","go-microservice","golang","goquery","gorilla-mux","gorilla-websocket","html-css","real-time-data","web-scraping","websocket-updates"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/estnafinema0.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2025-02-02T19:13:58.000Z","updated_at":"2025-02-07T21:43:03.000Z","dependencies_parsed_at":"2025-02-02T23:31:34.156Z","dependency_job_id":null,"html_url":"https://github.com/estnafinema0/Github-Trends-Aggregator","commit_stats":null,"previous_names":["estnafinema0/github-trends-aggregator"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/estnafinema0%2FGithub-Trends-Aggregator","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/estnafinema0%2FGithub-Trends-Aggregator/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/estnafinema0%2FGithub-Trends-Aggregator/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/estnafinema0%2FGithub-Trends-Aggregator/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/estnafinema0","download_url":"https://codeload.github.com/estnafinema0/Github-Trends-Aggregator/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248131753,"owners_count":21052921,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data-aggregation","full-stack","go","go-microservice","golang","goquery","gorilla-mux","gorilla-websocket","html-css","real-time-data","web-scraping","websocket-updates"],"created_at":"2025-02-15T15:43:00.565Z","updated_at":"2025-04-10T00:20:04.874Z","avatar_url":"https://github.com/estnafinema0.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# GitHub Trends Aggregator\n\n**GitHub Trends Aggregator** is a full-stack real-time web service that aggregates trending GitHub repositories, processes and displays them dynamically, offers filtering by language, sorting by stars, forks, current period stars, interest score. \n\n---\n\n- [GitHub Trends Aggregator](#github-trends-aggregator)\n  - [Overview](#overview)\n  - [Features](#features)\n  - [Architecture](#architecture)\n  - [Endpoints](#endpoints)\n  - [Installation](#installation)\n    - [Running the Server](#running-the-server)\n    - [Interactions in the terminal](#interactions-in-the-terminal)\n    - [Docker](#docker)\n  - [Technologies Used](#technologies-used)\n\n## Overview\n\nGitHub Trends Aggregator periodically scrapes the [GitHub Trending page](https://github.com/trending) using a resilient HTML parser and processes repository data to extract relevant metrics such as stars, forks, and a custom \"interest score\". The data is stored in an in-memory store and is made accessible via a RESTful API and WebSocket for live updates. An interactive client-side interface built with HTML and CSS presents the data in an *user-friendly format*, complete with filtering, sorting.\n\n---\n\n## Features\n\n- **Data Aggregation:**  \n  - Scrapes GitHub Trending HTML with a resilient parser using [goquery](https://github.com/PuerkitoBio/goquery). Presented in the module [fetcher](server/fetcher/fetcher.go). [Goroutine executing](server/scheduler/scheduler.go) the scraping process at one-minute intervals.  \n  - Extracts essential repository information (author, name, description, language, stars, forks, period stars) and computes an additional *interest score*.\n\n- **RESTful API Endpoints:**  \n  - `GET /trends` — Retrieve the list of trending repositories with support for filtering (by programming language), sorting (stars, forks, period stars, interest score).  \n  - `GET /trends/{id}` — Get detailed information about a specific repository (presented in a json format).  \n\n- **Real-time Updates:**  \n  - WebSocket support (`/ws` endpoint) to broadcast updates to connected clients immediately upon data changes.\n\n- **Go Concurrency:**  \n  - **Goroutines:** scheduled data fetching and handling multiple WebSocket connections.\n  - **Mutex Locking:** The in-memory data store is protected by sync.RWMutex methods.\n \n---\n\n## Architecture\n\n- **Backend:**  \n  - Written in Go, it consists of a microservice that performs HTML scraping, data processing, and exposes RESTful endpoints.\n  - Utilizes [Gorilla Mux](https://github.com/gorilla/mux) for routing and [Gorilla WebSocket](https://github.com/gorilla/websocket) for real-time communication.\n  - In-memory [store](server/store/store.go) for rapid development; easily extendable to persistent databases such as PostgreSQL.\n\n- **Frontend:**  \n\n  ![Trends page](images/image.png)\n  Dynamic interface with filtering and sorting.\n---\n\n## Endpoints\n\n- **GET /trends**  \n  Retrieves trending repositories. Supports query parameters:  \n  - `language` — Filter by programming language.  \n  - `sort_by` — Sort by `stars`, `forks`, `current_period_stars`, or `interest_score`.  \n\n- **GET /trends/{id}**  \n  Returns detailed information for a specific repository identified by `id` presented in a json format.\n\n- **WebSocket /ws**  \n  Establish a real-time connection to receive live updates whenever new trends are broadcast.\n\n---\n\n## Installation\n\nTo run the server, you need to have Go 1.18+ installed on Linux.\n\n### Running the Server\n\nRun Makefile from the root directory:\n   ```bash\n   make main\n   ```\n   The server will start on port `8080` (browser: `http://localhost:8080`).\n\n   Or run the server manually from the server directory:\n   ```bash\n   go get\n   go mod tidy\n   go run main.go\n   ```\n\n### Interactions in the terminal\n\n1) Get detailed information about a specific repository:\n```bash\ncurl http://localhost:8080/trends/1 | jq\n```\n`jq` will show the json in a readable format.\n\nThe response will be something like this:\n\n```json\n{\n  \"id\": \"oumi-ai/oumi\",\n  \"secondary\": 1,\n  \"author\": \"oumi-ai\",\n  \"name\": \"oumi\",\n  \"url\": \"https://github.com/oumi-ai/oumi\",\n  \"description\": \"Everything you need to build state-of-the-art foundation models, end-to-end.\",\n  \"language\": \"Python\",\n  \"stars\": 3549,\n  \"forks\": 0,\n  \"current_period_stars\": 1350,\n  \"updated_at\": \"2025-02-03T21:07:22.533614942+03:00\",\n  \"interest_score\": 3549\n}\n```\n\n2) Connect to the WebSocket via terminal:\n\nYou need to install `wscat` first:\n```bash\nnpm install -g wscat\n```\n\nThen connect to the WebSocket:\n```bash\nwscat -c ws://localhost:8080/ws\n```\n\n3) The other endpoinds have a nice representation in the browser: \n\n\n![Main page](images/image2.png)\n\n### Docker\n\nTo run the server in Docker, you need to have Docker installed on your machine.\n\nFrom the root directory:\n```bash\ndocker image build -t my_image1 ./server\ndocker container run -it -p 8080:8080 my_image1\n```\nAfter that, you can access the server at `http://localhost:8080`.\n\nAfter you're done, you can remove the container and the image:\n```bash\ndocker container rm -f my_container1\ndocker image rm -f my_image1\n```\n\n## Technologies Used\n\n- **Backend:** Go, Gorilla Mux, Gorilla WebSocket, goquery.\n- **Frontend:** HTML, CSS.\n\n---\n\nFeel free to explore the code. Any questions, feedback or suggestions are always welcome!\n\nHave a good coding day!\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Festnafinema0%2Fgithub-trends-aggregator","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Festnafinema0%2Fgithub-trends-aggregator","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Festnafinema0%2Fgithub-trends-aggregator/lists"}