{"id":22616002,"url":"https://github.com/lonelyhentxi/minellius","last_synced_at":"2025-04-11T12:14:41.037Z","repository":{"id":118797855,"uuid":"151424784","full_name":"lonelyhentxi/minellius","owner":"lonelyhentxi","description":"A experimental github data analysis solution. Group project of comp-3002, 2018 fall, hitsz. ","archived":false,"fork":false,"pushed_at":"2018-12-30T04:26:07.000Z","size":10064,"stargazers_count":9,"open_issues_count":0,"forks_count":0,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-04-10T09:56:19.609Z","etag":null,"topics":["angular","assignment","data-analysis","electron","github","nestjs","ng-zorro-antd","typescript"],"latest_commit_sha":null,"homepage":"","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/lonelyhentxi.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-10-03T14:20:42.000Z","updated_at":"2021-03-23T10:10:55.000Z","dependencies_parsed_at":"2023-06-09T07:46:26.158Z","dependency_job_id":null,"html_url":"https://github.com/lonelyhentxi/minellius","commit_stats":null,"previous_names":[],"tags_count":3,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lonelyhentxi%2Fminellius","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lonelyhentxi%2Fminellius/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lonelyhentxi%2Fminellius/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lonelyhentxi%2Fminellius/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/lonelyhentxi","download_url":"https://codeload.github.com/lonelyhentxi/minellius/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248399103,"owners_count":21097295,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["angular","assignment","data-analysis","electron","github","nestjs","ng-zorro-antd","typescript"],"created_at":"2024-12-08T19:10:22.512Z","updated_at":"2025-04-11T12:14:41.028Z","avatar_url":"https://github.com/lonelyhentxi.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003ch1 align=\"center\"\u003e\n  \u003cbr\u003e\n    \u003cimg src=\"./doc/favicon.png\" alt=\"Markdownify\" width=\"200\"\u003e\n  \u003cbr\u003e\n  Minellius\n  \u003cbr\u003e\n\u003c/h1\u003e\n\u003cp align=\"center\"\u003e\n\u003cb\u003e\nA experimental github data analysis solution\u003cbr\u003e\nGroup project of COMP-3002, 2018 fall, HITsz.\n\u003c/b\u003e\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n\u003cimg src=\"https://img.shields.io/travis/lonelyhentai/minellius.svg\" alt=\"build\"\u003e\n\u003cimg src=\"https://img.shields.io/github/tag/lonelyhentai/minellius.svg?colorB=green\" alt=\"tag\"\u003e\n\u003cimg src=\"https://img.shields.io/github/license/lonelyhentai/minellius.svg\" alt=\"license\"\u003e\n  \u003cimg src=\"https://img.shields.io/github/languages/top/lonelyhentai/minellius.svg?colorB=2b7489\" alt=\"language\"\u003e\n  \u003cimg src=\"https://img.shields.io/github/repo-size/lonelyhentai/minellius.svg\" alt=\"size\"\u003e\n  \u003cimg src=\"https://img.shields.io/github/issues/lonelyhentai/minellius.svg\"\n  alt=\"issue\"\n  \u003e\n\u003c/p\u003e\n\n## 👀 Overview\n\n\u003cb\u003eMinellius, a experimental github data analysis solution\u003c/b\u003e, which designed to combine data collection, persistence, integration, analysis, visualization, real-time monitoring, trend tracking, originating from a group project of COMP-3002, 2018 fall, HITsz.\n\nMinellius aims to help users understand the history, current, trend, ups and downs of github and even the open source world, enhance the sense of the times and the cultural aesthetics of the community; adjust the path of developers' learning and self-improvement to better integrate and participate in the development of community resources and culture.\n\n## 📑 Table of Contents\n\n\u003cul\u003e\n\u003cli\u003e\u003ca href=\"#overview\"\u003e\u0026#x1F440; Overview\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"#table-of-contents\"\u003e\u0026#x1F4D1; Table of Contents\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"#how-it-works\"\u003e\u0026#x1F528; How It Works\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"#%EF%B8%8F-features\"\u003e\u0026#x2B50;\u0026#xFE0F; Features\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"#example-snapshot\"\u003e\u0026#x1F453; Example Snapshot\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"#installation-usage\"\u003e\u0026#x1F527; Installation \u0026amp; Usage\u003c/a\u003e\n\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"#platform-support\"\u003e\u0026#x1F4BB; Platform Support\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"#language-support\"\u003e\u0026#x1F30F; Language Support\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"#acknowledgement\"\u003e\u0026#x1F44D; Acknowledgement\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"#source-code-reference-main\"\u003e\u0026#x1F449; Source Code Reference (Main)\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"#%EF%B8%8F-faq\"\u003e\u0026#x2754;\u0026#xFE0F; FAQ\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"#docs\"\u003e\u0026#x1F4D9; Docs\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"#%EF%B8%8F-resource-reference\"\u003e\u0026#x1F3F7;\u0026#xFE0F; Resource \u0026amp; Reference\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"#contributors\"\u003e\u0026#x1F606; Contributors\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"#license\"\u003e\u0026#x1F4DC; License\u003c/a\u003e\u003c/li\u003e\n\u003c/ul\u003e\n\n## 🔨 How It Works\n\nCrawl data from [Gharchive](https://www.gharchive.org/) and [Github Api](https://api.github.com/), then Aggregate, store, and visualize them.\n\n## ⭐️ Features\n\n1. Data Visualization with Charts and Tables\n2. Streaming Crawling and Analysis, Minimal Processing Space Requirement\n3. Basic User System with Bearer JWT Auth and Github Oauth Auth\n4. Secure Store and Transmission\n5. Continuous Integration\n6. Auto Recovery and Email Alert\n7. Autogen Documents\n8. I18n (Multi-language)\n9. Multi-platform\n10. Antd Design\n\n## 👓 Example Snapshot\n\n**Current Snapshot**\n\n![snapshot current](doc/snapshot-current.png)\n\n**Period Snapshot**\n\n![snapshot period](doc/snapshot-period.png)\n\n## 🔧 Installation \u0026 Usage\n\n### Client\n\n**From Binary (Recommended)**\n\nPlease see [release page](https://github.com/lonelyhentai/minellius/releases), and download the latest version for your platform. By default, we provide `windows version (portable exe)` and `linux version (AppImage)`. And because of project deadline limitation, we only provide `1280x720px` client.\n\n**From Source Code (Not Recommended)**\n\n1. First, enter into `client` directory and install the dependencies.Sure, you need [node.js](https://nodejs.org/en/) environment and **avaiable network (for china users)**.\n\n```sh\ncd client\nnpm install\n```\n\n2. Then build client for your platform.\n\n```sh\n# auto select suitable platform (recommended)\nnpm run electron:local\n# or manually select platform\nnpm run electron:windows\nnpm run electron:linux\nnpm run electron:mac # no test\n```\n\n3. You can find output in `client/release` directory, then copy and use it.\n\n**From Source Code To Dev (Only to developers)**\n\nAgain, install then start hot-reload dev server on http://localhost:4200 with electron. If you need chrome devtools, edit `client/src/main.ts`, set `devTools` to be `true` and  restart it.\n\n```sh\ncd client\nnpm install\nnpm run start\n```\n\n### Server\n\n\u003e **Important: Target server of our client has been set as our server, if you need deploy your server, you should edit it in the client source and rebuild the client as well.**\n\n**With Docker-compose (Recommended)**\n\n0. Sure, you need [docker](https://docs.docker.com/install/) and [docker-compose](https://docs.docker.com/compose/install/) environment.\n\n1. First, copy `docker-compose.sample.yaml` to `docker-compose.yaml` and edit environment variables in it or export them to your current shell.\n\n2. Copy `monitor/.docker-alertd.sample.yaml` to `monitor/.docker-alertd.yaml`. Edit alert email information in it.\n\n\u003e 3. Optional: We recommend postgresql database to deploy our service, if you use it, you can create a new database and import init tables from `deploy/database/postgresql/minellius.sql`. Then you can skip later database import.  \n\n\u003e 4. Optional: Because of huge amount of data, our 100GB database has no more space, so we cancel its auto start function in docker mode. To crawl period data, you should manually edit `crawler/Dockfile` and add `crawler/src/go.py` to the cmd field. For new fishs to docker or needing more information, see [Dockerfile Reference](https://docs.docker.com/engine/reference/builder/).\n\n5. Then easily build and start.\n\n```sh\ndocker-compose build\ndocker-compose up\n```\n\n6. Waiting `minellius-server` container init the database tables, you need import init insert data from `deploy/database/common`. If you use postgresql, you can easily import all (see step 3) `.sql` files, else you need manually import one by one now.\n\n7. If you need https and http2.0, install and start a [nginx web server](http://nginx.org/), copy our configuration file to its vhost directory, editting to fit yourself. And if you need free https cert, visit [Let's encrypt](https://letsencrypt.org/).\n\n**Manually Deploy (Not Recommended)**\n\n\u003e **Important: if you insist on manually deployment, you will lose the automatic email alert, and the automatic restart of the crawler will depends all on the third-party monitoring tool you use.**\n\n0. export need enviroment variables, see in `docker-compose.sample.yaml` environments fields.\n\n\u003e 1. Optional: We recommend postgresql database to deploy our service, if you use it, you can create a new database and import init tables from `deploy/database/postgresql/minellius.sql`. Then you can skip later database import.    \n\n2. `minellius-server`, ensuring you have [node.js](https://nodejs.org/en/) installed.\n\n```sh\ncd server\nnpm install\nnpm run start:prod\n# or for developement use\nnpm run start:dev\n```\n\n4. database import - waiting `minellius-server` container init the database tables, you need import init insert data from `deploy/database/common`. If you use postgresql, you can easily import all (see step 3) `.sql` files, else you need manually import one by one now.\n\n5. `minellius-crawler`, ensuring you have [basic python 3.6+ tools](https://www.python.org/) installed. And you need daemon tools to help you run them as services. Without docker, we recommend `pm2`.\n\n```sh\ncd crawler/\npip install -r requirements.txt\ncd src\npython current.py\n# if you need period\npython go.py\n```\n\n6. If you need https and http2.0, install and start a [nginx web server](http://nginx.org/), copy our configuration file to its vhost directory, editting to fit yourself. And if you need free https cert, visit [Let's encrypt](https://letsencrypt.org/).\n\n## 💻 Platform Support\n| Windows | Linux | Mac |\n| :---------: | :---------: | :---------: |\n| 10+\u003cbr\u003e(7+ possible) | 3.20+ With AppImage | Need manually build, no test now |\n\n## 🌏 Language Support\n\n1. English\n2. 简体中文\n\n\u0026nbsp;\u0026nbsp;\u0026nbsp;More languages are coming soon...\n\n## 👍 Acknowledgement\n\n\u003c!-- ALL-Acknowledgement-LIST:START - Do not remove or modify this section --\u003e\n\u003c!-- prettier-ignore --\u003e\n| [\u003cimg src=\"https://avatars3.githubusercontent.com/u/13409222?s=200\u0026v=4\" width=\"100px;\"/\u003e\u003cbr /\u003e\u003csub\u003e\u003cb\u003eElectron\u003c/b\u003e\u003c/sub\u003e](https://electronjs.org)\u003cbr /\u003e        | [\u003cimg src=\"https://angular.io/assets/images/logos/angular/angular.svg\" width=\"100px;\"/\u003e\u003cbr /\u003e\u003csub\u003e\u003cb\u003eAngular\u003c/b\u003e\u003c/sub\u003e](https://angular.io)\u003cbr /\u003e | [\u003cimg src=\"https://ng.ant.design/assets/img/logo.svg\" width=\"100px;\"/\u003e\u003cbr /\u003e\u003csub\u003e\u003cb\u003eNG-Zorro\u003c/b\u003e\u003c/sub\u003e](https://ng.ant.design)\u003cbr /\u003e          |  [\u003cimg src=\"https://nestjs.com/logo-small.5d4d1c8f.svg\" width=\"100px;\"/\u003e\u003cbr /\u003e\u003csub\u003e\u003cb\u003eNestjs\u003c/b\u003e\u003c/sub\u003e](https://nestjs.com)\u003cbr /\u003e  |  [\u003cimg src=\"https://raw.githubusercontent.com/github/explore/6c6508f34230f0ac0d49e847a326429eefbfc030/topics/docker/docker.png\" width=\"100px;\"/\u003e\u003cbr /\u003e\u003csub\u003e\u003cb\u003eDocker\u003c/b\u003e\u003c/sub\u003e](https://www.docker.com/)\u003cbr /\u003e |  \n| :-----------------------------------------------------------------------------------------------------------------------------------------------------------------: | :-----------------------------------------------------------------------------------------------------------------------------------------------------------------------: | :-------------------------------------------------------------------------------------------------------------------------------------------------------------------: | :-------------------------------------------------------------------------------------------------------------------------------------------------------------: | :------------------------------------------------------------------------------------------------------------------------------------------------------------: | \n\u003c!-- ALL-Acknowledgement-LIST:END --\u003e\n\n## 👉 Source Code Reference (Main)\n\n1. client\n    - infra - angular + electron + ant zorro\n    - components - `client/src/app/**/*.component.ts`\n    - providers \u0026 services - `client/src/app/providers/*.service.ts`\n    - dtos - `client/src/app/dtos/*.dto.ts`\n    - functools - `client/src/app/functools/*.functool.ts` \u0026 innerCode\n    - directives - `client/src/app/directives/*.directive.ts`\n    - interfaces - `client/src/app/interfaces/*.interface.ts` \u0026 innerCode\n    - animations - `client/src/app/animations/*.animations.ts`\n    - style - `client/src/app/**/*.scss` \u0026 innerCode\n    - module - `client/src/app/**/*.module.ts`\n    - router - `client/src/app/app-routing.module.ts` \u0026 routeService \u0026 routerService\n    - guards - `client/src/app/guards/auth.guard.ts`\n    - current - `client/src/app/components/current*.*` \u0026 `client/src/app/providers/current.service.ts`\n    - period - `client/src/app/components/period*.*` \u0026 `client/src/app/providers/period.service.ts`\n    - user - `client/src/app/componets/log*.*` \u0026 `client/src/app/providers/user.service.ts` \u0026 `client/src/app/components/control.component.ts`\n    - helper - `client/src/app/components/helper`\n    - i18n - `TranslateService` \u0026 `src/app/providers/custom-translate.service.ts` \u0026 `src/assets/i18n/*.json`\n    - introduction - `src/app/components/{home,video}.components.*`\n    - img \u0026 video \u0026 geo - `src/assets`\n    - test - `**/*.spec.ts`\n    - bootstrap - `main{.ts,.js}` | `index.html`\n    - config \n    - ...\n2. server\n    - nodejs + nest + native python + postgresql\n    - auth - `server/src/auth/**/*` \u0026 `server/src/role/**/*`\n    - user \u0026 group \u0026 permission ... - `server/src/role/**/*`\n    - data - `server/src/data/**/*`\n    - database - `server/src/database/**/*` \u0026 `deploy/database/*` \u0026 `**/*.entities.ts` \u0026 `TypeOrmModule` \u0026 `**/*.service.ts`\n    - config - `**/*-config.interface.ts` \u0026\u0026 `**/*-config{.*}` \u0026\u0026 environments variables \u0026 `**/*{.json,.yaml,.yml}` ...\n    - https \u0026 http2.0 - `minellius.nginx.sample.conf` + nginx proxy pass\n    - monitor - `monitor/.docker-alertd.yaml` + docker-alertd + `docker-compose.yaml` + docker-compose + docker + pm2.json + pm2\n    - cralwer - `crawler/src/*.py`\n    - document - `server/src/**/*.dto.ts` \u0026 `server/src/**/*.controller.ts` \u0026 swagger\n    - controller - `server/src/**/*.controller.ts`\n    - service - `server/src/**/*.service.ts`\n    - module - `server/src/**/*.module.ts`\n    - ...\n\n\n## ❔️ FAQ\n\n1. The range of the period event data you provide?\n  \u003e We provide data from 2015.01.01 to 2018.12.26. But since we don't have a low-cost way to store such a large amount of data, after the end of the semester, we will only provide the last month of data as the basic demo of our application service. And we will provide download link for that data, you can find them in resource section.\n\n## 📙 Docs\n\n* [Server Api Document](https://minellius.evernightfireworks.com/document)\n* [Initiation Document (Chinese Assignment)](./doc/initiation_document.md)\n* [Software_Requirements_Specification.pdf (Chinese Assignment)](./doc/Software_Requirements_Specification.pdf)\n\n## 🏷️ Resource \u0026 Reference\n\n1. [Course Information](http://liaoqing.me/comp3002.html)\n2. [Zipped Sql File from 2015.01.01 to 2018.01.26 Download](https://pan.baidu.com/s/17WEgYuZ4G_X6Ofig3c-alg)\n3. [Promotional Video (Short Version)](https://pan.baidu.com/s/12Gwn7sP0eknQO9S_LTAAxA)\n4. [Promotional Video (Full Version)](https://pan.baidu.com/s/1OakbX0BhmJC7Dv-BEC7icg)\n5. [Presentation Slides (Chinese Assignment)](https://pan.baidu.com/s/13AlfCgq6PM63Oo_okMUcSg)\n\n## 😆 Contributors\n\n\u003c!-- ALL-CONTRIBUTORS-LIST:START - Do not remove or modify this section --\u003e\n\u003c!-- prettier-ignore --\u003e\n| [\u003cimg src=\"https://avatars3.githubusercontent.com/u/23011677?v=3\" width=\"100px;\"/\u003e\u003cbr /\u003e\u003csub\u003e\u003cb\u003eZhou Yeheng\u003c/b\u003e\u003c/sub\u003e](https://github.com/lonelyhentai)\u003cbr /\u003e        | [\u003cimg src=\"https://avatars3.githubusercontent.com/u/43856818?v=3\" width=\"100px;\"/\u003e\u003cbr /\u003e\u003csub\u003e\u003cb\u003eCui Tianyu\u003c/b\u003e\u003c/sub\u003e](https://github.com/HITFZ-cty)\u003cbr /\u003e | [\u003cimg src=\"https://avatars0.githubusercontent.com/u/41630003?v=3\" width=\"100px;\"/\u003e\u003cbr /\u003e\u003csub\u003e\u003cb\u003eCao Weihan\u003c/b\u003e\u003c/sub\u003e](https://github.com/Cwhdly)\u003cbr /\u003e          |  [\u003cimg src=\"https://avatars0.githubusercontent.com/u/33949370?v=3\" width=\"100px;\"/\u003e\u003cbr /\u003e\u003csub\u003e\u003cb\u003eLin jintao\u003c/b\u003e\u003c/sub\u003e](https://github.com/dreamerlin)\u003cbr /\u003e  |  [\u003cimg src=\"https://avatars0.githubusercontent.com/u/44575939?v=3\" width=\"100px;\"/\u003e\u003cbr /\u003e\u003csub\u003e\u003cb\u003eFang An\u003c/b\u003e\u003c/sub\u003e](https://github.com/anhit)\u003cbr /\u003e |  \n| :-----------------------------------------------------------------------------------------------------------------------------------------------------------------: | :-----------------------------------------------------------------------------------------------------------------------------------------------------------------------: | :-------------------------------------------------------------------------------------------------------------------------------------------------------------------: | :-------------------------------------------------------------------------------------------------------------------------------------------------------------: | :------------------------------------------------------------------------------------------------------------------------------------------------------------: | \n\u003c!-- ALL-CONTRIBUTORS-LIST:END --\u003e\n\n## 📜 License\nThis software is licensed under the [MIT](./LICENSE.txt) \n\n```text\nMIT License\n\nCopyright (c) 2018 Zhou Yeheng, Cui Tianyu, Cao Weihan, Lin Jintao, Fang An\n\nPermission is hereby granted, free of charge, to any person obtaining a copy\nof this software and associated documentation files (the \"Software\"), to deal\nin the Software without restriction, including without limitation the rights\nto use, copy, modify, merge, publish, distribute, sublicense, and/or sell\ncopies of the Software, and to permit persons to whom the Software is\nfurnished to do so, subject to the following conditions:\n\nThe above copyright notice and this permission notice shall be included in all\ncopies or substantial portions of the Software.\n\nTHE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\nIMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\nFITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\nAUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\nLIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\nOUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\nSOFTWARE.\n```","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flonelyhentxi%2Fminellius","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flonelyhentxi%2Fminellius","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flonelyhentxi%2Fminellius/lists"}